@ohos-ports/514labs-kafka-javascript 0.0.0-stage → 1.7.1-patch-beta.0

package/.claude/settings.local.json

@@ -0,0 +1,12 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(npm pack:*)",
+      "Bash(grep:*)",
+      "Bash(ls:*)",
+      "Bash(tar:*)",
+      "Bash(cat:*)",
+      "Bash(node --version:*)"
+    ]
+  }
+}

package/.editorconfig

@@ -0,0 +1,5 @@
+[*]
+indent_style = space
+indent_size = 2
+trim_trailing_whitespace = true
+insert_final_newline = true

package/.envrc

@@ -0,0 +1 @@
+use flake

package/CHANGELOG.md

@@ -0,0 +1,380 @@
+# confluent-kafka-javascript 1.7.0
+
+v1.7.0 is a feature release. It is supported for all usage.
+
+### Enhancements
+
+1. References librdkafka v2.12.1. Refer to the [librdkafka v2.12.1 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.12.1) for more information.
+2. Configurable batch size through the `js.consumer.max.batch.size` property 
+   and cache size through the `js.consumer.max.cache.size.per.worker.ms` 
+   property (#393).
+3. Statistics callback now available when using the promisified API with all
+   client types (#399).
+4. Fix for at-least-once guarantee not ensured in case a seek happens on one
+partition and there are messages being fetched about other partitions (#393).
+5. Avoid returning a negative lag in case there is no cached offset for 
+   the HWM (#406).
+
+# confluent-kafka-javascript 1.6.0
+
+v1.6.0 is a feature release. It is supported for all usage.
+
+### Enhancements
+
+1. References librdkafka v2.12.0. Refer to the [librdkafka v2.12.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.12.0) for more information.
+2. OAuth OIDC method example for Kafka metadata based authentication with
+  an Azure IMDS endpoint using an attached managed identity as principal (#377).
+
+
+# confluent-kafka-javascript 1.5.0
+
+v1.5.0 is a feature release. It is supported for all usage.
+
+### Enhancements
+
+1. Adds support for `highWatermark`, `offsetLag()`, and `offsetLagLow()` in `eachBatch` callback (#317).
+
+
+## Fixes
+
+1. Fix issue of delay of up to 5s in receiving messages after pause and resume, or seek (#285, #363).
+
+
+# confluent-kafka-javascript v1.4.1
+
+v1.4.1 is a maintenance release. It is supported for all usage.
+
+## Enhancements
+
+1. References librdkafka v2.11.1. Refer to the [librdkafka v2.11.1 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.11.1) for more information.
+
+
+# confluent-kafka-javascript v1.4.0
+
+v1.4.0 is a feature release. It is supported for all usage.
+
+## Enhancements
+
+1. References librdkafka v2.11.0. Refer to the [librdkafka v2.11.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.11.0) for more information.
+2. [KIP-848] `describeGroups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and it defaults to being undefined (#329).
+3. [KIP-848] Admin API for listing consumer groups now has an optional filter to return only groups of given types (#328).
+4. Add support for Node v24 pre-built binaries (@weyert, #307, #337).
+
+
+# confluent-kafka-javascript v1.3.2
+
+v1.3.2 is a maintenance release. It is supported for all usage.
+
+## Enhancements
+
+1. References librdkafka v2.10.1. Refer to the [librdkafka v2.10.1 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.10.1) for more information.
+2. Support for schema id in header (#303)
+3. Add CEL support for Data Quality rules (#313)
+
+## Fixes
+
+1. Fix missing await during JSON deserialization (#301)
+2. Fix possible NPE in CSFLE executor (#305)
+
+
+# confluent-kafka-javascript v1.3.1
+
+v1.3.1 is a maintenance release. It is supported for all usage.
+
+## Fixes
+
+1. Avoid a race condition that causes 100% usage of a CPU core when
+   consuming with `partitionsConsumedConcurrently > 1` and all messages
+   are consumed (#300)
+2. Fix type definition for `Kafka()` constructor and for passing topic configurations
+   to the `producer()` and `consumer()` in the promisified API (@davidkhala, #297, #321).
+
+
+# confluent-kafka-javascript v1.3.0
+
+v1.3.0 is a feature release. It is supported for all usage.
+
+## Enhancements
+
+1. References librdkafka v2.10.0. Refer to the [librdkafka v2.10.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.10.0) for more information.
+
+## Fixes
+
+1. Support Protobuf oneof fields in Data Contract rules (#261)
+2. Ensure use of DEK object is thread-safe (#268)
+3. Fix token fetch to return after successful request (#278)
+4. Add loose OAuth authorization (#289)
+
+
+# confluent-kafka-javascript v1.2.0
+
+v1.2.0 is a feature release. It is supported for all usage.
+
+## Enhancements
+
+1. Add support for an Admin API to fetch topic offsets by timestamp (#206).
+2. References librdkafka v2.8.0. Refer to the [librdkafka v2.8.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.8.0) for more information.
+
+## Fixes
+
+1. Fixes an issue where the `eachBatch` callback was being called for the same partition concurrently (#224).
+2. Uses a Debian based builder instead of an Ubuntu based builder to ensure compatibility with older, but supported versions of Linux (#228).
+
+
+# confluent-kafka-javascript v1.1.0
+
+v1.1.0 is a feature release. It is supported for all usage.
+
+## Fixes
+
+1. Ensure algorithm query param is passed for CSFLE (#230)
+2. Handle records nested in arrays/maps when searching for tags (#231)
+
+
+# confluent-kafka-javascript v1.0.0
+
+v1.0.0 is a feature release. It is supported for all usage.
+
+## Enhancements
+
+1. Add support for an Admin API to fetch topic offsets (#156).
+2. Add support for Node v23 pre-built binaries (#158).
+3. Add KafkaJS-compatible errors to promisified Admin API (createTopics, deleteGroups, deleteTopicRecords) (#159).
+4. Include error types within Type definitions for promisified API (#210).
+
+## Fixes
+
+1. Fixes an issue where `uv_async_init` was being called off the event loop thread,
+   causing the node process to hang (#190).
+
+
+# confluent-kafka-javascript v0.6.1
+
+v0.6.1 is a limited availability maintenance release. It is supported for all usage.
+
+## Fixes
+
+### Schema Registry
+
+1. Sync aws-sdk packages to the same versioning.
+
+2. Minor enhancement to allow AWS profile to be specified.
+
+3. Add ability to override disable flag and actions on a rule.
+
+# confluent-kafka-javascript v0.6.0
+
+v0.6.0 is a limited availability maintenance release. It is supported for all usage.
+
+## Fixes
+
+### Schema Registry
+
+1. Add AWS AssumeRole support to AWS KMS. You can now specify a role arn, and optional
+role session name and optional role external id.
+
+2. Ensure different key ids use different client instances.
+
+# confluent-kafka-javascript v0.5.2
+
+v0.5.2 is a limited availability maintenance release. It is supported for all usage.
+
+Note: v0.5.2 has no code changes from v0.5.1, version is bumped to sync with @confluentinc/schemaregistry.
+
+
+# confluent-kafka-javascript v0.5.1
+
+v0.5.1 is a limited availability maintenance release. It is supported for all usage.
+
+## Fixes
+
+1. Fix an issue where `sendOffsets` wasn't working correctly in the transactional
+   producer (#172).
+
+
+# confluent-kafka-javascript v0.5.0
+
+v0.5.0 is a limited availability feature release. It is supported for all usage.
+
+Note: v0.5.0 was not released because of the bug mentioned in v0.5.1 section. Instead,
+      v0.5.1 was released with the fix directly.
+
+## Enhancements
+
+1. Add support for an Admin API to delete records.(#141).
+2. Add support for an Admin API to describe topics.(#155).
+3. Add support for dependent Admin client (#153).
+4. References librdkafka v2.6.1. Refer to the [librdkafka v2.6.1 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.6.1) for more information.
+
+## Fixes
+
+1. Fixes an issue with unresolved raced Promises leaking in the consumer (#151).
+2. Removes schemaregistry dependencies from root package.json to prevent issues with Node 18 (#162).
+
+
+# confluent-kafka-javascript v0.4.0
+
+v0.4.0 is a limited availability feature release. It is supported for all usage.
+
+## Enhancements
+
+1. Fixes an issue where headers were not passed correctly to the `eachBatch` callback (#130).
+2. Add support for an Admin API to list a consumer group's offsets (#49).
+3. Reduce consumer poll timeout to nil and add wakeups for new messages. This improves
+   the consumer efficiency, and resolves issues while running multiple consumers within
+   the same node process (#135).
+4. Add pre-built binaries for macOS (Intel).
+
+
+# confluent-kafka-javascript v0.3.0
+
+v0.3.0 is a limited availability feature release. It is supported for all usage.
+
+## Enhancements
+
+1. References librdkafka v2.6.0. Refer to the [librdkafka v2.6.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.6.0) for more information.
+1. Minor optimization to reduce schema ID lookups (#123).
+
+
+# confluent-kafka-javascript v0.2.1
+
+v0.2.1 is a limited availability release. It is supported for all usage.
+
+## Features
+
+1. Update README, docs, and examples for Confluent's Schema Registry client.
+
+
+# confluent-kafka-javascript v0.2.0
+
+v0.2.0 is a limited availability release. It is supported for all usage.
+
+## Features
+
+1. Switch to using `librdkafka` on the latest released tag `v2.5.3` instead of `master`.
+
+
+# confluent-kafka-javascript v0.1.17-devel
+
+v0.1.17-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add a commitCb method to the callback-based API which allows committing asynchronously.
+2. Pass assign/unassign functions to the rebalance callback in the promisified API, allowing
+   the user to control the assignment of partitions, or pause just after a rebalance.
+3. Remove store from promisified API and let the library handle all the stores.
+4. Add JavaScript-level debug logging to the client for debugging issues within the binding.
+5. Various fixes for performance and robustness of the consumer cache.
+6. Remove `consumerGroupId` argument from the `sendOffsets` method of the transactional producer,
+   and instead, only allow using a `consumer`.
+
+## Fixes
+
+1. Do not modify RegExps which don't start with a ^, instead, throw an error so
+   that there is no unexpected behaviour for the user (Issue [#64](https://github.com/confluentinc/confluent-kafka-javascript/issues/64)).
+2. Do not mutate arguments in run, pause and resume (Issue [#61](https://github.com/confluentinc/confluent-kafka-javascript/issues/61)).
+3. Fix a segmentation fault in `listGroups` when passing `matchConsumerGroupStates` as undefined.
+
+
+# confluent-kafka-javascript v0.1.16-devel
+
+v0.1.16-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add per-partition concurrency to consumer.
+2. Add true `eachBatch` support to consumer.
+3. Add a `leaderEpoch` field to the topic partitions where required (listing, committing, etc.).
+
+
+# confluent-kafka-javascript v0.1.15-devel
+
+v0.1.15-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add Node v22 builds and bump librdkafka version on each version bump of this library.
+
+
+# confluent-kafka-javascript v0.1.14-devel
+
+v0.1.14-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add metadata to offset commit and offset store (non-promisified API).
+2. Add types for logger and loglevel to configuration.
+3. Add Producer polling from background thread. This improves performance for cases when send is awaited on.
+4. Enable consume optimization from v0.1.13-devel (Features #2) by default for the promisified API.
+
+## Bug Fixes
+
+1. Fix issues with the header conversions from promisified API to the non-promisified API to match
+   the type signature and allow Buffers to be passed as header values in the C++ layer.
+
+
+# confluent-kafka-javascript v0.1.13-devel
+
+v0.1.13-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add support for `storeOffsets` in the consumer API.
+2. Add optimization while consuming, in cases where the size of messages pending in our subscription is less than the consumer cache size.
+
+## Bug Fixes
+
+1. Fix memory leak in incremental assign (@martijnimhoff, #35).
+2. Fix various issues with typings, and reconcile typings, JavaScript code, and MIGRATION.md to be consistent.
+
+
+# confluent-kafka-javascript v0.1.12-devel
+
+v0.1.12-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add support for `listTopics` in the Admin API.
+2. Add support for OAUTHBEARER token refresh callback for both promisified and non promisified API.
+
+## Bug Fixes
+
+1. Fix aliasing bug between `NodeKafka::Conf` and `RdKafka::ConfImpl`.
+2. Fix issue where `assign/unassign` were called instead of `incrementalAssign/incrementalUnassign` while using
+   the Cooperative Sticky assigner, and setting the `rebalance_cb` as a boolean rather than as a function.
+3. Fix memory leaks in Dispatcher and Conf (both leaked memory at client close).
+4. Fix type definitions and make `KafkaJS` and `RdKafka` separate namespaces, while maintaining compatibility
+   with node-rdkafka's type definitions.
+
+
+# confluent-kafka-javascript v0.1.11-devel
+
+v0.1.11-devel is a pre-production, early-access release.
+
+## Features
+
+1. Add support for `eachBatch` in the Consumer API (partial support for API compatibility).
+2. Add support for `listGroups`, `describeGroups` and `deleteGroups` in the Admin API.
+
+
+# confluent-kafka-javascript v0.1.10-devel
+
+v0.1.10-devel is a pre-production, early-access release.
+
+## Features
+
+1. Pre-built binaries for Windows (x64) added on an experimental basis.
+
+
+# confluent-kafka-javascript v0.1.9-devel
+
+v0.1.9-devel is a pre-production, early-access release.
+
+## Features
+
+1. Pre-built binaries for Linux (both amd64 and arm64, both musl and glibc), for macOS (m1), for node versions 18, 20 and 21.
+2. Promisified API for Consumer, Producer and Admin Client.
+3. Allow passing topic configuration properties via the global configuration block.
+4. Remove dependencies with security issues.
+5. Support for the Cooperative Sticky assignor.

package/CONTRIBUTING.md

@@ -0,0 +1,257 @@
+# Contributing to `confluent-kafka-javascript`
+
+:+1::tada: First off, thanks for taking the time to contribute! :tada::+1:
+
+The following is a set of guidelines for contributing to `confluent-kafka-javascript`
+which is hosted by [Confluent Inc.](https://github.com/confluentinc)
+on GitHub. This document lists rules, guidelines, and help getting started,
+so if you feel something is missing feel free to send a pull request.
+
+## What should I know before I get started?
+
+### Contributor Agreement
+
+Required (please follow instructions after making any Pull Requests).
+
+## How can I contribute?
+
+### Reporting Bugs
+
+Please use __Github Issues__ to report bugs. When filling out an issue report,
+make sure to copy any related code and stack traces so we can properly debug.
+We need to be able to reproduce a failing test to be able to fix your issue
+most of the time, so a custom written failing test is very helpful.
+
+Please also note the Kafka broker version that you are using and how many
+replicas, partitions, and brokers you are connecting to, because some issues
+might be related to Kafka. A list of `librdkafka` configuration key-value pairs
+also helps.
+
+Adding the property `debug` in your `librdkafka` configuration will help us. A list of
+possible values is available [here](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md),
+but you can set it to `all` if verbose logs are okay.
+
+### Suggesting Enhancements
+
+Please use __Github Issues__ to suggest enhancements. We are happy to consider
+any extra functionality or features to the library, as long as they add real
+and related value to users. Describing your use case and why such an addition
+helps the user base can help guide the decision to implement it into the
+library's core.
+
+### Pull Requests
+
+* Include new test cases (either end-to-end or unit tests) with your change.
+* Follow our style guides.
+* Make sure all tests are still passing and the linter does not report any issues.
+* End files with a new line.
+* Document the new code in the comments (if it is JavaScript) so the
+  documentation generator can update the reference documentation.
+* Avoid platform-dependent code.
+  <br>**Note:** If making modifications to the underlying C++, please use built-in
+  precompiler directives to detect such platform specificities. Use `Nan`
+  whenever possible to abstract node/v8 version incompatibility.
+* Make sure your branch is up to date and rebased.
+* Squash extraneous commits unless their history truly adds value to the library.
+
+## Styleguides
+
+### General style guidelines
+
+Download the [EditorConfig](http://editorconfig.org) plugin for your preferred
+text editor to automate the application of the following guidelines:
+
+* Use 2-space indent (no tabs).
+* Do not leave trailing whitespace on lines.
+* Files should end with a final newline.
+
+Also, adhere to the following not enforced by EditorConfig:
+
+* Limit lines to 80 characters in length. A few extra (<= 5) is fine if it helps
+  readability, use good judgement.
+* Use `lf` line endings. (git's `core.autocrlf` setting can help)
+
+### Git Commit Messages
+
+Commit messages should adhere to the guidelines in tpope's
+[A Note About Git Commit Messages](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
+
+In short:
+
+* Use the imperative mood. ("Fix bug", not "Fixed bug" or "Fixes bug")
+* Limit the first line to 50 characters or less, followed by a blank line
+  and detail paragraphs (limit detail lines to about 72 characters).
+* Reference issue numbers or pull requests whenever possible.
+
+### JavaScript Styleguide
+
+* All callbacks should follow the standard Node.js callback signature.
+* Your JavaScript should properly pass the linter (`make jslint` and `make eslint`).
+
+### C++ Styleguide
+
+* Class member variables should be prefixed with `m_`.
+* Use a comment when pointer ownership has changed hands.
+* Your C++ should properly pass the `cpplint.py` in the `make lint` test.
+
+### Specs Styleguide
+
+* Write JavaScript tests by using the `mocha` testing framework for the
+  non-promisified API and `jest` for the promisified API.
+* All `mocha` tests should use exports syntax.
+* All `mocha` test files should be suffixed with `.spec.js` instead of `.js`.
+* Unit tests should mirror the JavaScript files they test (for example,
+  `lib/client.js` is tested in `test/client.spec.js`).
+* Unit tests should have no outside service dependencies. Any time a dependency,
+  like Kafka, exists, you should create an end-to-end test.
+* You may mock a connection in a unit test if it is reliably similar to its real
+  variant.
+
+### Documentation Styleguide
+
+* Write all JavaScript documentation in jsdoc-compatible inline comments.
+* Each docblock should have references to return types and parameters. If an
+  object is a parameter, you should also document any required subproperties.
+* Use `@see` to reference similar pieces of code.
+* Use comments to document your code when its intent may be difficult to understand.
+* All documentation outside of the code should be in Github-compatible markdown.
+* Make good use of font variations like __bold__ and *italics*.
+* Use headers and tables of contents when they make sense.
+
+## Editor
+
+Using Visual Studio code to develop on `confluent-kafka-javascript`. If you use it you can configure the C++ plugin to resolve the paths needed to inform your intellisense. This is the config file I am using on a mac to resolve the required paths:
+
+`c_cpp_properties.json`
+```
+{
+    "configurations": [
+        {
+            "name": "Mac",
+            "includePath": [
+                "${workspaceFolder}/**",
+                "${workspaceFolder}",
+                "${workspaceFolder}/src",
+                "${workspaceFolder}/node_modules/nan",
+                "${workspaceFolder}/deps/librdkafka/src",
+                "${workspaceFolder}/deps/librdkafka/src-cpp",
+                "/usr/local/include/node",
+                "/usr/local/include/node/uv"
+            ],
+            "defines": [],
+            "macFrameworkPath": [
+                "/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks"
+            ],
+            "compilerPath": "/usr/bin/clang",
+            "cStandard": "c11",
+            "cppStandard": "c++17",
+            "intelliSenseMode": "clang-x64"
+        }
+    ],
+    "version": 4
+}
+```
+
+## Tests
+
+This project includes three types of tests in this project:
+* end-to-end integration tests (`mocha`)
+* unit tests (`mocha`)
+* integration tests for promisified API (`jest`)
+
+You can run all types of tests by using `Makefile`. Doing so calls `mocha` or `jest` in your locally installed `node_modules` directory.
+
+* Before you run the tests, be sure to init and update the submodules:
+  1. `git submodule init`
+  2. `git submodule update`
+* To run the unit tests, you can run `make lint` or `make test`.
+* To run the promisified integration tests, you can use `make promisified_test`.
+  You must have a running Kafka installation available. By default, the test tries to connect to `localhost:9092`;
+  however, you can supply the `KAFKA_HOST` environment variable to override this default behavior.
+* To run the integration tests, you can use `make e2e`.
+  You must have a running Kafka installation available. By default, the test tries to connect to `localhost:9092`;
+  however, you can supply the `KAFKA_HOST` environment variable to override this default behavior. Run `make e2e`.
+
+## Debugging
+
+### Debugging C++
+
+Use `gdb` for debugging (as shown in the following example).
+
+```
+node-gyp rebuild --debug
+
+gdb node
+(gdb) set args "path/to/file.js"
+(gdb) run
+[output here]
+```
+
+You can add breakpoints and so on after that.
+
+### Debugging and Profiling JavaScript
+
+Run the code with the `--inspect` flag, and then open `chrome://inspect` in Chrome and connect to the debugger.
+
+Example:
+
+```
+node --inspect path/to/file.js
+```
+
+## Updating librdkafka version
+
+The librdkafka should be periodically updated to the latest release in https://github.com/confluentinc/librdkafka/releases
+
+Steps to update:
+1. Update the `librdkafka` property in [`package.json`](https://github.com/confluentinc/confluent-kafka-javascript/blob/master/package.json) to the desired version.
+
+1. Update the librdkafka git submodule to that versions release commit (example below)
+
+    ```bash
+    cd deps/librdkafka
+    git checkout 063a9ae7a65cebdf1cc128da9815c05f91a2a996 # for version 1.8.2
+    ```
+
+    If you get an error during that checkout command, double check that the submodule was initialized / cloned! You may need to run `git submodule update --init --recursive`
+
+1. Update [`config.d.ts`](https://github.com/confluentinc/confluent-kafka-javascript/blob/master/config.d.ts) and [`errors.d.ts`](https://github.com/confluentinc/confluent-kafka-javascript/blob/master/errors.d.ts) TypeScript definitions by running:
+    ```bash
+    node ci/librdkafka-defs-generator.js
+    ```
+
+1. Run `npm install` to build with the new version and fix any build errors that occur.
+
+1. Run unit tests: `npm run test`
+
+1. Change the librdkafka version in `semaphore.yml`
+
+1. Update the version numbers referenced in the [`README.md`](https://github.com/confluentinc/confluent-kafka-javascript/blob/master/README.md) file to the new version.
+
+## Releasing
+
+1. Increment the `version` in `package.json`. Change the version in `util.js` too.
+If it's needed to change librdkafka version, see the **Updating librdkafka version** section.
+
+1. Run `npm run prepack` to verify package version.
+
+1. Run `npm install` to update the `package-lock.json` file.
+
+1. Create a PR and merge the above changes, and tag the merged commit with the new version. This should be the same string as `version` in `package.json`.
+
+1. The CI will run on the tag, which will create the release artifacts in Semaphore CI.
+
+1. Create a new GitHub release with the tag, and upload the release artifacts from Semaphore CI.
+   The release title should be the same string as `version` in `package.json`.
+
+1. build schemaregistry project to prepare for the release:
+   `(cd schemaregistry && npm run build)`
+
+1. Clear any file that isn't ignored by `.npmignore`
+
+1. In case it's a release candidate add `--tag rc` to the following commands,
+   to avoid setting it as latest one
+
+1. Publish Kafka client, with `--dry-run` first: `npm publish --dry-run --workspace=. --otp=<otp_here>`
+
+1. Publish Schema Registry client, with `--dry-run` first: `npm publish --dry-run --workspace=schemaregistry --otp=<otp_here>`