karafka 2.0.0.rc6 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +86 -2
- data/Gemfile.lock +1 -1
- data/README.md +2 -2
- data/lib/karafka/version.rb +1 -1
- data.tar.gz.sig +0 -0
- metadata +3 -3
- metadata.gz.sig +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 06f96b3ba14b7910d3cb23a90bdc135927f1483d5af4d079d59b3d6940d391b9
|
|
4
|
+
data.tar.gz: 2d936d4ddac360e229004c05f6ee51d004018e3c4cf4203067a99e6df93df92e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: caeb9bcf1f0301f31442025176f3da631aa8ff9c21164e112b9ce3112372309d4b7fe59863cf7cf511e6a6bc4448f61448f3bc2024b101d4ae2668a358e5bbe9
|
|
7
|
+
data.tar.gz: cfc422fae74512c2142ef36f9d4b5c3024bf318d79dc83456e27553aebc177cec7a3772f6378160c8ecdf5a3b2626d862ba91d823fae6e30cbb593d4198474a9
|
checksums.yaml.gz.sig
CHANGED
|
Binary file
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,91 @@
|
|
|
1
1
|
# Karafka framework changelog
|
|
2
2
|
|
|
3
|
+
## 2.0.0 (2022-08-5)
|
|
4
|
+
|
|
5
|
+
This changelog describes changes between `1.4` and `2.0`. Please refer to appropriate release notes for changes between particular `rc` releases.
|
|
6
|
+
|
|
7
|
+
Karafka 2.0 is a **major** rewrite that brings many new things to the table but also removes specific concepts that happened not to be as good as I initially thought when I created them.
|
|
8
|
+
|
|
9
|
+
Please consider getting a Pro version if you want to **support** my work on the Karafka ecosystem!
|
|
10
|
+
|
|
11
|
+
For anyone worried that I will start converting regular features into Pro: This will **not** happen. Anything free and fully OSS in Karafka 1.4 will **forever** remain free. Most additions and improvements to the ecosystem are to its free parts. Any feature that is introduced as a free and open one will not become paid.
|
|
12
|
+
|
|
13
|
+
### Additions
|
|
14
|
+
|
|
15
|
+
This section describes **new** things and concepts introduced with Karafka 2.0.
|
|
16
|
+
|
|
17
|
+
Karafka 2.0:
|
|
18
|
+
|
|
19
|
+
- Introduces multi-threaded support for [concurrent work](https://github.com/karafka/karafka/wiki/Concurrency-and-multithreading) consumption for separate partitions as well as for single partition work via [Virtual Partitions](https://github.com/karafka/karafka/wiki/Pro-Virtual-Partitions).
|
|
20
|
+
- Introduces [Active Job adapter](https://github.com/karafka/karafka/wiki/Active-Job) for using Karafka as a jobs backend with Ruby on Rails Active Job.
|
|
21
|
+
- Introduces fully automatic integration end-to-end [test suite](https://github.com/karafka/karafka/tree/master/spec/integrations) that checks any case I could imagine.
|
|
22
|
+
- Introduces [Virtual Partitions](https://github.com/karafka/karafka/wiki/Pro-Virtual-Partitions) for ability to parallelize work of a single partition.
|
|
23
|
+
- Introduces [Long-Running Jobs](https://github.com/karafka/karafka/wiki/Pro-Long-Running-Jobs) to allow for work that would otherwise exceed the `max.poll.interval.ms`.
|
|
24
|
+
- Introduces the [Enhanced Scheduler](https://github.com/karafka/karafka/wiki/Pro-Enhanced-Scheduler) that uses a non-preemptive LJF (Longest Job First) algorithm instead of a a FIFO (First-In, First-Out) one.
|
|
25
|
+
- Introduces [Enhanced Active Job adapter](https://github.com/karafka/karafka/wiki/Pro-Enhanced-Active-Job) that is optimized and allows for strong ordering of jobs and more.
|
|
26
|
+
- Introduces seamless [Ruby on Rails integration](https://github.com/karafka/karafka/wiki/Integrating-with-Ruby-on-Rails-and-other-frameworks) via `Rails::Railte` without need for any extra configuration.
|
|
27
|
+
- Provides `#revoked` [method](https://github.com/karafka/karafka/wiki/Consuming-messages#shutdown-and-partition-revocation-handlers) for taking actions upon topic revocation.
|
|
28
|
+
- Emits underlying async errors emitted from `librdkafka` via the standardized `error.occurred` [monitor channel](https://github.com/karafka/karafka/wiki/Error-handling-and-back-off-policy#error-tracking).
|
|
29
|
+
- Replaces `ruby-kafka` with `librdkafka` as an underlying driver.
|
|
30
|
+
- Introduces official [EOL policies](https://github.com/karafka/karafka/wiki/Versions-Lifecycle-and-EOL).
|
|
31
|
+
- Introduces [benchmarks](https://github.com/karafka/karafka/tree/master/spec/benchmarks) that can be used to profile Karafka.
|
|
32
|
+
- Introduces a requirement that the end user code **needs** to be [thread-safe](https://github.com/karafka/karafka/wiki/FAQ#does-karafka-require-gems-to-be-thread-safe).
|
|
33
|
+
- Introduces a [Pro subscription](https://github.com/karafka/karafka/wiki/Build-vs.-Buy) with a [commercial license](https://github.com/karafka/karafka/blob/master/LICENSE-COMM) to fund further ecosystem development.
|
|
34
|
+
|
|
35
|
+
### Deletions
|
|
36
|
+
|
|
37
|
+
This section describes things that are **no longer** part of the Karafka ecosystem.
|
|
38
|
+
|
|
39
|
+
Karafka 2.0:
|
|
40
|
+
|
|
41
|
+
- Removes topics mappers concept completely.
|
|
42
|
+
- Removes pidfiles support.
|
|
43
|
+
- Removes daemonization support.
|
|
44
|
+
- Removes support for using `sidekiq-backend` due to introduction of [multi-threading](https://github.com/karafka/karafka/wiki/Concurrency-and-multithreading).
|
|
45
|
+
- Removes the `Responders` concept in favour of WaterDrop producer usage.
|
|
46
|
+
- Removes completely all the callbacks in favour of finalizer method `#shutdown`.
|
|
47
|
+
- Removes single message consumption mode in favour of [documentation](https://github.com/karafka/karafka/wiki/Consuming-messages#one-at-a-time) on how to do it easily by yourself.
|
|
48
|
+
|
|
49
|
+
### Changes
|
|
50
|
+
|
|
51
|
+
This section describes things that were **changed** in Karafka but are still present.
|
|
52
|
+
|
|
53
|
+
Karafka 2.0:
|
|
54
|
+
|
|
55
|
+
- Uses only instrumentation that comes from Karafka. This applies also to notifications coming natively from `librdkafka`. They are now piped through Karafka prior to being dispatched.
|
|
56
|
+
- Integrates WaterDrop `2.x` tightly with autoconfiguration inheritance and an option to redefine it.
|
|
57
|
+
- Integrates with the `karafka-testing` gem for RSpec that also has been updated.
|
|
58
|
+
- Updates `cli info` to reflect the `2.0` details.
|
|
59
|
+
- Stops validating `kafka` configuration beyond minimum as the rest is handled by `librdkafka`.
|
|
60
|
+
- No longer uses `dry-validation`.
|
|
61
|
+
- No longer uses `dry-monitor`.
|
|
62
|
+
- No longer uses `dry-configurable`.
|
|
63
|
+
- Lowers general external dependencies three **heavily**.
|
|
64
|
+
- Renames `Karafka::Params::BatchMetadata` to `Karafka::Messages::BatchMetadata`.
|
|
65
|
+
- Renames `Karafka::Params::Params` to `Karafka::Messages::Message`.
|
|
66
|
+
- Renames `#params_batch` in consumers to `#messages`.
|
|
67
|
+
- Renames `Karafka::Params::Metadata` to `Karafka::Messages::Metadata`.
|
|
68
|
+
- Renames `Karafka::Fetcher` to `Karafka::Runner` and align notifications key names.
|
|
69
|
+
- Renames `StdoutListener` to `LoggerListener`.
|
|
70
|
+
- Reorganizes [monitoring and logging](https://github.com/karafka/karafka/wiki/Monitoring-and-logging) to match new concepts.
|
|
71
|
+
- Notifies on fatal worker processing errors.
|
|
72
|
+
- Contains updated install templates for Rails and no-non Rails.
|
|
73
|
+
- Changes how the routing style (`0.5`) behaves. It now builds a single consumer group instead of one per topic.
|
|
74
|
+
- Introduces changes that will allow me to build full web-UI in the upcoming `2.1`.
|
|
75
|
+
- Contains updated example apps.
|
|
76
|
+
- Standardizes error hooks for all error reporting (`error.occurred`).
|
|
77
|
+
- Changes license to `LGPL-3.0`.
|
|
78
|
+
- Introduces a `karafka-core` dependency that contains common code used across the ecosystem.
|
|
79
|
+
- Contains updated [wiki](https://github.com/karafka/karafka/wiki) on everything I could think of.
|
|
80
|
+
|
|
81
|
+
### What's ahead
|
|
82
|
+
|
|
83
|
+
Karafka 2.0 is just the beginning.
|
|
84
|
+
|
|
85
|
+
There are several things in the plan already for 2.1 and beyond, including a web dashboard, at-rest encryption, transactions support, and more.
|
|
86
|
+
|
|
3
87
|
## 2.0.0.rc6 (2022-08-05)
|
|
4
|
-
- Update licenser to use a gem based approach based on `karafka
|
|
88
|
+
- Update licenser to use a gem based approach based on `karafka-license`.
|
|
5
89
|
- Do not mark intermediate jobs as consumed when Karafka runs Enhanced Active Job with Virtual Partitions.
|
|
6
90
|
- Improve development experience by adding fast cluster state changes refresh (#944)
|
|
7
91
|
- Improve the license loading.
|
|
@@ -59,7 +143,7 @@
|
|
|
59
143
|
- Add more integration specs related to polling limits.
|
|
60
144
|
- Remove auto-detection of re-assigned partitions upon rebalance as for too fast rebalances it could not be accurate enough. It would also mess up in case of rebalances that would happen right after a `#seek` was issued for a partition.
|
|
61
145
|
- Optimize the removal of pre-buffered lost partitions data.
|
|
62
|
-
- Always
|
|
146
|
+
- Always run `#revoked` when rebalance with revocation happens.
|
|
63
147
|
- Evict executors upon rebalance, to prevent race-conditions.
|
|
64
148
|
- Align topics names for integration specs.
|
|
65
149
|
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
[](http://badge.fury.io/rb/karafka)
|
|
5
5
|
[](https://slack.karafka.io)
|
|
6
6
|
|
|
7
|
-
**Note**: All of the documentation here refers to Karafka `2.0.0
|
|
7
|
+
**Note**: All of the documentation here refers to Karafka `2.0.0` or higher. If you are looking for the documentation for Karafka `1.4`, please click [here](https://github.com/karafka/wiki/tree/1.4).
|
|
8
8
|
|
|
9
9
|
## About Karafka
|
|
10
10
|
|
|
@@ -55,7 +55,7 @@ We also maintain many [integration specs](https://github.com/karafka/karafka/tre
|
|
|
55
55
|
1. Add and install Karafka:
|
|
56
56
|
|
|
57
57
|
```bash
|
|
58
|
-
bundle add karafka
|
|
58
|
+
bundle add karafka
|
|
59
59
|
|
|
60
60
|
bundle exec karafka install
|
|
61
61
|
```
|
data/lib/karafka/version.rb
CHANGED
data.tar.gz.sig
CHANGED
|
Binary file
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: karafka
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.0.0
|
|
4
|
+
version: 2.0.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Maciej Mensfeld
|
|
@@ -287,9 +287,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
287
287
|
version: 2.7.0
|
|
288
288
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
289
289
|
requirements:
|
|
290
|
-
- - "
|
|
290
|
+
- - ">="
|
|
291
291
|
- !ruby/object:Gem::Version
|
|
292
|
-
version:
|
|
292
|
+
version: '0'
|
|
293
293
|
requirements: []
|
|
294
294
|
rubygems_version: 3.3.7
|
|
295
295
|
signing_key:
|
metadata.gz.sig
CHANGED
|
Binary file
|