karafka 0.5.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 9c40b95f3636ecc4ded9cbbfbb575129e5593560
+  data.tar.gz: 2ee0650b0244b772141af9566cfd7c41c0a8632d
+SHA512:
+  metadata.gz: c850800bd7b372abe43df6d3179be95d69296b0e6369dc14bea3431f8fe76e3b33fbb6db360e0dd08218bacea183dfd31671c64ac6f3a122f897f2013653a765
+  data.tar.gz: 08f57ac78f4c00d0426d17a6a24044cf393f173a356b87865d605b7699b3333424159ae58f9f5ff8cf0bf0d540559d24d19abd69e76d3f5243253c66a25f56b1

data/.gitignore

@@ -0,0 +1,68 @@
+# bundler state
+/.bundle
+/vendor/bundle/
+/vendor/ruby/
+/ruby/
+app.god
+
+# minimal Rails specific artifacts
+db/*.sqlite3
+/log/development.log
+/log/production.log
+/log/test.log
+/tmp/*
+*.gem
+*.~
+
+# various artifacts
+**.war
+*.rbc
+*.sassc
+.rspec
+.redcar/
+.capistrano/
+.sass-cache
+/config/god/sidekiq.rb
+/config/puma.rb
+/coverage.data
+/coverage/
+/doc/api/
+/doc/app/
+/doc/yard
+/doc/features.html
+/doc/specs.html
+/spec/tmp/*
+/cache
+/capybara*
+/capybara-*.html
+/gems
+/specifications
+rerun.txt
+pickle-email-*.html
+
+# If you find yourself ignoring temporary files generated by your text editor
+# or operating system, you probably want to add a global ignore instead:
+#   git config --global core.excludesfile ~/.gitignore_global
+#
+# Here are some files you may want to ignore globally:
+
+# scm revert files
+**.orig
+
+# Mac finder artifacts
+.DS_Store
+
+# Netbeans project directory
+/nbproject
+
+# RubyMine project files
+.idea
+
+# Textmate project files
+/*.tmproj
+
+# vim artifacts
+**.swp
+
+# documentation
+.yardoc

data/.ruby-gemset

@@ -0,0 +1 @@
+karafka

data/.ruby-version

@@ -0,0 +1 @@
+2.3.1

data/.travis.yml

@@ -0,0 +1,6 @@
+language: ruby
+rvm:
+  - 2.3.0
+  - 2.3.1
+script:
+  - bundle exec rake

data/CHANGELOG.md

@@ -0,0 +1,202 @@
+# Karafka framework changelog
+
+## 0.5.0-beta
+- Removed Zookeeper totally as dependency
+- Better group and partition rebalancing
+- Automatic thread management (no need for tunning) - each topic is a separate actor/thread
+- Moved from Poseidon into Ruby-Kafka
+- No more max_concurrency setting
+- After you define your App class and routes (and everything else) you need to add execute App.boot!
+- Manual consuming is no longer available (no more karafka consume command)
+- Karafka topics CLI is no longer available. No Zookeeper - no global topic discovery
+- Dropped ZK as dependency
+- karafka info command no longer prints details about Zookeeper
+- Better shutdown
+- No more autodiscovery via Zookeeper - instead, the whole cluster will be discovered directly from Kafka
+- No more support for Kafka 0.8
+- Support for Kafka 0.9
+- No more need for ActorCluster, since now we have a single thread (and Kafka connection) per topic
+- Ruby 2.2.* support dropped
+- Using App name as a Kafka client_id
+- Automatic Capistrano integration
+- Responders support for handling better responses pipelining and better responses flow description and design (see README for more details)
+- Gem dump
+- Readme updates
+- karafka flow CLI command for printing the application flow
+- Some internal refactorings
+
+## 0.4.2
+- #87 - Reconsume mode with crone for better Rails/Rack integration
+- Moved Karafka server related stuff into separate Karafka::Server class
+- Renamed Karafka::Runner into Karafka::Fetcher
+- Gem dump
+- Added chroot option to Zookeeper options
+- Moved BROKERS_PATH into config from constant
+- Added Karafka consume CLI action for a short running single consumption round
+- Small fixes to close broken connections
+- Readme updates
+
+## 0.4.1
+- Explicit throw(:abort) required to halt before_enqueue (like in Rails 5)
+- #61 - Autodiscover Kafka brokers based on Zookeeper data
+- #63 - Graceful shutdown with current offset state during data processing
+- #65 - Example of NewRelic monitor is outdated
+- #71 - Setup should be executed after user code is loaded
+- Gem dump x3
+- Rubocop remarks
+- worker_timeout config option has been removed. It now needs to be defined manually by the framework user because WorkerGlass::Timeout can be disabled and we cannot use Karafka settings on a class level to initialize user code stuff
+- Moved setup logic under setup/Setup namespace
+- Better defaults handling
+- #75 - Kafka and Zookeeper options as a hash
+- #82 - Karafka autodiscovery fails upon caching of configs
+- #81 - Switch config management to dry configurable
+- Version fix
+- Dropped support for Ruby 2.1.*
+- Ruby dump to 2.3.1
+
+## 0.4.0
+- Added WaterDrop gem with default configuration
+- Refactoring of config logic to simplify adding new dependencies that need to be configured based on #setup data
+- Gem dump
+- Readme updates
+- Renamed cluster to actor_cluster for method names
+- Replaced SidekiqGlass with generic WorkerGlass lib
+- Application bootstrap in app.rb no longer required
+- Karafka.boot needs to be executed after all the application files are loaded (template updated)
+- Small loader refactor (no API changes)
+- Ruby 2.3.0 support (default)
+- No more rake tasks
+- Karafka CLI instead of rake tasks
+- Worker cli command allows passing additional options directly to Sidekiq
+- Renamed concurrency to max_concurrency - it describes better what happens - Karafka will use this number of threads only when required
+- Added wait_timeout that allows us to tune how long should we wait on a single socket connection (single topic) for new messages before going to next one (this applies to each thread separately)
+- Rubocop remarks
+- Removed Sinatra and Puma dependencies
+- Karafka Cli internal reorganization
+- Karafka Cli routes task
+- #37 - warn log for failed parsing of a message
+- #43 - wrong constant name
+- #44 - Method name conflict
+- #48 - Cannot load such file -- celluloid/current
+- #46 - Loading application
+- #45 - Set up monitor in config
+- #47 - rake karafka:run uses app.rb only
+- #53 - README update with Sinatra/Rails integration description
+- #41 - New Routing engine
+- #54 - Move Karafka::Workers::BaseWorker to Karafka::BaseWorker
+- #55 - ApplicationController and ApplicationWorker
+
+## 0.3.2
+- Karafka::Params::Params lazy load merge keys with string/symbol names priorities fix
+
+## 0.3.1
+- Renamed Karafka::Monitor to Karafka::Process to represent a Karafka process wrapper
+- Added Karafka::Monitoring that allows to add custom logging and monitoring with external libraries and systems
+- Moved logging functionality into Karafka::Monitoring default monitoring
+- Added possibility to provide own monitoring as long as in responds to #notice and #notice_error
+- Standarized logging format for all logs
+
+## 0.3.0
+- Switched from custom ParserError for each parser to general catching of Karafka::Errors::ParseError and its descendants
+- Gem dump
+- Fixed #32 - now when using custom workers that does not inherit from Karafka::BaseWorker perform method is not required. Using custom workers means that the logic that would normally lie under #perform, needs to be executed directly from the worker.
+- Fixed #31 - Technically didn't fix because this is how Sidekiq is meant to work, but provided possibility to assign custom interchangers that allow to bypass JSON encoding issues by converting data that goes to Redis to a required format (and parsing it back when it is fetched)
+- Added full parameters lazy load - content is no longer loaded during #perform_async if params are not used in before_enqueue
+- No more namespaces for Redis by default (use separate DBs)
+
+## 0.1.21
+- Sidekiq 4.0.1 dump
+- Gem dump
+- Added direct celluloid requirement to Karafka (removed from Sidekiq)
+
+## 0.1.19
+- Internal call - schedule naming change
+- Enqueue to perform_async naming in controller to follow Sidekiqs naming convention
+- Gem dump
+
+## 0.1.18
+- Changed Redis configuration options into a single hash that is directly passed to Redis setup for Sidekiq
+- Added config.ru to provide a Sidekiq web UI (see README for more details)
+
+## 0.1.17
+- Changed Karafka::Connection::Cluster tp Karafka::Connection::ActorCluster to distinguish between a single thread actor cluster for multiple topic connection and a future feature that will allow process clusterization.
+- Add an ability to use user-defined parsers for a messages
+- Lazy load params for before callbacks
+- Automatic loading/initializng all workers classes during startup (so Sidekiq won't fail with unknown workers exception)
+- Params are now private to controller
+- Added bootstrap method to app.rb
+
+## 0.1.16
+- Cluster level error catching for all exceptions so actor is not killer
+- Cluster level error logging
+- Listener refactoring (QueueConsumer extracted)
+- Karafka::Connection::QueueConsumer to wrap around fetching logic - technically we could replace Kafka with any other messaging engine as long as we preserve the same API
+- Added debug env for debugging purpose in applications
+
+## 0.1.15
+- Fixed max_wait_ms vs socket_timeout_ms issue
+- Fixed closing queue connection after Poseidon::Errors::ProtocolError failure
+- Fixed wrong logging file selection based on env
+- Extracted Karafka::Connection::QueueConsumer object to wrap around queue connection
+
+## 0.1.14
+- Rake tasks for listing all the topics on Kafka server (rake kafka:topics)
+
+## 0.1.13
+- Ability to assign custom workers and use them bypassing Karafka::BaseWorker (or its descendants)
+- Gem dump
+
+## 0.1.12
+- All internal errors went to Karafka::Errors namespace
+
+## 0.1.11
+- Rescuing all the "before Sidekiq" processing so errors won't affect other incoming messages
+- Fixed dying actors after connection error
+- Added a new app status - "initializing"
+- Karafka::Status model cleanup
+
+## 0.1.10
+- Added possibility to specify redis namespace in configuration (failover to app name)
+- Renamed redis_host to redis_url in configuration
+
+## 0.1.9
+- Added worker logger
+
+## 0.1.8
+- Droped local env suppot in favour of [Envlogic](https://github.com/karafka/envlogic) - no changes in API
+
+## 0.1.7
+- Karafka option for Redis hosts (not localhost only)
+
+## 0.1.6
+- Added better concurency by clusterization of listeners
+- Added graceful shutdown
+- Added concurency that allows to handle bigger applications with celluloid
+- Karafka controllers no longer require group to be defined (created based on the topic and app name)
+- Karafka controllers no longer require topic to be defined (created based on the controller name)
+- Readme updates
+
+## 0.1.5
+- Celluloid support for listeners
+- Multi target logging (STDOUT and file)
+
+## 0.1.4
+- Renamed events to messages to follow Apache Kafka naming convention
+
+## 0.1.3
+- Karafka::App.logger moved to Karafka.logger
+- README updates (Usage section was added)
+
+## 0.1.2
+- Logging to log/environment.log
+- Karafka::Runner
+
+## 0.1.1
+- README updates
+- Raketasks updates
+- Rake installation task
+- Changelog file added
+
+## 0.1.0
+- Initial framework code
+

data/Gemfile

@@ -0,0 +1,8 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development, :test do
+  gem 'polishgeeks-dev-tools'
+  gem 'timecop'
+end

data/Gemfile.lock

@@ -0,0 +1,216 @@
+PATH
+  remote: .
+  specs:
+    karafka (0.5.0)
+      activesupport (~> 5.0)
+      celluloid (~> 0.17)
+      dry-configurable (~> 0.1.7)
+      envlogic (~> 1.0)
+      rake (~> 11.3)
+      ruby-kafka (= 0.3.15)
+      sidekiq (~> 4.2)
+      thor (~> 0.19)
+      waterdrop (~> 0.3)
+      worker-glass (~> 0.2)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activesupport (5.0.0.1)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (~> 0.7)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+    addressable (2.4.0)
+    ast (2.3.0)
+    axiom-types (0.1.1)
+      descendants_tracker (~> 0.0.4)
+      ice_nine (~> 0.11.0)
+      thread_safe (~> 0.3, >= 0.3.1)
+    brakeman (3.3.3)
+    bundler-audit (0.5.0)
+      bundler (~> 1.2)
+      thor (~> 0.18)
+    celluloid (0.17.3)
+      celluloid-essentials
+      celluloid-extras
+      celluloid-fsm
+      celluloid-pool
+      celluloid-supervision
+      timers (>= 4.1.1)
+    celluloid-essentials (0.20.5)
+      timers (>= 4.1.1)
+    celluloid-extras (0.20.5)
+      timers (>= 4.1.1)
+    celluloid-fsm (0.20.5)
+      timers (>= 4.1.1)
+    celluloid-pool (0.20.5)
+      timers (>= 4.1.1)
+    celluloid-supervision (0.20.6)
+      timers (>= 4.1.1)
+    codeclimate-engine-rb (0.3.1)
+      virtus (~> 1.0)
+    coderay (1.1.1)
+    coercible (1.0.0)
+      descendants_tracker (~> 0.0.1)
+    colorize (0.8.1)
+    concurrent-ruby (1.0.2)
+    connection_pool (2.2.0)
+    descendants_tracker (0.0.4)
+      thread_safe (~> 0.3, >= 0.3.1)
+    diff-lcs (1.2.5)
+    docile (1.1.5)
+    dry-configurable (0.1.7)
+      concurrent-ruby (~> 1.0)
+    envlogic (1.0.3)
+      activesupport
+    equalizer (0.0.11)
+    erubis (2.7.0)
+    faker (1.6.6)
+      i18n (~> 0.5)
+    flay (2.8.0)
+      erubis (~> 2.7.0)
+      path_expander (~> 1.0)
+      ruby_parser (~> 3.0)
+      sexp_processor (~> 4.0)
+    flog (4.4.0)
+      path_expander (~> 1.0)
+      ruby_parser (~> 3.1, > 3.1.0)
+      sexp_processor (~> 4.4)
+    haml (4.0.7)
+      tilt
+    haml_lint (0.18.1)
+      haml (~> 4.0)
+      rake (>= 10, < 12)
+      rubocop (>= 0.36.0)
+      sysexits (~> 1.1)
+    hitimes (1.2.4)
+    i18n (0.7.0)
+    ice_nine (0.11.2)
+    json (2.0.2)
+    launchy (2.4.3)
+      addressable (~> 2.3)
+    method_source (0.8.2)
+    minitest (5.9.1)
+    null-logger (0.1.3)
+    parser (2.3.1.2)
+      ast (~> 2.2)
+    path_expander (1.0.0)
+    polishgeeks-dev-tools (1.3.2)
+      brakeman
+      bundler-audit
+      faker
+      haml_lint
+      pry
+      rspec
+      rubocop
+      rubocop-rspec
+      rubycritic
+      shoulda
+      simplecov
+      timecop
+      yard
+    powerpack (0.1.1)
+    pry (0.10.4)
+      coderay (~> 1.1.0)
+      method_source (~> 0.8.1)
+      slop (~> 3.4)
+    rack (2.0.1)
+    rack-protection (1.5.3)
+      rack
+    rainbow (2.1.0)
+    rake (11.3.0)
+    redis (3.3.1)
+    reek (4.1.0)
+      codeclimate-engine-rb (~> 0.3.1)
+      parser (~> 2.3.1, >= 2.3.1.2)
+      rainbow (~> 2.0)
+    rspec (3.5.0)
+      rspec-core (~> 3.5.0)
+      rspec-expectations (~> 3.5.0)
+      rspec-mocks (~> 3.5.0)
+    rspec-core (3.5.2)
+      rspec-support (~> 3.5.0)
+    rspec-expectations (3.5.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.5.0)
+    rspec-mocks (3.5.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.5.0)
+    rspec-support (3.5.0)
+    rubocop (0.42.0)
+      parser (>= 2.3.1.1, < 3.0)
+      powerpack (~> 0.1)
+      rainbow (>= 1.99.1, < 3.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.0, >= 1.0.1)
+    rubocop-rspec (1.5.3)
+      rubocop (>= 0.42.0)
+    ruby-kafka (0.3.15)
+    ruby-progressbar (1.8.1)
+    ruby_parser (3.8.2)
+      sexp_processor (~> 4.1)
+    rubycritic (2.9.2)
+      colorize
+      flay (= 2.8.0)
+      flog (= 4.4.0)
+      launchy (= 2.4.3)
+      parser (= 2.3.1.2)
+      reek (= 4.1.0)
+      ruby_parser (~> 3.8)
+      virtus (~> 1.0)
+    sexp_processor (4.7.0)
+    shoulda (3.5.0)
+      shoulda-context (~> 1.0, >= 1.0.1)
+      shoulda-matchers (>= 1.4.1, < 3.0)
+    shoulda-context (1.2.1)
+    shoulda-matchers (2.8.0)
+      activesupport (>= 3.0.0)
+    sidekiq (4.2.2)
+      concurrent-ruby (~> 1.0)
+      connection_pool (~> 2.2, >= 2.2.0)
+      rack-protection (~> 1.5)
+      redis (~> 3.2, >= 3.2.1)
+    simplecov (0.12.0)
+      docile (~> 1.1.0)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.0)
+    slop (3.6.0)
+    sysexits (1.2.0)
+    thor (0.19.1)
+    thread_safe (0.3.5)
+    tilt (2.0.5)
+    timecop (0.8.1)
+    timers (4.1.1)
+      hitimes
+    tzinfo (1.2.2)
+      thread_safe (~> 0.1)
+    unicode-display_width (1.1.0)
+    virtus (1.0.5)
+      axiom-types (~> 0.1)
+      coercible (~> 1.0)
+      descendants_tracker (~> 0.0, >= 0.0.3)
+      equalizer (~> 0.0, >= 0.0.9)
+    waterdrop (0.3.2)
+      bundler
+      connection_pool
+      null-logger
+      rake
+      ruby-kafka
+    worker-glass (0.2.2)
+      activesupport
+      null-logger
+    yard (0.9.5)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.2)
+  karafka!
+  polishgeeks-dev-tools
+  timecop
+
+BUNDLED WITH
+   1.12.5