worker_tools 0.1.2 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: f8781038bd01108da06e6c7449cb29faaa45fdd8
-  data.tar.gz: eed78607ccda5d5b0ed7ad5030a42b8dc286fcea
+SHA256:
+  metadata.gz: 2d73219c29b55493f0be76ce047dbe7f72259f4efaab15d1528230bdfc1920b5
+  data.tar.gz: a8ef47af79183aa6e6b465a0cbc6c7396ccccf3cb92574c386b956ebd3415056
 SHA512:
-  metadata.gz: cc5633debc66182adff1af028ae0034937be1d369abbaabdbc731da5e792480db6074b602e951efaf4ec1889d0586fd175b44df4945eff8819d637cdfec171c8
-  data.tar.gz: d6dcb02514729dcf459df1eaee5f40df1a792731336aea56a8b92541418163227b17144174285075e5365f98546c0a2607534c5b62845bee1e0982c94dd9bb39
+  metadata.gz: 30287518cdce7bb9536d802fffdaa54b893bfcc811b03f1c226ea59b45787c0f5b4090b82796984e5c77741679bd035620cd3e38aecb4ebb0f1dcac9de50cb16
+  data.tar.gz: 18c1c3ff22097f710cea838c6cf4a8058c35343d7f76312e063771ef2dc02c6ad0fbdace866c1b3dc9e21ab3545ea91f041ff16f4d540402be601c44c834d0dc

data/.travis.yml

@@ -2,4 +2,5 @@ sudo: false
 language: ruby
 rvm:
   - 2.3.1
-before_install: gem install bundler -v 1.14.6
+  - 2.7.0
+before_install: gem install bundler -v 2.1.4

data/CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+## [1.0.0] - 2022-05-20
+
+Compared to 0.2.1
+
+### New
+
+- Namespaced errors and non failure logic
+- Support for status running and complete_with_warnings
+- Benchmark wrapper
+- Counters wrapper
+- Notes instead of information field
+- Filter for slack errors (`slack_error_notifiable`)
+- Model attachments convention
+- Complete specification of csv open arguments
+
+### BREAKING CHANGES
+
+Instead of writing the final csv or xlsx to a folder, the gem assumes that the model provides an add_attachment method.
+
+Both csv and xlsx output modules use entry hashes for content (`csv_output_entries`, `xlsx_output_entries`). The mapper methods `csv_output_row_values` and `xlsx_output_row_values` do not need (in most cases) to be defined, there is a default now. See the complete examples in the README.
+
+These methods were renamed
+
+- `xlsx_output_values` => `xlsx_output_row_values`
+- `xlsx_insert_headers` => `xlsx_output_insert_headers`
+- `xlsx_insert_rows` => ` xlsx_output_insert_rows`
+- `xlsx_iterators` => `xlsx_output_iterators`
+- `xlsx_style_columns` => `xlsx_output_style_columns`
+- `xlsx_write_sheet` => `xlsx_output_write_sheet`
+
+These methods were removed
+
+- `add_info` in favor of `add_note`
+- `create_model_if_not_available`, a model is always created.
+- `format_log_message`, `format_info_message` in favor of `format_message`
+- `csv_output_target`
+- `cvs_output_target_folder`
+- `csv_output_target_file_name`
+- `csv_ouput_ensure_target_folder`
+- `csv_output_write_target`
+- `xlsx_output_target`
+- `xlsx_output_target_folder`
+- `xlsx_ensure_output_target_folder`
+- `xlsx_write_output_target`
+
+[1.0.0]: https://github.com/i22-digitalagentur/worker-tools/compare/0.2.1...1.0.0

data/README.md

@@ -1,13 +1,53 @@
 # WorkerTools
 
-[![Build Status](https://travis-ci.org/i22-digitalagentur/worker-tools.svg?branch=master)](https://travis-ci.org/i22-digitalagentur/worker-tools)
+[![Build Status][build-badge]][build-url]
+[![MIT License][license-shield]][license-url]
+[![Release][release-shield]][release-url]
+![Maintenance][maintained-shield]
+
+<br>
+
+<details open="open">
+  <summary>Table of Contents</summary>
+  <ol>
+    <li>
+      <a href="#about-the-project">About The Project</a>
+    </li>
+    <li>
+      <a href="#installation">Installation</a>
+    </li>
+    <li><a href="#conventions">Conventions</a></li>
+    <li><a href="#module-basics">Module 'Basics'</a></li>
+    <li><a href="#module-recorder">Module 'Recorder'</a></li>
+    <li><a href="#module-slackerrornotifier">Module 'SlackErrorNotifier'</a></li>
+    <li><a href="#wrappers">Wrappers</a></li>
+    <li><a href="#module-notes">Module 'Notes'</a></li>
+    <li><a href="#attachments">Attachments</a></li>
+    <li>
+      <a href="#complete-examples">Complete Examples</a>
+      <ul>
+        <li><a href="#xlsx-input-example">XLSX Input Example</a></li>
+        <li><a href="#csv-input-example">CSV Input Example</a></li>
+        <li><a href="#csv-output-example">CSV Output Example</a></li>
+        <li><a href="#xlsx-output-example">XLSX Output Example</a></li>
+      </ul>
+    </li>
+    <li><a href="#changelog">Changelog</a></li>
+    <li><a href="#requirements">Requirements</a></li>
+    <li><a href="#contributing">Contributing</a></li>
+    <li><a href="#license">License</a></li>
+    <li><a href="#acknowledgement">Acknowledgement</a></li>
+  </ol>
+</details>
+
+## About The Project
 
 WorkerTools is a collection of modules meant to speed up how we write background tasks following a few basic patterns. The structure of plain independent modules with limited abstraction allows to define and override a few methods according to your needs without requiring a deep investment in the library.
 
 These modules provide some features and conventions to address the following points with little configuration on your part:
 
 - How to save the state the task.
-- How to save information relevant to the admins / customers.
+- How to save notes relevant to the admins / customers.
 - How to log the details
 - How to handle exceptions and send notifications
 - How to process CSV files (as input and output)
@@ -32,13 +72,20 @@ Or install it yourself as:
 
 ## Conventions
 
-Most of the modules require an ActiveRecord model to keep track of the state, information, and files related to the job. The class of this model is typically an Import, Export, Report.. or something more generic like a JobEntry.
+Most of the modules require an ActiveRecord model to keep track of the state, notes, and files related to the job. The class of this model is typically an Import, Export, Report.. or something more generic like a JobEntry.
 
 An example of this model for an Import using Paperclip would be something like this:
 
 ```ruby
 class Import < ApplicationRecord
-  enum state: { waiting: 0, complete: 1, failed: 2 }
+  enum state: %w[
+    waiting
+    complete
+    complete_with_warnings
+    failed
+    running
+  ].map { |e| [e, e] }.to_h
+
   enum kind: { foo: 0, bar: 1 }
 
   has_attached_file :attachment
@@ -48,7 +95,9 @@ class Import < ApplicationRecord
   end
 ```
 
-The state `complete` and `failed` are used by the modules. Both `state` and `kind` could be an enum or just a string field.  Whether you have one, none or many attachments, and which library you use to handle it's up to you.
+The state `complete` and `failed` are used by the modules. Both `state` and `kind` could be an enum or just a string field. Whether you have one, none or many attachments, and which library you use to handle it's up to you.
+
+The state `complete_with_warnings` indicates that the model contains notes that did not lead to a failure but should get some attention. By default those levels are `warning` and `errors` and can be customized.
 
 In this case the migration would be something like this:
 
@@ -56,9 +105,10 @@ In this case the migration would be something like this:
   def change
     create_table :imports do |t|
       t.integer :kind, null: false
-      t.integer :state, default: 0, null: false
-      t.text :information
+      t.string :state, default: 'waiting', null: false
+      t.json :notes, default: []
       t.json  :options, default: {}
+      t.json :meta, default: {}
 
       t.string :attachment_file_name
       t.integer :attachment_file_size
@@ -66,7 +116,7 @@ In this case the migration would be something like this:
 
       t.timestamps
     end
- ```
+```
 
 ## Module 'Basics'
 
@@ -95,12 +145,13 @@ end
 
 The basics module contains a `perform` method, which is the usual entry point for ApplicationJob and Sidekiq. It can receive the id of the model, the model instance, or nothing, in which case it will attempt to create this model on its own.
 
+By default errors subclassed from WorkerTools::Errors::Invalid (such as those related to wrong headers in the input modules) will not raise and mark the model as failed. The method `non_failure_error?` lets you modifiy this behaviour.
 
 ## Module 'Recorder'
 
-Provides some methods to manage a log and the `information` field of the model. The main methods are `add_info`, `add_log`, and `record` (which both logs and appends the message to the information field). See all methods in [recorder](/lib/worker_tools/recorder.rb)
+Provides some methods to manage a log and the `notes` field of the model. The main methods are `add_info`, `add_log`, and `record` (which both logs and appends the message to the notes field). See all methods in [recorder](/lib/worker_tools/recorder.rb)
 
-This module has a _recoder_ wrapper that will register the exception details into the log and information field in case of error:
+This module has a _recoder_ wrapper that will register the exception details into the log and notes field in case of error:
 
 ```ruby
 class MyImporter
@@ -124,30 +175,45 @@ If you only want the logger functions, without worrying about persisting a model
   end
 ```
 
-## Module RocketchatErrorNotifier
+## Module SlackErrorNotifier
 
-[rocketchat_error_notifier](/lib/worker_tools/rocketchat_error_notifier.rb)
+Provides a Slack error notifier wrapper. To do this, you need to define SLACK_NOTIFIER_WEBHOOK as well as SLACK_NOTIFIER_CHANNEL. Then you need to include the SlackErrorNotifier module in your class and append slack_error_notifier to your wrappers. Below you can see an example.
 
-## Module SlackErrorNotifier
+```ruby
+  class MyImporter
+    include WorkerTools::SlackErrorNotifier
+
+    wrappers :slack_error_notifier
+
+    def perform
+      with_wrapper_logger do
+        # do stuff
+      end
+    end
+  end
+```
 
-[slack_error_notifier](/lib/worker_tools/slack_error_notifier.rb)
+See all methods in [slack_error_notifier](/lib/worker_tools/slack_error_notifier.rb)
 
 ## Module CSV Input
 
-[csv_input](/lib/worker_tools/csv_input.rb)
+See all methods in [csv_input](/lib/worker_tools/csv_input.rb)
 
 ## Module CSV Output
 
-[csv_output](/lib/worker_tools/csv_output.rb)
+See all methods in [csv_output](/lib/worker_tools/csv_output.rb)
 
 ## Module XLSX Input
 
-[xlsx_input](/lib/worker_tools/xlsx_input.rb)
+See all methods in [xlsx_input](/lib/worker_tools/xlsx_input.rb)
 
+## Module XLSX Output
+
+See all methods in [xlsx_output](/lib/worker_tools/xlsx_output.rb)
 
 ## Wrappers
 
-In the [basics module](/lib/worker_tools/basics.rb), `perform` calls your custom method `run` to do the actual work of the task, and wraps it around any methods expecting a block that you might have had defined using `wrappers`.  That gives us a systematic way to add logic depending on the output of `run` and any exceptions that might arise, such as logging the error and context, sending a chat notification, retrying under some circumstances, etc.
+In the [basics module](/lib/worker_tools/basics.rb), `perform` calls your custom method `run` to do the actual work of the task, and wraps it around any methods expecting a block that you might have had defined using `wrappers`. That gives us a systematic way to add logic depending on the output of `run` and any exceptions that might arise, such as logging the error and context, sending a chat notification, retrying under some circumstances, etc.
 
 The following code
 
@@ -203,13 +269,285 @@ def perform(model_id)
 end
 ```
 
+## Counter
+
+There is a counter wrapper that you can use to add custom counters to the meta attribute. To do this, you need to complete the following tasks:
+
+- include WorkerTools::Counters to your class
+- add :counters to the wrappers method props
+- call counters method with your custom counters
+  You can see an example below. After that, you can access your custom counters via the meta attribute.
+
+```ruby
+class MyImporter
+  include WorkerTools::Counters
+  wrappers :counters
+  counters :foo, :bar
+
+  def run
+    example_foo_counter_methods
+  end
+
+  def example_foo_counter_methods
+    # you can use the increment helper
+    10.times { increment_foo }
+
+    # the counter works like a regular accessor, you can read it and modify it
+    # directly
+    self.bar = 100
+    puts bar # => 100
+  end
+
+  # ..
+end
+```
+
+## Benchmark
+
+There is a benchmark wrapper that you can use to record the benchmark. The only thing you need to do is to include the benchmark module and append the name to the wrapper array. Below you can see an example of the integration.
+
+```ruby
+class MyImporter
+  include WorkerTools::Benchmark
+  wrappers :benchmark
+
+  def run
+    # do stuff
+  end
+
+  # ..
+end
+```
+
+## Module 'Notes'
+
+If you use ActiveRecord you may need to modify the serializer as well as deserializer from the note attribute. After that you can easily serialize hashes and array of hashes with indifferent access. For that purpose the gem provides two utility methods. (HashWithIndifferentAccessType, SerializedArrayType). There is an example of how you can use it.
+
+```ruby
+  class ServiceTask < ApplicationRecord
+
+    attribute :notes, SerializedArrayType.new(type: HashWithIndifferentAccessType.new)
+  end
+```
+
+See all methods in [utils](/lib/worker_tools/utils)
+
+## Attachments
+
+The modules that generate a file expect the model to provide an `add_attachment` method with following signature:
+
+```ruby
+  def add_attachment(file, file_name: nil, content_type: nil)
+    # your logic
+  end
+```
+
+You can skip this convention by overwriting the module related method, for example after including `CsvOutput`
+
+```ruby
+def csv_output_add_attachment
+  # default implementation
+  # model.add_attachment(csv_output_tmp_file, file_name: csv_output_file_name, content_type: 'text/csv')
+
+  # your method
+  ftp_upload(csv_output_tmp_file)
+end
+```
+
+## Complete Examples
+
+### XLSX Input Example
+
+```ruby
+class XlsxInputExample
+  include Sidekiq::Worker
+  include WorkerTools::Basics
+  include WorkerTools::Recorder
+  include WorkerTools::XlsxInput
+
+  wrappers %i[basics recorder]
+
+  def model_class
+    Import
+  end
+
+  def model_kind
+    'xlsx_input_example'
+  end
+
+  def run
+    xlsx_input_foreach.each { |row| SomeModel.create!(row) }
+  end
+
+  def xlsx_input_columns
+    {
+      foo: 'Your Foo',
+      bar: 'Your Bar'
+    }
+  end
+end
+```
+
+### CSV Input Example
+
+```ruby
+class CsvInputExample
+  include Sidekiq::Worker
+  include WorkerTools::Basics
+  include WorkerTools::Recorder
+  include WorkerTools::CsvInput
+
+  wrappers %i[basics recorder]
+
+  def model_class
+    Import
+  end
+
+  def model_kind
+    'csv_input_example'
+  end
+
+  def csv_input_columns
+    {
+      flavour: 'Flavour',
+      number: 'Number'
+    }
+  end
+
+  def run
+    csv_input_foreach.map { |row| do_something row_to_attributes(row) }
+  end
+
+  def row_to_attributes(row)
+    {
+      flavour: row['flavour'].downcase,
+      number: row['number'].to_i * 10
+    }
+  end
+end
+```
+
+### CSV Output Example
+
+```ruby
+# More complex example with CsvOutput
+class CsvOutputExample
+  include Sidekiq::Worker
+  include WorkerTools::Basics
+  include WorkerTools::CsvOutput
+  include WorkerTools::Recorder
+
+  wrappers %i[basics recorder]
+
+  def model_class
+    Report
+  end
+
+  def model_kind
+    'csv_out_example'
+  end
+
+  def model_file_name
+    "#{model_kind}-#{Date.current}.csv"
+  end
+
+  def run
+    csv_output_write_file
+  end
+
+  def csv_output_column_headers
+    @csv_output_column_headers ||= {
+      foo: 'Foo',
+      bar: 'Bar'
+    }
+  end
+
+  def csv_output_entries
+    @csv_output_entries ||= User.includes(...).lazy.map do |user|
+      {
+        foo: user.foo,
+        bar: user.bar
+      }
+    end
+  end
+
+end
+```
+
+### XLSX Output Example
+
+```ruby
+# ExampleXlsxOutput
+class XlsxOutputExample
+  include Sidekiq::Worker
+  include WorkerTools::Basics
+  include WorkerTools::Recorder
+  include WorkerTools::XlsxOutput
+
+  wrappers %i[basics recorder]
+
+  def model_class
+    Export
+  end
+
+  def model_kind
+    'xlsx_output_example'
+  end
+
+  def run
+    xlsx_output_write_file
+  end
+
+  def xlsx_output_column_headers
+    @xlsx_output_column_headers ||= {
+      foo: 'Foo',
+      bar: 'Bar'
+    }
+  end
+
+  def xlsx_output_entries
+    @xlsx_output_entries ||= SomeArray.map do |entry|
+      {
+        foo: user.foo,
+        bar: user.bar
+      }
+    end
+  end
+end
+```
+
+## Changelog
+
+See [CHANGELOG](CHANGELOG.md)
+
+## Requirements
+
+- ruby > 2.3.1
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/i22-digitalagentur/worker_tools.
+Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.
 
+1. Fork the Project
+2. Create your Feature Branch (`git checkout -b feature/new_feature`)
+3. Commit your Changes (`git commit -m 'feat: Add new feature'`)
+4. Push to the Branch (`git push origin feature/new_feature`)
+5. Open a Pull Request
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+The gem is available under the MIT License. See `LICENSE` for more information.
+
+## Acknowledgement
+
+- [Img Shields](https://shields.io)
+
+<!--shield-styles-->
 
+[build-badge]: https://travis-ci.org/i22-digitalagentur/worker-tools.svg?branch=master
+[build-url]: https://travis-ci.org/i22-digitalagentur/worker-tools
+[maintained-shield]: https://img.shields.io/badge/Maintained%3F-yes-green.svg?style=flat
+[release-shield]: https://img.shields.io/github/release/i22-digitalagentur/coverage-badge-creator.svg?style=flat
+[release-url]: https://github.com/i22-digitalagentur/worker-tools/releases/
+[license-shield]: https://img.shields.io/badge/License-MIT-yellow.svg?style=flat
+[license-url]: https://github.com/i22-digitalagentur/worker-tools/blob/master/LICENSE

data/lib/worker_tools/basics.rb

@@ -6,7 +6,6 @@ module WorkerTools
 
     included do
       attr_writer :model
-      attr_accessor :information
 
       def self.wrappers(*args)
         @wrappers ||= args.flatten
@@ -33,6 +32,7 @@ module WorkerTools
 
     def perform(model_id = nil)
       @model_id = model_id
+
       with_wrappers(wrapper_methods) do
         run
       end
@@ -42,32 +42,36 @@ module WorkerTools
       self.class.read_wrappers.map do |wrapper|
         symbolized_method = "with_wrapper_#{wrapper}".to_sym
         raise "Missing wrapper #{wrapper}" unless respond_to?(symbolized_method)
+
         symbolized_method
       end
     end
 
     def with_wrapper_basics(&block)
+      save_state_without_validate('running')
       block.yield
       finalize
     # this time we do want to catch Exception to attempt to handle some of the
     # critical errors.
     # rubocop:disable Lint/RescueException
-    rescue Exception
+    rescue Exception => e
+      return finalize if non_failure_error?(e)
+
       # rubocop:enable Lint/RescueException
-      model.state = 'failed'
-      model.save!(validate: false)
+      save_state_without_validate('failed')
       raise
     end
 
     def finalize
-      model.update_attributes!(
-        state: 'complete',
-        information: information
-      )
+      mark_with_warnings = model.notes.any? do |note|
+        complete_with_warnings_note_levels.include?(note.with_indifferent_access[:level].to_s)
+      end
+
+      model.update!(state: mark_with_warnings ? :complete_with_warnings : :complete)
     end
 
-    def create_model_if_not_available
-      false
+    def complete_with_warnings_note_levels
+      %w[error warning]
     end
 
     def model
@@ -76,17 +80,29 @@ module WorkerTools
 
     def with_wrappers(wrapper_symbols, &block)
       return yield if wrapper_symbols.blank?
+
       current_wrapper_symbol = wrapper_symbols.shift
       send(current_wrapper_symbol) { with_wrappers(wrapper_symbols, &block) }
     end
 
+    def non_failure_error?(error)
+      error.is_a?(WorkerTools::Errors::Invalid)
+      # or add your list
+      # [WorkerTools::Errors::Invalid, SomeOtherError].any? { |k| e.is_a?(k) }
+    end
+
     private
 
+    def save_state_without_validate(state)
+      model.state = state
+      model.save!(validate: false)
+    end
+
     def find_model
       @model_id ||= nil
       return @model_id if @model_id.is_a?(model_class)
       return model_class.find(@model_id) if @model_id
-      raise 'Model not available' unless create_model_if_not_available
+
       t = model_class.new
       t.kind = model_kind if t.respond_to?(:kind=)
       t.save!(validate: false)

data/lib/worker_tools/benchmark.rb

@@ -0,0 +1,15 @@
+module WorkerTools
+  module Benchmark
+    extend ActiveSupport::Concern
+
+    included do
+      attr_accessor :benchmark
+
+      def with_wrapper_benchmark(&block)
+        @benchmark = ::Benchmark.measure(&block)
+
+        model.meta['duration'] = @benchmark.real.round if model.respond_to?(:meta)
+      end
+    end
+  end
+end

data/lib/worker_tools/counters.rb

@@ -0,0 +1,40 @@
+module WorkerTools
+  module Counters
+    extend ActiveSupport::Concern
+
+    included do
+      def self.counters(*args)
+        @counters ||= args.flatten
+        add_counter_methods
+      end
+
+      def self.read_counters
+        @counters || []
+      end
+
+      def self.add_counter_methods
+        @counters.each do |name|
+          # ex `inserts`
+          define_method(name) { model.meta[name] }
+
+          # ex `inserts=`
+          define_method("#{name}=") { |value| model.meta[name] = value }
+
+          # ex `increment_inserts`
+          define_method("increment_#{name}") { model.meta[name] += 1 }
+        end
+      end
+
+      def with_wrapper_counters(&block)
+        reset_counters
+        block.call
+      end
+
+      def reset_counters
+        self.class.read_counters.each do |name|
+          model.meta[name] = 0
+        end
+      end
+    end
+  end
+end