job-iteration 0.9.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e042aee693c7b308e483fd6276a9966894dacf75
+  data.tar.gz: bd8fe09135913110159faaaa7498e0786877650a
+SHA512:
+  metadata.gz: ef8ca88e85af8844864511165062f1dc75d23b34d8161849ae7db55ddb9f8e70bf53c58cb96465800cb82c717eb34b945c6a55a6f8d8e88861c661a302723f69
+  data.tar.gz: 2854303407e9d67bbc727970b0ee979f74a63917039ffa356efba86ed6b8f3b4eb29a42d56735e9516404d203cd890af8850f84e979f83ff1a581aafb76819d8

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+.ruby-version
+.rubocop-http---shopify-github-io-ruby-style-guide-rubocop-yml

data/.rubocop.yml

@@ -0,0 +1,13 @@
+inherit_from:
+  - http://shopify.github.io/ruby-style-guide/rubocop.yml
+
+AllCops:
+  TargetRubyVersion: 2.4.4
+  Exclude:
+    - 'vendor/bundle/**/*'
+Lint/HandleExceptions:
+  Exclude:
+    - lib/job-iteration.rb
+Style/GlobalVars:
+  Exclude:
+    - lib/job-iteration/integrations/resque.rb

data/.travis.yml

@@ -0,0 +1,14 @@
+services:
+  - mysql
+  - redis-server
+language: ruby
+rvm:
+  - 2.4.4
+  - 2.5.1
+before_install:
+  - gem install bundler -v 1.16.0
+  - mysql -e 'CREATE DATABASE job_iteration_test;'
+script:
+  - bundle exec rake test
+  - bundle exec rubocop
+

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at shatrov@me.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in job-iteration.gemspec
+gemspec
+
+# for integration testing
+gem 'sidekiq'
+gem 'resque'
+
+gem 'activerecord'
+gem 'mysql2', '~> 0.4.4'
+gem 'globalid'
+gem 'i18n'
+gem 'redis'
+gem 'database_cleaner'
+
+gem 'pry'
+gem 'mocha'
+
+gem 'rubocop'

data/Gemfile.lock

@@ -0,0 +1,113 @@
+PATH
+  remote: .
+  specs:
+    job-iteration (0.9.0)
+      activejob (~> 5.2)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activejob (5.2.0)
+      activesupport (= 5.2.0)
+      globalid (>= 0.3.6)
+    activemodel (5.2.0)
+      activesupport (= 5.2.0)
+    activerecord (5.2.0)
+      activemodel (= 5.2.0)
+      activesupport (= 5.2.0)
+      arel (>= 9.0)
+    activesupport (5.2.0)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 0.7, < 2)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+    arel (9.0.0)
+    ast (2.4.0)
+    coderay (1.1.2)
+    concurrent-ruby (1.0.5)
+    connection_pool (2.2.2)
+    database_cleaner (1.7.0)
+    globalid (0.4.1)
+      activesupport (>= 4.2.0)
+    i18n (1.0.1)
+      concurrent-ruby (~> 1.0)
+    jaro_winkler (1.5.1)
+    metaclass (0.0.4)
+    method_source (0.9.0)
+    minitest (5.11.3)
+    mocha (1.5.0)
+      metaclass (~> 0.0.1)
+    mono_logger (1.1.0)
+    multi_json (1.13.1)
+    mustermann (1.0.2)
+    mysql2 (0.4.10)
+    parallel (1.12.1)
+    parser (2.5.1.0)
+      ast (~> 2.4.0)
+    powerpack (0.1.2)
+    pry (0.11.3)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    rack (2.0.5)
+    rack-protection (2.0.3)
+      rack
+    rainbow (3.0.0)
+    rake (10.5.0)
+    redis (4.0.1)
+    redis-namespace (1.6.0)
+      redis (>= 3.0.4)
+    resque (1.27.4)
+      mono_logger (~> 1.0)
+      multi_json (~> 1.0)
+      redis-namespace (~> 1.3)
+      sinatra (>= 0.9.2)
+      vegas (~> 0.1.2)
+    rubocop (0.57.2)
+      jaro_winkler (~> 1.5.1)
+      parallel (~> 1.10)
+      parser (>= 2.5)
+      powerpack (~> 0.1)
+      rainbow (>= 2.2.2, < 4.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.0, >= 1.0.1)
+    ruby-progressbar (1.9.0)
+    sidekiq (5.1.3)
+      concurrent-ruby (~> 1.0)
+      connection_pool (~> 2.2, >= 2.2.0)
+      rack-protection (>= 1.5.0)
+      redis (>= 3.3.5, < 5)
+    sinatra (2.0.3)
+      mustermann (~> 1.0)
+      rack (~> 2.0)
+      rack-protection (= 2.0.3)
+      tilt (~> 2.0)
+    thread_safe (0.3.6)
+    tilt (2.0.8)
+    tzinfo (1.2.5)
+      thread_safe (~> 0.1)
+    unicode-display_width (1.4.0)
+    vegas (0.1.11)
+      rack (>= 1.0.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activerecord
+  bundler (~> 1.16)
+  database_cleaner
+  globalid
+  i18n
+  job-iteration!
+  minitest (~> 5.0)
+  mocha
+  mysql2 (~> 0.4.4)
+  pry
+  rake (~> 10.0)
+  redis
+  resque
+  rubocop
+  sidekiq
+
+BUNDLED WITH
+   1.16.1

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Shopify
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,191 @@
+# Job Iteration API
+
+Meet Iteration, an extension for [ActiveJob](https://github.com/rails/rails/tree/master/activejob) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
+
+## Background
+
+Imagine the following job:
+
+```ruby
+class SimpleJob < ActiveJob::Base
+  def perform
+    User.find_each do |user|
+      user.notify_about_something
+    end
+  end
+end
+```
+
+The job would run fairly quickly when you only have a hundred User records. But as the number of records grows, it will take longer for a job to iterate over all Users. Eventually, there will be millions of records to iterate and the job will end up taking hours and days.
+
+With frequent deploys and worker restarts, it would mean that a job will be either lost of started from the beginning. Some records (especially those in the beginning of the relation) will be processed more than once.
+
+Cloud environments are also unpredictable, and there's no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and will restart it in 5 minutes? What if Kubernetes pod is getting [evicted](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)? Again, all job progress will be lost.
+
+Software that is designed for high availability [must be friendly](https://12factor.net/disposability) to interruptions that come from the infrastructure. That's exactly what Iteration brings to ActiveJob. It's been developed at Shopify to safely process long-running jobs, in Cloud, and has been working in production since May 2017.
+
+We recommend you to watch a [conference talk](https://www.youtube.com/watch?v=XvnWjsmAl60) about the ideas and history behind Iteration API.
+
+## Getting started
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'job-iteration'
+```
+
+And then execute:
+
+    $ bundle
+
+In the job, include `JobIteration::Iteration` module and start describing the job with two methods (`build_enumerator` and `each_iteration`) instead of `perform`:
+
+```ruby
+class NotifyUsersJob < ActiveJob::Base
+  include JobIteration::Iteration
+
+  def build_enumerator(cursor:)
+    enumerator_builder.active_record_on_records(
+      User.all,
+      cursor: cursor,
+    )
+  end
+
+  def each_iteration(user)
+    user.notify_about_something
+  end
+end
+```
+
+`each_iteration` will be called for each `User` model in `User.all` relation. The relation will be ordered by primary key, exactly like `find_each` does.
+
+Check out more examples of Iterations:
+
+```ruby
+class BatchesJob < ActiveJob::Iteration
+  def build_enumerator(product_id, cursor:)
+    enumerator_builder.active_record_on_batches(
+      Product.find(product_id).comments,
+      cursor: cursor,
+      batch_size: 100,
+    )
+  end
+
+  def each_iteration(batch_of_comments, product_id)
+    # batch_of_comments will contain batches of 100 records
+    Comment.where(id: batch_of_comments.map(&:id)).update_all(deleted: true)
+  end
+end
+```
+
+```ruby
+class ArrayJob < ActiveJob::Iteration
+  def build_enumerator(cursor:)
+    enumerator_builder.array(['build', 'enumerator', 'from', 'any', 'array'], cursor: cursor)
+  end
+
+  def each_iteration(array_element)
+    # use array_element
+  end
+end
+```
+
+```ruby
+class CsvJob < ActiveJob::Iteration
+  def build_enumerator(import_id, cursor:)
+    import = Import.find(import_id)
+    JobIteration::CsvEnumerator.new(import.csv).rows(cursor: cursor)
+  end
+
+  def each_iteration(csv_row)
+    # insert csv_row to database
+  end
+end
+```
+
+## Guides
+
+* [Iteration: how it works](guides/iteration-how-it-works.md).
+* [Best practices](guides/best-practices.md).
+
+## Requirements
+
+ActiveJob is the primary requirement for Iteration. While there's nothing that prevents it, Iteration is not yet compatible with [vanilla](https://github.com/mperham/sidekiq/wiki/Active-Job) Sidekiq API.
+
+### API
+
+Iteration job must respond to `build_enumerator` and `each_iteration` methods. `build_enumerator` must return [Enumerator](http://ruby-doc.org/core-2.5.1/Enumerator.html) object that respects the `cursor` value.
+
+### Sidekiq adapter
+
+Unless you are running on Heroku, we recommend you to tune Sidekiq's [timeout](https://github.com/mperham/sidekiq/wiki/Deployment#overview) option from the default 8 seconds to 25-30 seconds, to allow the last `each_iteration` to complete and gracefully shutdown.
+
+### Resque adapter
+
+There a few configuration assumptions that are required for Iteration to work with Resque. `GRACEFUL_TERM` must be enabled (giving the job ability to gracefully interrupt), and `FORK_PER_JOB` is recommended to be disabled (set to `false`).
+
+## FAQ
+
+**Why can't I just iterate in `#perform` method and do whatever I want?** You can, but then your job has to comply with a long list of requirements, such as the ones above. This creates leaky abstractions more easily, when instead we can expose a more powerful abstraction for developers--without exposing the underlying infrastructure.
+
+**What happens when my job is interrupted?** A checkpoint will be persisted to Redis after the current `each_iteration`, and the job will be re-enqueued. Once it's popped off the queue, the worker will work off from the next iteration.
+
+**What happens with retries?** An interruption of a job does not count as a retry. The iteration of job that caused the job to fail will be retried and progress will continue from there on.
+
+**What happens if my iteration takes a long time?** We recommend that a single `each_iteration` should take no longer than 30 seconds. In the future, this may raise.
+
+**Why is it important that `each_iteration` takes less than 30 seconds?** When the job worker is scheduled for restart or shutdown, it gets a notice to finish remaining unit of work. To guarantee that no progress is lost we need to make sure that `each_iteration` completes within a reasonable amount of time.
+
+**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job. We may expose primitives in the future to do this more effectively, but this is not terribly common today. We recommend to read https://goo.gl/UobaaU to learn more about nested operations.
+
+**Why do I use have to use this ugly helper in `build_enumerator`? Why can't you automatically infer it?** This is how the first version of the API worked. We checked the type of object returned by `build_enumerable`, and whether it was ActiveRecord Relation or an Array, we used the matching adapter. This caused opaque type branching in Iteration internals and it didn’t allow developers to craft their own Enumerators and control the cursor value. We made a decision to _always_ return Enumerator instance from `build_enumerator`. Now we provide explicit helpers to convert ActiveRecord Relation or an Array to Enumerator, and for more complex iteration flows developers can build their own `Enumerator` objects.
+
+**What is the difference between Enumerable and Enumerator?** We recomend [this post](http://blog.arkency.com/2014/01/ruby-to-enum-for-enumerator/) to learn more about Enumerators in Ruby.
+
+**My job has a complex flow. How do I write my own Enumerator?** Iteration API takes care of persisting the cursor (that you may use to calculate an offset) and controlling the job state. The power of Enumerator object is that you can use the cursor in any way you want. One example is a cursorless job that pops records from a datastore until the job is interrupted:
+
+```ruby
+class MyJob < ActiveJob::Base
+  include JobIteration::Iteration
+
+  def build_enumerator(cursor:)
+    Enumerator.new do
+      Redis.lpop("mylist") # or: Kafka.poll(timeout: 10.seconds)
+    end
+  end
+
+  def each_iteration(element_from_redis)
+    # ...
+  end
+end
+```
+
+## Credits
+
+This project would not be possible without these individuals (in alphabetical order):
+
+* Daniella Niyonkuru
+* Emil Stolarsky
+* Florian Weingarten
+* Guillaume Malette
+* Hormoz Kheradmand
+* Mohamed-Adam Chaieb
+* Simon Eskildsen
+
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies. Then, run `bundle exec rake test` to run the tests.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/Shopify/job-iteration. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Job::Iteration project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/kirs/job-iteration/blob/master/CODE_OF_CONDUCT.md).