carrierwave_backgrounder 0.4.3 → 1.0.0.beta.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c08d7bdb01034545507f0ef13305aa12764e2771148c4ddc22f1291fbba8f7f9
-  data.tar.gz: 21d4f4a1cd200c5b05f9b967d4fda436e7d11752ba6fb6307f619a7b32ae1924
+  metadata.gz: 29bf0a9e2454432c72a9799addeb1f6f059fb910766d946bb999d8b616d9d078
+  data.tar.gz: 9ab201113526a8c1cb9529f5361b6023349f7f73959ad13c075c7468d68a86c1
 SHA512:
-  metadata.gz: 683938ec5318b2727ae40e177a717fc6135f479f397cbdd39e0b3639a70ff3508ef98406ced8fd203b6d34d4017a84d2a2eab64f164d69bd8d24c64e2ff1786c
-  data.tar.gz: 44b9e651e679fa450dfbe9620e58e5472c0be9bc20bd7f4ce29894de28c30c7f6d5c04f7dd9aa56fc4277bd8d0a3ff0205aa287fb31439e8bfaa5d6158ccbe0b
+  metadata.gz: f7dee9bd5893f749599fb804e40400f7d4b54cba1e0ea295962a9442cfbcb02b1d08fd643096d8d242adc2911e65cfcccb97e4f7ef23922d298cada2a2ec8575
+  data.tar.gz: e9465ab71a2fe0fc263121f9fc0aee60475cfe4a0f0bec2675d9194abceeaa6372272ed9bacf3867a671b7d4169c5bd51ab414897e06173c52f7aa3e67f700d8

data/.gitignore

@@ -1,5 +1,6 @@
 *.gem
 .bundle
+*.sqlite3
 Gemfile.lock
 pkg/*
-vendor/*
+vendor/*

data/.travis.yml

@@ -1,12 +1,16 @@
 language: ruby
 
 rvm:
- - 2.3
- - 2.4
- - 2.5
- - 2.6
+ - 2.7
+ - 3.0
+ - 3.1
+ - 3.2
  - ruby-head
 
+env:
+- QUEUE_ADAPTER='sidekiq'
+- QUEUE_ADAPTER='active_job'
+
 before_install:
  - yes | gem update --system --force
  - gem install bundler

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+## 1.0.0-beta
+
+### enhancements
+  * Add native support for ActiveJob [lardawge]
+  * Add support for multi-upload [lardawge]
+  * Add rails app for testing so we can replicate real world conditions [lardawge]
+
+### bugfixes
+  * Fix issue where a new job would be queued when a upload was removed
+
+### breaking changes
+  * Remove support for Rails prior to after_commit
+  * Remove support for queueing systems other than ActiveJob and Sidekiq.
+    This will simplify support and offload the heavy lifting to ActiveJob
 
 ## 0.4.3
 

data/Gemfile

@@ -1,4 +1,4 @@
 source "http://rubygems.org"
 
 # Specify your gem's dependencies in carrierwave_backgrounder.gemspec
-gemspec
+gemspec

data/README.md

@@ -1,12 +1,17 @@
 # CarrierWave Backgrounder
 
-[![Build Status](https://secure.travis-ci.org/lardawge/carrierwave_backgrounder.png)](http://travis-ci.org/lardawge/carrierwave_backgrounder)
+[![Build Status](https://app.travis-ci.com/lardawge/carrierwave_backgrounder.svg?branch=master)](https://app.travis-ci.com/lardawge/carrierwave_backgrounder)
 [![Code Climate](https://codeclimate.com/github/lardawge/carrierwave_backgrounder.png)](https://codeclimate.com/github/lardawge/carrierwave_backgrounder)
 
-I like CarrierWave. That being said, I don't like tying up app instances waiting for images to process.
+NOTICE: Version 1.0.0 contains breaking changes if you are coming from an earlier version.
+The most notible change is the removal of queue backend options other than active_job and sidekiq.
+If you are using other backends, switch over to active_job which should support your preference.
+If you are using Sidekiq, there is nothing to change.
 
-This gem addresses that by offloading processing or storage/processing to a background task.
-We currently support Delayed Job, Resque, Sidekiq, SuckerPunch, Girl Friday, Qu, and Queue Classic.
+I am a fan of CarrierWave. That being said, I don't like tying up requests waiting for images to process.
+
+This gem addresses that by offloading processing or storaging/processing to a background task.
+We currently support ActiveJob and Sidekiq.
 
 ## Background options
 
@@ -15,18 +20,17 @@ There are currently two offerings for backgrounding upload tasks which are as fo
 ```ruby
 # This stores the original file with no processing/versioning.
 # It will upload the original file to s3.
-# This was developed to use where you do not have control over the cache location such as Heroku.
+# This was developed to use in cases where you do not have access to the cached location such as Heroku.
 
 Backgrounder::ORM::Base::process_in_background
 ```
 
 ```ruby
-# This does nothing to the file after it is cached which makes it super fast.
+# This does nothing to the file after it is cached which makes it fast.
 # It requires a column in the database which stores the cache location set by carrierwave so the background job can access it.
 # The drawback to using this method is the need for a central location to store the cached files.
-# Heroku may deploy workers on separate servers from where your dyno cached the files.
 #
-# IMPORTANT: Only use this method if you have full control over your tmp storage directory.
+# IMPORTANT: Only use this method if you have full control over your tmp storage directory and can mount it on every deployed server.
 
 Backgrounder::ORM::Base::store_in_background
 ```
@@ -35,6 +39,8 @@ Backgrounder::ORM::Base::store_in_background
 
 These instructions assume you have previously set up [CarrierWave](https://github.com/jnicklas/carrierwave) and your queuing lib of choice.
 
+Take a look at the [Rails app](spec/support/dummy_app) to see examples of setup.
+
 In Rails, add the following your Gemfile:
 
 ```ruby
@@ -43,31 +49,41 @@ gem 'carrierwave_backgrounder'
 
 Run the generator which will create an initializer in config/initializers.
 ```bash
-  rails g carrierwave_backgrounder:install
+rails g carrierwave_backgrounder:install
 ```
 
-You can pass additional configuration options to Girl Friday and Sidekiq:
+You can pass additional configuration options to Sidekiq:
 
 ```ruby
 CarrierWave::Backgrounder.configure do |c|
-  c.backend :girl_friday, queue: :awesome_queue, size: 3, store: GirlFriday::Store::Redis
+  c.backend :sidekiq, queue: :awesome_queue, size: 3
 end
 ```
 
-**IMPORTANT FOR SIDEKIQ BACKEND** - Custom queue should be defined inside the Sidekiq configuration otherwise jobs won't be processed:
+**IMPORTANT FOR SIDEKIQ BACKEND** - carrierwave (default queue name) should be added to your queue list or it will not run:
 
 ```yml
 :queues:
-  - [awesome_queue, 1]
+  - [carrierwave, 1]
   - default
 ```
 
-In your CarrierWave uploader file:
+In your CarrierWave uploader file you will need to add a cache directory as well as change cache_storage to File:
 
 ```ruby
 class AvatarUploader < CarrierWave::Uploader::Base
   include ::CarrierWave::Backgrounder::Delay
 
+  # This is required if you are using S3 or any other remote storage.
+  # CarrierWave's default is to store the cached file remotely which is slow and uses bandwidth.
+  # By setting this to File, it will only store on saving of the record.
+  cache_storage CarrierWave::Storage::File
+
+  # It is recommended to set this to a persisted location if you are using `::store_in_background`.
+  def cache_dir
+    "path/that/persists"
+  end
+
   #etc...
 end
 ```
@@ -82,7 +98,9 @@ process_in_background :avatar
 ```
 
 Optionally you can add a column to the database which will be set to `true` when
-the background processing is started and to `false` when the background processing is complete.
+the background processing has started and to `false` when the background processing is complete.
+
+This is set to true in the after_commit hook when the job is created. It is very useful if you are waiting to notify the user of completion.
 
 ```ruby
 add_column :users, :avatar_processing, :boolean, null: false, default: false
@@ -97,7 +115,7 @@ mount_uploader :avatar, AvatarUploader
 store_in_background :avatar
 ```
 
-Add a column to the model you want to background which will store the temp file location:
+Add a column to the model which will store the temp file location:
 
 ```ruby
 add_column :users, :avatar_tmp, :string
@@ -115,23 +133,37 @@ If you need to process/store the upload immediately:
 This must be set before you assign an upload:
 
 ```ruby
-# In a controller
 @user = User.new
 @user.process_avatar_upload = true
 @user.attributes = params[:user]
 ```
 
 ### Override worker
-To override the worker in cases where additional methods need to be called or you have app specific requirements, pass the worker class as the
-second argument:
+To override the worker in cases where additional methods need to be called or you have app specific requirements, pass the worker class as the second argument:
 
 ```ruby
 process_in_background :avatar, MyParanoidWorker
 ```
 
-Then create a worker that subclasses carrierwave_backgrounder's worker:
+Then create a worker that subclasses carrierwave_backgrounder's worker.
+Each method, #store_in_background and #process_in_background has there own worker.
+
+#### For Sidekiq
+`process_in_background` subclass `::CarrierWave::Workers::ProcessAsset`
+`store_in_background` subclass `::CarrierWave::Workers::StoreAsset`
+
+#### For ActiveJob
+`process_in_background` subclass `::CarrierWave::Workers::ActiveJob::ProcessAsset`
+`store_in_background` subclass `::CarrierWave::Workers::ActiveJob::StoreAsset`
 
 ```ruby
+# Sidekiq Example
+
+class User < ActiveRecord::Base
+  mount_uploader :avatar, AvatarUploader
+  process_in_background, :avatar, MyParanoidWorker
+end
+
 class MyParanoidWorker < ::CarrierWave::Workers::ProcessAsset
   # ...or subclass CarrierWave::Workers::StoreAsset if you're using store_in_background
 
@@ -143,29 +175,18 @@ class MyParanoidWorker < ::CarrierWave::Workers::ProcessAsset
 end
 ```
 
-### ActiveJob
-Use overriden worker that inherits from ActiveJob::Base and includes relevant worker mixin:
 ```ruby
-class MyActiveJobWorker < ActiveJob::Base
-  include ::CarrierWave::Workers::ProcessAssetMixin
-  # ... or include ::CarrierWave::Workers::StoreAssetMixin
+# ActiveJob Example
+
+class User < ActiveRecord::Base
+  mount_uploader :avatar, AvatarUploader
+  process_in_background, :avatar, MyActiveJobWorker
+end
 
+class MyActiveJobWorker < ::CarrierWave::Workers::ActiveJob::StoreAsset
   after_perform do
     # your code here
   end
-
-  # Sometimes job gets performed before the file is uploaded and ready.
-  # You can define how to handle that case by overriding `when_not_ready` method
-  # (by default it does nothing)
-  def when_not_ready
-    retry_job
-  end
-end
-```
-Don't forget to set `active_job` as a backend in the config:
-```ruby
-CarrierWave::Backgrounder.configure do |c|
-  c.backend :active_job, queue: :carrierwave
 end
 ```
 
@@ -174,35 +195,6 @@ We use the after_commit hook when using active_record. This creates a problem wh
 if you're using transactional fixtures. One solution to the problem is to use the [TestAfterCommit gem](https://github.com/grosser/test_after_commit).
 There are various other solutions in which case google is your friend.
 
-### Uploaders mounted on mongoid embedded documents
-The workers fetch the document with the mounted uploader using the model class name and id. Uploads on embedded documents
-cannot be obtained this way. If the position of the document in the root document structure is known, a workaround is to override the embedded models
-find method like this:
-
-```ruby
-class SomeRootDocument
-  include Mongoid::Document
-
-  embeds_many :embedded_documents
-end
-
-class EmbeddedDocument
-  include Mongoid::Document
-
-  embedded_in :some_root_document
-
-  mount_uploader :image, ImageUploader
-  process_in_background :image
-
-  def self.find(id)
-    bson_id = Moped::BSON::ObjectId.from_string(id) # needed for Mongoid 3
-
-    root = SomeRootDocument.where('embedded_documents._id' => bson_id).first
-    root.embedded_documents.find(id)
-  end
-end
-```
-
 ## License
 
 Copyright (c) 2011 Larry Sprock

data/carrierwave_backgrounder.gemspec

@@ -17,8 +17,13 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_dependency "carrierwave", [">= 0.5", "< 2.2"]
+  s.add_dependency "carrierwave", ["> 2.0", "< 4.0"]
+  s.add_dependency "rails", ["> 6.0", "< 8.0"]
 
-  s.add_development_dependency "rspec", ["~> 3.10.0"]
+  s.add_development_dependency "rspec", ["~> 3.12"]
   s.add_development_dependency "rake"
+  s.add_development_dependency "rspec-rails"
+  s.add_development_dependency "sqlite3"
+  s.add_development_dependency "sidekiq"
+  s.add_development_dependency "pry"
 end

data/lib/backgrounder/orm/activemodel.rb

@@ -10,8 +10,8 @@ module CarrierWave
         private
 
         def _define_shared_backgrounder_methods(mod, column, worker)
-          before_save :"set_#{column}_processing", :if => :"enqueue_#{column}_background_job?"
-          send _supported_callback, :"enqueue_#{column}_background_job", :if => :"enqueue_#{column}_background_job?"
+          before_save :"set_#{column}_processing", if: :"enqueue_#{column}_background_job?"
+          after_commit :"enqueue_#{column}_background_job", if: :"enqueue_#{column}_background_job?"
 
           super
 
@@ -19,15 +19,15 @@ module CarrierWave
             options = self.class.uploader_options[column] || {}
             serialization_column = options[:mount_on] || column
 
-            send(:"#{serialization_column}_changed?") ||              # after_save support
             previous_changes.has_key?(:"#{serialization_column}") ||  # after_commit support
-            send(:"remote_#{column}_url").present? ||                 # Remote upload support
+            remote_url_present? ||                                    # Remote upload support
             send(:"#{column}_cache").present?                         # Form failure support
           end
-        end
 
-        def _supported_callback
-          respond_to?(:after_commit) ? :after_commit : :after_save
+          define_method :remote_url_present? do
+            !!(send(:"remote_#{column}_url").present? if respond_to?(:"remote_#{column}_url")) ||  # Remote upload support for a single file
+              !!(send(:"remote_#{column}_urls").present? if respond_to?(:"remote_#{column}_urls"))   # Remote upload support for multiple files
+          end
         end
       end # ActiveModel
 

data/lib/backgrounder/orm/base.rb

@@ -35,16 +35,17 @@ module CarrierWave
         # which can be used to check if processing is complete.
         #
         #   def self.up
-        #     add_column :users, :avatar_processing, :boolean
+        #     add_column :users, :avatar_processing, :boolean, null: false, default: false
         #   end
         #
-        def process_in_background(column, worker=::CarrierWave::Workers::ProcessAsset)
+        def process_in_background(column, worker_klass=nil)
           attr_accessor :"process_#{column}_upload"
 
+          worker = worker_klass || "#{CarrierWave::Backgrounder.worker_klass}::ProcessAsset"
           mod = Module.new
           include mod
 
-          _define_shared_backgrounder_methods(mod, column, worker)
+          _define_shared_backgrounder_methods(mod, column, worker )
         end
 
         ##
@@ -69,19 +70,16 @@ module CarrierWave
         #     store_in_background :avatar, CustomWorker
         #   end
         #
-        def store_in_background(column, worker=::CarrierWave::Workers::StoreAsset)
+        def store_in_background(column, worker_klass=nil)
           attr_accessor :"process_#{column}_upload"
 
+          worker = worker_klass || "#{CarrierWave::Backgrounder.worker_klass}::StoreAsset"
+
           mod = Module.new
           include mod
           mod.class_eval  <<-RUBY, __FILE__, __LINE__ + 1
-            def remove_#{column}=(value)
-              super
-              self.process_#{column}_upload = true
-            end
-
             def write_#{column}_identifier
-              super and return if process_#{column}_upload
+              super and return if process_#{column}_upload || remove_#{column}
               self.#{column}_tmp = #{column}_cache if #{column}_cache
             end
 
@@ -105,11 +103,19 @@ module CarrierWave
             end
 
             def enqueue_#{column}_background_job?
-              !remove_#{column}? && !process_#{column}_upload && #{column}_updated?
+              !remove_#{column}? && !process_#{column}_upload && #{column}_present? && #{column}_updated?
+            end
+
+            def #{column}_mounted_as
+              #{column}.is_a?(Array) ? #{column}.first.mounted_as : #{column}.mounted_as
+            end
+
+            def #{column}_present?
+              #{column}.is_a?(Array) ? #{column}.present? : #{column}.file.present?
             end
 
             def enqueue_#{column}_background_job
-              CarrierWave::Backgrounder.enqueue_for_backend(#{worker}, self.class.name, id.to_s, #{column}.mounted_as)
+              CarrierWave::Backgrounder.enqueue_for_backend(#{worker}, self.class.name, id.to_s, #{column}_mounted_as)
             end
           RUBY
         end

data/lib/backgrounder/support/backends.rb

@@ -26,57 +26,13 @@ module CarrierWave
             worker.perform_later(*args.map(&:to_s))
           end
 
-          def enqueue_delayed_job(worker, *args)
-            worker_args = {}
-            if ::Delayed::Job.new.respond_to?(:queue)
-              worker_args[:queue] = queue_options[:queue] if queue_options[:queue]
-              worker_args[:priority] = queue_options[:priority] if queue_options[:priority]
-              ::Delayed::Job.enqueue worker.new(*args), worker_args
-            else
-              worker_args[:priority] = queue_options[:priority] if queue_options[:priority]
-              ::Delayed::Job.enqueue worker.new(*args), worker_args
-              if queue_options[:queue]
-                ::Rails.logger.warn("Queue name given but no queue column exists for Delayed::Job")
-              end
-            end
-          end
-
-          def enqueue_resque(worker, *args)
-            worker.instance_variable_set('@queue', queue_options[:queue] || :carrierwave)
-            ::Resque.enqueue worker, *args
-          end
-
           def enqueue_sidekiq(worker, *args)
             override_queue_name = worker.sidekiq_options['queue'] == 'default' || worker.sidekiq_options['queue'].nil?
             args = sidekiq_queue_options(override_queue_name, 'class' => worker, 'args' => args.map(&:to_s))
             worker.client_push(args)
           end
 
-          def enqueue_girl_friday(worker, *args)
-            @girl_friday_queue ||= GirlFriday::WorkQueue.new(queue_options.delete(:queue) || :carrierwave, queue_options) do |msg|
-              worker = msg[:worker]
-              worker.perform
-            end
-            @girl_friday_queue << { :worker => worker.new(*args) }
-          end
-
-          def enqueue_sucker_punch(worker, *args)
-            worker.new.async.perform(*args)
-          end
-
-          def enqueue_qu(worker, *args)
-            worker.instance_variable_set('@queue', queue_options[:queue] || :carrierwave)
-            ::Qu.enqueue worker, *args
-          end
-
-          def enqueue_qc(worker, *args)
-            class_name, subject_id, mounted_as = args
-            ::QC.enqueue "#{worker.name}.perform", class_name, subject_id, mounted_as.to_s
-          end
-
-          def enqueue_immediate(worker, *args)
-            worker.new(*args).perform
-          end
+          private
 
           def sidekiq_queue_options(override_queue_name, args)
             if override_queue_name && queue_options[:queue]

data/lib/backgrounder/version.rb

@@ -1,5 +1,5 @@
 module CarrierWave
   module Backgrounder
-    VERSION = "0.4.3"
+    VERSION = "1.0.0.beta.2"
   end
 end

data/lib/backgrounder/workers/active_job/process_asset.rb

@@ -0,0 +1,14 @@
+# encoding: utf-8
+require 'backgrounder/workers/process_asset_mixin'
+
+module CarrierWave
+  module Workers
+    module ActiveJob
+
+      class ProcessAsset < ::ActiveJob::Base
+        include CarrierWave::Workers::ProcessAssetMixin
+      end
+
+    end # ActiveJob
+  end # Workers
+end # Backgrounder

data/lib/backgrounder/workers/active_job/store_asset.rb

@@ -0,0 +1,14 @@
+# encoding: utf-8
+require 'backgrounder/workers/store_asset_mixin'
+
+module CarrierWave
+  module Workers
+    module ActiveJob
+
+      class StoreAsset < ::ActiveJob::Base
+        include CarrierWave::Workers::StoreAssetMixin
+      end
+
+    end # ActiveJob
+  end # Workers
+end # Backgrounder

data/lib/backgrounder/workers/process_asset.rb

@@ -1,4 +1,6 @@
 # encoding: utf-8
+require 'backgrounder/workers/process_asset_mixin'
+
 module CarrierWave
   module Workers
 

data/lib/backgrounder/workers/process_asset_mixin.rb

@@ -11,17 +11,26 @@ module CarrierWave
 
       def perform(*args)
         record = super(*args)
+        asset = record.send(:"#{column}")
 
-        if record && record.send(:"#{column}").present?
-          record.send(:"process_#{column}_upload=", true)
-          if record.send(:"#{column}").recreate_versions! && record.respond_to?(:"#{column}_processing")
-            record.update_attribute :"#{column}_processing", false
-          end
-        else
-          when_not_ready
+        return unless record && asset_present?(asset)
+
+        recreate_asset_versions!(asset)
+
+        if record.respond_to?(:"#{column}_processing")
+          record.update_attribute :"#{column}_processing", false
         end
       end
 
+      private
+
+      def recreate_asset_versions!(asset)
+        asset.is_a?(Array) ? asset.map(&:recreate_versions!) : asset.recreate_versions!
+      end
+
+      def asset_present?(asset)
+        asset.is_a?(Array) ? asset.present? : asset.file.present?
+      end
     end # ProcessAssetMixin
 
   end # Workers

data/lib/backgrounder/workers/store_asset.rb

@@ -1,4 +1,6 @@
 # encoding: utf-8
+require 'backgrounder/workers/store_asset_mixin'
+
 module CarrierWave
   module Workers
 

data/lib/backgrounder/workers/store_asset_mixin.rb

@@ -9,34 +9,29 @@ module CarrierWave
         base.extend CarrierWave::Workers::ClassMethods
       end
 
-      attr_reader :cache_path, :tmp_directory
-
       def perform(*args)
         record = super(*args)
 
-        if record && record.send(:"#{column}_tmp")
-          store_directories(record)
-          record.send :"process_#{column}_upload=", true
-          record.send :"#{column}_tmp=", nil
-          record.send :"#{column}_processing=", false if record.respond_to?(:"#{column}_processing")
-          File.open(cache_path) { |f| record.send :"#{column}=", f }
-          if record.save!
-            FileUtils.rm_r(tmp_directory, :force => true)
-          end
-        else
-          when_not_ready
-        end
+        return unless record && record.send(:"#{column}_tmp")
+
+        record.send :"process_#{column}_upload=", true
+        record.send :"#{column}_cache=", record.send(:"#{column}_tmp")  # Set the cache path
+        cache_assets! record.send(:"#{column}")                         # Trigger version creation
+        store_assets! record.send(:"#{column}")                         # Store the files
+        record.send :"#{column}_tmp=", nil
+        record.send :"#{column}_processing=", false if record.respond_to?(:"#{column}_processing")
+        record.save!
       end
 
       private
 
-      def store_directories(record)
-        asset, asset_tmp = record.send(:"#{column}"), record.send(:"#{column}_tmp")
-        cache_directory  = File.expand_path(asset.cache_dir, asset.root)
-        @cache_path      = File.join(cache_directory, asset_tmp)
-        @tmp_directory   = File.join(cache_directory, asset_tmp.split("/").first)
+      def cache_assets!(asset)
+        asset.is_a?(Array) ? asset.map(&:cache!) : asset.cache!
       end
 
+      def store_assets!(asset)
+        asset.is_a?(Array) ? asset.map(&:store!) : asset.store!
+      end
     end # StoreAssetMixin
 
   end # Workers

data/lib/backgrounder/workers.rb

@@ -1,6 +1,4 @@
 require 'backgrounder/workers/base'
 require 'backgrounder/workers/class_methods'
-require 'backgrounder/workers/process_asset_mixin'
-require 'backgrounder/workers/store_asset_mixin'
 require 'backgrounder/workers/process_asset'
 require 'backgrounder/workers/store_asset'