maintenance_tasks 1.0.0 → 1.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d591c21fb76efef72185bf183b0f4b0bc0a5b97a2606e083f6fbfc11325e029b
-  data.tar.gz: 6195855524490ebc663f1970134b6aed2abe7887345b6c17d2d0dccea9d81fb4
+  metadata.gz: c673afea63371c8c8f69ccda520b66fb1391d17ff6f241eb5610e45c60493561
+  data.tar.gz: a1b914d6fdf7391bda0691444f8af2e263c71313c295bf2fd38e6e655f661a99
 SHA512:
-  metadata.gz: 3fa1e498d50830521cabeb8babda612a9762438cd1fdccf5cecf776ae069ec85cdbaedbded208bdfc1e72479b69ffb76a39b999e4c2f8d810811dc8672f0b5cc
-  data.tar.gz: 2d9aa8e0c134baa3864f2cd5e9fe53944a5d13a89c21caab26dcd394b3eb8845af62397f2d1be880c9f27d71cbe77db3865d16c7da53f0b70cc64cc7bf4153a4
+  metadata.gz: d161e823b16c12abd127d725cccf985d0bca50253cb40937d265685d6078730beff127913eea7973ff23211576d5d143508ac61968ebb2460f1c12fc89ff5d82
+  data.tar.gz: 46de8ac9e6819829de191a0fcfaf40c48b2922f1cc3024df58a41a7e6763658149f3f32baa168045bae6c00790da1b789647d36bad880714bdf63e75d67aeb32

data/README.md

@@ -3,11 +3,16 @@
 A Rails engine for queuing and managing maintenance tasks.
 
 ## Table of Contents
+
+* [Demo](#demo)
 * [Installation](#installation)
+  * [Active Job Dependency](#active-job-dependency)
 * [Usage](#usage)
   * [Creating a Task](#creating-a-task)
-    * [Considerations when writing Tasks](#considerations-when-writing-tasks)
+  * [Creating a CSV Task](#creating-a-csv-task)
+  * [Considerations when writing Tasks](#considerations-when-writing-tasks)
   * [Writing tests for a Task](#writing-tests-for-a-task)
+  * [Writing tests for a CSV Task](#writing-tests-for-a-csv-task)
   * [Running a Task](#running-a-task)
   * [Monitoring your Task's status](#monitoring-your-tasks-status)
   * [How Maintenance Tasks runs a Task](#how-maintenance-tasks-runs-a-task)
@@ -17,22 +22,23 @@ A Rails engine for queuing and managing maintenance tasks.
     * [Customizing the maintenance tasks module](#customizing-the-maintenance-tasks-module)
     * [Customizing the underlying job class](#customizing-the-underlying-job-class)
     * [Customizing the rate at which task progress gets updated](#customizing-the-rate-at-which-task-progress-gets-updated)
+    * [Customizing which Active Storage service to use](#customizing-which-active-storage-service-to-use)
 * [Upgrading](#upgrading)
 * [Contributing](#contributing)
 * [Releasing new versions](#releasing-new-versions)
 
-## Installation
+## Demo
 
-Add this line to your application's Gemfile:
+Watch this demo video to see the gem in action:
 
-```ruby
-gem 'maintenance_tasks'
-```
+[![Link to demo video](static/demo.png)](https://www.youtube.com/watch?v=BTuvTQxlFzs)
 
-And then execute:
+## Installation
+
+To install the gem and run the install generator, execute:
 
 ```bash
-$ bundle
+$ bundle add maintenance_tasks
 $ rails generate maintenance_tasks:install
 ```
 
@@ -40,6 +46,22 @@ The generator creates and runs a migration to add the necessary table to your
 database. It also mounts Maintenance Tasks in your `config/routes.rb`. By
 default the web UI can be accessed in the new `/maintenance_tasks` path.
 
+In case you use an exception reporting service (e.g. Bugsnag) you might want to
+define an error handler. See [Customizing the error
+handler](#customizing-the-error-handler) for more information.
+
+### Active Job Dependency
+
+The Maintenance Tasks framework relies on ActiveJob behind the scenes to run
+Tasks. The default queuing backend for ActiveJob is
+[asynchronous][async-adapter]. It is **strongly recommended** to change this to
+a persistent backend so that Task progress is not lost during code or
+infrastructure changes. For more information on configuring a queuing backend,
+take a look at the [ActiveJob documentation][active-job-docs].
+
+[async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html
+[active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend
+
 ## Usage
 
 ### Creating a Task
@@ -81,7 +103,45 @@ module Maintenance
 end
 ```
 
-#### Considerations when writing Tasks
+### Creating a CSV Task
+
+You can also write a Task that iterates on a CSV file. Note that writing CSV
+Tasks **requires ActiveStorage to be configured**. Ensure that the dependency
+is specified in your application's Gemfile, and that you've followed the
+[setup instuctions][setup].
+
+[setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
+
+Generate a CSV Task by running:
+
+```bash
+$ rails generate maintenance_tasks:task import_posts --csv
+```
+
+The generated task is a subclass of `MaintenanceTasks::Task` that implements:
+
+* `process`: do the work of your maintenance task on a `CSV::Row`
+
+```ruby
+# app/tasks/maintenance/import_posts_task.rb
+module Maintenance
+  class ImportPostsTask < MaintenanceTasks::Task
+    csv_collection
+
+    def process(row)
+      Post.create!(title: row["title"], content: row["content"])
+    end
+  end
+end
+```
+
+```csv
+# posts.csv
+title,content
+My Title,Hello World!
+```
+
+### Considerations when writing Tasks
 
 MaintenanceTasks relies on the queue adapter configured for your application to
 run the job which is processing your Task. The guidelines for writing Task may
@@ -120,7 +180,7 @@ module Maintenance
     test "#process performs a task iteration" do
       post = Post.new
 
-      Maintenance::UpdatePostsTask.new.process(post)
+      Maintenance::UpdatePostsTask.process(post)
 
       assert_equal 'New content!', post.content
     end
@@ -128,6 +188,32 @@ module Maintenance
 end
 ```
 
+### Writing tests for a CSV Task
+
+You should write tests for your `#process` method in a CSV Task as well. It
+takes a `CSV::Row` as an argument. You can pass a row, or a hash with string
+keys to `#process` from your test.
+
+```ruby
+# app/tasks/maintenance/import_posts_task_test.rb
+module Maintenance
+  class ImportPostsTaskTest < ActiveSupport::TestCase
+    test "#process performs a task iteration" do
+      assert_difference -> { Post.count } do
+        Maintenance::UpdatePostsTask.process({
+          'title' => 'My Title',
+          'content' => 'Hello World!',
+        })
+      end
+
+      post = Post.last
+      assert_equal 'My Title', post.title
+      assert_equal 'Hello World!', post.content
+    end
+  end
+end
+```
+
 ### Running a Task
 
 You can run your new Task by accessing the Web UI and clicking on "Run".
@@ -138,11 +224,26 @@ Alternatively, you can run your Task in the command line:
 $ bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask
 ```
 
-You can also run a Task in Ruby by sending `run` with a Task name to a Runner
-instance:
+To run a Task that processes CSVs from the command line, use the --csv option:
+
+```bash
+$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv 'path/to/my_csv.csv'
+```
+
+You can also run a Task in Ruby by sending `run` with a Task name to Runner:
+
+```ruby
+MaintenanceTasks::Runner.run(name: 'Maintenance::UpdatePostsTask')
+```
+
+To run a Task that processes CSVs using the Runner, provide a Hash containing an
+open IO object and a filename to `run`:
 
 ```ruby
-MaintenanceTasks::Runner.new.run('Maintenance::UpdatePostsTask')
+MaintenanceTasks::Runner.run(
+  name: 'Maintenance::ImportPostsTask'
+  csv_file: { io: File.open('path/to/my_csv.csv'), filename: 'my_csv.csv' }
+)
 ```
 
 ### Monitoring your Task's status
@@ -224,17 +325,39 @@ be placed in a `maintenance_tasks.rb` initializer.
 Exceptions raised while a Task is performing are rescued and information about
 the error is persisted and visible in the UI.
 
-If your application uses Bugsnag to monitor errors, the gem will automatically
-notify Bugsnag of any errors raised while a Task is performing.
-
-If you want to integrate with another exception monitoring service or customize
-error handling, a callback can be defined:
+If you want to integrate with an exception monitoring service (e.g. Bugsnag),
+you can define an error handler:
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
-MaintenanceTasks.error_handler = ->(error) { MyErrorMonitor.notify(error) }
+MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
+  Bugsnag.notify(error) do |notification|
+    notification.add_tab(:task, task_context)
+  end
+end
 ```
 
+The error handler should be a lambda that accepts three arguments:
+
+* `error`: The exception that was raised.
+* `task_context`: A hash with additional information about the Task and the
+  error:
+  * `task_name`: The name of the Task that errored
+  * `started_at`: The time the Task started
+  * `ended_at`: The time the Task errored
+  Note that `task_context` may be empty if the Task produced an error before any
+  context could be gathered (for example, if deserializing the job to process
+  your Task failed).
+* `errored_element`: The element, if any, that was being processed when the
+  Task raised an exception. If you would like to pass this object to your
+  exception monitoring service, make sure you **sanitize the object** to avoid
+  leaking sensitive data and **convert it to a format** that is compatible with
+  your bug tracker. For example, Bugsnag only sends the id and class name of
+  ActiveRecord objects in order to protect sensitive data. CSV rows, on the
+  other hand, are converted to strings and passed raw to Bugsnag, so make sure
+  to filter any personal data from these objects before adding them to a
+  report.
+
 #### Customizing the maintenance tasks module
 
 `MaintenanceTasks.tasks_module` can be configured to define the module in which
@@ -281,6 +404,36 @@ MaintenanceTasks.ticker_delay = 2.seconds
 
 If no value is specified, it will default to 1 second.
 
+#### Customizing which Active Storage service to use
+
+The Active Storage framework in Rails 6.1 and up supports multiple storage
+services per environment. To specify which service to use,
+`MaintenanceTasks.active_storage_service` can be configured with the service's
+key, as specified in your application's `config/storage.yml`:
+
+```yaml
+# config/storage.yml
+user_data:
+  service: GCS
+  credentials: <%= Rails.root.join("path/to/user/data/keyfile.json") %>
+  project: "my-project"
+  bucket: "user-data-bucket"
+
+internal:
+  service: GCS
+  credentials: <%= Rails.root.join("path/to/internal/keyfile.json") %>
+  project: "my-project"
+  bucket: "internal-bucket"
+```
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+MaintenanceTasks.active_storage_service = :internal
+```
+
+There is no need to configure this option if your application uses only one
+storage service per environment.
+
 ## Upgrading
 
 Use bundler to check for and upgrade to newer versions. After installing a new
@@ -302,15 +455,20 @@ pull requests. You can find the contribution guidelines on
 
 ## Releasing new versions
 
-This gem is published to packagecloud. The procedure to publish a new version:
+Updates should be added to the latest draft release on GitHub as Pull Requests
+are merged.
+
+Once a release is ready, follow these steps:
 
 * Update `spec.version` in `maintenance_tasks.gemspec`.
 * Run `bundle install` to bump the `Gemfile.lock` version of the gem.
 * Open a PR and merge on approval.
-* Create a [release on GitHub][release] with a version number that matches the
-  version defined in the gemspec.
 * Deploy via [Shipit][shipit] and see the new version on
   <https://rubygems.org/gems/maintenance_tasks>.
+* Ensure the release has documented all changes and publish it.
+* Create a new [draft release on GitHub][release] with the title
+  'Upcoming Release'. The tag version can be left blank. This will be the
+  starting point for documenting changes related to the next release.
 
 [release]: https://help.github.com/articles/creating-releases/
 [shipit]: https://shipit.shopify.io/shopify/maintenance_tasks/rubygems

data/Rakefile

@@ -1,29 +1,29 @@
 # frozen_string_literal: true
 begin
-  require 'bundler/setup'
+  require "bundler/setup"
 rescue LoadError
-  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+  puts "You must `gem install bundler` and `bundle install` to run rake tasks"
 end
 
-require 'rdoc/task'
+require "rdoc/task"
 RDoc::Task.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title    = 'MaintenanceTasks'
-  rdoc.options << '--line-numbers'
-  rdoc.rdoc_files.include('README.md')
-  rdoc.rdoc_files.include('lib/**/*.rb')
+  rdoc.rdoc_dir = "rdoc"
+  rdoc.title    = "MaintenanceTasks"
+  rdoc.options << "--line-numbers"
+  rdoc.rdoc_files.include("README.md")
+  rdoc.rdoc_files.include("lib/**/*.rb")
 end
 
-APP_RAKEFILE = File.expand_path('test/dummy/Rakefile', __dir__)
-load('rails/tasks/engine.rake')
+APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
+load("rails/tasks/engine.rake")
 
-load('rails/tasks/statistics.rake')
+load("rails/tasks/statistics.rake")
 
-require 'bundler/gem_tasks'
+require "bundler/gem_tasks"
 
-require 'rubocop/rake_task'
+require "rubocop/rake_task"
 RuboCop::RakeTask.new
 
-task(test: 'app:test')
-task('test:system' => 'app:test:system')
-task(default: ['db:setup', 'test', 'test:system', 'rubocop'])
+task(test: "app:test")
+task("test:system" => "app:test:system")
+task(default: ["db:test:prepare", "test", "test:system", "rubocop"])

data/app/controllers/maintenance_tasks/application_controller.rb

@@ -3,11 +3,9 @@
 module MaintenanceTasks
   # Base class for all controllers used by this engine.
   #
-  # @api private
+  # Can be extended to add different authentication and authorization code.
   class ApplicationController < ActionController::Base
-    include Pagy::Backend
-
-    BULMA_CDN = 'https://cdn.jsdelivr.net'
+    BULMA_CDN = "https://cdn.jsdelivr.net"
 
     content_security_policy do |policy|
       policy.style_src(BULMA_CDN)
@@ -17,10 +15,9 @@ module MaintenanceTasks
     before_action do
       request.content_security_policy_nonce_generator ||=
         ->(_request) { SecureRandom.base64(16) }
-      request.content_security_policy_nonce_directives = ['style-src']
+      request.content_security_policy_nonce_directives = ["style-src"]
     end
 
     protect_from_forgery with: :exception
   end
-  private_constant :ApplicationController
 end

data/app/controllers/maintenance_tasks/runs_controller.rb

@@ -6,17 +6,7 @@ module MaintenanceTasks
   #
   # @api private
   class RunsController < ApplicationController
-    before_action :set_run, except: :index
-
-    # Shows a full list of Runs.
-    def index
-      query = Run.all.order(id: :desc)
-      if params[:task_name].present?
-        task_name = Run.sanitize_sql_like(params[:task_name])
-        query = query.where('task_name LIKE ?', "%#{task_name}%")
-      end
-      @pagy, @runs = pagy(query)
-    end
+    before_action :set_run
 
     # Updates a Run status to paused.
     def pause
@@ -40,5 +30,4 @@ module MaintenanceTasks
       @run = Run.find(params.fetch(:id))
     end
   end
-  private_constant :RunsController
 end

data/app/controllers/maintenance_tasks/tasks_controller.rb

@@ -7,7 +7,7 @@ module MaintenanceTasks
   #
   # @api private
   class TasksController < ApplicationController
-    before_action :set_refresh, only: [:index, :show]
+    before_action :set_refresh, only: [:index]
 
     # Renders the maintenance_tasks/tasks page, displaying
     # available tasks to users, grouped by category.
@@ -19,12 +19,16 @@ module MaintenanceTasks
     # Shows running and completed instances of the Task.
     def show
       @task = TaskData.find(params.fetch(:id))
-      @pagy, @previous_runs = pagy(@task.previous_runs)
+      set_refresh if @task.last_run&.active?
+      @runs_page = RunsPage.new(@task.previous_runs, params[:cursor])
     end
 
     # Runs a given Task and redirects to the Task page.
     def run
-      task = Runner.new.run(name: params.fetch(:id))
+      task = Runner.run(
+        name: params.fetch(:id),
+        csv_file: params[:csv_file]
+      )
       redirect_to(task_path(task))
     rescue ActiveRecord::RecordInvalid => error
       redirect_to(task_path(error.record.task_name), alert: error.message)
@@ -38,5 +42,4 @@ module MaintenanceTasks
       @refresh = 3
     end
   end
-  private_constant :TasksController
 end

data/app/helpers/maintenance_tasks/application_helper.rb

@@ -4,17 +4,6 @@ module MaintenanceTasks
   #
   # @api private
   module ApplicationHelper
-    include Pagy::Frontend
-
-    # Renders pagination for the page, if there is more than one page present.
-    #
-    # @param pagy [Pagy] the pagy instance containing pagination details,
-    #   including the number of pages the results are spread across.
-    # @return [String] the HTML to render for pagination.
-    def pagination(pagy)
-      raw(pagy_bulma_nav(pagy)) if pagy.pages > 1
-    end
-
     # Renders a time element with the given datetime, worded as relative to the
     # current time.
     #
@@ -24,44 +13,9 @@ module MaintenanceTasks
     # @param datetime [ActiveSupport::TimeWithZone] the time to be presented.
     # @return [String] the HTML to render with the relative datetime in words.
     def time_ago(datetime)
-      time_tag(datetime, title: datetime.utc.iso8601, class: 'is-clickable') do
-        time_ago_in_words(datetime) + ' ago'
-      end
-    end
-
-    # Fix stylesheet_link_tag to handle integrity when preloading.
-    # To be reverted once fixed upstream in Rails.
-    def stylesheet_link_tag(*sources)
-      return super unless respond_to?(:send_preload_links_header, true)
-      options = sources.extract_options!.stringify_keys
-      path_options = options.extract!('protocol', 'host', 'skip_pipeline')
-        .symbolize_keys
-      preload_links = []
-      crossorigin = options.delete('crossorigin')
-      crossorigin = 'anonymous' if crossorigin == true
-      nopush = options['nopush'].nil? ? true : options.delete('nopush')
-      integrity = options['integrity']
-
-      sources_tags = sources.uniq.map do |source|
-        href = path_to_stylesheet(source, path_options)
-        preload_link = "<#{href}>; rel=preload; as=style"
-        preload_link += "; crossorigin=#{crossorigin}" unless crossorigin.nil?
-        preload_link += "; integrity=#{integrity}" unless integrity.nil?
-        preload_link += '; nopush' if nopush
-        preload_links << preload_link
-        tag_options = {
-          'rel' => 'stylesheet',
-          'media' => 'screen',
-          'crossorigin' => crossorigin,
-          'href' => href,
-        }.merge!(options)
-        tag(:link, tag_options)
+      time_tag(datetime, title: datetime.utc.iso8601, class: "is-clickable") do
+        time_ago_in_words(datetime) + " ago"
       end
-
-      send_preload_links_header(preload_links)
-
-      safe_join(sources_tags)
     end
   end
-  private_constant :ApplicationHelper
 end