sidekiq-undertaker 1.1.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b83cd4a266136182c00e82e85f9173d9c8b53722302ea8bc6b440706b31b12f2
-  data.tar.gz: 478bb085a406e62c4f9b6eb2b4cd8e465b39dafaffa1a827c17669ffca488c83
+  metadata.gz: fb255bb741814575247793b7ddfdd7dfa6fe5c0d86a21df3b33f315209beef53
+  data.tar.gz: fc6b4970d6e7d2c8410525c76fd97fc1c98943a0f9bb381807946829e38180ab
 SHA512:
-  metadata.gz: fe769f3ef5497ae2848dcd3bd858495152b813a72970cd6b311b1655c7d65e025e64cd2e88a808f4f23cced4ee2d4cc19cc7daf598df119a4127d5cbd276de16
-  data.tar.gz: bad9876a8e994490b9461f443cd93b02da7363dc9c2b1b7a6ea936741c3d4865b99b58e7328a34cdae8d66224d52bedb02affeefbe9737ba23f1825d3bc8dc46
+  metadata.gz: '0162683d05be1edf9d3dc362a7b7c543fc4d0d6891945540637cc03dcb919a1780003ed55503f3cbc3af8ab0ef3a03ee28bfe6d6575dac4610ee623dd373b3b1'
+  data.tar.gz: 610f3d98e380193606e37c74953dd53b00c79ae42fd6f091f940f27e9b61d55305df546a207676b8139a8d41719ba0fd3336cf87b5f35ec0b16b43283796c64a

data/.github/workflows/ruby-build.yml

@@ -39,7 +39,7 @@ jobs:
           - 'truffleruby+graalvm'
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Set up Ruby
     # To automatically get bug fixes and new Ruby versions for ruby/setup-ruby,
     # change this to (see https://github.com/ruby/setup-ruby#versioning):

data/CHANGELOG.md

@@ -0,0 +1,104 @@
+# Changelog
+
+## Unreleased
+
+## [1.3.0] - 2022-04-27
+### Added
+- Added option to export and re-import dead jobs
+
+## [1.2.0] - 2022-02-03
+### Changed
+- Updated sidekiq dependency to 6.4.0
+
+## [1.1.1] - 2022-02-03
+### Added
+- Added CHANGELOG.md
+
+## [1.1.0] - 2022-01-19
+### Added
+- Added renovate.json
+- Added ruby-build github workflow
+- Added .rspec config file
+
+### Changed
+- Updated support for Sidekiq 6.2.2 and higher
+- Updated travis CI config to use latest ruby versions
+- Updated to use GitHub-native Dependabot
+- Updated spech_helper
+- Updated approvals to version 0.0.25
+
+## [1.0.2] - 2021-01-27
+### Added
+- Consider also explicitly stating that PRs must pass RuboCop to be accepted
+- Added rubocop badge to README.md
+
+### Changed
+- Upgraded rubocop and rt_rubocop_defaults
+- Updated rake requirement from ~> 12.3 to ~> 13.0
+- Updated Travis CI to use JRuby-9.2.10.0
+- Updated README to clarify RuboCop install
+
+## [1.0.1] - 2020-02-03
+### Added
+- Added ruby 2.7.0 to travis file and mark ruby 2.4.9 as optional
+
+### Changed
+- Updated codeclimate.yml to enable rubocop
+
+### Fixed
+- Fixed pagination on morgue page
+- Fixed rubocopo issues
+
+## [1.0.0] - 2019-12-24
+### Added
+- Added RubyGems badge
+- Added custom locales
+
+### Changed
+- Renamed gem from sidekiq-cleaner to sidekiq-undertaker
+- Updated do not show 'all' as job or error name in titles
+- Updated gemspec, .travis.yml and README.md
+- Updated rename buckets and add the 'older' bucket
+- Refactored move sort into JobAggregator
+- Renamed JobAggregator to JobDistributor
+- Updated translations
+- Updated add '/filter' and '/morgue' prefixes to URLs
+- Renamed Helpers to APIHelpers
+- Moved view files to top level web folder
+- Renamed views
+- Moved WebExtension::Helper into separate file
+
+### Removed
+- Removed before install steps from .travis.yml
+
+### Fixed
+- Updated README.md to fix broken members link
+- Fixed approvals specs after sidekiq ref update
+- Fixed sidekiq dependency in gemspec
+- Fixed broken links in filter view
+- Fixed Travis link
+- Fixed paths in views
+
+## [0.3.6] - 2019-12-04
+### Added
+- Added Test Coverage Badge
+- Added CodeClimate configuration
+- Added support for JRuby
+- Added .travis.yml
+- Added support for csrf and allowed this to be used with newer versions of sidekiq
+
+### Changed
+- Updated travis config
+- Updated Rubocop and RtRubocopDefaults gems
+- Updated README.md
+- Updated gemspec
+- Renamed approvals specs to shorten file path
+- Updated approval specs to exclude Sidekiq::VERSION
+- Removed celluloid dependency
+- Allowed developers to use other Rubies
+- Removed .ruby-version and added .ruby-version and .rvmrc to the .gitignore
+
+### Fixed
+- Fixed "Retry All" and "Delete All" function
+- Fixed Csrf and sidekiq version compatibility
+- Fixed Cleaner Redirect Bug

data/README.md

@@ -45,24 +45,24 @@ Or install it yourself as:
 
 The filter page shows a table with time-buckets as columns and rows for each job class.
 
-![Sidekiq Undertaker](Demo_Filter.png)
+![Sidekiq Undertaker](assets/Undertaker_demo_all_jobs.png)
 
 #### Job Filter View
 
 For each job class, you can drill down to view error distribution based on
 error class.
 
-![Sidekiq Undertaker](Demo_Job_Filter.png)
+![Sidekiq Undertaker](assets/Undertaker_demo_1_job_all_errors.png)
 
 #### Morgue View
 Finally, click on the individual error counts to display details of the
 errors in a list form.
 
-![Sidekiq Undertaker](Demo_Morgue_1_Job.png)
+![Sidekiq Undertaker](assets/Undertaker_demo_1_job_1_error.png)
 
 The morgue view can, for example, also show an error distribution over all job classes.
 
-![Sidekiq Undertaker](Demo_Morgue_all.png)
+![Sidekiq Undertaker](assets/Undertaker_demo_all_jobs_1_error.png)
 
 ## Contributing
 
@@ -81,6 +81,9 @@ this fork was renamed to `sidekiq-undertaker`.
 
 The [Sidekiq-Cleaner](https://github.com/HackingHabits/sidekiq-cleaner) gem was originally created by [Madan Thangavelu](https://github.com/HackingHabits).
 [Tout](https://github.com/Tout/sidekiq-cleaner) and [TheWudu](https://github.com/TheWudu/sidekiq-cleaner) also contributed to it.
+[Zlatko Rednjak](https://github.com/Rednjak) added the initial version of the `CHANGELOG.md`.
+[Lucas Dell'Isola](https://github.com/ldellisola) added the export/import feature for dead jobs.
+
 For the complete list of network members have a look at the [fork overview](https://github.com/ThomasKoppensteiner/sidekiq-undertaker/network/members).
 
 ## Alternative Projects

data/lib/sidekiq/undertaker/version.rb

@@ -2,6 +2,6 @@
 
 module Sidekiq
   module Undertaker
-    VERSION = "1.1.0"
+    VERSION = "1.3.0"
   end
 end

data/lib/sidekiq/undertaker/web_extension/api_helpers.rb

@@ -2,11 +2,16 @@
 
 require "sidekiq/undertaker/job_distributor"
 require "sidekiq/undertaker/job_filter"
+require "json"
+require "zip"
 
 module Sidekiq
   module Undertaker
     module WebExtension
+      # rubocop:disable Metrics/ModuleLength
       module APIHelpers
+        EXPORT_CHUNK_SIZE = 2000
+
         def show_filter
           store_request_params
 
@@ -59,22 +64,25 @@ module Sidekiq
         def post_undertaker
           raise ::ArgumentError.new("Key missing") unless params["key"]
 
-          params["key"].each do |key|
-            job = Sidekiq::DeadSet.new.fetch(*parse_params(key)).first
-            if job
-              if params["retry"]
-                job.retry
-              elsif params["delete"]
-                job.delete
-              end
-            end
-          end
+          jobs = params["key"].map { |k| Sidekiq::DeadSet.new.fetch(*parse_params(k)).first }.compact
 
-          redirect redirect_path(request)
+          handle_selected_jobs jobs
         rescue ::ArgumentError
           bad_request
         end
 
+        def handle_selected_jobs(jobs)
+          return send_data(*prepare_data(jobs.map(&:item), EXPORT_CHUNK_SIZE)) if params["export"]
+
+          if params["retry"]
+            jobs.each(&:retry)
+          elsif params["delete"]
+            jobs.each(&:delete)
+          end
+
+          redirect redirect_path(request)
+        end
+
         def post_undertaker_job_class_error_class_buckent_name_delete
           store_request_params
           @dead_jobs = Sidekiq::Undertaker::JobFilter.filter_dead_jobs(params)
@@ -96,6 +104,28 @@ module Sidekiq
           redirect redirect_path(request)
         end
 
+        def post_undertaker_job_class_error_class_buckent_name_export
+          store_request_params
+
+          @dead_jobs = Sidekiq::Undertaker::JobFilter.filter_dead_jobs(params)
+          send_data(*prepare_data(@dead_jobs.map { |j| j.job.item }, EXPORT_CHUNK_SIZE))
+        end
+
+        def post_import_jobs
+          file = params["upload_file"]
+          raise ::ArgumentError.new("The file is not a json") if file.nil? || file[:type] != "application/json"
+
+          data = params["upload_file"][:tempfile].read
+          dead_set = Sidekiq::DeadSet.new
+
+          JSON.parse(data).each do |job|
+            dead_set.kill(Sidekiq.dump_json(job))
+          end
+          redirect redirect_path(request)
+        rescue StandardError
+          bad_request
+        end
+
         def render_result(template)
           render(:erb, File.read(File.join(view_path, template)))
         end
@@ -112,13 +142,40 @@ module Sidekiq
 
         def redirect_path(request)
           path = request.referer ? URI.parse(request.referer).path : request.path
-          path.gsub("/delete", "").gsub("/retry", "")
+          path.gsub("/delete", "").gsub("/retry", "").gsub("/export", "")
+        end
+
+        def prepare_data(data, chunk_size)
+          filename = Time.now.strftime("%Y-%m-%d_%H-%M").to_s
+          return [data.to_json, "application/json", "#{filename}.json"] if data.length <= chunk_size
+
+          filename = "#{@req_job_class}_#{filename}"
+          zip = Zip::OutputStream.write_buffer do |file|
+            data.each_slice(chunk_size).each_with_index do |chunk, index|
+              file.put_next_entry("#{filename}_part-#{index + 1}.json")
+              file.write chunk.to_json
+            end
+          end
+
+          [zip.string, "application/zip", "#{filename}.zip"]
+        end
+
+        def send_data(data, content_type = "text/plain", file_name = "attachment.txt")
+          [
+            200,
+            {
+              "Content-Type"        => content_type,
+              "Content-Disposition" => "attachment; filename=\"#{file_name}\""
+            },
+            [data]
+          ]
         end
 
         def bad_request
           [400, { "Content-Type" => "text/plain" }, ["Bad Request"]]
         end
       end
+      # rubocop:enable Metrics/ModuleLength
     end
   end
 end

data/lib/sidekiq/undertaker/web_extension.rb

@@ -12,7 +12,6 @@ module Sidekiq
         app.get "/undertaker/filter" do
           show_filter
         end
-
         app.get "/undertaker/filter/:job_class/:bucket_name" do
           show_filter_by_job_class_bucket_name
         end
@@ -32,6 +31,14 @@ module Sidekiq
         app.post "/undertaker/morgue/:job_class/:error_class/:bucket_name/retry" do
           post_undertaker_job_class_error_class_buckent_name_retry
         end
+
+        app.post "/undertaker/morgue/:job_class/:error_class/:bucket_name/export" do
+          post_undertaker_job_class_error_class_buckent_name_export
+        end
+
+        app.post "/undertaker/import_jobs" do
+          post_import_jobs
+        end
       end
       # rubocop:enable Metrics/MethodLength
     end

data/sidekiq-undertaker.gemspec

@@ -21,6 +21,7 @@ Gem::Specification.new do |spec|
   spec.metadata      = {
     "homepage_uri"          => "https://github.com/ThomasKoppensteiner/sidekiq-undertaker",
     "source_code_uri"       => "https://github.com/ThomasKoppensteiner/sidekiq-undertaker",
+    "changelog_uri"         => "https://github.com/ThomasKoppensteiner/sidekiq-undertaker/blob/master/CHANGELOG.md",
     "bug_tracker_uri"       => "https://github.com/ThomasKoppensteiner/sidekiq-undertaker/issues",
     "build_status_uri"      => "https://travis-ci.org/ThomasKoppensteiner/sidekiq-undertaker",
     "rubygems_mfa_required" => "true"
@@ -54,5 +55,6 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "sinatra", "~> 2.0"
   spec.add_development_dependency "timecop", "~> 0.9"
 
-  spec.add_runtime_dependency "sidekiq", ">= 6.2.2", "< 6.3"
+  spec.add_runtime_dependency "rubyzip"
+  spec.add_runtime_dependency "sidekiq", ">= 6.4", "< 7"
 end

data/spec/fixtures/approvals/sidekiq_undertaker_webextension/show_filter/when_filter_page_is_called/behaves_like_a_page/the_displayed_page_is_correct_for_sidekiqv6.approved.txt

@@ -18,7 +18,7 @@
     <meta name="google" content="notranslate" />
     
   </head>
-  <body class="admin" data-poll-path="" data-locale="en">
+  <body class="admin" data-locale="en">
     <div class="navbar navbar-default navbar-fixed-top">
   <div class="container-fluid">
     <div class="navbar-header" data-navbar="static">
@@ -29,9 +29,8 @@
       </button>
       <div class="navbar-toggle collapsed navbar-livereload">
         
-  
-    <a id="live-poll" class="btn btn-primary" href="/sidekiq/undertaker/filter?poll=true">Live Poll</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
         
@@ -95,9 +94,8 @@
         <li class="navbar-livereload">
           <div class="poll-wrapper">
             
-  
-    <a id="live-poll" class="btn btn-primary" href="/sidekiq/undertaker/filter?poll=true">Live Poll</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
             
@@ -114,40 +112,40 @@
           <div class="col-sm-12 summary_bar">
             <ul class="list-unstyled summary row">
   <li class="processed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtProcessed" class="count">0</span>
     <span class="desc">Processed</span>
   </li>
   <li class="failed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtFailed" class="count">0</span>
     <span class="desc">Failed</span>
   </li>
   <li class="busy col-sm-1">
     <a href="/sidekiq/busy">
-      <span class="count">0</span>
+      <span id="txtBusy" class="count">0</span>
       <span class="desc">Busy</span>
     </a>
   </li>
   <li class="enqueued col-sm-1">
     <a href="/sidekiq/queues">
-      <span class="count">0</span>
+      <span id="txtEnqueued" class="count">0</span>
       <span class="desc">Enqueued</span>
     </a>
   </li>
   <li class="retries col-sm-1">
     <a href="/sidekiq/retries">
-      <span class="count">0</span>
+      <span id="txtRetries" class="count">0</span>
       <span class="desc">Retries</span>
     </a>
   </li>
   <li class="scheduled col-sm-1">
     <a href="/sidekiq/scheduled">
-      <span class="count">0</span>
+      <span id="txtScheduled" class="count">0</span>
       <span class="desc">Scheduled</span>
     </a>
   </li>
   <li class="dead col-sm-1">
     <a href="/sidekiq/morgue">
-      <span class="count">4</span>
+      <span id="txtDead" class="count">4</span>
       <span class="desc">Dead</span>
     </a>
   </li>
@@ -213,6 +211,22 @@
   
 </table>
 
+<header class="row header">
+  <div class="col-sm-12">
+    <h3>
+      Import Jobs
+    </h3>
+  </div>
+</header>
+
+<form enctype="multipart/form-data" method="post" action='/sidekiq/undertaker/import_jobs'>
+  <input type='hidden' name='authenticity_token' value='stubbed-csrf-token'/>
+    <input type="file" id=upload_file" name="upload_file" >
+    <button class="btn btn-danger" style="margin-top: 10px" type="submit">
+      Import
+    </button>
+</form>
+
           </div>
         </div>
       </div>
@@ -228,7 +242,7 @@
             <p class="navbar-text redis-url" title="redis://127.0.0.1:6379/0">redis://127.0.0.1:6379/0</p>
           </li>
           <li>
-            <p class="navbar-text server-utc-time">20:57:00 UTC</p>
+            <p id="serverUtcTime" class="navbar-text server-utc-time">20:57:00 UTC</p>
           </li>
           <li>
             <p class="navbar-text"><a style="color: gray;" href="https://github.com/mperham/sidekiq/wiki">docs</a></p>

data/spec/fixtures/approvals/sidekiq_undertaker_webextension/show_filter/when_job_classbucket_page_is_called/behaves_like_a_page/the_displayed_page_is_correct_for_sidekiqv6.approved.txt

@@ -18,7 +18,7 @@
     <meta name="google" content="notranslate" />
     
   </head>
-  <body class="admin" data-poll-path="" data-locale="en">
+  <body class="admin" data-locale="en">
     <div class="navbar navbar-default navbar-fixed-top">
   <div class="container-fluid">
     <div class="navbar-header" data-navbar="static">
@@ -29,9 +29,8 @@
       </button>
       <div class="navbar-toggle collapsed navbar-livereload">
         
-  
-    <a id="live-poll" class="btn btn-primary" href="/sidekiq/undertaker/filter/HardWorker/1_hour?poll=true">Live Poll</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
         
@@ -95,9 +94,8 @@
         <li class="navbar-livereload">
           <div class="poll-wrapper">
             
-  
-    <a id="live-poll" class="btn btn-primary" href="/sidekiq/undertaker/filter/HardWorker/1_hour?poll=true">Live Poll</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
             
@@ -114,40 +112,40 @@
           <div class="col-sm-12 summary_bar">
             <ul class="list-unstyled summary row">
   <li class="processed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtProcessed" class="count">0</span>
     <span class="desc">Processed</span>
   </li>
   <li class="failed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtFailed" class="count">0</span>
     <span class="desc">Failed</span>
   </li>
   <li class="busy col-sm-1">
     <a href="/sidekiq/busy">
-      <span class="count">0</span>
+      <span id="txtBusy" class="count">0</span>
       <span class="desc">Busy</span>
     </a>
   </li>
   <li class="enqueued col-sm-1">
     <a href="/sidekiq/queues">
-      <span class="count">0</span>
+      <span id="txtEnqueued" class="count">0</span>
       <span class="desc">Enqueued</span>
     </a>
   </li>
   <li class="retries col-sm-1">
     <a href="/sidekiq/retries">
-      <span class="count">0</span>
+      <span id="txtRetries" class="count">0</span>
       <span class="desc">Retries</span>
     </a>
   </li>
   <li class="scheduled col-sm-1">
     <a href="/sidekiq/scheduled">
-      <span class="count">0</span>
+      <span id="txtScheduled" class="count">0</span>
       <span class="desc">Scheduled</span>
     </a>
   </li>
   <li class="dead col-sm-1">
     <a href="/sidekiq/morgue">
-      <span class="count">4</span>
+      <span id="txtDead" class="count">4</span>
       <span class="desc">Dead</span>
     </a>
   </li>
@@ -229,7 +227,7 @@
             <p class="navbar-text redis-url" title="redis://127.0.0.1:6379/0">redis://127.0.0.1:6379/0</p>
           </li>
           <li>
-            <p class="navbar-text server-utc-time">20:57:00 UTC</p>
+            <p id="serverUtcTime" class="navbar-text server-utc-time">20:57:00 UTC</p>
           </li>
           <li>
             <p class="navbar-text"><a style="color: gray;" href="https://github.com/mperham/sidekiq/wiki">docs</a></p>

data/spec/fixtures/approvals/sidekiq_undertaker_webextension/show_filter/when_job_classbucket_page_is_polled/behaves_like_a_page/the_displayed_page_is_correct_for_sidekiqv6.approved.txt

@@ -18,7 +18,7 @@
     <meta name="google" content="notranslate" />
     
   </head>
-  <body class="admin" data-poll-path="/sidekiq/undertaker/filter/HardWorker/1_hour" data-locale="en">
+  <body class="admin" data-locale="en">
     <div class="navbar navbar-default navbar-fixed-top">
   <div class="container-fluid">
     <div class="navbar-header" data-navbar="static">
@@ -29,9 +29,8 @@
       </button>
       <div class="navbar-toggle collapsed navbar-livereload">
         
-  
-    <a id="live-poll" class="btn btn-primary active" href="/sidekiq/undertaker/filter/HardWorker/1_hour">Stop Polling</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
         
@@ -95,9 +94,8 @@
         <li class="navbar-livereload">
           <div class="poll-wrapper">
             
-  
-    <a id="live-poll" class="btn btn-primary active" href="/sidekiq/undertaker/filter/HardWorker/1_hour">Stop Polling</a>
-  
+    <a class="live-poll-start live-poll btn btn-primary">Live Poll</a>
+    <a class="live-poll-stop live-poll btn btn-primary active">Stop Polling</a>
 
 
             
@@ -114,40 +112,40 @@
           <div class="col-sm-12 summary_bar">
             <ul class="list-unstyled summary row">
   <li class="processed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtProcessed" class="count">0</span>
     <span class="desc">Processed</span>
   </li>
   <li class="failed col-sm-1">
-    <span class="count">0</span>
+    <span id="txtFailed" class="count">0</span>
     <span class="desc">Failed</span>
   </li>
   <li class="busy col-sm-1">
     <a href="/sidekiq/busy">
-      <span class="count">0</span>
+      <span id="txtBusy" class="count">0</span>
       <span class="desc">Busy</span>
     </a>
   </li>
   <li class="enqueued col-sm-1">
     <a href="/sidekiq/queues">
-      <span class="count">0</span>
+      <span id="txtEnqueued" class="count">0</span>
       <span class="desc">Enqueued</span>
     </a>
   </li>
   <li class="retries col-sm-1">
     <a href="/sidekiq/retries">
-      <span class="count">0</span>
+      <span id="txtRetries" class="count">0</span>
       <span class="desc">Retries</span>
     </a>
   </li>
   <li class="scheduled col-sm-1">
     <a href="/sidekiq/scheduled">
-      <span class="count">0</span>
+      <span id="txtScheduled" class="count">0</span>
       <span class="desc">Scheduled</span>
     </a>
   </li>
   <li class="dead col-sm-1">
     <a href="/sidekiq/morgue">
-      <span class="count">4</span>
+      <span id="txtDead" class="count">4</span>
       <span class="desc">Dead</span>
     </a>
   </li>
@@ -229,7 +227,7 @@
             <p class="navbar-text redis-url" title="redis://127.0.0.1:6379/0">redis://127.0.0.1:6379/0</p>
           </li>
           <li>
-            <p class="navbar-text server-utc-time">20:57:00 UTC</p>
+            <p id="serverUtcTime" class="navbar-text server-utc-time">20:57:00 UTC</p>
           </li>
           <li>
             <p class="navbar-text"><a style="color: gray;" href="https://github.com/mperham/sidekiq/wiki">docs</a></p>