s3_migration 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e1e5306758296ed3929c545d4607b6e05e439ffb274df1a4bb6f5af23b9ad803
+  data.tar.gz: 9b0c6999c349179acf6e3da35608510442efdf67863950af29387f5e6f217d8c
+SHA512:
+  metadata.gz: e74181493a0e761de5be0376a20f4f40dc8916b0bda396e72073b4c3c9463cd04f36443fc7fd635983882d8cff43fdd5b3c61d696fa383ffa3321baf9912b4b3
+  data.tar.gz: 49aaf2bf5292519ab484b0b44ea2b679a323ba5d99a7b2b28c609fd3af8a7490bbb22cf4e1e36cef62c20554d9c47bf71b90d425ed1848b918185e03c2635f41

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Max Kadel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,107 @@
+# s3_migration
+
+Move a Hyrax application's original files from disk storage to S3, with every copy verified and every step reversible.
+
+Original files often sit on a shared or network volume that costs far more per GB than object storage. This gem copies them to an S3 bucket through the [valkyrie-shrine](https://github.com/samvera-labs/valkyrie-shrine) adapter, checks each copy before the file's identifier is switched, and then helps you quarantine, roll back or purge the disk copies from a manifest, never by deleting a directory.
+
+## Requirements
+
+- Ruby 3.2 or newer, Rails 7.x and Hyrax 5.x, with Valkyrie as the metadata layer.
+- Originals on a plain `Valkyrie::Storage::Disk` adapter (`disk://` ids). The versioned disk adapter that `rails generate hyrax:install` sets up is not supported; the tasks stop with a message explaining why.
+- A `valkyrie-shrine` adapter registered as `:repository_s3`, for the bucket.
+
+## Installation
+
+```ruby
+gem 's3_migration', github: 'samvera-labs/s3_migration'
+```
+
+The rake tasks load with the Rails app; there is nothing to generate.
+
+## Usage
+
+Every task that changes anything is a dry run unless you set `APPLY=true`, and the migration itself reports what it would do with `DRY_RUN=true`.
+
+| Task | What it does |
+| --- | --- |
+| `s3_migration:report` | Read-only: files and gigabytes by use and storage scheme. |
+| `s3_migration:migrate` | Copy originals to S3, verify them, then switch their identifiers. The disk copy is never touched. |
+| `s3_migration:verify` | Read-only audit of what is already in S3. Get zero problems before deleting anything. |
+| `s3_migration:quarantine` | Re-check each migrated file against the manifest, then move its disk copy into `.quarantine`. |
+| `s3_migration:rollback` | Put a disk copy and its original identifier back, and remove the unused upload. |
+| `s3_migration:purge` | Delete quarantined copies older than `MIN_AGE_DAYS` (default 14). |
+
+Options: `DRY_RUN`, `INLINE` (run in the foreground instead of queueing jobs), `LIMIT=<n>` files, `APPLY`, `VERIFY_SHA=false`, `MIN_AGE_DAYS=<positive number>`, `S3_MIGRATION_QUEUE`, and `PROGRESS_EVERY=<n>` (`0` turns progress lines off).
+
+The full walkthrough, including creating the bucket, registering the adapter and reclaiming disk space safely, is in [docs/migrating-files-to-s3.md](docs/migrating-files-to-s3.md).
+
+### Multitenant applications
+
+The tasks work on whichever repository the app is pointed at, so run them once per tenant. If one bucket serves several tenants, set `S3Migration.scope_provider` to a callable returning the current tenant's label; manifest entries, and therefore quarantine, rollback and purge, are then limited to that tenant.
+
+## Development
+
+The specs run in a small Rails app under `spec/internal` that boots Hyrax with the Postgres Valkyrie adapter. They need only Postgres, not Fedora, Solr, Redis or S3.
+
+### Running the specs
+
+1. Start Postgres. [docker-compose.yaml](docker-compose.yaml) runs one on `localhost:5432` with user and password `postgres`, matching the default connection:
+
+   ```bash
+   docker compose up -d --wait postgres
+   ```
+
+   If port 5432 is taken (for example by a local Postgres), pick another and point the specs at it:
+
+   ```bash
+   POSTGRES_PORT=5433 docker compose up -d --wait postgres
+   export DATABASE_URL=postgres://postgres:postgres@localhost:5433/s3_migration_test
+   ```
+
+   Any other Postgres works too: set `DATABASE_URL` to it. Without `DATABASE_URL` the specs use `postgres://postgres:postgres@localhost:5432/s3_migration_test`.
+
+2. Install the gems (Ruby 3.2 or newer):
+
+   ```bash
+   bundle install
+   ```
+
+3. Run the specs. The first run creates the `s3_migration_test` database and loads `spec/internal/db/schema.rb` into it.
+
+   ```bash
+   bundle exec rspec                                  # whole suite
+   bundle exec rspec spec/s3_migration/file_migrator_spec.rb       # one file
+   bundle exec rspec spec/s3_migration/file_migrator_spec.rb:42    # one example, by line
+   bundle exec rspec --only-failures                  # re-run what failed last time
+   ```
+
+   A full run fails unless line coverage is 100%; the report is written to `coverage/index.html`. A run of a single file always reports low coverage, so ignore that failure when you filter.
+
+4. Lint with `bundle exec rubocop`, then stop Postgres when you are done:
+
+   ```bash
+   docker compose down        # add -v to delete the database volume too
+   ```
+
+### Troubleshooting
+
+- `PG::ConnectionBad` or `connection refused`: Postgres is not up yet, or `DATABASE_URL` points somewhere else. Check `docker compose ps`.
+- Schema changes: after editing `spec/internal/db/schema.rb`, run `docker compose down -v` and start again so the specs reload it into a fresh database.
+
+CI (GitHub Actions) runs rubocop and the specs on each push and pull request, and requires both to pass.
+
+Every pull request needs one release-note label, which the `Verify` check enforces: `patch-ver` (bug fix), `minor-ver` (new backward-compatible feature), `major-ver` (breaking change), `dependencies` (dependency bump, set by Dependabot) or `ignore-for-release` (CI, docs, tooling).
+
+### Releasing
+
+On an up-to-date, clean `main`, bump `S3Migration::VERSION` in `lib/s3_migration/version.rb` and merge that change. Then, signed in to rubygems.org (`gem signin`; the gem requires MFA):
+
+```bash
+bundle exec rake release
+```
+
+This tags `vX.Y.Z`, pushes the tag and publishes the gem. `bundle exec rake build` builds it into `pkg/` without publishing.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

data/docs/migrating-files-to-s3.md

@@ -0,0 +1,74 @@
+# Migrating original files from disk to S3
+
+Without an S3 storage adapter, Valkyrie writes original files to the disk adapter (`storage/files` by default) as `disk://` identifiers. That directory often ends up on a shared or network volume (NFS, EFS, a SAN, a mounted block device), which can cost far more per GB than object storage, and which has to be sized, backed up and shared between every app and worker process. This guide moves new and existing originals to S3 (or an S3-compatible service such as MinIO) so they stop living on that volume.
+
+Only originals move. Derivatives, uploads and similar stay on the shared storage directories.
+
+## Requirements
+
+- Originals are on a plain `Valkyrie::Storage::Disk` adapter (`disk://` ids), registered as `:disk`. If it is registered under another name, set `S3Migration::SourceStorage.adapter_name = :that_name`. Apps on a versioned disk adapter (`versiondisk://` ids, which `rails generate hyrax:install` sets up) or on Fedora are not supported: they keep file versions and deletion markers that a copy to S3 would lose. The tasks stop with a clear message in that case.
+- A Hyrax 5 application on Rails 7. The tasks read file sets and their original files through `Hyrax.query_service` and `Hyrax.custom_queries`, so Valkyrie must be the metadata layer (Wings-only apps are not covered).
+- The destination is a [valkyrie-shrine](https://github.com/samvera-labs/valkyrie-shrine) S3 adapter registered as `:repository_s3` (step 2).
+
+## 1. Create the bucket
+
+Use a private bucket with versioning on and public access blocked. Putting each client or deployment in its own bucket keeps access, lifecycle and cost accounting separate. Place it close to the app servers rather than to end users: the app reads files from the bucket and streams them to users, so distance between the app and the bucket adds latency and, on some providers, transfer cost.
+
+## 2. Register the S3 adapter and make it the default
+
+The app needs a Valkyrie storage adapter registered as `:repository_s3` (the generated `config/initializers/1_valkyrie.rb` has a commented example using the `REPOSITORY_S3_*` variables) and set as `Valkyrie.config.storage_adapter` so new files go there. Give the app read/write/delete and list access to the bucket, either through the credentials your platform provides (instance role, workload identity, environment credentials) or through the adapter's key options. Restart every process that writes files (web and background workers), and clear caches (including any IIIF/derivative caches).
+
+New files now get `shrine://` identifiers. Existing files keep working from disk until migrated.
+
+## 3. Migrate existing files
+
+`s3_migration:migrate` enqueues one background job per file set (`S3Migration::MigrateFilesJob` lists the disk files, `S3Migration::MigrateFileSetJob` moves each one), so speed scales with the number of job workers. Each job uploads the file, reads it back from S3 and compares its SHA-256 with the disk file, and only then switches the identifier; on a mismatch the upload is removed and the file stays on disk. The S3 adapter also verifies each upload on its own, comparing an MD5 of the source file with the object's ETag, which reads the source file one more time. Before uploading, the disk file is also checked against the checksum Hyrax recorded when the file was characterized (an MD5, optionally written as `urn:md5:`, `urn:sha1:` or `urn:sha256:`); a mismatch is reported as `suspect` and the file is not migrated. Transient errors retry. The disk copy is never touched, and the task can be re-run safely (migrated files are skipped).
+
+Start with a dry run, which runs inline and prints what it would do. It reads every file to check checksums, so it takes a while on large repositories:
+
+```bash
+DRY_RUN=true bundle exec rake s3_migration:migrate
+```
+
+Then a small real batch, run inline so you can read the output, followed by the full run, which enqueues jobs:
+
+```bash
+INLINE=true LIMIT=5 bundle exec rake s3_migration:migrate
+bundle exec rake s3_migration:migrate
+```
+
+Run these wherever you normally run rake tasks against the app (a console on an app host, a container, an orchestrator job).
+
+Things to know when running at scale:
+
+- Set `S3_MIGRATION_QUEUE` to run the jobs on a dedicated queue (your workers must be listening to it) so ingest and derivative jobs are not starved.
+- Per-file results are logged as `[S3Migration] <file set id> <status>`.
+- File sets are not re-indexed: the migration changes only where the bytes are stored, so the search index (including extracted full text) is left exactly as it was. Re-indexing would also re-read every file from S3 and re-run text extraction.
+- File sets with no original file (for example empty file sets left by an import) are reported as `NO_ORIGINAL` and skipped; nothing is moved or deleted for them.
+- Long inline runs, reports and audits print a progress line every 500 items with the elapsed time and rate. Change the interval with `PROGRESS_EVERY=<n>`, or set it to `0` to turn it off.
+- Every migrated file also gets a manifest object in the bucket (see step 5).
+- The live site reads from the same storage while this runs, so start with few workers and watch the storage's throughput and latency.
+
+### Applications that partition one repository (for example multitenant apps)
+
+The tasks work on the repository the app is currently pointed at, so run them once per partition with whatever mechanism your app has for that. If one bucket serves several partitions, set `S3Migration.scope_provider` to a callable returning a label for the current partition (for example in an initializer); manifest entries are then stored per label, so the quarantine and rollback steps only touch the current partition's files.
+
+## 4. Verify
+
+- Audit what is in S3 (read-only; checks the object exists, the recorded size and the stored checksum): `bundle exec rake s3_migration:verify`. Expect `Problems: 0`.
+- A newly created work's original has a `shrine://` identifier.
+- Old and new works still display their files (viewer/IIIF, downloads) after clearing caches.
+- Thumbnails are unaffected (they are derivatives).
+- A re-run of the migration reports nothing left to migrate.
+
+## 5. Reclaim the disk space
+
+Every migrated file is recorded in a manifest object in the bucket (`_migration-manifests/<file set id>.json`: file set and file IDs, old disk id, new S3 id, size, SHA-256). Deletion is manifest-driven, never `rm -rf storage/files`.
+
+1. Take a snapshot or backup of the storage volume, and keep bucket versioning on.
+2. `bundle exec rake s3_migration:report` lists what is still on disk by file use and storage scheme. The migration only moves originals; extracted text, other file versions and similar may still be `disk://` and are not covered by the manifest.
+3. Dry-run, then apply: `bundle exec rake s3_migration:quarantine` (add `APPLY=true`). For each file it re-checks that the database still points at the S3 id and that the S3 object matches the manifest size and SHA-256, then moves the disk copy to `<storage>/.quarantine` (a rename within the same volume).
+4. Soak (at least a week or two, through a backup cycle), re-checking the site. To undo, `bundle exec rake s3_migration:rollback` (dry-run by default, `APPLY=true`) checks the disk copy against the manifest, restores it and the original identifier, then removes the S3 upload and manifest entry that are no longer used (bucket versioning keeps the removed object recoverable).
+5. Permanently delete: `bundle exec rake s3_migration:purge` (dry-run by default; `APPLY=true`, `MIN_AGE_DAYS=14`, which must be a positive number). Like every step above it works from the manifest: it deletes only the quarantined copies of files in the manifest scope it runs under (see `S3Migration.scope_provider`), and leaves anything else in `.quarantine`.
+
+Do a non-production environment first and production after, checking again between each. Storage is usually billed or provisioned by stored bytes, so the saving only appears after step 5 (and, on volumes you size by hand, after you shrink them). Derivatives often remain on the volume; size them separately. If your storage offers cheaper tiers for rarely read files, enabling them is a lower-risk way to cut cost while you work through the steps above.

data/lib/s3_migration/auditor.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Read-only check of one original file already in S3: the object must be readable, match the
+  # recorded size, and match the stored checksum when one is present.
+  class Auditor
+    Result = Struct.new(:status, :problems, keyword_init: true)
+
+    def initialize(original:, storage: Valkyrie::StorageAdapter)
+      @original = original
+      @storage = storage
+    end
+
+    # @return [Result] status is :ok, :problem or :not_in_s3
+    def call
+      return Result.new(status: :not_in_s3, problems: []) unless original.file_identifier.to_s.start_with?('shrine://')
+
+      s3_file = storage.find_by(id: original.file_identifier)
+      problems = size_problems(s3_file) + checksum_problems(s3_file)
+      Result.new(status: problems.empty? ? :ok : :problem, problems:)
+    end
+
+    private
+
+    attr_reader :original, :storage
+
+    def size_problems(s3_file)
+      recorded = original.recorded_size.first
+      return [] unless recorded && s3_file.size.to_i != recorded.to_i
+
+      ["size #{s3_file.size} != recorded #{recorded}"]
+    end
+
+    def checksum_problems(s3_file)
+      stored = StoredChecksum.for(original)
+      return [] unless stored
+
+      actual = s3_file.checksum(digests: [stored.digest_class.new]).first
+      actual == stored.digest ? [] : ["#{stored.algorithm} #{actual} != stored #{stored.digest}"]
+    end
+  end
+end

data/lib/s3_migration/file_lock.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Holds a lock on one file set while a block runs, so two jobs never migrate it at the same time (a double
+  # enqueue, or a retry while the first attempt is still running). It is the Redis lock Hyrax takes for works.
+  # A job that cannot get the lock in time raises Hyrax::LockManager::UnableToAcquireLockError and is retried.
+  module FileLock
+    DEFAULT_TTL_SECONDS = 900
+
+    def self.synchronize(file_set_id, &)
+      manager.lock("s3_migration:#{file_set_id}", &)
+    end
+
+    # The lock expires on its own if a worker dies holding it. S3_MIGRATION_LOCK_TTL (seconds) must exceed the
+    # time to upload the largest original.
+    def self.manager
+      ttl = ENV.fetch('S3_MIGRATION_LOCK_TTL', DEFAULT_TTL_SECONDS).to_i * 1000
+      Hyrax::LockManager.new(ttl, Hyrax.config.lock_retry_count, Hyrax.config.lock_retry_delay)
+    end
+  end
+end

data/lib/s3_migration/file_migrator.rb

@@ -0,0 +1,148 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Copies one original file from the disk storage adapter to the S3 repository adapter and
+  # switches its identifier only after the copy is verified. Never touches the disk copy.
+  # Unexpected errors are raised (so a job can retry); an upload the database does not point at is removed.
+  class FileMigrator
+    Result = Struct.new(:status, :message, :entry, keyword_init: true)
+
+    CHUNK_SIZE = 1_048_576
+
+    # @param file_set [Valkyrie::Resource] a file set
+    # @param original [Hyrax::FileMetadata] the file set's original file
+    def initialize(file_set:, original:, dry_run: false, s3_storage: nil, manifest: nil)
+      @file_set = file_set
+      @original = original
+      @dry_run = dry_run
+      @s3_storage = s3_storage || Valkyrie::StorageAdapter.find(:repository_s3)
+      @manifest = manifest
+    end
+
+    # @return [Result] status is one of :migrated, :would_migrate, :already_migrated,
+    #   :unrecognised, :missing, :suspect, :mismatch or :raced
+    def call
+      skipped = precheck
+      return skipped if skipped
+      return result(:would_migrate) if dry_run
+
+      FileLock.synchronize(file_set.id) { migrate }
+    end
+
+    private
+
+    attr_reader :file_set, :original, :dry_run, :s3_storage
+
+    def manifest
+      @manifest ||= Manifest.new
+    end
+
+    def identifier
+      original.file_identifier.to_s
+    end
+
+    def disk_path
+      SourceStorage.path_for(identifier)
+    end
+
+    def filename
+      Array(original.original_filename).first || File.basename(disk_path)
+    end
+
+    def precheck
+      return result(:already_migrated) if identifier.start_with?('shrine://')
+      return result(:unrecognised, identifier) unless SourceStorage.handles?(identifier)
+      return result(:missing, disk_path) unless File.exist?(disk_path)
+
+      suspect = stored_checksum_mismatch
+      result(:suspect, "disk file does not match stored checksum #{suspect}") if suspect
+    end
+
+    def stored_checksum
+      return @stored_checksum if defined?(@stored_checksum)
+
+      @stored_checksum = StoredChecksum.for(original)
+    end
+
+    def stored_checksum_mismatch
+      stored = stored_checksum
+      stored.to_s if stored && disk_digests[:stored] != stored.digest
+    end
+
+    # One pass feeds both digests. The S3 adapter then reads the file again to upload it, and once more to
+    # verify the upload (its ETag check).
+    def disk_digests
+      @disk_digests ||= begin
+        sha256 = Digest::SHA256.new
+        stored = stored_checksum&.digest_class&.new
+        File.open(disk_path, 'rb') do |io|
+          while (chunk = io.read(CHUNK_SIZE))
+            sha256 << chunk
+            stored << chunk if stored
+          end
+        end
+        { sha256: sha256.hexdigest, stored: stored&.hexdigest }
+      end
+    end
+
+    def migrate
+      uploaded = nil
+      disk_sha256 = disk_digests[:sha256]
+      uploaded = upload_original
+      outcome = if copy_matches?(uploaded, disk_sha256)
+                  switch_identifier(uploaded, disk_sha256)
+                else
+                  result(:mismatch, 'S3 copy does not match disk (size/SHA-256)')
+                end
+    ensure
+      discard_upload(uploaded) if uploaded && outcome&.status != :migrated
+    end
+
+    def switch_identifier(uploaded, disk_sha256)
+      current = Hyrax.query_service.find_by(id: original.id)
+      unless SourceStorage.handles?(current.file_identifier)
+        return result(:raced, 'already migrated by another process')
+      end
+
+      entry = build_entry(uploaded, disk_sha256)
+      manifest.write(entry)
+      persist(current, uploaded)
+      result(:migrated, nil, entry)
+    end
+
+    # The adapter checks the copy inside upload, so a failure there leaves an object under a key we never see.
+    def upload_original
+      watch = UploadWatch.new(s3_storage:, file_set_id: file_set.id)
+      File.open(disk_path, 'rb') do |io|
+        s3_storage.upload(file: io, original_filename: filename, resource: file_set)
+      end
+    rescue StandardError
+      watch&.discard_new
+      raise
+    end
+
+    def discard_upload(uploaded)
+      UploadCleanup.new(file_set_id: file_set.id, file_metadata_id: original.id, s3_storage:,
+                        manifest:).call(uploaded.id)
+    end
+
+    def copy_matches?(uploaded, disk_sha256)
+      s3_storage.find_by(id: uploaded.id).valid?(size: File.size(disk_path), digests: { 'sha256' => disk_sha256 })
+    end
+
+    def persist(current, uploaded)
+      current.file_identifier = uploaded.id
+      Hyrax.persister.save(resource: current)
+    end
+
+    def build_entry(uploaded, disk_sha256)
+      { file_set_id: file_set.id.to_s,
+        file_metadata_id: original.id.to_s, old_id: identifier, new_id: uploaded.id.to_s,
+        size: File.size(disk_path), sha256: disk_sha256, at: Time.current.utc.iso8601 }
+    end
+
+    def result(status, message = nil, entry = nil)
+      Result.new(status:, message:, entry:)
+    end
+  end
+end

data/lib/s3_migration/manifest.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # One JSON object per migrated file set under PREFIX in the repository bucket, written through
+  # the S3 storage adapter, so parallel jobs never share a buffer and no separate S3 configuration
+  # is needed. Entries are kept per S3Migration.scope when one is set.
+  class Manifest
+    PREFIX = '_migration-manifests'
+
+    def initialize(storage: nil, scope: S3Migration.scope)
+      @shrine = (storage || Valkyrie::StorageAdapter.find(:repository_s3)).shrine
+      @scope = scope
+    end
+
+    def write(entry)
+      shrine.upload(StringIO.new(entry.to_json), key_for(entry.fetch(:file_set_id)))
+    end
+
+    def delete(file_set_id)
+      shrine.delete(key_for(file_set_id))
+    end
+
+    def read(file_set_id)
+      parse(shrine.open(key_for(file_set_id)))
+    rescue Shrine::FileNotFound
+      nil
+    end
+
+    def each_entry
+      return enum_for(:each_entry) unless block_given?
+
+      shrine.bucket.objects(prefix: "#{[shrine.prefix, PREFIX, scope].compact.join('/')}/").each do |object|
+        yield parse(object.get.body)
+      end
+    end
+
+    private
+
+    attr_reader :shrine, :scope
+
+    def key_for(file_set_id)
+      "#{[PREFIX, scope].compact.join('/')}/#{file_set_id}.json"
+    end
+
+    def parse(io)
+      JSON.parse(io.read, symbolize_names: true)
+    end
+  end
+end

data/lib/s3_migration/migrate_file_set_job.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Moves one file set's original file from disk to S3. Unexpected errors are raised so the job retries;
+  # verification failures (suspect, mismatch, ...) are logged and left on disk.
+  class MigrateFileSetJob < ActiveJob::Base
+    queue_as { ENV.fetch('S3_MIGRATION_QUEUE', 'default') }
+    retry_on StandardError, wait: :polynomially_longer, attempts: 5
+
+    def perform(file_set_id, dry_run: false)
+      file_set = Hyrax.query_service.find_by(id: file_set_id)
+      original = Hyrax.custom_queries.find_original_file(file_set:)
+      return unless original
+
+      result = S3Migration::FileMigrator.new(file_set:, original:, dry_run:).call
+      Rails.logger.info("[S3Migration] #{file_set_id} #{result.status} #{result.message}".strip)
+      result
+    rescue Valkyrie::Persistence::ObjectNotFoundError
+      Rails.logger.info("[S3Migration] #{file_set_id} not found; skipping")
+    end
+  end
+end

data/lib/s3_migration/migrate_files_job.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Enqueues a MigrateFileSetJob for every file set whose original is still on disk.
+  class MigrateFilesJob < ActiveJob::Base
+    queue_as { ENV.fetch('S3_MIGRATION_QUEUE', 'default') }
+
+    def perform(limit: nil)
+      enqueued = 0
+      S3Migration.each_file_set do |file_set|
+        break if limit && enqueued >= limit
+        next unless on_disk?(file_set)
+
+        MigrateFileSetJob.perform_later(file_set.id.to_s)
+        enqueued += 1
+      end
+      Rails.logger.info("[S3Migration] enqueued #{enqueued} file sets")
+      enqueued
+    end
+
+    private
+
+    def on_disk?(file_set)
+      on_error = ->(fs, error) { Rails.logger.warn("[S3Migration] #{fs.id} has no readable original: #{error.class}") }
+      S3Migration.original_for(file_set, on_error:)&.file_identifier.then { |identifier| identifier && S3Migration::SourceStorage.handles?(identifier) }
+    end
+  end
+end

data/lib/s3_migration/progress.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module S3Migration
+  # Prints a progress line every PROGRESS_EVERY items (default 500; 0 turns it off), so long runs show
+  # signs of life, how fast they are going, and what they have found so far.
+  class Progress
+    # @param detail [#call, nil] returns extra text for each line, e.g. running counts
+    def initialize(every: ENV.fetch('PROGRESS_EVERY', 500).to_i, out: $stdout, detail: nil, clock: lambda {
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    })
+      @every = every
+      @out = out
+      @detail = detail
+      @clock = clock
+      @started = clock.call
+      @count = 0
+    end
+
+    def tick
+      @count += 1
+      report if @every.positive? && (@count % @every).zero?
+    end
+
+    private
+
+    def report
+      elapsed = @clock.call - @started
+      rate = @count / [elapsed, 0.001].max
+      line = format('  ... %<count>d examined (%<rate>.1f/s, %<elapsed>s elapsed)',
+                    count: @count, rate:, elapsed: duration(elapsed))
+      line += "  #{@detail.call}" if @detail
+      @out.puts(line)
+      @out.flush
+    end
+
+    def duration(seconds)
+      minutes, secs = seconds.to_i.divmod(60)
+      hours, minutes = minutes.divmod(60)
+      return format('%<hours>d:%<minutes>02d:%<secs>02d', hours:, minutes:, secs:) if hours.positive?
+
+      format('%<minutes>d:%<secs>02d', minutes:, secs:)
+    end
+  end
+end