active_storage-async_variants 0.8.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -0
- data/CLAUDE.md +13 -7
- data/README.md +30 -57
- data/Rakefile +1 -20
- data/app/controllers/active_storage/async_variants/callbacks_controller.rb +28 -19
- data/config/routes.rb +0 -17
- data/db/migrate/20260304000001_add_state_columns_to_variant_records.rb +9 -0
- data/gemfiles/rails_7.2.gemfile +0 -4
- data/gemfiles/rails_8.0.gemfile +0 -4
- data/gemfiles/rails_8.1.gemfile +0 -4
- data/lib/active_storage/async_variants/attachment_extension.rb +19 -2
- data/lib/active_storage/async_variants/blob_extension.rb +3 -1
- data/lib/active_storage/async_variants/named_variant_scan.rb +55 -0
- data/lib/active_storage/async_variants/preview_extension.rb +52 -37
- data/lib/active_storage/async_variants/process_job.rb +53 -17
- data/lib/active_storage/async_variants/reflection_extension.rb +2 -2
- data/lib/active_storage/async_variants/registry.rb +1 -1
- data/lib/active_storage/async_variants/transformer.rb +32 -1
- data/lib/active_storage/async_variants/variant_record_extension.rb +15 -0
- data/lib/active_storage/async_variants/variant_with_record_extension.rb +11 -30
- data/lib/active_storage/async_variants/variation_extension.rb +8 -5
- data/lib/active_storage/async_variants/version.rb +1 -1
- data/lib/active_storage/async_variants.rb +31 -34
- metadata +4 -47
- data/app/assets/javascripts/progress-bar.js +0 -347
- data/app/assets/javascripts/retry.js +0 -11
- data/app/assets/stylesheets/progress.css +0 -35
- data/app/assets/stylesheets/retry.css +0 -98
- data/app/controllers/active_storage/async_variants/states_controller.rb +0 -71
- data/app/views/active_storage/async_variants/states/_failed.html.erb +0 -44
- data/app/views/active_storage/async_variants/states/_pending.html.erb +0 -1
- data/app/views/active_storage/async_variants/states/_processed.html.erb +0 -5
- data/app/views/active_storage/async_variants/states/_processing.html.erb +0 -15
- data/app/views/active_storage/async_variants/states/show.html.erb +0 -10
- data/features/retry_flow.feature +0 -20
- data/features/retry_visibility.feature +0 -23
- data/features/state_rendering.feature +0 -40
- data/features/step_definitions/dummy_steps.rb +0 -144
- data/features/support/env.rb +0 -28
- data/lib/active_storage/async_variants/asset_tag_helper_extension.rb +0 -59
- data/lib/active_storage/async_variants/helper.rb +0 -80
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: fffff13cd7b602bc0139ae60dff32c4d38d573b738edaa244f21df5dfa331723
|
|
4
|
+
data.tar.gz: 1251c74b70fd15eaaf1e48e494b7fc9d773f2989f83e1b99ca6862fdaec6c144
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2bfef2ba7b3d4576d20a867095c7e75f8f36ab122d0718901856ce0bab8330521f5edeb73a0e75a12f3ed20250fe28c25304b4823929167daa910d240a7b9477
|
|
7
|
+
data.tar.gz: f0e5e7a9cfe3d75ff7afe960fc7675447b466dc6e8500b7db6fca3c6a24cd8d0dfa837e938b216898a8ee8d95f470c1aa80b03be7c407e8e0ee7a63a46d74775
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,26 @@
|
|
|
1
|
+
## [0.10.0]
|
|
2
|
+
|
|
3
|
+
- **Breaking:** Async previews now persist the same graph stock Active Storage builds. For a previewable blob (a video), an image-format variant is a `Preview`: the extracted frame is attached as the source blob's `preview_image`, and the variant record hangs off the *frame* blob, not the source. `ProcessJob` resolves the attachment's `representation` and takes the preview path when stock would. External transformers receive a frame placeholder (`byte_size: 0, checksum: "0"`) to write into; the success callback reconciles both placeholders. Read video stills through `representation`/`preview`, not `variant`.
|
|
4
|
+
- **Breaking:** Naming a `transformer:` is now the only async opt-in. The `async:` option is gone and raises `ArgumentError`. The built-in `ActiveStorage::AsyncVariants::Standard` transformer replaces a bare `async: true` (the stock variant pipeline, run in the background). A transformer's kind follows from what it overrides: `Transformer#standard?` (neither), `#inline?` (`process`), `#external?` (`initiate`).
|
|
5
|
+
- **Breaking:** Attaching a file enqueues an async variant only when it declares Active Storage's `preprocessed:` option (`true`, a method name, or a proc evaluated against the record). Previously every variant naming a `transformer:` was enqueued on every attach, whatever the file — so an image upload also ran the attachment's video transcode. Stock Active Storage no longer sees async variants when preprocessing, so `preprocessed: true` never also queues a `TransformJob` or a local `PreviewImageJob`.
|
|
6
|
+
- Migration note: existing variant records for video stills live on the source blob under the old structure. Move them onto the frame blob (`UPDATE active_storage_variant_records SET blob_id = <frame>`), attaching a placeholder frame where none exists; the `variation_digest` is unchanged. Records whose output is itself a video (transcodes) stay on the source. Replace `async: true` with `transformer: ActiveStorage::AsyncVariants::Standard` where no transformer was named, and drop it where one was. Add `preprocessed: true` (or a condition) to each async variant that should process on attach.
|
|
7
|
+
- `Preview` speaks the same progress protocol as `VariantWithRecord`: `async_state`, `progress`, `progress_known?`, `progress_rate`, `last_heartbeat_at`, `error`.
|
|
8
|
+
- `async_record` is public on both `VariantWithRecord` and `Preview`, and both gain `retry!` (drop a failed record, then `enqueue!`), so callers need not know where a representation keeps its record.
|
|
9
|
+
- `NamedVariantScan` compares declarations structurally instead of instantiating `attachment.variant`, which raises for videos.
|
|
10
|
+
- Fix: declaring `preprocessed:` alongside a `transformer:` no longer breaks the declaration-time registry warm-up (a proc raised on `Marshal.dump`; other values registered the wrong digest).
|
|
11
|
+
- MirrorService: the success callback mirrors reconciled variant blobs to the mirrors, as Rails does for direct uploads, and `bucket_backed?` resolves through the primary service.
|
|
12
|
+
- Frame placeholder creation is serialized on the blob row: every image variant of one video reaches it concurrently, and a second `preview_image` attachment would be invisible to `has_one`, stranding the records hanging off it.
|
|
13
|
+
- Ships the `state`/`error`/`attempts` migration for `active_storage_variant_records` (guarded with `if_not_exists`, so apps that added the columns themselves can install it harmlessly), and builds the test schema from `db/migrate` so a missing shipped migration fails the suite.
|
|
14
|
+
- Fix: `ProcessJob` skips a variant record that is already processed or whose run is still heartbeating. Previously every enqueue re-dispatched the variant — attaching an already-processed blob to another record, a retry, or a duplicate job paid for a second render and flipped processed media back to processing.
|
|
15
|
+
- Fix: the success callback applies the reported `byte_size` and `checksum` whenever they arrive, not only over the `0`/`"0"` placeholders, since a re-render writes over the record's existing output blob. Reported `preview_image_*` values land only on a frame, never on a variant's source blob.
|
|
16
|
+
- Fix: destroying a variant record purges any output a racing render attached after the destroy read its attachments, so a purged source no longer leaves outputs pointing at a deleted record.
|
|
17
|
+
|
|
18
|
+
## [0.9.0]
|
|
19
|
+
|
|
20
|
+
- **Breaking:** Split the opinionated UI out into a separate gem, [`active_storage-async_variants-ui`](https://github.com/botandrose/active_storage-async_variants-ui). This gem is now UI-free plumbing — the async state machine, `ProcessJob`, the heartbeat watchdog, the external-service callback endpoint, and the `Transformer` base class. Its only runtime dependency is `activestorage` (no more `turbo-rails` / `isolate_assets`).
|
|
21
|
+
- Moved to the UI gem: `image_tag`/`video_tag` with `async:`/`direct:`, the `<turbo-frame>` state endpoint and partials, the circular progress bar, the filename-bearing placeholder route, the retry affordance, and the `cdn_host` / `parent_controller` / `retry_visible_if` configuration. A variant's `.url` (serving the original until processed) is unchanged.
|
|
22
|
+
- Migration note: apps that relied on the rendering helpers should add `gem "active_storage-async_variants-ui"`. Apps using `active_storage-crucible` or only the variant pipeline need no change.
|
|
23
|
+
|
|
1
24
|
## [0.8.0]
|
|
2
25
|
|
|
3
26
|
- The processing progress bar now advances smoothly between polls: variants expose a `rate` (percent-per-second, derived from the record's `created_at`, last heartbeat, and reported progress) that the `@botandrose/progress-bar` element uses to optimistically creep forward, instead of only stepping on each refresh. Includes a migration adding a `created_at` column to `active_storage_variant_records`; run it before deploying.
|
data/CLAUDE.md
CHANGED
|
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|
|
4
4
|
|
|
5
5
|
## What This Is
|
|
6
6
|
|
|
7
|
-
A Rails engine gem that extends Active Storage with async-safe variant processing. Solves the problem where slow transformations (e.g., video transcoding) block requests or fail silently.
|
|
7
|
+
A Rails engine gem that extends Active Storage with async-safe variant processing. Solves the problem where slow transformations (e.g., video transcoding) block requests or fail silently. Naming a `transformer:` on a variant definition opts it into async processing.
|
|
8
8
|
|
|
9
9
|
## Commands
|
|
10
10
|
|
|
@@ -19,17 +19,23 @@ bundle exec rspec spec/active_storage/async_variants_spec.rb -e "description" #
|
|
|
19
19
|
|
|
20
20
|
The gem works by prepending extension modules onto Active Storage classes:
|
|
21
21
|
|
|
22
|
-
- **`VariationExtension`** → `ActiveStorage::Variation` — extracts
|
|
23
|
-
- **`AttachmentExtension`** → `ActiveStorage::Attachment` — hooks into `transform_variants_later` to enqueue `ProcessJob` for variants
|
|
22
|
+
- **`VariationExtension`** → `ActiveStorage::Variation` — extracts the `transformer:` option from variant config before passing the rest to standard Active Storage
|
|
23
|
+
- **`AttachmentExtension`** → `ActiveStorage::Attachment` — hooks into `transform_variants_later` to enqueue `ProcessJob` for async variants whose `preprocessed:` holds for the record, hiding async variants from stock preprocessing
|
|
24
24
|
- **`VariantWithRecordExtension`** → `ActiveStorage::VariantWithRecord` — overrides URL generation to serve the original blob while not ready; adds state query methods (`processed?`, `processing?`, `pending?`, `failed?`)
|
|
25
|
-
- **`
|
|
25
|
+
- **`PreviewExtension`** → `ActiveStorage::Preview` — blocks the synchronous preview transform for async variants and resolves state/URLs through the stock preview graph
|
|
26
|
+
- **`ProcessJob`** — background job that resolves the attachment's representation (variant vs preview), determines transformer type (inline vs external), and processes accordingly
|
|
27
|
+
|
|
28
|
+
### Preview Structure
|
|
29
|
+
|
|
30
|
+
Async previews persist the same graph stock Active Storage builds: the extracted frame is attached as the source blob's `preview_image`, and the variant record hangs off the *frame* blob, not the source. External transformers get a frame placeholder (`byte_size: 0, checksum: "0"`) the service writes into; the success callback reconciles both the variant and frame placeholders. Inline/default transformers extract the real frame up front via the stock previewer.
|
|
26
31
|
|
|
27
32
|
### Transformer Types
|
|
28
33
|
|
|
29
|
-
|
|
30
|
-
- **External**: implements `initiate(source_url:, destination_url:, callback_url:, **options)` → frees worker immediately, external service POSTs to callback URL when done
|
|
34
|
+
Naming a `transformer:` is the async opt-in; the kind is determined by what the transformer overrides (`Transformer#standard?`/`#inline?`/`#external?`):
|
|
31
35
|
|
|
32
|
-
|
|
36
|
+
- **Standard** (`ActiveStorage::AsyncVariants::Standard`): overrides neither → runs the stock Active Storage variant pipeline in the background. Replaces the old bare `async: true` (no transformer) declaration, which now raises.
|
|
37
|
+
- **Inline**: overrides `process(file, **options)` → blocks worker, returns `{ io:, content_type:, filename: }`
|
|
38
|
+
- **External**: overrides `initiate(source_url:, callback_url:, **options)` → frees worker immediately, external service POSTs to callback URL when done
|
|
33
39
|
|
|
34
40
|
### Callback Endpoint
|
|
35
41
|
|
data/README.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Extends Active Storage with pluggable per-variant transformers, async-safe variant processing, and failure handling.
|
|
4
4
|
|
|
5
|
+
This gem is the UI-free plumbing. A variant's `.url` serves the original while it is pending/processing/failed, and the processed variant once ready — nothing renders a spinner or progress bar. For the optional turbo-frame UI (`image_tag`/`video_tag` with `async:`/`direct:`, a circular progress bar, and a retry affordance), add the companion gem [`active_storage-async_variants-ui`](https://github.com/botandrose/active_storage-async_variants-ui).
|
|
6
|
+
|
|
5
7
|
## The Problem
|
|
6
8
|
|
|
7
9
|
Active Storage's variant system assumes transformations are fast and reliable -- like generating an image thumbnail. But some transformations are slow (transcoding a 1GB video to 720p VP9) and fallible (the transcode may permanently fail). When you use `process: :later`, Active Storage enqueues a background job, but if the variant is requested before the job finishes, it falls through to synchronous processing -- blocking the request for minutes or timing out entirely. And if the transformation fails, the error bubbles up with no tracking or retry limits.
|
|
@@ -19,7 +21,7 @@ bin/rails db:migrate
|
|
|
19
21
|
|
|
20
22
|
## Usage
|
|
21
23
|
|
|
22
|
-
|
|
24
|
+
Name a `transformer:` on any variant to opt it into the async pipeline:
|
|
23
25
|
|
|
24
26
|
```ruby
|
|
25
27
|
class User < ApplicationRecord
|
|
@@ -27,102 +29,73 @@ class User < ApplicationRecord
|
|
|
27
29
|
attachable.variant :web,
|
|
28
30
|
transformer: VideoTranscoder,
|
|
29
31
|
codec: "vp9",
|
|
30
|
-
resolution: "720p"
|
|
31
|
-
async: true
|
|
32
|
+
resolution: "720p"
|
|
32
33
|
end
|
|
33
34
|
end
|
|
34
35
|
```
|
|
35
36
|
|
|
36
|
-
The presence of
|
|
37
|
+
The presence of a `transformer:` is what opts a variant into async processing. Without it, variants behave exactly as they do in standard Active Storage. The transformer also says *how* the work happens — there are three kinds, distinguished by what they implement:
|
|
37
38
|
|
|
38
39
|
```ruby
|
|
39
40
|
has_one_attached :video do |attachable|
|
|
40
|
-
#
|
|
41
|
+
# External: hands off to a remote service (implements #initiate)
|
|
41
42
|
attachable.variant :web,
|
|
42
43
|
transformer: VideoTranscoder,
|
|
43
|
-
codec: "vp9"
|
|
44
|
-
async: true
|
|
44
|
+
codec: "vp9"
|
|
45
45
|
|
|
46
|
-
#
|
|
46
|
+
# Standard: the stock Active Storage variant pipeline, run in the background
|
|
47
|
+
# (e.g. a large image resize that's too slow for an inline request)
|
|
47
48
|
attachable.variant :thumbnail,
|
|
48
49
|
resize_to_limit: [200, 200],
|
|
49
|
-
|
|
50
|
+
transformer: ActiveStorage::AsyncVariants::Standard
|
|
50
51
|
|
|
51
|
-
#
|
|
52
|
+
# Inline: custom processing in the worker (implements #process)
|
|
52
53
|
attachable.variant :watermarked,
|
|
53
54
|
transformer: WatermarkStamper
|
|
54
55
|
end
|
|
55
56
|
```
|
|
56
57
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
```erb
|
|
60
|
-
<%= video_tag user.video.variant(:web).url %>
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
While the variant is still processing (or has failed), this serves the original video. Once processing completes, it serves the transcoded variant.
|
|
64
|
-
|
|
65
|
-
## `image_tag` / `video_tag` with `async:` and `direct:`
|
|
66
|
-
|
|
67
|
-
For a polish layer that shows a progress bar while a variant is being processed, polls for completion in the background, and optionally serves the finished variant straight from your CDN, the gem adds two options to `image_tag` and `video_tag`:
|
|
68
|
-
|
|
69
|
-
```erb
|
|
70
|
-
<%# Variant URL with the async client wired up: a progress bar while pending,
|
|
71
|
-
polls for completion, swaps to the real image when ready. %>
|
|
72
|
-
<%= image_tag user.avatar.variant(:web), async: true %>
|
|
58
|
+
A variant with no `transformer:` is a plain synchronous Active Storage variant — the gem stays out of the way.
|
|
73
59
|
|
|
74
|
-
|
|
75
|
-
routing through Rails. While pending/failed, falls back to the
|
|
76
|
-
Rails representation URL (which serves the original). %>
|
|
77
|
-
<%= image_tag user.avatar.variant(:web), direct: true %>
|
|
60
|
+
### Processing on attach
|
|
78
61
|
|
|
79
|
-
|
|
80
|
-
polling for completion. %>
|
|
81
|
-
<%= image_tag user.avatar.variant(:web), async: true, direct: true %>
|
|
62
|
+
An async variant is processed when a file is attached only if it says so with Active Storage's own `preprocessed:` option. It takes `true`, a method name, or a proc, evaluated against the record — so one attachment that accepts both images and videos can transcode only the videos:
|
|
82
63
|
|
|
83
|
-
|
|
84
|
-
|
|
64
|
+
```ruby
|
|
65
|
+
has_one_attached :media do |attachable|
|
|
66
|
+
attachable.variant :thumb, resize_to_limit: [150, 150], format: "webp",
|
|
67
|
+
transformer: Crucible, preprocessed: true
|
|
68
|
+
attachable.variant :transcoded, resize_to_limit: [1280, 720], format: "mp4",
|
|
69
|
+
transformer: Crucible, preprocessed: :video?
|
|
70
|
+
end
|
|
85
71
|
```
|
|
86
72
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
### Configure the direct URL host
|
|
73
|
+
An async variant without `preprocessed:` is never enqueued on attach; call `enqueue!` on it when you want it. Stock Active Storage never preprocesses an async variant itself — the gem's `ProcessJob` is the only thing that does.
|
|
90
74
|
|
|
91
|
-
|
|
75
|
+
In views, use the same Active Storage helpers:
|
|
92
76
|
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
ActiveStorage::AsyncVariants.cdn_host = "https://d1234abcd.cloudfront.net"
|
|
77
|
+
```erb
|
|
78
|
+
<%= video_tag user.video.variant(:web).url %>
|
|
96
79
|
```
|
|
97
80
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
### JavaScript
|
|
101
|
-
|
|
102
|
-
No manual wiring is required. The async state partials are self-contained `<turbo-frame>`s. The only requirement is that the host app loads **Turbo** -- the gem depends on `turbo-rails`, which a default Rails app already includes.
|
|
81
|
+
While the variant is still processing (or has failed), this serves the original video. Once processing completes, it serves the transcoded variant.
|
|
103
82
|
|
|
104
83
|
## Configuration
|
|
105
84
|
|
|
106
|
-
Set options in an initializer. The `configure` block groups them (each is also a plain accessor, e.g. `ActiveStorage::AsyncVariants.
|
|
85
|
+
Set options in an initializer. The `configure` block groups them (each is also a plain accessor, e.g. `ActiveStorage::AsyncVariants.heartbeat_interval = …`):
|
|
107
86
|
|
|
108
87
|
```ruby
|
|
109
88
|
# config/initializers/active_storage_async_variants.rb
|
|
110
89
|
ActiveStorage::AsyncVariants.configure do |config|
|
|
111
|
-
config.cdn_host = "https://d1234abcd.cloudfront.net"
|
|
112
90
|
config.heartbeat_interval = 5.seconds
|
|
113
91
|
config.heartbeat_stale_after = 60.seconds
|
|
114
|
-
config.parent_controller = "ApplicationController"
|
|
115
|
-
config.retry_visible_if { current_user&.admin? }
|
|
116
92
|
end
|
|
117
93
|
```
|
|
118
94
|
|
|
119
95
|
| Option | Default | Purpose |
|
|
120
96
|
|--------|---------|---------|
|
|
121
|
-
| `
|
|
122
|
-
| `heartbeat_interval` | `5.seconds` | Expected cadence of progress heartbeats; the processing `<turbo-frame>` re-polls at this rate. |
|
|
97
|
+
| `heartbeat_interval` | `5.seconds` | Expected cadence of progress heartbeats from external transformers. (The UI gem's processing `<turbo-frame>` re-polls at this rate.) |
|
|
123
98
|
| `heartbeat_stale_after` | `60.seconds` | A processing variant with no heartbeat for this long is marked `failed`. Must exceed `heartbeat_interval`. |
|
|
124
|
-
| `parent_controller` | `"ActionController::Base"` | Base class for the gem's controllers, so the retry view can reach your app's `current_user`. Set as a String. |
|
|
125
|
-
| `retry_visible_if` | off | Block (run in the view context) gating the failed-state retry affordance. |
|
|
126
99
|
|
|
127
100
|
## Writing a Transformer
|
|
128
101
|
|
|
@@ -229,7 +202,7 @@ variant.error # => error message string, or nil
|
|
|
229
202
|
|
|
230
203
|
## Placeholders
|
|
231
204
|
|
|
232
|
-
There is nothing to configure. A variant's `.url` serves the original while it is pending, processing, or failed, and the processed variant once ready.
|
|
205
|
+
There is nothing to configure. A variant's `.url` serves the original while it is pending, processing, or failed, and the processed variant once ready. (For a circular progress bar while processing and a red error glyph once failed, add the [`active_storage-async_variants-ui`](https://github.com/botandrose/active_storage-async_variants-ui) gem.)
|
|
233
206
|
|
|
234
207
|
## Failure Handling
|
|
235
208
|
|
|
@@ -248,7 +221,7 @@ variant.error # => "ffmpeg exited with status 1: ..."
|
|
|
248
221
|
### External transformer flow
|
|
249
222
|
|
|
250
223
|
1. User uploads a file to an attachment that has async variants defined
|
|
251
|
-
2. After attachment, a background job is enqueued for each async variant
|
|
224
|
+
2. After attachment, a background job is enqueued for each async variant whose `preprocessed:` holds for the record
|
|
252
225
|
3. `VariantRecord` is created with state `pending`
|
|
253
226
|
4. The job calls the transformer's `initiate` method with presigned source/destination URLs and a signed callback URL, then exits -- the worker is free
|
|
254
227
|
5. The external service processes the file, uploads the result to the destination URL
|
data/Rakefile
CHANGED
|
@@ -5,23 +5,4 @@ require "rspec/core/rake_task"
|
|
|
5
5
|
|
|
6
6
|
RSpec::Core::RakeTask.new(:spec)
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
task :cucumber do
|
|
10
|
-
sh "bundle exec cucumber"
|
|
11
|
-
end
|
|
12
|
-
|
|
13
|
-
desc "Run rspec and cucumber"
|
|
14
|
-
task all: [:spec, :cucumber]
|
|
15
|
-
|
|
16
|
-
namespace :vendor do
|
|
17
|
-
desc "Pull the latest @botandrose/progress-bar from unpkg.com into app/assets"
|
|
18
|
-
task :progress_bar do
|
|
19
|
-
require "open-uri"
|
|
20
|
-
url = "https://unpkg.com/@botandrose/progress-bar"
|
|
21
|
-
dest = File.expand_path("app/assets/javascripts/progress-bar.js", __dir__)
|
|
22
|
-
File.write(dest, URI.open(url).read)
|
|
23
|
-
puts "Vendored #{url} -> #{dest}"
|
|
24
|
-
end
|
|
25
|
-
end
|
|
26
|
-
|
|
27
|
-
task default: :all
|
|
8
|
+
task default: :spec
|
|
@@ -4,9 +4,6 @@ module ActiveStorage
|
|
|
4
4
|
module AsyncVariants
|
|
5
5
|
class CallbacksController < ActionController::API
|
|
6
6
|
def create
|
|
7
|
-
variant_record_id = ActiveStorage.verifier.verify(params[:token], purpose: :async_variant_callback)
|
|
8
|
-
variant_record = ActiveStorage::VariantRecord.find(variant_record_id)
|
|
9
|
-
|
|
10
7
|
case params[:status]
|
|
11
8
|
when "success"
|
|
12
9
|
variant_record.update!(state: "processed")
|
|
@@ -29,33 +26,45 @@ module ActiveStorage
|
|
|
29
26
|
|
|
30
27
|
private
|
|
31
28
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
29
|
+
def variant_record
|
|
30
|
+
@variant_record ||= begin
|
|
31
|
+
variant_record_id = ActiveStorage.verifier.verify(params[:token], purpose: :async_variant_callback)
|
|
32
|
+
ActiveStorage::VariantRecord.find(variant_record_id)
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
|
|
36
36
|
def apply_reported_metadata(variant_record, params)
|
|
37
37
|
reconcile(variant_record.image.blob, params[:byte_size], params[:checksum])
|
|
38
|
-
|
|
38
|
+
if frame?(variant_record.blob)
|
|
39
|
+
reconcile(variant_record.blob, params[:preview_image_byte_size], params[:preview_image_checksum])
|
|
40
|
+
end
|
|
39
41
|
end
|
|
40
42
|
|
|
41
|
-
#
|
|
42
|
-
# this is idempotent and never overwrites a real source blob's metadata.
|
|
43
|
+
# A re-render writes over a reused blob, so the reported values describe the object now stored.
|
|
43
44
|
def reconcile(blob, byte_size, checksum)
|
|
44
45
|
return unless blob
|
|
45
46
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
attrs[:byte_size] = bytes
|
|
47
|
+
if positive_int?(byte_size)
|
|
48
|
+
blob.byte_size = byte_size
|
|
49
49
|
end
|
|
50
|
-
if checksum.present?
|
|
51
|
-
|
|
50
|
+
if checksum.present?
|
|
51
|
+
blob.checksum = checksum
|
|
52
52
|
end
|
|
53
|
-
blob.
|
|
53
|
+
blob.save!
|
|
54
|
+
|
|
55
|
+
if blob.checksum != "0"
|
|
56
|
+
blob.mirror_later
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# preview_image_* must never land on a variant's source blob.
|
|
61
|
+
def frame?(blob)
|
|
62
|
+
blob.checksum == "0" ||
|
|
63
|
+
ActiveStorage::Attachment.exists?(record_type: "ActiveStorage::Blob", name: "preview_image", blob_id: blob.id)
|
|
54
64
|
end
|
|
55
65
|
|
|
56
|
-
def positive_int(value)
|
|
57
|
-
|
|
58
|
-
int.positive? ? int : nil
|
|
66
|
+
def positive_int?(value)
|
|
67
|
+
value.to_i.positive?
|
|
59
68
|
end
|
|
60
69
|
end
|
|
61
70
|
end
|
data/config/routes.rb
CHANGED
|
@@ -4,21 +4,4 @@ Rails.application.routes.draw do
|
|
|
4
4
|
post "/active_storage/async_variants/callbacks/:token",
|
|
5
5
|
to: "active_storage/async_variants/callbacks#create",
|
|
6
6
|
as: :active_storage_async_variant_callback
|
|
7
|
-
|
|
8
|
-
get "/active_storage/async_variants/states/:signed_blob_id/:variation_key",
|
|
9
|
-
to: "active_storage/async_variants/states#show",
|
|
10
|
-
as: :async_variant_state
|
|
11
|
-
post "/active_storage/async_variants/states/:signed_blob_id/:variation_key/retry",
|
|
12
|
-
to: "active_storage/async_variants/states#retry",
|
|
13
|
-
as: :async_variant_state_retry
|
|
14
|
-
|
|
15
|
-
# Tiny 1x1 GIF whose URL carries the media's filename, so the processing/failed
|
|
16
|
-
# box reserves layout without fetching the original through a representation.
|
|
17
|
-
get "/active_storage/async_variants/placeholder/:filename",
|
|
18
|
-
to: "active_storage/async_variants/states#placeholder",
|
|
19
|
-
as: :async_variant_placeholder,
|
|
20
|
-
constraints: { filename: %r{[^/]+} },
|
|
21
|
-
format: false
|
|
22
|
-
|
|
23
|
-
ActiveStorage::AsyncVariants::Assets.draw(self, "/active_storage/async_variants/assets")
|
|
24
7
|
end
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
class AddStateColumnsToVariantRecords < ActiveRecord::Migration[7.2]
|
|
4
|
+
def change
|
|
5
|
+
add_column :active_storage_variant_records, :state, :string, default: "pending", if_not_exists: true
|
|
6
|
+
add_column :active_storage_variant_records, :error, :text, if_not_exists: true
|
|
7
|
+
add_column :active_storage_variant_records, :attempts, :integer, default: 0, if_not_exists: true
|
|
8
|
+
end
|
|
9
|
+
end
|
data/gemfiles/rails_7.2.gemfile
CHANGED
data/gemfiles/rails_8.0.gemfile
CHANGED
data/gemfiles/rails_8.1.gemfile
CHANGED
|
@@ -5,22 +5,39 @@ module ActiveStorage
|
|
|
5
5
|
module AttachmentExtension
|
|
6
6
|
private
|
|
7
7
|
|
|
8
|
+
# Stock would also preprocess an async variant: a TransformJob, or a PreviewImageJob that runs the local previewer.
|
|
8
9
|
def transform_variants_later
|
|
9
|
-
super
|
|
10
|
+
without_async_variants { super }
|
|
10
11
|
enqueue_async_variant_jobs
|
|
11
12
|
end
|
|
12
13
|
|
|
14
|
+
def named_variants
|
|
15
|
+
return super unless @without_async_variants
|
|
16
|
+
super.reject { |_name, named_variant| async_variant?(named_variant) }
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def without_async_variants
|
|
20
|
+
@without_async_variants = true
|
|
21
|
+
yield
|
|
22
|
+
ensure
|
|
23
|
+
@without_async_variants = false
|
|
24
|
+
end
|
|
25
|
+
|
|
13
26
|
def enqueue_async_variant_jobs
|
|
14
27
|
return unless blob.bucket_backed?
|
|
15
28
|
|
|
16
29
|
named_variants.each do |name, named_variant|
|
|
17
|
-
next unless named_variant.
|
|
30
|
+
next unless async_variant?(named_variant) && named_variant.preprocessed?(record)
|
|
18
31
|
|
|
19
32
|
ActiveStorage::AsyncVariants::ProcessJob.perform_later(
|
|
20
33
|
record, self.name, name.to_s,
|
|
21
34
|
)
|
|
22
35
|
end
|
|
23
36
|
end
|
|
37
|
+
|
|
38
|
+
def async_variant?(named_variant)
|
|
39
|
+
named_variant.transformations.key?(:transformer)
|
|
40
|
+
end
|
|
24
41
|
end
|
|
25
42
|
end
|
|
26
43
|
end
|
|
@@ -7,8 +7,10 @@ module ActiveStorage
|
|
|
7
7
|
# processing workers (Crucible, etc.) can reach via presigned URLs. The
|
|
8
8
|
# Disk and Test services return false; the gem then defers to vanilla
|
|
9
9
|
# ActiveStorage rather than trying to enqueue jobs or serve fallbacks.
|
|
10
|
+
# MirrorService doesn't delegate :bucket, so resolve through its primary.
|
|
10
11
|
def bucket_backed?
|
|
11
|
-
service.respond_to?(:
|
|
12
|
+
resolved = service.respond_to?(:primary) ? service.primary : service
|
|
13
|
+
resolved.respond_to?(:bucket)
|
|
12
14
|
end
|
|
13
15
|
end
|
|
14
16
|
end
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActiveStorage
|
|
4
|
+
module AsyncVariants
|
|
5
|
+
# Finds the async named-variant declaration whose transformations match a
|
|
6
|
+
# variation, starting from a blob's attachments. Used to recover the
|
|
7
|
+
# :transformer option when the Registry is cold, and by
|
|
8
|
+
# Preview/VariantWithRecord enqueue paths to locate the attachment to
|
|
9
|
+
# dispatch ProcessJob against.
|
|
10
|
+
#
|
|
11
|
+
# Walks one level up through a preview_image attachment so a lookup from a
|
|
12
|
+
# video's extracted frame still finds the named variant declared on the
|
|
13
|
+
# parent record's source-video field.
|
|
14
|
+
#
|
|
15
|
+
# Candidates are compared structurally (order-insensitive, JSON-normalized)
|
|
16
|
+
# against the declared transformations, never by instantiating
|
|
17
|
+
# attachment.variant -- which raises InvariableError for previewable blobs.
|
|
18
|
+
module NamedVariantScan
|
|
19
|
+
module_function
|
|
20
|
+
|
|
21
|
+
def find(blob, variation, depth: 0)
|
|
22
|
+
blob.attachments.each do |attachment|
|
|
23
|
+
if attachment.name == "preview_image" && attachment.record_type == "ActiveStorage::Blob" && depth < 1
|
|
24
|
+
source = ActiveStorage::Blob.find_by(id: attachment.record_id)
|
|
25
|
+
result = source && find(source, variation, depth: depth + 1)
|
|
26
|
+
return result if result
|
|
27
|
+
next
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
attachment.send(:named_variants).each do |name, named_variant|
|
|
31
|
+
candidate = ActiveStorage::Variation.wrap(named_variant.transformations)
|
|
32
|
+
next unless candidate.async_options[:transformer]
|
|
33
|
+
return [attachment, name, candidate.async_options] if matches?(candidate, variation)
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
nil
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# A format-less declaration matches any target format, because runtime
|
|
40
|
+
# resolution (Blob#variant's default_to) adds a blob-dependent format:
|
|
41
|
+
# the declaration never carries.
|
|
42
|
+
def matches?(candidate, variation)
|
|
43
|
+
declared = normalize(candidate.transformations)
|
|
44
|
+
target = normalize(variation.transformations)
|
|
45
|
+
declared_format = declared.delete("format")
|
|
46
|
+
target_format = target.delete("format")
|
|
47
|
+
declared == target && (declared_format.nil? || declared_format == target_format)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def normalize(transformations)
|
|
51
|
+
JSON.parse(transformations.to_json)
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|