hls 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +145 -0
- data/CLAUDE.md +448 -0
- data/README.md +283 -4
- data/examples/directory.rb +16 -42
- data/lib/generators/hls/install/install_generator.rb +42 -0
- data/lib/generators/hls/install/templates/application_video.rb +28 -0
- data/lib/generators/hls/install/templates/initializer.rb +14 -0
- data/lib/generators/hls/video/templates/video.rb +44 -0
- data/lib/generators/hls/video/video_generator.rb +37 -0
- data/lib/hls/application_video.rb +636 -0
- data/lib/hls/cache.rb +31 -0
- data/lib/hls/codecs.rb +94 -0
- data/lib/hls/directory.rb +38 -0
- data/lib/hls/encode_job.rb +44 -0
- data/lib/hls/input.rb +93 -0
- data/lib/hls/instrumentation.rb +40 -0
- data/lib/hls/lock.rb +50 -0
- data/lib/hls/manifest.rb +193 -0
- data/lib/hls/railtie.rb +40 -0
- data/lib/hls/state.rb +106 -0
- data/lib/hls/storage.rb +132 -0
- data/lib/hls/testing.rb +263 -0
- data/lib/hls/uploader.rb +211 -0
- data/lib/hls/version.rb +1 -1
- data/lib/hls.rb +29 -201
- data/plans/00-goal.md +112 -0
- data/plans/01-profile-dsl.md +105 -0
- data/plans/02-upload-pipeline.md +140 -0
- data/plans/03-reader-and-manifest.md +128 -0
- data/plans/04-codec-portability.md +90 -0
- data/plans/05-server-migration.md +126 -0
- data/plans/06-activestorage-adapter.md +255 -0
- metadata +36 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e08ad4995abd5e93b9dd89b2971c3d6cb698c774cd495d683356c4ec8cc1f2c2
|
|
4
|
+
data.tar.gz: f1fa71e889f4002caa0b51c5fb11048f1b7fcf82fbe68935eef1c6bb9ac436cc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3ef35475f72800b5d9d01841a48b83e84c525d517ad94e9a3127dccf4dd05c82676854accc6a0cc4e19d7fc5bb1622daac1f93cf207126a735974cd674b07c37
|
|
7
|
+
data.tar.gz: 06c7ddb78a322e5728f5e6f03faa53e3e67833c1c5fe857f6ce3fa66d2900efcea91f578ca44aff7570e83b947f3262a0e8e9080f3c675fdb9085dd316d9fef7
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.2.0] - 2026-05-06
|
|
11
|
+
|
|
12
|
+
### Changed (breaking)
|
|
13
|
+
|
|
14
|
+
- **`config_digest` now sorts hash keys before SHA256ing the payload.**
|
|
15
|
+
The serialized form is stable across Ruby versions and hash insertion
|
|
16
|
+
order. *One-time effect on existing deployments:* the next `process`
|
|
17
|
+
run will re-encode every video once, since previously-recorded
|
|
18
|
+
digests no longer match the new representation. Subsequent runs are
|
|
19
|
+
no-ops as before.
|
|
20
|
+
|
|
21
|
+
- **`HLS::Cache` groups the playlist cache backend with its TTL.** The
|
|
22
|
+
separate `manifest_cache` + `manifest_cache_ttl` class settings (and
|
|
23
|
+
the `cache_ttl:` kwarg on `HLS::Manifest.new`) are gone. Configure
|
|
24
|
+
one object on the profile:
|
|
25
|
+
|
|
26
|
+
class ApplicationVideo < HLS::ApplicationVideo
|
|
27
|
+
def self.cache = HLS::Cache.new(backend: Rails.cache, ttl: 5.minutes)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
`HLS::Manifest` accepts an `HLS::Cache` or any object responding to
|
|
31
|
+
`fetch(key, &block)` (raw `Rails.cache` still works — it just uses
|
|
32
|
+
the cache's own default TTL).
|
|
33
|
+
|
|
34
|
+
- **`HLS::Storage::S3` is now the default storage adapter** and owns
|
|
35
|
+
`bucket_name` + `signing_ttl`. The polymorphic `bucket` setting on
|
|
36
|
+
`HLS::ApplicationVideo` (and the matching `signing_ttl` setting and
|
|
37
|
+
`resolve_bucket` method) are removed. Configure profiles like:
|
|
38
|
+
|
|
39
|
+
class ApplicationVideo < HLS::ApplicationVideo
|
|
40
|
+
def self.storage = HLS::Storage::S3.new(
|
|
41
|
+
bucket_name: ENV.fetch("VIDEO_S3_BUCKET_NAME"),
|
|
42
|
+
signing_ttl: 1.hour
|
|
43
|
+
)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
`HLS::Manifest` and `HLS::Uploader` now take `storage:` instead of
|
|
47
|
+
`bucket:` (and `Manifest` no longer takes `expires_in:` — it reads
|
|
48
|
+
it from `storage.signing_ttl`).
|
|
49
|
+
|
|
50
|
+
- **The Railtie no longer fires the `:hls_application_video` load
|
|
51
|
+
hook**, no longer registers a `Rails.application.config.hls` config
|
|
52
|
+
bag, and no longer copies values onto profile classes via a
|
|
53
|
+
separate initializer. Settings live on the profile classes directly
|
|
54
|
+
(Zeitwerk reloads handle dev-mode freshness). Initializers shrink to
|
|
55
|
+
one line: `HLS.s3_resource = Aws::S3::Resource.new(...)`.
|
|
56
|
+
|
|
57
|
+
### Added
|
|
58
|
+
|
|
59
|
+
- **`HLS::EncodeJob` retry policy.** `discard_on` for `HLS::Lock::Busy`
|
|
60
|
+
(another worker is doing it) and `HLS::State::CorruptError` (operator
|
|
61
|
+
intervention required) — both are poison messages that ActiveJob's
|
|
62
|
+
default retry-everything-five-times behavior wastes work on. Other
|
|
63
|
+
errors continue to follow the host app's default retry policy.
|
|
64
|
+
|
|
65
|
+
### Removed
|
|
66
|
+
|
|
67
|
+
- **Dropped `parallel` and `bigdecimal` gem dependencies.** Neither was
|
|
68
|
+
used in `lib/`. The Uploader does its own bounded threading via
|
|
69
|
+
`Queue` + `Thread.new`, and nothing in the gem touches BigDecimal.
|
|
70
|
+
|
|
71
|
+
### Added
|
|
72
|
+
|
|
73
|
+
- **Config-aware encode idempotency.** The state sidecar now records
|
|
74
|
+
a `config_digest` alongside `input_digest` — a SHA256 of the
|
|
75
|
+
encode-affecting profile config (renditions, posters, codecs,
|
|
76
|
+
bitrates, segment_duration, bits_per_pixel). `process` re-runs
|
|
77
|
+
ffmpeg when *either* the input bytes or the profile config has
|
|
78
|
+
changed since the last successful encode. Previously, bumping
|
|
79
|
+
`audio_bitrate` or adding a rendition was a silent no-op on
|
|
80
|
+
re-run. Settings that don't affect output bytes (`storage`,
|
|
81
|
+
`cache`, `ffmpeg_timeout`, `variant_uri`) are excluded.
|
|
82
|
+
- **Rails generators.** `bin/rails g hls:install` scaffolds
|
|
83
|
+
`config/initializers/hls.rb` and `app/videos/application_video.rb`.
|
|
84
|
+
`bin/rails g hls:video NAME` writes a per-content-type profile under
|
|
85
|
+
`app/videos/` with a sensible default ladder, an active hero poster,
|
|
86
|
+
and commented hints for the common per-profile overrides.
|
|
87
|
+
|
|
88
|
+
- **ffmpeg stderr capture.** When ffmpeg fails, the tail of its stderr
|
|
89
|
+
is included in `HLS::Error` so failures are diagnosable without
|
|
90
|
+
re-running with verbose logging.
|
|
91
|
+
- **Input validation.** `HLS::Input#validate!` raises early when the
|
|
92
|
+
input has no video stream (audio-only files, malformed media), and
|
|
93
|
+
`#video?` exposes the same predicate. The encode pipeline calls
|
|
94
|
+
`validate!` before invoking ffmpeg.
|
|
95
|
+
- **Lock file in output dir.** `HLS::Lock` provides advisory file
|
|
96
|
+
locking via `flock`. `process` acquires it before encoding/uploading;
|
|
97
|
+
a second concurrent worker for the same output dir gets
|
|
98
|
+
`HLS::Lock::Busy` instead of corrupting state.
|
|
99
|
+
- **Encode bundle verification.** `verify_encode!` walks the just-
|
|
100
|
+
encoded output, asserting the master + variants + segments + posters
|
|
101
|
+
all exist and are non-empty, before recording state and uploading.
|
|
102
|
+
- **ActiveSupport::Notifications hooks.** Events `encode.hls`,
|
|
103
|
+
`poster.hls`, `verify.hls`, `upload_object.hls`, `upload_retry.hls`,
|
|
104
|
+
and `process.hls` published when AS is loaded; pure-Ruby usage is a
|
|
105
|
+
no-op. See README for payload keys.
|
|
106
|
+
- **Configurable ffmpeg timeout.** `ffmpeg_timeout` class setting (in
|
|
107
|
+
seconds) terminates a stuck ffmpeg with SIGTERM then SIGKILL. `nil`
|
|
108
|
+
default preserves prior behavior.
|
|
109
|
+
- **S3 retry with backoff.** Uploader retries transient failures
|
|
110
|
+
(network errors, 503, RequestTimeout, SlowDown, InternalError) up to
|
|
111
|
+
`max_retries` times with exponential backoff. Permanent errors
|
|
112
|
+
(NoSuchBucket, 403) fail fast.
|
|
113
|
+
- **Threaded uploader.** Bounded-concurrency parallel uploads via a
|
|
114
|
+
worker pool. Default `concurrency: 4`. Set to `1` for serial.
|
|
115
|
+
- **GOP scales with segment_duration.** Keyframe interval is now
|
|
116
|
+
`framerate × segment_duration` instead of a hardcoded 180. Each HLS
|
|
117
|
+
segment starts on a keyframe regardless of the configured segment
|
|
118
|
+
length, fixing seek stalls on non-default `segment_duration`.
|
|
119
|
+
- **Storage adapter pattern.** `bucket` accepts any object responding
|
|
120
|
+
to `object(key)` that yields a duck-typed object with `get`, `put`,
|
|
121
|
+
and `presigned_url`. `HLS::Storage::Memory` ships as a no-network
|
|
122
|
+
adapter for tests. Documented MinIO setup in README.
|
|
123
|
+
- **Pluggable Manifest cache.** Read-side Manifest accepts a `cache:`
|
|
124
|
+
object (an `HLS::Cache` wrapping a `Rails.cache`-shaped backend, or
|
|
125
|
+
any object responding to `fetch(key, &block)`) to cut S3 GETs for
|
|
126
|
+
hot videos.
|
|
127
|
+
- **`resolve_variants_under` RSpec matcher.** Public test helper that
|
|
128
|
+
catches the variant-URI-doubling bug class by simulating RFC 3986
|
|
129
|
+
resolution against the master URL.
|
|
130
|
+
|
|
131
|
+
### Changed
|
|
132
|
+
|
|
133
|
+
- Variant URIs in the master playlist are now generated by an
|
|
134
|
+
overridable `variant_uri(path:, variant_index:)` class method.
|
|
135
|
+
Default returns `<basename(path)>/<variant_index>.m3u8`, which
|
|
136
|
+
resolves cleanly under a `/videos/*path/:id.m3u8` route shape.
|
|
137
|
+
- Cache-Control on uploaded `.m3u8` files relaxed from `no-cache` to
|
|
138
|
+
`public, max-age=300` so a CDN can edge-cache playlists between
|
|
139
|
+
re-encodes.
|
|
140
|
+
- ffprobe / ffmpeg subprocess calls switched from backticks / `system`
|
|
141
|
+
to `Open3.capture3` for safer argument handling and stderr capture.
|
|
142
|
+
|
|
143
|
+
## [0.1.0]
|
|
144
|
+
|
|
145
|
+
- Initial release.
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,448 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
Guide for AI assistants and human maintainers working on this gem.
|
|
4
|
+
The README is for *users* of the gem; this file is for people
|
|
5
|
+
*changing* it.
|
|
6
|
+
|
|
7
|
+
## What this gem actually is
|
|
8
|
+
|
|
9
|
+
A Rails-friendly Ruby library for taking a video file, encoding it as
|
|
10
|
+
an HLS bundle (multi-rendition multiplex + posters), and serving it
|
|
11
|
+
from a private S3-compatible bucket via pre-signed URLs.
|
|
12
|
+
|
|
13
|
+
The user-facing surface is small:
|
|
14
|
+
- `HLS::ApplicationVideo` — DSL base class for `app/videos/*.rb`
|
|
15
|
+
profile classes
|
|
16
|
+
- `HLS::Manifest` — read-side, returns signed playlists
|
|
17
|
+
- `HLS::Input` — ffprobe wrapper
|
|
18
|
+
- `HLS::Testing` — RSpec helpers for verifying user-defined profiles
|
|
19
|
+
|
|
20
|
+
Everything else (`Uploader`, `State`, `Codecs`, `Railtie`,
|
|
21
|
+
`EncodeJob`) is plumbing that profile classes orchestrate.
|
|
22
|
+
|
|
23
|
+
## Architecture & layering
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
L4 Framework integration railtie.rb, encode_job.rb
|
|
27
|
+
L3 Orchestration application_video.rb
|
|
28
|
+
L2 Storage manifest.rb (read), uploader.rb (write)
|
|
29
|
+
L1 Utilities codecs, input, directory, state, testing,
|
|
30
|
+
version
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Lower layers don't reach up. `Manifest` doesn't know `ApplicationVideo`
|
|
34
|
+
exists; `Uploader` doesn't know about `Manifest`. `ApplicationVideo`
|
|
35
|
+
is the only file that composes the layers, and it's the only file
|
|
36
|
+
users normally subclass.
|
|
37
|
+
|
|
38
|
+
The Railtie is *optional* — the gem works in plain Ruby. The Railtie
|
|
39
|
+
file is only required when `defined?(Rails::Railtie)` is true at
|
|
40
|
+
load time (see `lib/hls.rb` bottom).
|
|
41
|
+
|
|
42
|
+
## File map
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
lib/hls.rb Top-level module, error class, S3 resource accessor, requires
|
|
46
|
+
lib/hls/version.rb Version constant
|
|
47
|
+
lib/hls/application_video.rb DSL + encode/poster/upload orchestration
|
|
48
|
+
lib/hls/manifest.rb Reads bundle from S3, returns signed M3u8 playlists
|
|
49
|
+
lib/hls/uploader.rb Walks output dir, parallel idempotent S3 upload with retries
|
|
50
|
+
lib/hls/state.rb JSON sidecar: input digest, encoded_at, per-key etags
|
|
51
|
+
lib/hls/lock.rb Advisory flock around the output dir during process
|
|
52
|
+
lib/hls/instrumentation.rb ActiveSupport::Notifications wrapper (no-op without AS)
|
|
53
|
+
lib/hls/storage.rb Storage protocol doc + Memory adapter for tests
|
|
54
|
+
lib/hls/codecs.rb Logical → explicit codec resolution per host
|
|
55
|
+
lib/hls/input.rb ffprobe wrapper (Open3, raises on failure)
|
|
56
|
+
lib/hls/directory.rb Source-walking helper for batch encoding scripts
|
|
57
|
+
lib/hls/testing.rb Public RSpec helpers + matcher (only when RSpec defined)
|
|
58
|
+
lib/hls/railtie.rb Autoloads app/videos/, applies config.hls defaults
|
|
59
|
+
lib/hls/encode_job.rb ActiveJob wrapper around profile.process
|
|
60
|
+
|
|
61
|
+
lib/generators/hls/install/ `bin/rails g hls:install` — initializer + ApplicationVideo
|
|
62
|
+
lib/generators/hls/video/ `bin/rails g hls:video NAME` — per-content-type profile
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Each lib file has a matching spec under `spec/hls/`. Cross-cutting
|
|
66
|
+
specs (`process_spec.rb`, `poster_dsl_spec.rb`) live alongside.
|
|
67
|
+
End-to-end specs that shell out to ffmpeg live under `spec/integration/`.
|
|
68
|
+
|
|
69
|
+
## The pipeline
|
|
70
|
+
|
|
71
|
+
`profile.process` acquires a file lock on the output dir, then runs:
|
|
72
|
+
|
|
73
|
+
1. **Probe** — `HLS::Input` shells out to `ffprobe` for width / height
|
|
74
|
+
/ codec / duration / framerate. Lazy and memoized per Input
|
|
75
|
+
instance. `validate!` is called before encode to fail fast on
|
|
76
|
+
audio-only inputs.
|
|
77
|
+
2. **Encode** — `ApplicationVideo#encode!` builds an `ffmpeg` command
|
|
78
|
+
from the rendition declarations + codec resolution. One ffmpeg
|
|
79
|
+
invocation produces all renditions of the multiplex via
|
|
80
|
+
`-filter_complex split` + `-var_stream_map`. Subject to
|
|
81
|
+
`ffmpeg_timeout` and stderr is captured into `HLS::Error` on failure.
|
|
82
|
+
3. **Poster** — separate ffmpeg invocation if any `poster ...`
|
|
83
|
+
declarations exist. Multiple outputs from one decode pass.
|
|
84
|
+
4. **Verify** — `verify_encode!` walks the output and asserts all
|
|
85
|
+
expected files exist and are non-empty. Bails BEFORE recording state
|
|
86
|
+
so a failed verify doesn't claim the bundle is encoded.
|
|
87
|
+
5. **Upload** — `HLS::Uploader` walks the output dir, computes MD5 of
|
|
88
|
+
each file, skips files whose recorded digest in `state.json`
|
|
89
|
+
matches. PUTs each file with appropriate Content-Type and
|
|
90
|
+
Cache-Control. Default `concurrency: 4` parallel workers, with
|
|
91
|
+
bounded retries on transient errors. Records etag back into state.
|
|
92
|
+
|
|
93
|
+
Idempotency is enforced at two levels:
|
|
94
|
+
- **Encode level**: skipped when the input's SHA256 digest matches
|
|
95
|
+
what's recorded in state.json AND the master playlist file actually
|
|
96
|
+
exists. Both checks are necessary — see "Common gotchas" below.
|
|
97
|
+
- **Upload level**: per-file MD5 compared to recorded digest.
|
|
98
|
+
|
|
99
|
+
## Class-level DSL pattern
|
|
100
|
+
|
|
101
|
+
`HLS::ApplicationVideo` uses a small custom inheritable-attribute
|
|
102
|
+
helper (`class_setting`) instead of pulling in ActiveSupport's
|
|
103
|
+
`class_attribute`. This keeps the gem usable without Rails.
|
|
104
|
+
|
|
105
|
+
The pattern: each setting is a singleton method that reads with no
|
|
106
|
+
args, writes with one. Reads walk the class hierarchy via `superclass`
|
|
107
|
+
until they find a set value or hit the default.
|
|
108
|
+
|
|
109
|
+
`renditions` and `posters` accumulate into per-class arrays. The
|
|
110
|
+
`inherited` callback dups parent declarations into subclasses, so
|
|
111
|
+
subclass mutations don't leak back to the parent.
|
|
112
|
+
|
|
113
|
+
Avoid the temptation to swap this for `class_attribute` unless we
|
|
114
|
+
later decide to take a hard dep on activesupport. Right now the gem's
|
|
115
|
+
only Rails-flavored deps are dev-only (railties, activejob).
|
|
116
|
+
|
|
117
|
+
## Storage layout
|
|
118
|
+
|
|
119
|
+
ffmpeg writes the bundle into `output/`:
|
|
120
|
+
|
|
121
|
+
```
|
|
122
|
+
<output>/
|
|
123
|
+
├── index.m3u8 master playlist
|
|
124
|
+
├── 0/ first variant (highest rendition)
|
|
125
|
+
│ ├── index.m3u8
|
|
126
|
+
│ └── 0.ts, 1.ts, ...
|
|
127
|
+
├── 1/ second variant
|
|
128
|
+
│ ├── index.m3u8
|
|
129
|
+
│ └── 0.ts, 1.ts, ...
|
|
130
|
+
├── 2/...
|
|
131
|
+
├── hero.jpg if `poster :hero` declared
|
|
132
|
+
├── thumbnail.jpg if `poster :thumbnail` declared
|
|
133
|
+
└── .hls-state.json written by us, not uploaded
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Variant names default to integer indices (`0/`, `1/`, `2/`) because
|
|
137
|
+
that's ffmpeg's default `%v` template substitution. Named variants via
|
|
138
|
+
`-var_stream_map name:high,...` is a known follow-up not yet built.
|
|
139
|
+
|
|
140
|
+
Bucket layout mirrors local layout under a `key_prefix`:
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
<bucket>/<key_prefix>/index.m3u8
|
|
144
|
+
<bucket>/<key_prefix>/0/index.m3u8
|
|
145
|
+
<bucket>/<key_prefix>/0/0.ts
|
|
146
|
+
...
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
## Read side: how URI rewriting works
|
|
150
|
+
|
|
151
|
+
The encoded master playlist points at variants by their relative path:
|
|
152
|
+
|
|
153
|
+
```
|
|
154
|
+
0/index.m3u8
|
|
155
|
+
1/index.m3u8
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
But the host app's controller wants to serve variants by their
|
|
159
|
+
*number* under a stable URL like `/videos/<id>/<variant>.m3u8`. So
|
|
160
|
+
`Manifest#master_playlist` rewrites:
|
|
161
|
+
|
|
162
|
+
```
|
|
163
|
+
0/index.m3u8 → <path>/0.m3u8
|
|
164
|
+
1/index.m3u8 → <path>/1.m3u8
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
The variant playlist (returned by `Variant#playlist`) goes a step
|
|
168
|
+
further: each segment URI gets rewritten to a pre-signed S3 URL the
|
|
169
|
+
player can fetch directly without proxying through the app server.
|
|
170
|
+
|
|
171
|
+
**Critical**: `master_playlist` must NOT mutate the cached raw
|
|
172
|
+
playlist. We had a regression where it did, which broke `variants`
|
|
173
|
+
when both were called in the same request (`File.dirname` of an
|
|
174
|
+
already-rewritten URI returns nonsense). There's a test pinning this.
|
|
175
|
+
|
|
176
|
+
## Codec resolution
|
|
177
|
+
|
|
178
|
+
`video_codec :h264` is a *logical* codec. At command-build time, the
|
|
179
|
+
gem picks the best available encoder by:
|
|
180
|
+
|
|
181
|
+
1. Checking `RbConfig::CONFIG["host_os"]` for darwin / linux / etc.
|
|
182
|
+
2. Querying `ffmpeg -encoders` once per process and caching the result
|
|
183
|
+
3. Walking the per-platform priority list (videotoolbox first on
|
|
184
|
+
macOS; nvenc → qsv → libx264 on Linux)
|
|
185
|
+
|
|
186
|
+
To pin a specific encoder regardless of host, pass a string:
|
|
187
|
+
`video_codec "libx264"`. Tests stub `HLS::Codecs.platform` and
|
|
188
|
+
`available_encoders` to make platform-dependent paths deterministic.
|
|
189
|
+
|
|
190
|
+
## Common gotchas
|
|
191
|
+
|
|
192
|
+
### Variant URIs are RELATIVE to the master playlist URL
|
|
193
|
+
|
|
194
|
+
The master playlist served by the controller lives at e.g.
|
|
195
|
+
`/videos/<path>/<id>.m3u8`. Variant URIs inside it are RFC 3986
|
|
196
|
+
relative references — the player resolves each one against the
|
|
197
|
+
master's URL. A variant URI of `<id>/0.m3u8` resolves to
|
|
198
|
+
`/videos/<path>/<id>/0.m3u8` (good). A variant URI of
|
|
199
|
+
`<path>/<id>/0.m3u8` resolves to
|
|
200
|
+
`/videos/<path>/<path>/<id>/0.m3u8` — **the path doubles** —
|
|
201
|
+
because resolution drops the master URL's filename and joins
|
|
202
|
+
relative to the parent directory. We hit this once: it produced
|
|
203
|
+
`Aws::S3::Errors::NoSuchKey` when the player tried to fetch the
|
|
204
|
+
doubled URL.
|
|
205
|
+
|
|
206
|
+
`HLS::Manifest#master_playlist` produces the URI via the configured
|
|
207
|
+
`variant_uri` callable, defaulting to
|
|
208
|
+
`<basename(path)>/<index>.m3u8`. This default works for
|
|
209
|
+
`/videos/*path/:id/:variant.m3u8` Rails routes. For any other URL
|
|
210
|
+
shape, override `self.variant_uri(path:, variant_index:)` on the
|
|
211
|
+
profile class.
|
|
212
|
+
|
|
213
|
+
There's a regression test in `spec/hls/manifest_spec.rb` that
|
|
214
|
+
simulates the player's URL resolution with `URI#+`. Keep it. The
|
|
215
|
+
`HLS::Testing` matcher `resolve_variants_under(url)` is the
|
|
216
|
+
generalization users can run on their own profile specs.
|
|
217
|
+
|
|
218
|
+
### `path:` gem deps don't hot-reload
|
|
219
|
+
|
|
220
|
+
When the host app pins `gem "hls", path: "../hls"` for development,
|
|
221
|
+
gem source changes don't reload across the Rails process. After
|
|
222
|
+
changing gem code, fully restart the Rails server (Ctrl+C, `bin/dev`
|
|
223
|
+
again). Zeitwerk's reloader watches `app/`, not gems.
|
|
224
|
+
|
|
225
|
+
### `hls.js` caches master playlists in-memory
|
|
226
|
+
|
|
227
|
+
Even after a server-side fix lands, the player keeps the previously-
|
|
228
|
+
fetched master playlist in memory across plays in the same tab. If
|
|
229
|
+
debugging URL rewriting, hard-reload the browser tab (Cmd+Shift+R)
|
|
230
|
+
or open a private window to bypass it. The `Cache-Control: public,
|
|
231
|
+
max-age=300` we set on uploaded m3u8s is also strong enough to keep
|
|
232
|
+
old playlists around briefly.
|
|
233
|
+
|
|
234
|
+
### "Encoded but files missing"
|
|
235
|
+
|
|
236
|
+
If state.json exists but the output directory's been wiped (ephemeral
|
|
237
|
+
worker, manual cleanup, etc.), the naive `state.encoded?` check passes
|
|
238
|
+
but the upload step would try to walk an empty directory. The fix
|
|
239
|
+
(in `ApplicationVideo#encoded?`, private) checks BOTH state AND that
|
|
240
|
+
the master playlist is on disk. Keep this dual check — there's a test
|
|
241
|
+
for it.
|
|
242
|
+
|
|
243
|
+
### Empty-string buckets
|
|
244
|
+
|
|
245
|
+
`ENV.fetch("VIDEO_S3_BUCKET_NAME", "")` returns `""` when the env var
|
|
246
|
+
is missing. Empty string is truthy in Ruby. `resolve_bucket` treats
|
|
247
|
+
both `nil` and `""` as "no bucket configured" and raises. Don't
|
|
248
|
+
add another path that bypasses this check.
|
|
249
|
+
|
|
250
|
+
### No Rails config bag, no load hook
|
|
251
|
+
|
|
252
|
+
The Railtie does only two things: register `app/videos` as an
|
|
253
|
+
autoload path and lazy-require `HLS::EncodeJob` when ActiveJob
|
|
254
|
+
loads. There is intentionally no `Rails.application.config.hls.*`
|
|
255
|
+
bag and no `:hls_application_video` load hook to subscribe to.
|
|
256
|
+
|
|
257
|
+
Why? Because Zeitwerk already reloads `app/videos/*.rb` in dev, the
|
|
258
|
+
class body of `ApplicationVideo` re-runs on every reload — so any
|
|
259
|
+
configuration written there (env-var-derived or otherwise) refreshes
|
|
260
|
+
automatically. A load hook would be redundant.
|
|
261
|
+
|
|
262
|
+
Profile classes own their own configuration via the class-level DSL
|
|
263
|
+
plus `def self.storage = ...` overrides. The only module-level state
|
|
264
|
+
the gem keeps is `HLS.s3_resource` for the AWS SDK client.
|
|
265
|
+
|
|
266
|
+
### Storage is one object that owns bucket + signing TTL
|
|
267
|
+
|
|
268
|
+
Earlier versions had two separate class settings (`bucket` accepting
|
|
269
|
+
String/Bucket/duck-typed thing, plus `signing_ttl`) and a
|
|
270
|
+
`resolve_bucket` switch that picked an adapter based on the value
|
|
271
|
+
type. That all collapsed into one `storage` setting that takes any
|
|
272
|
+
object satisfying the protocol (`signing_ttl` + `object(key)`):
|
|
273
|
+
|
|
274
|
+
- `HLS::Storage::S3` is the default. It owns `bucket_name`,
|
|
275
|
+
`signing_ttl`, and an optional `s3_resource` override.
|
|
276
|
+
- `HLS::Storage::Memory` is the test/dev adapter.
|
|
277
|
+
- Any duck-typed object (future ActiveStorage adapter, custom
|
|
278
|
+
backend) plugs in the same way.
|
|
279
|
+
|
|
280
|
+
`HLS::Manifest` and `HLS::Uploader` only ever talk through `storage`
|
|
281
|
+
— they don't know what kind of bucket or backend is behind it.
|
|
282
|
+
|
|
283
|
+
### State sidecar corruption
|
|
284
|
+
|
|
285
|
+
A malformed state.json would silently re-encode + re-upload everything
|
|
286
|
+
without telling the operator. We deliberately raise `HLS::State::CorruptError`
|
|
287
|
+
instead of recovering. If you hit this, delete the state file
|
|
288
|
+
intentionally rather than working around it in the gem.
|
|
289
|
+
|
|
290
|
+
### ffmpeg multithreading on macOS
|
|
291
|
+
|
|
292
|
+
On macOS with `h264_videotoolbox`, multiple concurrent ffmpeg
|
|
293
|
+
processes contend on the Media Engine and macOS's videotoolbox
|
|
294
|
+
session limits. `HLS_PARALLEL=1` is the safe default. Bumping it
|
|
295
|
+
helps on Linux with libx264 only if you have many small videos to
|
|
296
|
+
batch.
|
|
297
|
+
|
|
298
|
+
### Cache-control on m3u8
|
|
299
|
+
|
|
300
|
+
VOD playlists are immutable once written. We use `public, max-age=300`
|
|
301
|
+
(not `no-cache`) so CDNs can edge-cache them. A redeploy of a bundle
|
|
302
|
+
takes effect within 5 min. Don't tighten this without thinking about
|
|
303
|
+
CDN cost.
|
|
304
|
+
|
|
305
|
+
### GOP must equal framerate × segment_duration
|
|
306
|
+
|
|
307
|
+
ffmpeg's `-g` (GOP size) sets how many frames between keyframes. HLS
|
|
308
|
+
players seek to segment boundaries and need each segment to start with
|
|
309
|
+
a keyframe. Computed in `ApplicationVideo#gop_size` from
|
|
310
|
+
`input.framerate * segment_duration`. Hardcoding it (the previous bug)
|
|
311
|
+
caused stalls when segment_duration was set to anything other than the
|
|
312
|
+
"normal" value. Don't reintroduce a constant here.
|
|
313
|
+
|
|
314
|
+
### Lock file collides with stale interrupted runs only via Busy
|
|
315
|
+
|
|
316
|
+
`process` writes `.hls-lock` and `flock(LOCK_EX | LOCK_NB)`s it. A
|
|
317
|
+
second process gets `HLS::Lock::Busy` *immediately* — we don't wait.
|
|
318
|
+
The lock file itself stays on disk after release; only the kernel-level
|
|
319
|
+
advisory lock is dropped. The state.json + .hls-lock + .DS_Store + ._*
|
|
320
|
+
patterns are all skipped by the uploader.
|
|
321
|
+
|
|
322
|
+
### Storage protocol is duck-typed
|
|
323
|
+
|
|
324
|
+
`bucket` accepts anything responding to `object(key)` whose return
|
|
325
|
+
value implements `get` / `put(body:, content_type:, cache_control:)` /
|
|
326
|
+
`presigned_url(:get, expires_in:)`. The `Aws::S3::Bucket` already
|
|
327
|
+
matches; `HLS::Storage::Memory` is a test double; anything else is the
|
|
328
|
+
host app's responsibility. `resolve_bucket` returns duck-typed buckets
|
|
329
|
+
unchanged — it only special-cases String (via `HLS.s3_resource`) and
|
|
330
|
+
Aws::S3::Bucket (passthrough).
|
|
331
|
+
|
|
332
|
+
## Running tests
|
|
333
|
+
|
|
334
|
+
```sh
|
|
335
|
+
bundle exec rspec # everything (~35s, ffmpeg integration)
|
|
336
|
+
bundle exec rspec --exclude-pattern "spec/integration/**" # unit only (~1s)
|
|
337
|
+
bundle exec rspec spec/hls/manifest_spec.rb # one file
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
The integration specs (`spec/integration/`) actually shell out to
|
|
341
|
+
ffmpeg. They generate a 12-second test source via
|
|
342
|
+
`HLS::Testing.generate_test_video` and verify output dimensions,
|
|
343
|
+
playlist structure, and segment counts. Slow but they're the test
|
|
344
|
+
that catches command-building bugs the stubbed unit tests miss.
|
|
345
|
+
|
|
346
|
+
The Rails-flavored specs (`spec/hls/railtie_spec.rb` and
|
|
347
|
+
`spec/integration/rails_pipeline_spec.rb`) share a single Rails app
|
|
348
|
+
boot via `spec/support/dummy_rails_app.rb`. Rails freezes
|
|
349
|
+
`autoload_paths` after `initialize!`, so booting twice in one process
|
|
350
|
+
crashes — always go through `DummyRailsApp.boot!`.
|
|
351
|
+
|
|
352
|
+
## Adding things
|
|
353
|
+
|
|
354
|
+
### A new codec
|
|
355
|
+
|
|
356
|
+
Edit `lib/hls/codecs.rb`:
|
|
357
|
+
- Add to `H264` if it's an h264 variant
|
|
358
|
+
- Add to `H264_PRIORITY[platform]` for auto-resolution
|
|
359
|
+
- Add a `case` arm in `ApplicationVideo#video_codec_options` for
|
|
360
|
+
encoder-specific ffmpeg flags (preset, profile, tune, etc.)
|
|
361
|
+
- Add a spec covering both the resolution and the command-shape
|
|
362
|
+
|
|
363
|
+
### A new class-level setting
|
|
364
|
+
|
|
365
|
+
Add `class_setting :name, default: ...` near the bottom of the class
|
|
366
|
+
body in `application_video.rb`. If host apps should be able to
|
|
367
|
+
override it via `config.hls.name = ...`, add a corresponding line in
|
|
368
|
+
the Railtie's `apply_config` initializer.
|
|
369
|
+
|
|
370
|
+
### A new ffmpeg arg in the encode command
|
|
371
|
+
|
|
372
|
+
Edit `ApplicationVideo#command` (or one of `video_maps` / `audio_maps`).
|
|
373
|
+
**Add a unit test** asserting the new arg appears in the right place,
|
|
374
|
+
and check whether the integration spec's golden assertions need
|
|
375
|
+
updating.
|
|
376
|
+
|
|
377
|
+
### A new test helper
|
|
378
|
+
|
|
379
|
+
Add to `lib/hls/testing.rb`. Helpers can shell out (ffmpeg/ffprobe are
|
|
380
|
+
hard deps anyway) but should not require Rails. Custom matchers go
|
|
381
|
+
inside the `if defined?(RSpec::Matchers)` block at the bottom of that
|
|
382
|
+
file.
|
|
383
|
+
|
|
384
|
+
### A new Rails generator
|
|
385
|
+
|
|
386
|
+
Live under `lib/generators/hls/<name>/`:
|
|
387
|
+
- `<name>_generator.rb` — subclass `Rails::Generators::{Base,NamedBase}`
|
|
388
|
+
- `templates/*.rb` (or `.tt`) — Thor templates; `<%= %>` interpolates
|
|
389
|
+
generator method results
|
|
390
|
+
|
|
391
|
+
Spec it under `spec/generators/`. Use the Rails helpers
|
|
392
|
+
(`Rails::Generators::Testing::{Behavior,Assertions}`) plus
|
|
393
|
+
`spec/support/minitest_shims.rb` so `assert_file` and friends work
|
|
394
|
+
inside RSpec. Always include a behavior test that `load`s the
|
|
395
|
+
generated file and asserts the resulting class actually works — text
|
|
396
|
+
matches don't catch wrong inheritance, missing renditions, etc.
|
|
397
|
+
|
|
398
|
+
## How this gem is used in `../server`
|
|
399
|
+
|
|
400
|
+
The sister project `../server` (beautifulruby.com) is the canonical
|
|
401
|
+
production consumer. Its `app/controllers/videos_controller.rb` is
|
|
402
|
+
~50 lines that delegates to `CourseVideo.manifest(path)`. Its
|
|
403
|
+
`config/initializers/hls.rb` configures the Tigris bucket. The legacy
|
|
404
|
+
`app/models/video.rb` was deleted during the integration in favor of
|
|
405
|
+
`HLS::Manifest`.
|
|
406
|
+
|
|
407
|
+
When making changes here that could affect the read side (Manifest,
|
|
408
|
+
URI rewriting, signed URLs), run the server suite too:
|
|
409
|
+
|
|
410
|
+
```sh
|
|
411
|
+
cd ../server && bundle exec rspec spec/requests/videos_spec.rb
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
## Plans directory
|
|
415
|
+
|
|
416
|
+
Long-form design docs live in `plans/`. The flow:
|
|
417
|
+
|
|
418
|
+
- `plans/00-goal.md` — the north star and master checklist
|
|
419
|
+
- `plans/0[1-5]-*.md` — implemented steps (all checked off)
|
|
420
|
+
- `plans/06-activestorage-adapter.md` — the next major chunk; not
|
|
421
|
+
built yet
|
|
422
|
+
|
|
423
|
+
The format of each plan is consistent: Goal / Preconditions / Work /
|
|
424
|
+
Acceptance. The acceptance checklists are runnable — they're meant
|
|
425
|
+
to be the gate for marking the step complete.
|
|
426
|
+
|
|
427
|
+
## Things deliberately NOT in the gem
|
|
428
|
+
|
|
429
|
+
Rejected to keep scope tight:
|
|
430
|
+
|
|
431
|
+
- Live HLS / DASH support — VOD only
|
|
432
|
+
- DRM beyond pre-signed URLs
|
|
433
|
+
- Custom segment naming via post-encode rewriting (ffmpeg's templating
|
|
434
|
+
covers what we need)
|
|
435
|
+
- Multi-profile fan-out from one source within a single `process` —
|
|
436
|
+
use multiple profile instances if you need both `WebVideo` and
|
|
437
|
+
`MobileVideo` outputs
|
|
438
|
+
|
|
439
|
+
## Release workflow
|
|
440
|
+
|
|
441
|
+
```sh
|
|
442
|
+
# 1. Bump lib/hls/version.rb
|
|
443
|
+
# 2. Move the [Unreleased] section in CHANGELOG.md under the new version
|
|
444
|
+
bundle exec rake release
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
This tags, pushes, and publishes to rubygems. There's no CI release
|
|
448
|
+
pipeline yet — releases are manual.
|