expo-video-subtitle 0.5.0 → 0.7.0

package/CHANGELOG.md

@@ -7,6 +7,93 @@ Changes inherited from the upstream base version are not repeated here — see t
 [upstream changelog](https://github.com/expo/expo/blob/main/packages/expo-video/CHANGELOG.md)
 for the history of everything that isn't subtitle related.
 
+## 0.7.0
+
+### Added
+
+- **ASS signs are drawn with the transforms the script asks for (Android).** A typeset sign is
+  placed on the artwork, and the artwork is usually not square to the screen: in one episode of
+  Jujutsu Kaisen S3, 3,723 events carry `\frz`, 1,304 of them past 5° and 104 past 45°, because the
+  placards they sit on are drawn at an angle. Rendered flat, the translation reads as pasted over
+  the picture instead of lying on it.
+
+  Media3 cannot do this and has no room to. `SubtitlePainter` contains no call to `rotate`, `skew`,
+  `Matrix` or `Camera`; `Cue` carries no rotation field, only `shearDegrees`, which the painter
+  ignores; and `SsaParser` discards every override block except `\pos`, `\move` and `\an`.
+
+  So signs leave Media3's hands entirely. `AssSignParser` splits a script in two — typeset events
+  to us, dialogue to the delegate, with the sign lines cut out of the copy the delegate parses, so
+  double-drawing is structurally impossible rather than merely avoided. Each sign rides the cue
+  stream as a zero-width placeholder carrying an `AssSignSpan`, which buys Media3's timing for free,
+  and `AssSignOverlay` draws it on a canvas in front of the captions.
+
+  Read and honoured: `\pos`, `\move` (at its destination), `\org`, `\an` and the legacy `\a`,
+  `\frz`, `\frx`, `\fry`, `\fscx`, `\fscy`, `\fs`, `\c` / `\1c`, `\alpha` / `\1a`, `\b`,
+  `\i`, `\u`, style `Alignment`, `Fontsize`, `PrimaryColour`, `OutlineColour`, `Outline`,
+  `ScaleX/Y`, `Angle` and the margins, plus `PlayResX` / `PlayResY` and VSFilter's 4:3 rule when one
+  is missing.
+
+  The overlay sits in the player's content frame, which is sized to the video rather than the view,
+  so positions survive letterboxing.
+
+### Known limitations
+
+- `\t` animation, `\fad`, `\clip`, `\blur`, `\be`, `\fax` shear, vector drawings and karaoke are
+  still dropped. Reproducing those means libass, not a larger subset of it.
+- Fonts fall back to the system face: a script naming `Bogart Semibold` gets the default at the
+  right size and weight, not that typeface.
+- `\frx` and `\fry` are applied with an unverified sign convention. They are almost always tiny —
+  3,363 of 5,126 `\frx` values and 2,715 of 3,579 `\fry` values in that episode are under 3° — so a
+  wrong sign is subtle rather than visible. `\frz` is what carries the effect and is verified
+  against the source line it came from.
+- iOS still draws signs flat. AVFoundation renders WebVTT itself and has no rotation to give.
+
+### Changed
+
+- The module now has JVM unit tests (`./gradlew :expo-video-subtitle:testDebugUnitTest`). The sign
+  parser is pure Kotlin with no Android imports precisely so it can be tested this way: its tag
+  scanner and colour handling fail by drawing a sign in the wrong place, which nothing else catches.
+
+## 0.6.0
+
+### Fixed
+
+- **ASS dialogue keeps working with `bottomOffset` and with cue stacking again (Android).** Media3's
+  `SsaParser` calls `setLine` and `setPosition` on *every* cue, including the bottom-centre dialogue
+  the author placed nothing on. `SubtitlePainter` applies `bottomPaddingFraction` only to cues whose
+  `line` is unset, so `SubtitleSource.bottomOffset` silently stopped moving anything on an `.ass`
+  track, and `StackingSubtitleParser` treated every cue as deliberately placed and left simultaneous
+  speakers drawing on top of each other.
+
+  A new `SsaCueNormalizer` clears the placement Media3 filled in. Recognising it is exact rather
+  than a guess: the values are a pure function of the cue's anchors, so a cue matching both defaults
+  for the bottom-centre anchors is precisely a cue with no positioning tag. Anything the author did
+  place — `\an8`, `\pos`, a style aligned elsewhere — keeps everything and still bypasses stacking.
+
+- **Signs keep the colours they were typeset in (Android).** The span filter dropped colour from
+  every cue so the app's own subtitle style would win, which is right for dialogue and wrong for a
+  sign: typesetting is drawn onto the artwork in colours chosen to sit on it, and the dark lettering
+  on the white placards in Jujutsu Kaisen S3E3 became white-on-white. Cues an ASS track positions
+  itself now keep their appearance; unpositioned dialogue still loses its own and follows the
+  viewer's settings. Only ASS opts in — it is the one format where positioning reliably means
+  "sign" rather than a cue setting on ordinary dialogue.
+
+- **Typesetters' working notes stay hidden.** A script ships its production notes on a style that
+  renders nothing: JJK S3E3 carries `Style: f0ld3r-invisible` with `Fontsize 0` and every colour
+  `&HFF……`, holding 158 events of "--mahito", "#1", "check in frame 6212". libass honours both
+  signals. Media3 turns a transparent primary colour into a `ForegroundColorSpan` with alpha 0,
+  which the span filter then strips, so the note came back in the viewer's colour at the viewer's
+  size; the iOS converter never read either field, and WebVTT cannot express opacity anyway. Events
+  on such a style are now dropped on both platforms, before anything can make them visible.
+
+### Known limitations
+
+- Sign **font size** is still the app's, not the script's. Media3 does read `Fontsize` into
+  `Cue.textSize`, but the filter clears it so the two subtitle views measure the same glyphs; giving
+  it back to signs needs the rounded-background view checked on a device first.
+- Blur, borders, shadows, rotation and `\t` animation are still dropped. Matching libass output
+  exactly needs libass, not Media3.
+
 ## 0.5.0
 
 ### Added

package/FORK.md

@@ -27,7 +27,11 @@ Measured against the published `expo-video@56.1.4` tarball. `+`/`-` are added/re
 | `android/…/utils/SubtitleSpanFilter.kt` | 135 | Strips colour, size, family and weight from every cue of every format, keeping italic/underline/strikethrough and all positioning, plus `FilteringSubtitleParser` which applies it to whatever a delegate emits. **Load-bearing fact, verified against Media3 1.9.0 bytecode:** `SubtitleViewUtils.removeAllEmbeddedStyling` touches only `clearWindowColor`, `getText`/`setText` and `setTextSize` — never `line`, `position` or `textAlignment`. That is why the filter can live in the parser and the view flag can stay on; if an upstream sync changes it, this whole design has to be revisited. Uses an **allowlist**, because a denylist misses `TextAppearanceSpan` (colour + size + family in one span). `StyleSpan` needs a rewrite rather than a keep/drop: `BOLD_ITALIC` has to come back as plain `ITALIC` over the same range. |
 | `android/…/records/SubtitleSource.kt` | 63 | Record → `MediaItem.SubtitleConfiguration` + MIME resolution. |
 | `android/…/records/SubtitleStyle.kt` | 152 | Record → `CaptionStyleCompat`; hex colour parsing; font-family resolution (`res/font` → `ReactFontManager`); the `applyEmbeddedStyles` flag, which `SubtitleUtils` reads directly rather than `toCaptionStyle` and which defaults to `true` since 0.5.0. |
-| `android/…/utils/StackingSubtitleParser.kt` | 308 | `SubtitleParser` for SubRip that segments the cue timeline and merges simultaneous cues into one stacked block, plus the `SubtitleParser.Factory` that routes SubRip to it and wraps every parser in `FilteringSubtitleParser`. SSA/ASS is deliberately not stacked: `SsaParser` sets both `line` and `position` on every cue, so they would all take the passthrough branch anyway. Assigns each cue a vertical slot it keeps for its whole time on screen — a finished cue's line is held blank (`\u200B`) so the cues above it do not drop. Merges through a `SpannableStringBuilder` so cue spans survive. |
+| `android/…/utils/StackingSubtitleParser.kt` | 342 | `SubtitleParser` that segments the cue timeline and merges simultaneous **unpositioned** cues into one stacked block, plus the `SubtitleParser.Factory` that routes SubRip and ASS/SSA to it and wraps every parser in `FilteringSubtitleParser`. ASS gets there through `SsaCueNormalizer` as the `prepare` step — without it `SsaParser` sets both `line` and `position` on every cue and they would all take the passthrough branch. Assigns each cue a vertical slot it keeps for its whole time on screen — a finished cue's line is held blank (`\u200B`) so the cues above it do not drop. Merges through a `SpannableStringBuilder` so cue spans survive. |
+| `android/…/utils/SsaCueNormalizer.kt` | 93 | Clears the `line`/`position` Media3's `SsaParser` fills in on cues the ASS author never placed, so `bottomOffset` and cue stacking work on `.ass` the way they already do on `.srt`. The test is exact rather than heuristic: those values are a pure function of the cue's anchors, so matching both defaults for the bottom-centre anchors *is* "no positioning tag". Also drops events on a style the author made invisible (`Fontsize 0`, or a primary colour whose inverted alpha byte is `FF`) — that is where typesetters park working notes, and nothing downstream can carry "invisible". |
+| `android/…/utils/AssSignParser.kt` | 505 | Splits an ASS script into the typeset events this package draws and a copy of the script with those lines removed for Media3's parser. One pass, no matching step, so an event is in exactly one of the two outputs and cannot be drawn twice. Pure Kotlin with no Android imports, which is what makes it unit-testable on the JVM. Reads `\pos`, `\move`, `\org`, `\an`, `\frz`, `\frx`, `\fry`, `\fscx/y`, `\fs`, `\c`, `\alpha`, `\b`, `\i`, `\u` and the style columns behind them. The tag scanner takes the longest name first — `\c` must not swallow `\clip`, `\b` must not swallow `\bord`, `\blur` or `\be` — and reads a leading digit as part of the name for `\1c` and `\4a`. |
+| `android/…/utils/AssSignOverlay.kt` | 235 | Draws those signs on a canvas in front of the captions, with `Matrix` for rotation and scale and `Camera` for the two 3-D axes. **Media3 cannot do this**: `SubtitlePainter` has no rotate, skew, Matrix or Camera call, and `Cue` has no rotation field. Sits in the same parent as `SubtitleView` — the content frame, sized to the video, which is why `\pos` survives letterboxing. Lays text out on change rather than per frame, because frame-by-frame typesetting changes the on-screen set every 40 ms. |
+| `android/…/utils/AssSignSpan.kt` | 24 | Carries one parsed sign through Media3's cue stream to the overlay, on a zero-width placeholder so `SubtitleView` draws nothing. Riding the cue stream is what buys the timing for free. |
 | `android/…/utils/RoundedSubtitleBackground.kt` | 352 | Draws `subtitleStyle.backgroundRadius` as rounded boxes, which Media3 cannot do at all: it paints `backgroundColor` as a framework `BackgroundColorSpan` and `windowColor` with `canvas.drawRect`, neither takes a radius, and `SubtitlePainter`/`CanvasSubtitleOutput` are package-private final while `SubtitleView` is final. A second `SubtitleView` is inserted behind `PlayerView`'s own, fed from `Player.Listener.onCues`, drawing a `LineBackgroundSpan` per line. **Its `EDGE_TYPE_NONE` is load-bearing** — `SubtitlePainter` copies the cue text for an edge pass and draws both layouts when the edge type is outline/raised/depressed, which would paint the box twice and darken a semi-transparent colour. State hangs off the view's `tag`, never a map in the companion object; see the `find` KDoc for why even a `WeakHashMap` leaks here. |
 | `FORK.md`, `LICENSE` | — | This guide; MIT licence (upstream copyright preserved). |
 | `eslint.config.js` | 4 | Verbatim copy of `expo-module-scripts/templates/eslint.config.js`. **Recreate it if a sync drops it** — `expo-module configure` treats it as an *optional* template and only syncs it when it already exists, so without this file `npm run lint` dies with "ESLint couldn't find an eslint.config.js". |

package/android/build.gradle

@@ -26,6 +26,11 @@ dependencies {
   implementation "androidx.media3:media3-ui:${androidxMedia3Version}"
   implementation "androidx.media3:media3-datasource-okhttp:${androidxMedia3Version}"
 
+  // The ASS sign parser is pure Kotlin with no Android imports, which is what makes it worth
+  // testing on the JVM: its tag scanner and colour handling are the parts that fail silently on a
+  // device, by drawing a sign in the wrong place rather than by crashing.
+  testImplementation 'junit:junit:4.13.2'
+
   def fragment_version = "1.8.9"
   implementation "androidx.fragment:fragment:$fragment_version"
   implementation "androidx.fragment:fragment-ktx:$fragment_version"

package/android/src/main/java/expo/modules/video/utils/AssSignOverlay.kt

@@ -0,0 +1,269 @@
+package expo.modules.video.utils
+
+import android.annotation.SuppressLint
+import android.content.Context
+import android.graphics.Camera
+import android.graphics.Canvas
+import android.graphics.Color
+import android.graphics.Matrix
+import android.graphics.Paint
+import android.graphics.Typeface
+import android.text.Layout
+import android.text.StaticLayout
+import android.text.TextPaint
+import android.view.View
+import android.view.ViewGroup
+import androidx.annotation.OptIn
+import androidx.media3.common.Player
+import androidx.media3.common.text.Cue
+import androidx.media3.common.text.CueGroup
+import androidx.media3.common.util.UnstableApi
+import androidx.media3.ui.PlayerView
+import androidx.media3.ui.SubtitleView
+
+/**
+ * Draws ASS typesetting — signs, placards, on-screen labels — with the transforms the script asks
+ * for.
+ *
+ * `SubtitleView` cannot: `SubtitlePainter` has no rotation, and `Cue` has no field to carry one.
+ * So [AssSignParser] takes those events out of Media3's cue stream, wraps each in an [AssSignSpan],
+ * and this view draws them itself.
+ *
+ * ### Where it sits
+ *
+ * As a sibling of `PlayerView`'s own `SubtitleView`, in front of it. That parent is the content
+ * frame, which is sized to the *video*, not to the player — which is the whole reason positions
+ * survive letterboxing. `\pos` is in `PlayResX`/`PlayResY` space and maps onto this view's bounds
+ * directly. [RoundedSubtitleBackground] takes the same route for the same reason.
+ *
+ * ### Why a canvas and not child views
+ *
+ * A `TextView` per sign would work — Android view transforms map onto ASS almost one for one — but
+ * a heavily typeset episode puts dozens of signs on screen across a scene and each would cost a
+ * measure and layout pass. Drawing straight to the canvas costs one `StaticLayout` per sign per
+ * frame it changes, and it is the only way to stroke the outline the way ASS means it: the border
+ * drawn *behind* the fill, not as a shadow.
+ */
+@SuppressLint("ViewConstructor")
+@OptIn(UnstableApi::class)
+internal class AssSignOverlay(context: Context) : View(context), Player.Listener {
+
+  private val camera = Camera()
+  private val matrix = Matrix()
+
+  private var signs: List<AssSign> = emptyList()
+  private var prepared: List<PreparedSign> = emptyList()
+  private var player: Player? = null
+
+  init {
+    setWillNotDraw(false)
+  }
+
+  override fun onCues(cueGroup: CueGroup) {
+    setSigns(cueGroup.cues)
+  }
+
+  private fun setSigns(cues: List<Cue>) {
+    val next = cues.mapNotNull(::signOf)
+    // Reference equality on the list contents is enough: the parser produces one AssSign per event
+    // and hands out the same instance for every segment the event spans.
+    if (next.size == signs.size && next.indices.all { next[it] === signs[it] }) return
+    signs = next
+    rebuild()
+  }
+
+  override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
+    super.onSizeChanged(w, h, oldw, oldh)
+    // Every measurement below is in view pixels, so all of it is stale after a resize — which
+    // happens on every rotation and on entering full screen.
+    rebuild()
+  }
+
+  /**
+   * Lays the signs out once per change rather than once per frame.
+   *
+   * Frame-by-frame typesetting changes the on-screen set every 40 ms, and `onDraw` can run more
+   * often than that for reasons of the view system's own. Text layout is the expensive half of
+   * drawing, and none of it depends on anything that varies between frames.
+   */
+  private fun rebuild() {
+    prepared = if (width == 0 || height == 0) emptyList() else signs.mapNotNull(::prepare)
+    invalidate()
+  }
+
+  fun attachTo(player: Player?) {
+    if (this.player === player) return
+    this.player?.removeListener(this)
+    this.player = player
+    player?.addListener(this)
+    // A listener only hears about the *next* cue change, so seed from what is on screen already.
+    setSigns(player?.currentCues?.cues ?: emptyList())
+  }
+
+  fun detach() {
+    player?.removeListener(this)
+    player = null
+    signs = emptyList()
+    rebuild()
+  }
+
+  override fun onDraw(canvas: Canvas) {
+    for (sign in prepared) {
+      draw(canvas, sign)
+    }
+  }
+
+  private fun draw(canvas: Canvas, sign: PreparedSign) {
+    canvas.save()
+    canvas.concat(sign.matrix)
+    canvas.translate(sign.left, sign.top)
+    sign.stroke?.draw(canvas)
+    sign.fill.draw(canvas)
+    canvas.restore()
+  }
+
+  /** Everything about one sign that does not change between frames. */
+  private class PreparedSign(
+    val matrix: Matrix,
+    val left: Float,
+    val top: Float,
+    val fill: StaticLayout,
+    val stroke: StaticLayout?,
+  )
+
+  private fun prepare(sign: AssSign): PreparedSign? {
+    // Script space to view space. Both axes are scaled independently so a script authored at a
+    // different aspect ratio than the video still lands where it was drawn.
+    val scaleToViewX = width / sign.playResX
+    val scaleToViewY = height / sign.playResY
+
+    val alpha = (sign.alpha * 255f).toInt().coerceIn(0, 255)
+    if (alpha == 0) return null
+
+    // A fresh paint per sign: StaticLayout keeps the reference it was built with, so a shared
+    // mutable one would repaint every sign already laid out in this pass.
+    val fillPaint = TextPaint(Paint.ANTI_ALIAS_FLAG)
+    fillPaint.applySign(sign, scaleToViewY)
+    fillPaint.color = Color.argb(alpha, Color.red(sign.colour or ALPHA_MASK), Color.green(sign.colour or ALPHA_MASK), Color.blue(sign.colour or ALPHA_MASK))
+
+    // Wrapping is the script's job, not ours: a sign is placed by hand and its line breaks are
+    // written into the text as \N. A width of "as wide as it wants" reproduces that.
+    val textWidth = Layout.getDesiredWidth(sign.text, fillPaint).toInt() + 1
+    val layout = StaticLayout.Builder
+      .obtain(sign.text, 0, sign.text.length, fillPaint, textWidth)
+      .setAlignment(Layout.Alignment.ALIGN_CENTER)
+      .setIncludePad(false)
+      .build()
+
+    val anchorX = sign.x * scaleToViewX
+    val anchorY = sign.y * scaleToViewY
+    // \pos names the point the cue's *alignment corner* sits on, so the box is offset from it.
+    val left = anchorX - when ((sign.alignment - 1) % 3) {
+      0 -> 0f
+      2 -> layout.width.toFloat()
+      else -> layout.width / 2f
+    }
+    val top = anchorY - when ((sign.alignment - 1) / 3) {
+      0 -> layout.height.toFloat()
+      2 -> 0f
+      else -> layout.height / 2f
+    }
+
+    // \org moves the centre of rotation away from the text, which is how a sign stays glued to a
+    // placard that is itself pivoting off-screen. Without it every rotation would spin about the
+    // text's own middle and drift off the artwork.
+    val pivotX = sign.pivotX * scaleToViewX
+    val pivotY = sign.pivotY * scaleToViewY
+
+    matrix.reset()
+    if (sign.rotationX != 0f || sign.rotationY != 0f) {
+      camera.save()
+      camera.rotateX(sign.rotationX)
+      camera.rotateY(sign.rotationY)
+      camera.getMatrix(matrix)
+      camera.restore()
+      // Camera transforms about the origin, so the pivot has to be walked to it and back.
+      matrix.preTranslate(-pivotX, -pivotY)
+      matrix.postTranslate(pivotX, pivotY)
+    }
+    matrix.postScale(sign.scaleX, sign.scaleY, pivotX, pivotY)
+    matrix.postRotate(sign.rotationZ, pivotX, pivotY)
+
+    val stroke = if (sign.outlineWidth <= 0f) null else {
+      val strokePaint = TextPaint(Paint.ANTI_ALIAS_FLAG)
+      strokePaint.style = Paint.Style.STROKE
+      strokePaint.applySign(sign, scaleToViewY)
+      strokePaint.strokeWidth = sign.outlineWidth * scaleToViewY * 2f
+      strokePaint.color = Color.argb(alpha, Color.red(sign.outlineColour or ALPHA_MASK), Color.green(sign.outlineColour or ALPHA_MASK), Color.blue(sign.outlineColour or ALPHA_MASK))
+      // The stroke is centred on the glyph edge, so half of it lands inside the letter. Drawing the
+      // fill afterwards covers that half and leaves the outline the width the script asked for.
+      StaticLayout.Builder
+        .obtain(sign.text, 0, sign.text.length, strokePaint, textWidth)
+        .setAlignment(Layout.Alignment.ALIGN_CENTER)
+        .setIncludePad(false)
+        .build()
+    }
+
+    return PreparedSign(Matrix(matrix), left, top, layout, stroke)
+  }
+
+  private fun TextPaint.applySign(sign: AssSign, scaleToViewY: Float) {
+    textSize = sign.fontSize * scaleToViewY
+    typeface = Typeface.create(
+      Typeface.DEFAULT,
+      when {
+        sign.bold && sign.italic -> Typeface.BOLD_ITALIC
+        sign.bold -> Typeface.BOLD
+        sign.italic -> Typeface.ITALIC
+        else -> Typeface.NORMAL
+      },
+    )
+    isUnderlineText = sign.underline
+  }
+
+  companion object {
+    /** ASS colours are stored without an alpha byte; opacity travels separately. */
+    private const val ALPHA_MASK = 0xFF000000.toInt()
+
+    private fun signOf(cue: Cue): AssSign? {
+      val text = cue.text as? android.text.Spanned ?: return null
+      return text.getSpans(0, text.length, AssSignSpan::class.java).firstOrNull()?.sign
+    }
+
+    /**
+     * Inserts the overlay in front of `PlayerView`'s subtitle view, or returns the one already
+     * there.
+     *
+     * Returns null when the player view has no subtitle view — a configuration this package does
+     * not produce, but `PlayerView` allows.
+     */
+    /** Removes the overlay from [playerView], dropping its listener registration with it. */
+    fun remove(playerView: PlayerView) {
+      val parent = playerView.subtitleView?.parent as? ViewGroup ?: return
+      for (index in 0 until parent.childCount) {
+        val overlay = parent.getChildAt(index) as? AssSignOverlay ?: continue
+        overlay.detach()
+        parent.removeView(overlay)
+        return
+      }
+    }
+
+    fun install(playerView: PlayerView): AssSignOverlay? {
+      val subtitleView: SubtitleView = playerView.subtitleView ?: return null
+      val parent = subtitleView.parent as? ViewGroup ?: return null
+
+      for (index in 0 until parent.childCount) {
+        (parent.getChildAt(index) as? AssSignOverlay)?.let { return it }
+      }
+
+      val overlay = AssSignOverlay(playerView.context)
+      // Directly in front of the captions: a sign belongs to the picture, and the dialogue box
+      // should not be able to cover it.
+      parent.addView(overlay, parent.indexOfChild(subtitleView) + 1, ViewGroup.LayoutParams(
+        ViewGroup.LayoutParams.MATCH_PARENT,
+        ViewGroup.LayoutParams.MATCH_PARENT,
+      ))
+      return overlay
+    }
+  }
+}