expo-video-subtitle 0.7.1 → 0.7.3

package/CHANGELOG.md

@@ -7,6 +7,57 @@ Changes inherited from the upstream base version are not repeated here — see t
 [upstream changelog](https://github.com/expo/expo/blob/main/packages/expo-video/CHANGELOG.md)
 for the history of everything that isn't subtitle related.
 
+## 0.7.3
+
+Hardening only. Nothing here changes what 0.7.2 draws when 0.7.2 works.
+
+### Changed
+
+- **The sign store falls back to the only script it holds when the track id does not match.** The
+  parser reads `Format.id` from the `Format` its factory was handed; the overlay reads it from the
+  selected text track. Those should be the same value, but if they ever are not, the failure is
+  silent and total — signs parse correctly and never appear, which is precisely how the previous two
+  delivery attempts failed. With one track loaded there is nothing to be ambiguous about. With
+  several it still declines, because a wrong guess would draw one language's typesetting over
+  another's.
+
+- **The overlay logs what it resolved.** Every way this subsystem has failed so far has been quiet:
+  a dropped span, a rejected timestamp, an id that did not match. One line under the `AssSignOverlay`
+  tag now says which text track was selected and how many signs were found for it, so the next
+  failure is one `logcat` away instead of a bisect.
+
+## 0.7.2
+
+### Fixed
+
+- **Signs did not render at all (Android).** 0.7.0 handed each sign to the overlay on a custom span
+  attached to a cue. Nothing survives that trip: `SubtitleExtractor` writes every `CuesWithTiming`
+  into the sample queue through `CueEncoder`, which calls `Cue.toSerializableBundle()` and then
+  `Parcel.marshall()`, and a span is only preserved if it is one of the three Media3 declares in
+  `CustomSpanBundler` or a framework `ParcelableSpan`. Anything else is dropped silently. The cue
+  arrived on the other side as the zero-width placeholder with nothing attached, so the overlay found
+  no signs — no crash, no log, just missing typesetting while the dialogue rendered normally.
+
+  Signs now go to `AssSignStore`, keyed by the track's `Format.id`, and the overlay reads the
+  player's position on each frame to decide what is on screen. That also removes two problems the
+  cue-stream route carried with it: 12,616 sign events would have put ~26,000 boundaries through the
+  timeline pass in `StackingSubtitleParser` — an O(n²) sweep over every entry at every boundary — and
+  written ~26,000 samples into the queue. The dialogue-only file left for Media3 holds 377 events.
+
+- **Scale was applied between the two rotations instead of before both.** Android's `Matrix.post*`
+  methods pre-multiply, so the calls apply in the order they are written, and the camera step ran
+  first. ASS scales a glyph and then turns it; with an anisotropic scale — this episode carries
+  `\fscx125.86` against `\fscy120.19` — the two do not commute, so the shape itself was wrong.
+
+- **The overlay re-laid-out every sign whenever any one of them changed.** Frame-by-frame typesetting
+  changes the on-screen set every 40 ms, so a scene holding ten placards rebuilt all ten `StaticLayout`s
+  on every tick. Layouts are now kept by sign identity and only new signs are laid out.
+
+- **A timestamp with no hour fell through to flat rendering.** `MM:SS.cc` is legal — Media3's own
+  `SSA_TIMECODE_PATTERN` makes the hour optional — but this parser required it and skipped the event,
+  which then stayed in the copy handed to Media3. Media3 reads `\pos` perfectly well and drew the
+  sign flat, with nothing in the logs to say it had happened.
+
 ## 0.7.1
 
 ### Fixed

package/FORK.md

@@ -30,8 +30,8 @@ Measured against the published `expo-video@56.1.4` tarball. `+`/`-` are added/re
 | `android/…/utils/StackingSubtitleParser.kt` | 342 | `SubtitleParser` that segments the cue timeline and merges simultaneous **unpositioned** cues into one stacked block, plus the `SubtitleParser.Factory` that routes SubRip and ASS/SSA to it and wraps every parser in `FilteringSubtitleParser`. ASS gets there through `SsaCueNormalizer` as the `prepare` step — without it `SsaParser` sets both `line` and `position` on every cue and they would all take the passthrough branch. Assigns each cue a vertical slot it keeps for its whole time on screen — a finished cue's line is held blank (`\u200B`) so the cues above it do not drop. Merges through a `SpannableStringBuilder` so cue spans survive. |
 | `android/…/utils/SsaCueNormalizer.kt` | 93 | Clears the `line`/`position` Media3's `SsaParser` fills in on cues the ASS author never placed, so `bottomOffset` and cue stacking work on `.ass` the way they already do on `.srt`. The test is exact rather than heuristic: those values are a pure function of the cue's anchors, so matching both defaults for the bottom-centre anchors *is* "no positioning tag". Also drops events on a style the author made invisible (`Fontsize 0`, or a primary colour whose inverted alpha byte is `FF`) — that is where typesetters park working notes, and nothing downstream can carry "invisible". |
 | `android/…/utils/AssSignParser.kt` | 505 | Splits an ASS script into the typeset events this package draws and a copy of the script with those lines removed for Media3's parser. One pass, no matching step, so an event is in exactly one of the two outputs and cannot be drawn twice. Pure Kotlin with no Android imports, which is what makes it unit-testable on the JVM. Reads `\pos`, `\move`, `\org`, `\an`, `\frz`, `\frx`, `\fry`, `\fscx/y`, `\fs`, `\c`, `\alpha`, `\b`, `\i`, `\u` and the style columns behind them. The tag scanner takes the longest name first — `\c` must not swallow `\clip`, `\b` must not swallow `\bord`, `\blur` or `\be` — and reads a leading digit as part of the name for `\1c` and `\4a`. |
-| `android/…/utils/AssSignOverlay.kt` | 235 | Draws those signs on a canvas in front of the captions, with `Matrix` for rotation and scale and `Camera` for the two 3-D axes. **Media3 cannot do this**: `SubtitlePainter` has no rotate, skew, Matrix or Camera call, and `Cue` has no rotation field. Sits in the same parent as `SubtitleView` — the content frame, sized to the video, which is why `\pos` survives letterboxing. Lays text out on change rather than per frame, because frame-by-frame typesetting changes the on-screen set every 40 ms. |
-| `android/…/utils/AssSignSpan.kt` | 24 | Carries one parsed sign through Media3's cue stream to the overlay, on a zero-width placeholder so `SubtitleView` draws nothing. Riding the cue stream is what buys the timing for free. |
+| `android/…/utils/AssSignOverlay.kt` | 330 | Draws those signs on a canvas in front of the captions, with `Matrix` for rotation and scale and `Camera` for the two 3-D axes. Drives itself from the player's clock on a `Choreographer` callback, because signs do not ride the cue stream and nothing else would tell it when one starts; 40 ms events need frame granularity anyway. **Media3 cannot do this**: `SubtitlePainter` has no rotate, skew, Matrix or Camera call, and `Cue` has no rotation field. Sits in the same parent as `SubtitleView` — the content frame, sized to the video, which is why `\pos` survives letterboxing. Lays text out on change rather than per frame, because frame-by-frame typesetting changes the on-screen set every 40 ms. |
+| `android/…/utils/AssSignStore.kt` | 61 | Hands parsed signs from the parser to the overlay, keyed by the track's `Format.id`. **Not** the cue stream: `CueEncoder` marshals every cue through `Parcel`, and a custom span that is neither a framework `ParcelableSpan` nor one of Media3's three `CustomSpanBundler` types is dropped without a word — which is how 0.7.0 shipped with signs that parsed correctly and never appeared. Staying out of the cue stream also keeps 12,616 sign events from putting ~26,000 boundaries through the O(n²) timeline pass. |
 | `android/…/utils/RoundedSubtitleBackground.kt` | 352 | Draws `subtitleStyle.backgroundRadius` as rounded boxes, which Media3 cannot do at all: it paints `backgroundColor` as a framework `BackgroundColorSpan` and `windowColor` with `canvas.drawRect`, neither takes a radius, and `SubtitlePainter`/`CanvasSubtitleOutput` are package-private final while `SubtitleView` is final. A second `SubtitleView` is inserted behind `PlayerView`'s own, fed from `Player.Listener.onCues`, drawing a `LineBackgroundSpan` per line. **Its `EDGE_TYPE_NONE` is load-bearing** — `SubtitlePainter` copies the cue text for an edge pass and draws both layouts when the edge type is outline/raised/depressed, which would paint the box twice and darken a semi-transparent colour. State hangs off the view's `tag`, never a map in the companion object; see the `find` KDoc for why even a `WeakHashMap` leaks here. |
 | `FORK.md`, `LICENSE` | — | This guide; MIT licence (upstream copyright preserved). |
 | `eslint.config.js` | 4 | Verbatim copy of `expo-module-scripts/templates/eslint.config.js`. **Recreate it if a sync drops it** — `expo-module configure` treats it as an *optional* template and only syncs it when it already exists, so without this file `npm run lint` dies with "ESLint couldn't find an eslint.config.js". |

package/android/src/main/java/expo/modules/video/utils/AssSignOverlay.kt

@@ -11,12 +11,14 @@ import android.graphics.Typeface
 import android.text.Layout
 import android.text.StaticLayout
 import android.text.TextPaint
+import android.view.Choreographer
 import android.view.View
 import android.view.ViewGroup
 import androidx.annotation.OptIn
+import androidx.media3.common.C
 import androidx.media3.common.Player
-import androidx.media3.common.text.Cue
-import androidx.media3.common.text.CueGroup
+import androidx.media3.common.Tracks
+import androidx.media3.common.util.Log
 import androidx.media3.common.util.UnstableApi
 import androidx.media3.ui.PlayerView
 import androidx.media3.ui.SubtitleView
@@ -25,9 +27,10 @@ import androidx.media3.ui.SubtitleView
  * Draws ASS typesetting — signs, placards, on-screen labels — with the transforms the script asks
  * for.
  *
- * `SubtitleView` cannot: `SubtitlePainter` has no rotation, and `Cue` has no field to carry one.
- * So [AssSignParser] takes those events out of Media3's cue stream, wraps each in an [AssSignSpan],
- * and this view draws them itself.
+ * `SubtitleView` cannot: `SubtitlePainter` has no call to `rotate`, `skew`, `Matrix` or `Camera`,
+ * and `Cue` has no rotation field, only `shearDegrees`, which the painter ignores. So
+ * [AssSignParser] takes those events out of the file Media3 parses and leaves them in
+ * [AssSignStore], and this view draws them.
  *
  * ### Where it sits
  *
@@ -36,93 +39,170 @@ import androidx.media3.ui.SubtitleView
  * survive letterboxing. `\pos` is in `PlayResX`/`PlayResY` space and maps onto this view's bounds
  * directly. [RoundedSubtitleBackground] takes the same route for the same reason.
  *
- * ### Why a canvas and not child views
+ * ### Why it drives itself from the clock
  *
- * A `TextView` per sign would work — Android view transforms map onto ASS almost one for one — but
- * a heavily typeset episode puts dozens of signs on screen across a scene and each would cost a
- * measure and layout pass. Drawing straight to the canvas costs one `StaticLayout` per sign per
- * frame it changes, and it is the only way to stroke the outline the way ASS means it: the border
- * drawn *behind* the fill, not as a shadow.
+ * Signs do not ride the cue stream — see [AssSignStore] for why they cannot — so nothing tells this
+ * view when one begins. It reads the player's position on each frame instead, which is also the only
+ * granularity that works: frame-by-frame typesetting writes one event per frame, and a sign on
+ * screen for 40 ms has to arrive and leave on the frame it was authored for.
  */
 @SuppressLint("ViewConstructor")
 @OptIn(UnstableApi::class)
 internal class AssSignOverlay(context: Context) : View(context), Player.Listener {
 
   private val camera = Camera()
-  private val matrix = Matrix()
 
-  private var signs: List<AssSign> = emptyList()
-  private var prepared: List<PreparedSign> = emptyList()
   private var player: Player? = null
+  private var script: AssScript? = null
+
+  private var active: List<AssSign> = emptyList()
+  private var prepared: List<PreparedSign> = emptyList()
+
+  /**
+   * Keyed by identity, because [AssSignParser] produces one instance per event and hands back the
+   * same one for every frame it is on screen. Without it, a scene holding ten placards while an
+   * eleventh animates would re-lay-out all eleven on every 40 ms tick.
+   */
+  private var layouts = HashMap<AssSign, PreparedSign>()
+
+  private var frameScheduled = false
+  private val onFrame = Choreographer.FrameCallback {
+    frameScheduled = false
+    tick()
+    schedule()
+  }
+
+  // -----------------------------------------------------------------------------------------
+  // Binding
+  // -----------------------------------------------------------------------------------------
+
+  fun attachTo(player: Player?) {
+    if (this.player !== player) {
+      this.player?.removeListener(this)
+      this.player = player
+      player?.addListener(this)
+    }
+    // Re-resolved unconditionally: the same player switches between text tracks, and this doubles
+    // as the first resolve, before `onTracksChanged` has had anything to say.
+    resolveScript()
+    schedule()
+  }
+
+  fun detach() {
+    player?.removeListener(this)
+    player = null
+    script = null
+    layouts = HashMap()
+    setActive(emptyList())
+  }
+
+  override fun onTracksChanged(tracks: Tracks) {
+    resolveScript()
+    schedule()
+  }
+
+  private fun resolveScript() {
+    val trackId = selectedTextTrackId()
+    val next = AssSignStore.get(trackId)
+    if (next === script) return
+    // Logged because every way this has failed so far has been silent. If signs go missing again,
+    // this line says whether the parser filed them and whether the overlay found them.
+    Log.d(TAG, "text track $trackId -> ${next?.signs?.size ?: "no"} sign(s)")
+    script = next
+    layouts = HashMap()
+    setActive(emptyList())
+  }
+
+  /**
+   * `Format.id` of the selected text track, which `SubtitleSource.toSubtitleConfiguration` sets to
+   * `expo-subtitle:<language>:<uri>` — the same value the parser filed its signs under.
+   */
+  private fun selectedTextTrackId(): String? {
+    val groups = player?.currentTracks?.groups ?: return null
+    for (group in groups) {
+      if (group.type != C.TRACK_TYPE_TEXT) continue
+      for (index in 0 until group.length) {
+        if (group.isTrackSelected(index)) return group.getTrackFormat(index).id
+      }
+    }
+    return null
+  }
+
+  // -----------------------------------------------------------------------------------------
+  // The clock
+  // -----------------------------------------------------------------------------------------
+
+  override fun onAttachedToWindow() {
+    super.onAttachedToWindow()
+    schedule()
+  }
+
+  override fun onDetachedFromWindow() {
+    super.onDetachedFromWindow()
+    Choreographer.getInstance().removeFrameCallback(onFrame)
+    frameScheduled = false
+  }
 
-  init {
-    setWillNotDraw(false)
+  private fun schedule() {
+    // No script means nothing to draw and no reason to hold a frame callback open. Paused playback
+    // still ticks, because a seek while paused has to move the signs with it.
+    if (frameScheduled || script == null || !isAttachedToWindow) return
+    frameScheduled = true
+    Choreographer.getInstance().postFrameCallback(onFrame)
   }
 
-  override fun onCues(cueGroup: CueGroup) {
-    setSigns(cueGroup.cues)
+  private fun tick() {
+    val script = this.script ?: return
+    val player = this.player ?: return
+    setActive(script.signsAt(player.contentPosition * 1000L))
   }
 
-  private fun setSigns(cues: List<Cue>) {
-    val next = cues.mapNotNull(::signOf)
-    // Reference equality on the list contents is enough: the parser produces one AssSign per event
-    // and hands out the same instance for every segment the event spans.
-    if (next.size == signs.size && next.indices.all { next[it] === signs[it] }) return
-    signs = next
+  private fun setActive(next: List<AssSign>) {
+    if (next.size == active.size && next.indices.all { next[it] === active[it] }) return
+    active = next
     rebuild()
   }
 
   override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
     super.onSizeChanged(w, h, oldw, oldh)
     // Every measurement below is in view pixels, so all of it is stale after a resize — which
-    // happens on every rotation and on entering full screen.
+    // happens on rotation and on entering full screen.
+    layouts = HashMap()
     rebuild()
   }
 
-  /**
-   * Lays the signs out once per change rather than once per frame.
-   *
-   * Frame-by-frame typesetting changes the on-screen set every 40 ms, and `onDraw` can run more
-   * often than that for reasons of the view system's own. Text layout is the expensive half of
-   * drawing, and none of it depends on anything that varies between frames.
-   */
+  /** Lays out only what is new on screen, and keeps the rest as it was. */
   private fun rebuild() {
-    prepared = if (width == 0 || height == 0) emptyList() else signs.mapNotNull(::prepare)
-    invalidate()
-  }
+    if (width == 0 || height == 0) {
+      prepared = emptyList()
+      invalidate()
+      return
+    }
 
-  fun attachTo(player: Player?) {
-    if (this.player === player) return
-    this.player?.removeListener(this)
-    this.player = player
-    player?.addListener(this)
-    // A listener only hears about the *next* cue change, so seed from what is on screen already.
-    setSigns(player?.currentCues?.cues ?: emptyList())
+    val next = HashMap<AssSign, PreparedSign>(active.size)
+    prepared = active.mapNotNull { sign ->
+      (layouts[sign] ?: prepare(sign))?.also { next[sign] = it }
+    }
+    layouts = next
+    invalidate()
   }
 
-  fun detach() {
-    player?.removeListener(this)
-    player = null
-    signs = emptyList()
-    rebuild()
-  }
+  // -----------------------------------------------------------------------------------------
+  // Drawing
+  // -----------------------------------------------------------------------------------------
 
   override fun onDraw(canvas: Canvas) {
     for (sign in prepared) {
-      draw(canvas, sign)
+      canvas.save()
+      canvas.concat(sign.matrix)
+      canvas.translate(sign.left, sign.top)
+      sign.stroke?.draw(canvas)
+      sign.fill.draw(canvas)
+      canvas.restore()
     }
   }
 
-  private fun draw(canvas: Canvas, sign: PreparedSign) {
-    canvas.save()
-    canvas.concat(sign.matrix)
-    canvas.translate(sign.left, sign.top)
-    sign.stroke?.draw(canvas)
-    sign.fill.draw(canvas)
-    canvas.restore()
-  }
-
-  /** Everything about one sign that does not change between frames. */
+  /** Everything about one sign that does not change while it is on screen. */
   private class PreparedSign(
     val matrix: Matrix,
     val left: Float,
@@ -141,16 +221,16 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
     if (alpha == 0) return null
 
     // A fresh paint per sign: StaticLayout keeps the reference it was built with, so a shared
-    // mutable one would repaint every sign already laid out in this pass.
+    // mutable one would repaint every sign already laid out.
     val fillPaint = TextPaint(Paint.ANTI_ALIAS_FLAG)
     fillPaint.applySign(sign, scaleToViewY)
-    fillPaint.color = Color.argb(alpha, Color.red(sign.colour or ALPHA_MASK), Color.green(sign.colour or ALPHA_MASK), Color.blue(sign.colour or ALPHA_MASK))
+    fillPaint.color = sign.colour.withAlpha(alpha)
 
     // Wrapping is the script's job, not ours: a sign is placed by hand and its line breaks are
     // written into the text as \N. A width of "as wide as it wants" reproduces that.
     val textWidth = Layout.getDesiredWidth(sign.text, fillPaint).toInt() + 1
-    // A multi-line sign wraps its lines the way its own alignment says, not always centred: a
-    // left-anchored placard reads wrong with its second line pulled to the middle.
+    // A multi-line sign wraps the way its own alignment says, not always centred: a left-anchored
+    // placard reads wrong with its second line pulled to the middle.
     val blockAlignment = when ((sign.alignment - 1) % 3) {
       0 -> Layout.Alignment.ALIGN_NORMAL
       2 -> Layout.Alignment.ALIGN_OPPOSITE
@@ -177,23 +257,31 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
     }
 
     // \org moves the centre of rotation away from the text, which is how a sign stays glued to a
-    // placard that is itself pivoting off-screen. Without it every rotation would spin about the
-    // text's own middle and drift off the artwork.
+    // placard pivoting off-screen. Without it every rotation would spin about the text's own middle
+    // and drift off the artwork.
     val pivotX = sign.pivotX * scaleToViewX
     val pivotY = sign.pivotY * scaleToViewY
 
-    matrix.reset()
+    // Order matters, and it is not the order the calls read in. Android's `post*` methods
+    // pre-multiply — `M.postRotate(r)` maps a point as `R * (M * p)` — so the first call written is
+    // the first one applied to the glyph. ASS scales the glyph and *then* turns it, and with an
+    // anisotropic scale (this episode carries fscx 125.86 against fscy 120.19) the two do not
+    // commute, so getting it backwards changes the shape rather than just the arithmetic.
+    val matrix = Matrix()
+    matrix.postScale(sign.scaleX, sign.scaleY, pivotX, pivotY)
     if (sign.rotationX != 0f || sign.rotationY != 0f) {
+      val cameraMatrix = Matrix()
       camera.save()
       camera.rotateX(sign.rotationX)
       camera.rotateY(sign.rotationY)
-      camera.getMatrix(matrix)
+      // `getMatrix` overwrites rather than concatenating, so it needs a matrix of its own.
+      camera.getMatrix(cameraMatrix)
       camera.restore()
-      // Camera transforms about the origin, so the pivot has to be walked to it and back.
-      matrix.preTranslate(-pivotX, -pivotY)
-      matrix.postTranslate(pivotX, pivotY)
+      // The camera transforms about the origin; walk the pivot there and back.
+      cameraMatrix.preTranslate(-pivotX, -pivotY)
+      cameraMatrix.postTranslate(pivotX, pivotY)
+      matrix.postConcat(cameraMatrix)
     }
-    matrix.postScale(sign.scaleX, sign.scaleY, pivotX, pivotY)
     matrix.postRotate(sign.rotationZ, pivotX, pivotY)
 
     val stroke = if (sign.outlineWidth <= 0f) null else {
@@ -201,9 +289,9 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
       strokePaint.style = Paint.Style.STROKE
       strokePaint.applySign(sign, scaleToViewY)
       strokePaint.strokeWidth = sign.outlineWidth * scaleToViewY * 2f
-      strokePaint.color = Color.argb(alpha, Color.red(sign.outlineColour or ALPHA_MASK), Color.green(sign.outlineColour or ALPHA_MASK), Color.blue(sign.outlineColour or ALPHA_MASK))
+      strokePaint.color = sign.outlineColour.withAlpha(alpha)
       // The stroke is centred on the glyph edge, so half of it lands inside the letter. Drawing the
-      // fill afterwards covers that half and leaves the outline the width the script asked for.
+      // fill over it leaves the outline the width the script asked for.
       StaticLayout.Builder
         .obtain(sign.text, 0, sign.text.length, strokePaint, textWidth)
         .setAlignment(blockAlignment)
@@ -211,7 +299,7 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
         .build()
     }
 
-    return PreparedSign(Matrix(matrix), left, top, layout, stroke)
+    return PreparedSign(matrix, left, top, layout, stroke)
   }
 
   private fun TextPaint.applySign(sign: AssSign, scaleToViewY: Float) {
@@ -228,22 +316,16 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
     isUnderlineText = sign.underline
   }
 
-  companion object {
-    /** ASS colours are stored without an alpha byte; opacity travels separately. */
-    private const val ALPHA_MASK = 0xFF000000.toInt()
+  /** ASS colours carry no alpha byte; opacity travels separately. */
+  private fun Int.withAlpha(alpha: Int): Int {
+    val opaque = this or OPAQUE
+    return Color.argb(alpha, Color.red(opaque), Color.green(opaque), Color.blue(opaque))
+  }
 
-    private fun signOf(cue: Cue): AssSign? {
-      val text = cue.text as? android.text.Spanned ?: return null
-      return text.getSpans(0, text.length, AssSignSpan::class.java).firstOrNull()?.sign
-    }
+  companion object {
+    private const val TAG = "AssSignOverlay"
+    private const val OPAQUE = 0xFF000000.toInt()
 
-    /**
-     * Inserts the overlay in front of `PlayerView`'s subtitle view, or returns the one already
-     * there.
-     *
-     * Returns null when the player view has no subtitle view — a configuration this package does
-     * not produce, but `PlayerView` allows.
-     */
     /** Removes the overlay from [playerView], dropping its listener registration with it. */
     fun remove(playerView: PlayerView) {
       val parent = playerView.subtitleView?.parent as? ViewGroup ?: return
@@ -255,6 +337,13 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
       }
     }
 
+    /**
+     * Inserts the overlay in front of `PlayerView`'s subtitle view, or returns the one already
+     * there.
+     *
+     * Returns null when the player view has no subtitle view — a configuration this package does not
+     * produce, but `PlayerView` allows.
+     */
     fun install(playerView: PlayerView): AssSignOverlay? {
       val subtitleView: SubtitleView = playerView.subtitleView ?: return null
       val parent = subtitleView.parent as? ViewGroup ?: return null
@@ -264,12 +353,13 @@ internal class AssSignOverlay(context: Context) : View(context), Player.Listener
       }
 
       val overlay = AssSignOverlay(playerView.context)
-      // Directly in front of the captions: a sign belongs to the picture, and the dialogue box
-      // should not be able to cover it.
-      parent.addView(overlay, parent.indexOfChild(subtitleView) + 1, ViewGroup.LayoutParams(
-        ViewGroup.LayoutParams.MATCH_PARENT,
-        ViewGroup.LayoutParams.MATCH_PARENT,
-      ))
+      // Directly in front of the captions: a sign belongs to the picture, and the dialogue box must
+      // not be able to cover it.
+      parent.addView(
+        overlay,
+        parent.indexOfChild(subtitleView) + 1,
+        ViewGroup.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT),
+      )
       return overlay
     }
   }

package/android/src/main/java/expo/modules/video/utils/AssSignParser.kt

@@ -232,13 +232,21 @@ internal object AssSignParser {
     )
   }
 
-  /** `H:MM:SS.cc`, centiseconds. */
+  /**
+   * `H:MM:SS.cc` or `MM:SS.cc`, centiseconds.
+   *
+   * The hour is optional because Media3's own `SSA_TIMECODE_PATTERN` makes it optional. Requiring it
+   * was worse than a parse failure: an event this rejected was never classified, so it stayed in the
+   * copy handed to Media3 — whose `SsaStyle.Overrides` reads `\pos` perfectly well and would draw
+   * the sign flat and unrotated, which is the exact failure this whole subsystem exists to remove,
+   * with nothing in the logs to say it had happened.
+   */
   private fun parseTimeUs(value: String): Long? {
     val parts = value.split(':')
-    if (parts.size != 3) return null
-    val h = parts[0].trim().toLongOrNull() ?: return null
-    val m = parts[1].trim().toLongOrNull() ?: return null
-    val s = parts[2].trim().toDoubleOrNull() ?: return null
+    if (parts.size !in 2..3) return null
+    val h = if (parts.size == 3) parts[0].trim().toLongOrNull() ?: return null else 0L
+    val m = parts[parts.size - 2].trim().toLongOrNull() ?: return null
+    val s = parts[parts.size - 1].trim().toDoubleOrNull() ?: return null
     return ((h * 3600 + m * 60) * 1_000_000L) + (s * 1_000_000.0).toLong()
   }
 
@@ -500,11 +508,39 @@ internal data class AssSign(
   val playResY: Float,
 )
 
-internal data class AssScript(
+internal class AssScript(
   val playResX: Float,
   val playResY: Float,
   /** Typeset events, drawn by [AssSignOverlay]. */
   val signs: List<AssSign>,
   /** The script with every sign, drawing and invisible event cut out, for Media3's own parser. */
   val dialogueOnly: String,
-)
+) {
+  /**
+   * Signs bucketed by the whole second they are on screen during, so the overlay can answer "what is
+   * showing now" once per frame without walking twelve thousand events.
+   *
+   * A bucket per second rather than a sorted sweep because playback is not monotonic — every seek
+   * would otherwise need a rebuild — and because frame-by-frame typesetting is almost entirely
+   * events shorter than one frame, so nearly every sign lands in exactly one bucket.
+   */
+  val signsBySecond: Map<Long, List<AssSign>> by lazy {
+    val buckets = HashMap<Long, MutableList<AssSign>>()
+    for (sign in signs) {
+      val first = sign.startUs / 1_000_000L
+      // `endUs` is exclusive, so an event ending exactly on a second boundary does not reach into
+      // the next bucket.
+      val last = (sign.endUs - 1).coerceAtLeast(sign.startUs) / 1_000_000L
+      for (second in first..last) {
+        buckets.getOrPut(second) { ArrayList(4) }.add(sign)
+      }
+    }
+    buckets
+  }
+
+  /** The signs on screen at [positionUs], in file order. */
+  fun signsAt(positionUs: Long): List<AssSign> {
+    val bucket = signsBySecond[positionUs / 1_000_000L] ?: return emptyList()
+    return bucket.filter { positionUs >= it.startUs && positionUs < it.endUs }
+  }
+}

package/android/src/main/java/expo/modules/video/utils/AssSignStore.kt

@@ -0,0 +1,70 @@
+package expo.modules.video.utils
+
+/**
+ * Hands parsed ASS typesetting from the subtitle parser to [AssSignOverlay].
+ *
+ * ### Why not the cue stream
+ *
+ * The obvious route — wrap each sign in a span on a cue and let Media3's timing carry it — does not
+ * survive the trip. `SubtitleExtractor` writes every `CuesWithTiming` into the sample queue through
+ * `CueEncoder`, which calls `Cue.toSerializableBundle()` and then `Parcel.marshall()`. Spans are
+ * only preserved if they are one of the three Media3 declares in `CustomSpanBundler` or a framework
+ * `ParcelableSpan`; anything else is dropped without a word. A custom span arrives on the other side
+ * as plain text with nothing attached, so the overlay saw no signs at all — no crash, no log, just
+ * an empty screen where the typesetting should be.
+ *
+ * Two further reasons not to go back to it even if a span could be smuggled through. A typeset
+ * episode holds 12,616 sign events, which would put ~26,000 boundaries into
+ * [StackingSubtitleParser]'s timeline pass — an O(n²) sweep over every entry at every boundary — and
+ * then write ~26,000 samples into the queue. Keeping signs out of the cue stream leaves the
+ * dialogue-only file at a few hundred events, which is the size that pass was written for.
+ *
+ * ### The key
+ *
+ * `Format.id`, which `SubtitleSource.toSubtitleConfiguration` sets to
+ * `expo-subtitle:<language>:<uri>` — stable, unique per track, and readable from both ends: the
+ * parser gets it from the `Format` its factory was given, and the overlay reads it off the selected
+ * text track. Without a match the overlay draws nothing rather than guessing, because guessing means
+ * drawing one language's typesetting over another's.
+ */
+internal object AssSignStore {
+
+  /**
+   * Enough for the track being watched plus a few the viewer switched away from. Each entry holds
+   * every sign in an episode — 12,616 in the file this was built against — so this is not a cache to
+   * let grow.
+   */
+  private const val MAX_TRACKS = 4
+
+  private val scripts = object : LinkedHashMap<String, AssScript>(MAX_TRACKS, 0.75f, true) {
+    override fun removeEldestEntry(eldest: MutableMap.MutableEntry<String, AssScript>): Boolean =
+      size > MAX_TRACKS
+  }
+
+  /** Called from the extractor's loading thread. */
+  @Synchronized
+  fun put(trackId: String, script: AssScript) {
+    scripts[trackId] = script
+  }
+
+  /**
+   * Called from the main thread, once per track change.
+   *
+   * Falls back to the only script on hand when the id does not match. The two ends read `Format.id`
+   * from different places — the parser from the `Format` its factory was handed, the overlay from
+   * the selected track — and if those ever disagree the failure is silent and total: signs parse
+   * perfectly and never appear, which is exactly how the previous delivery route failed. With one
+   * track loaded there is no ambiguity to protect against, so the safe answer is the obvious one.
+   * With several, a wrong guess would draw one language's typesetting over another's, so it declines.
+   */
+  @Synchronized
+  fun get(trackId: String?): AssScript? {
+    scripts[trackId]?.let { return it }
+    return scripts.values.singleOrNull()
+  }
+
+  @Synchronized
+  fun clear() {
+    scripts.clear()
+  }
+}

package/android/src/main/java/expo/modules/video/utils/RoundedSubtitleBackground.kt

@@ -191,9 +191,6 @@ class RoundedSubtitleBackground private constructor(private val backgroundView:
    */
   private fun toBackgroundCue(cue: Cue): Cue? {
     val text = cue.text ?: return null // Bitmap cues carry no text to put a box behind.
-    // A sign's cue is a zero-width placeholder standing in for text drawn by `AssSignOverlay`.
-    // There is no glyph here to sit behind, and the box would be the only thing on screen.
-    if (text is Spanned && text.getSpans(0, text.length, AssSignSpan::class.java).isNotEmpty()) return null
     val stripped = if (keepMetricSpans && text is Spanned) {
       SpannableString(text).also { spannable ->
         for (span in spannable.getSpans(0, spannable.length, Any::class.java)) {

package/android/src/main/java/expo/modules/video/utils/StackingSubtitleParser.kt

@@ -1,8 +1,6 @@
 package expo.modules.video.utils
 
-import android.text.SpannableString
 import android.text.SpannableStringBuilder
-import android.text.Spanned
 import androidx.annotation.OptIn
 import androidx.media3.common.C
 import androidx.media3.common.Format
@@ -77,24 +75,14 @@ open class StackingSubtitleParser(
 ) : SubtitleParser {
 
   /**
-   * A chance for a subclass to take some events out of the delegate's hands and supply its own cues
-   * for them. Returns the bytes the delegate should parse plus the cues the subclass is drawing.
-   *
-   * Default is to change nothing.
+   * A chance for a subclass to hand the delegate something other than the bytes it was given —
+   * [StackingSsaParser] removes the typeset events it draws itself. Default is to change nothing.
    */
   protected open fun split(data: ByteArray, offset: Int, length: Int): Split =
-    Split(data, offset, length, emptyList())
-
-  /** [split]'s result: what the delegate parses, and what the subclass contributes. */
-  protected class Split(
-    val data: ByteArray,
-    val offset: Int,
-    val length: Int,
-    val ownCues: List<PreparedCue>,
-  )
+    Split(data, offset, length)
 
-  /** One cue a subclass supplies, with the timing the delegate would otherwise have derived. */
-  protected class PreparedCue(val cue: Cue, val startUs: Long, val endUs: Long)
+  /** [split]'s result: the bytes the delegate should parse. */
+  protected class Split(val data: ByteArray, val offset: Int, val length: Int)
 
   // Every emitted segment is a complete description of what should be on screen for its interval,
   // so segments replace each other rather than being merged by the renderer.
@@ -113,9 +101,6 @@ open class StackingSubtitleParser(
   ) {
     val entries = mutableListOf<Entry>()
     val split = split(data, offset, length)
-    for (prepared in split.ownCues) {
-      entries.add(Entry(prepared.cue, prepared.startUs, prepared.endUs, entries.size))
-    }
     // Always ask the delegate for everything; `outputOptions` is applied to the merged result below,
     // because a segment can span cues that fall on both sides of the requested start time.
     delegate.parse(split.data, split.offset, split.length, SubtitleParser.OutputOptions.allCues()) { cuesWithTiming ->
@@ -320,64 +305,59 @@ class StackingSubripParser : StackingSubtitleParser(SubripParser())
 /**
  * ASS and SSA.
  *
- * Splits the script in two before anything else runs. [AssSignParser] takes the typeset events —
- * anything the author placed — and the delegate gets a copy with those lines removed, so the two
- * renderers can never draw the same event. Signs come back as cues carrying an [AssSignSpan], which
- * rides Media3's own timing all the way to [AssSignOverlay]; the cue text itself is a zero-width
- * character, so `SubtitleView` lays it out and draws nothing.
+ * Splits the script before anything else runs. [AssSignParser] takes the typeset events — anything
+ * the author placed — and the delegate gets a copy with those lines removed, so the two renderers
+ * can never draw the same event. The signs go to [AssSignStore] for [AssSignOverlay] to pick up;
+ * they deliberately do **not** ride the cue stream, because a custom span does not survive
+ * `CueEncoder`'s trip through `Parcel` and because twelve thousand of them would swamp the timeline
+ * pass above. See [AssSignStore] for the whole reasoning.
  *
- * Falls back to plain delegation when the script yields no signs, which is every `.ass` that is
- * only dialogue, and when the bytes are not decodable as text.
+ * Falls back to plain delegation when the script yields no signs, which is every `.ass` that is only
+ * dialogue, and when anything at all goes wrong.
  *
  * The delegate is built by the caller rather than constructed here because an SSA track muxed into
  * a container carries its `[V4+ Styles]` header in the format's initialization data, and only
  * `DefaultSubtitleParserFactory` knows to pass it on. A parser built without it resolves no styles.
  */
 @OptIn(UnstableApi::class)
-class StackingSsaParser(delegate: SubtitleParser) :
-  StackingSubtitleParser(delegate, SsaCueNormalizer::normalize) {
+class StackingSsaParser(
+  delegate: SubtitleParser,
+  /**
+   * `Format.id` of the track being parsed, which is how the overlay finds these signs again. Null
+   * means no lookup key, so the signs are parsed out of the delegate's copy but never drawn — the
+   * captions stay correct, the typesetting is simply absent.
+   */
+  private val trackId: String?,
+) : StackingSubtitleParser(delegate, SsaCueNormalizer::normalize) {
 
   override fun split(data: ByteArray, offset: Int, length: Int): Split {
     // Nothing in here is worth a crash, and the whole body is inside the guard rather than just the
-    // parse. This runs on the extractor's loading thread, where an exception does not fail the
+    // parse. This runs on the extractor's loading thread, where a thrown exception is not a failed
     // subtitle — it takes playback down with it, which is how a stray `}` in a regex turned into a
     // dead player on an episode that had been playing fine.
     //
     // A subtitle file is untrusted input from a scraper. The honest failure for one this cannot
     // read is captions drawn Media3's own way, flat, rather than no video at all.
     return runCatching {
+      if (trackId == null) return@runCatching Split(data, offset, length)
+
       val script = AssSignParser.parse(String(data, offset, length, Charsets.UTF_8))
       if (script.signs.isEmpty()) {
-        Split(data, offset, length, emptyList())
+        Split(data, offset, length)
       } else {
+        AssSignStore.put(trackId, script)
         val dialogue = script.dialogueOnly.toByteArray(Charsets.UTF_8)
-        Split(dialogue, 0, dialogue.size, script.signs.map(::toEntry))
+        Split(dialogue, 0, dialogue.size)
       }
     }.getOrElse { error ->
       Log.w(TAG, "Sign rendering unavailable for this track; falling back to Media3", error)
-      Split(data, offset, length, emptyList())
+      Split(data, offset, length)
     }
   }
 
   private companion object {
     const val TAG = "StackingSsaParser"
   }
-
-  private fun toEntry(sign: AssSign): PreparedCue {
-    val text = SpannableString(SIGN_CUE_PLACEHOLDER)
-    text.setSpan(AssSignSpan(sign), 0, text.length, Spanned.SPAN_EXCLUSIVE_EXCLUSIVE)
-
-    // Positioned so the stacking pass leaves it alone and the span filter keeps it intact. The
-    // values are only a rough placement for the invisible placeholder; the overlay reads the real
-    // coordinates off the span.
-    val cue = Cue.Builder()
-      .setText(text)
-      .setPosition((sign.x / sign.playResX).coerceIn(0f, 1f))
-      .setLine((sign.y / sign.playResY).coerceIn(0f, 1f), Cue.LINE_TYPE_FRACTION)
-      .build()
-
-    return PreparedCue(cue, sign.startUs, sign.endUs)
-  }
 }
 
 /**
@@ -407,7 +387,7 @@ class ExpoVideoSubtitleParserFactory : SubtitleParser.Factory {
     // reliably a sign — typeset to sit on the artwork, in colours chosen to match it — rather than
     // dialogue that happens to carry a cue setting.
     format.isSsa() -> FilteringSubtitleParser(
-      StackingSsaParser(defaultFactory.create(format)),
+      StackingSsaParser(defaultFactory.create(format), format.id),
       keepAppearanceOfPositionedCues = true,
     )
     else -> FilteringSubtitleParser(defaultFactory.create(format))

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "expo-video-subtitle",
-  "version": "0.7.1",
+  "version": "0.7.3",
   "description": "A cross-platform, performant video component for React Native and Expo with external subtitle sideloading and custom subtitle styling.",
   "keywords": [
     "expo",

package/android/src/main/java/expo/modules/video/utils/AssSignSpan.kt

@@ -1,23 +0,0 @@
-package expo.modules.video.utils
-
-/**
- * Carries one parsed ASS sign through Media3's cue stream to [AssSignOverlay].
- *
- * The cue itself is a single zero-width character, so `SubtitleView` lays it out and draws nothing
- * while the overlay reads the sign off this span and draws it properly. Riding the cue stream is
- * what buys the timing for free: Media3 already decides which cues are on screen at each moment,
- * and a sign is on screen exactly as long as its cue is.
- *
- * A marker span rather than a field on `Cue` because `Cue` is final and carries no room for one —
- * it has `shearDegrees` and nothing else transform-shaped, and `SubtitlePainter` ignores even that.
- */
-internal class AssSignSpan(val sign: AssSign)
-
-/**
- * The cue text a sign rides on.
- *
- * A zero-width space still occupies a line of height in `SubtitleView`, which is harmless for a
- * positioned cue and keeps the span anchored to something. Distinct from [HELD_SLOT_PLACEHOLDER]
- * only in intent; both are invisible.
- */
-internal const val SIGN_CUE_PLACEHOLDER = "​"