react-x11 2.52.0 → 2.54.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (115) hide show
  1. package/README.md +10 -10
  2. package/docs/README.md +160 -314
  3. package/docs/accessibility.md +176 -229
  4. package/docs/appearance.md +77 -176
  5. package/docs/click-to-component.md +18 -51
  6. package/docs/clipboard.md +40 -59
  7. package/docs/components.md +518 -1036
  8. package/docs/dbus.md +61 -133
  9. package/docs/debugging.md +152 -245
  10. package/docs/desktop-calendar.md +51 -88
  11. package/docs/desktop.md +222 -347
  12. package/docs/devtools.md +50 -167
  13. package/docs/drag-and-drop.md +126 -204
  14. package/docs/ecosystem/animation.md +91 -125
  15. package/docs/ecosystem/data-fetching.md +38 -53
  16. package/docs/ecosystem/dev-tooling.md +159 -202
  17. package/docs/ecosystem/forms.md +61 -97
  18. package/docs/ecosystem/headless.md +64 -89
  19. package/docs/ecosystem/i18n.md +18 -21
  20. package/docs/ecosystem/icons.md +93 -126
  21. package/docs/ecosystem/layout.md +42 -52
  22. package/docs/ecosystem/routing.md +18 -29
  23. package/docs/ecosystem/state.md +71 -88
  24. package/docs/ecosystem/testing.md +131 -157
  25. package/docs/ecosystem/theming.md +119 -165
  26. package/docs/ecosystem.md +145 -179
  27. package/docs/elements.md +1096 -1774
  28. package/docs/embedding.md +83 -132
  29. package/docs/events.md +412 -695
  30. package/docs/extending.md +784 -1326
  31. package/docs/eyedropper.md +63 -122
  32. package/docs/filedialog.md +71 -107
  33. package/docs/frame.md +195 -339
  34. package/docs/gl.md +65 -62
  35. package/docs/globalmenu.md +63 -146
  36. package/docs/macos.md +531 -3234
  37. package/docs/notifications.md +33 -47
  38. package/docs/packaging.md +166 -309
  39. package/docs/permissions.md +9 -14
  40. package/docs/react-features.md +81 -106
  41. package/docs/remote.md +32 -76
  42. package/docs/scale.md +72 -186
  43. package/docs/styling.md +329 -547
  44. package/docs/system.md +200 -280
  45. package/docs/testing.md +63 -83
  46. package/docs/typescript.md +6 -48
  47. package/docs/uri-schemes.md +67 -107
  48. package/docs/wayland-backend.md +200 -459
  49. package/docs/windows-integrations.md +75 -131
  50. package/docs/windows.md +343 -1367
  51. package/package.json +3 -2
  52. package/src/Reconciler.js +2 -2
  53. package/src/a11y.js +1 -1
  54. package/src/backend/context2d.js +3 -3
  55. package/src/bootstrap.js +1 -1
  56. package/src/cocoa/app.js +6 -6
  57. package/src/cocoa/bezels.js +1 -1
  58. package/src/cocoa/fonts.js +1 -1
  59. package/src/cocoa/globalmenu.js +1 -1
  60. package/src/cocoa/keymap.js +1 -1
  61. package/src/cocoa/panewindow.js +1 -1
  62. package/src/cocoa/player.js +1 -1
  63. package/src/cocoa/presenter.js +2 -2
  64. package/src/cocoa/promotion.js +1 -1
  65. package/src/cocoa/relaunch.js +1 -1
  66. package/src/cocoa/threaded.js +1 -1
  67. package/src/cocoa/video.js +1 -1
  68. package/src/cocoa/window.js +4 -4
  69. package/src/components/Menu.js +7 -2
  70. package/src/components/Switch.js +1 -1
  71. package/src/components/native.js +1 -1
  72. package/src/dnd.js +2 -2
  73. package/src/glnodes.js +1 -1
  74. package/src/grid.js +1 -1
  75. package/src/imagedecode-worker.js +32 -0
  76. package/src/imagedecode.js +140 -2
  77. package/src/layouts.js +1 -1
  78. package/src/nodes/animation.js +1 -1
  79. package/src/nodes/invalidate.js +1 -1
  80. package/src/nodes/layouthost.js +1 -1
  81. package/src/nodes/video.js +19 -8
  82. package/src/nodes/window/droptarget.js +1 -1
  83. package/src/ntk.d.ts +12 -0
  84. package/src/pacing.js +1 -1
  85. package/src/refresh/index.js +2 -1
  86. package/src/types/video.d.ts +5 -2
  87. package/src/videobackend.js +28 -0
  88. package/src/videoframes.js +29 -16
  89. package/src/wayland/app.js +27 -0
  90. package/src/wayland/video.js +376 -0
  91. package/src/win32/a11y.js +1 -1
  92. package/src/win32/app.js +2 -2
  93. package/src/win32/fonts.js +1 -1
  94. package/src/win32/glarea.js +4 -4
  95. package/src/win32/overlay.js +1 -1
  96. package/src/win32/panewindow.js +3 -3
  97. package/src/win32/shell.js +1 -1
  98. package/src/win32/window.js +2 -2
  99. package/src/windowid.js +1 -1
  100. package/src/xvideo.js +189 -0
  101. package/docs/architecture/accessibility.md +0 -470
  102. package/docs/architecture/animation.md +0 -707
  103. package/docs/architecture/container-queries.md +0 -525
  104. package/docs/architecture/custom-layout.md +0 -334
  105. package/docs/architecture/drag-and-drop.md +0 -941
  106. package/docs/architecture/element-layer-contents.md +0 -622
  107. package/docs/architecture/frame-pacing.md +0 -255
  108. package/docs/architecture/grid-layout.md +0 -358
  109. package/docs/architecture/protocol-efficiency.md +0 -357
  110. package/docs/architecture/video.md +0 -1149
  111. package/docs/architecture/windowed-regions.md +0 -854
  112. package/docs/glx.md +0 -137
  113. package/docs/wayland.md +0 -545
  114. package/docs/windows-embedding.md +0 -271
  115. package/docs/windows-gl.md +0 -221
@@ -1,622 +0,0 @@
1
- # Element-owned layer contents: a `Surface` that is its node's layer
2
-
3
- _Design record, 2026-09-07. Written against master at 7e95217 (react-x11
4
- 2.8.3 plus the frame pacer, ntk ^8.7.0, `@windowkit/appkit` ^0.6.0), for
5
- issue #499 — the fourth of the upstream seams in
6
- sidorares/react-x11-components#69 §6 and the last of them still unbuilt.
7
- **Nothing is implemented with this document**; §9 is the shape the
8
- implementing PR takes, and §10 is the spike its claims were measured on.
9
- The mechanism it extends is
10
- [layer promotion](../macos.md#layer-promotion-the-animated-few-on-their-own-layers-above-the-surface-presenter);
11
- the cost it removes is the one [frame pacing](frame-pacing.md) measured and
12
- then paced around._
13
-
14
- ---
15
-
16
- ## 0. TL;DR
17
-
18
- - An element with a **retained surface** — a terminal, a chart on a socket,
19
- a media frame — draws its pixels once into a `Surface` of its own and
20
- composites that bitmap over its box on every frame
21
- ([extending.md](../extending.md#an-element-that-covers-its-box)). On the
22
- Cocoa backend that composite, and the window swapchain's catch-up copy
23
- behind it, are **1.3ms of a 3.8ms frame** on an M1 Pro at 1800×1280 — to
24
- move pixels that already exist into a bitmap the window then hands to
25
- Core Animation, which had them all along.
26
- - If the element's surface **were** its layer, both disappear: the window's
27
- bitmap is never touched, so the window neither flips nor catches up, and
28
- the element's own flip is `layer.contents = iosurface` — **0.03ms,
29
- constant, whatever the size**. `<glarea>` already works this way
30
- (src/cocoa/glarea.js), and layer promotion (#483, src/cocoa/promotion.js)
31
- already gives a node a layer above the window's bitmap for animation.
32
- This is the same move for a node whose pixels are its own.
33
- - The render server scans out of **our** memory: a fill into the IOSurface
34
- a layer is showing is on the screen with no present call at all (§2,
35
- measured). So an element-owned surface needs the pair the window has —
36
- a back buffer to draw into, a flip, and a damage-sized catch-up copy so
37
- the new back buffer is not one frame stale.
38
- - The API is two pieces. A `Surface` asked for with `presentable: true` —
39
- the same `react-x11/ntk` `Surface` a component already allocates, asking
40
- for a swapchain instead of a bitmap. And
41
- `presentedSurface()` on the node — an accessor core asks every frame,
42
- like `opaqueRect()`, answering the surface and the rect it covers.
43
- **Declining is always safe**, on X11 and whenever the scene says no: the
44
- element keeps compositing through `ctx.drawImage(surface, …)`, which is
45
- the code it already has.
46
- - The policy is promotion's, unchanged: the layer is above all the 2D
47
- content, so a node is taken only while nothing painted after it reaches
48
- into its bounds and every clipping ancestor holds all of it — re-decided
49
- every frame, handed back the frame that finds it overlapped.
50
-
51
- ## 1. What a retained-surface element pays now
52
-
53
- The element class is the one
54
- [extending.md](../extending.md#scrolling-the-pixels-not-just-the-offset)
55
- already documents: it keeps its drawing in a `Surface`, draws into it when
56
- its data changes — a terminal writes the rows that arrived, a chart the
57
- sample that landed, and a pan is one `copyWithin` of the band that survives
58
- — and its `paint` is one line:
59
-
60
- ```js
61
- paint(ctx) {
62
- const box = this.contentBox();
63
- ctx.drawImage(this.surface, box.x, box.y);
64
- }
65
- ```
66
-
67
- On the Cocoa backend that line and its consequences are the frame:
68
-
69
- 1. **The composite.** `drawImage` is one `CGContextDrawImage` from the
70
- element's bitmap into the window's backing store, clipped to the pass.
71
- Full-area, 0.96ms at 1800×1280 on an M1 Pro (§7); the components PRD
72
- measured 1.7ms in a live terminal at 2000×1400.
73
- 2. **The window's flip.** `CocoaWindow.present` hands the backing store's
74
- IOSurface to the window's root layer — cheap, 0.03ms, but it happens
75
- because the element dirtied the bitmap.
76
- 3. **The window's catch-up copy.** After the flip the other buffer of the
77
- pair is one frame stale, so the flip memcpys the frame's damage across
78
- (src/cocoa/window.js). The element's rect is that damage: 0.31ms
79
- full-area here, 0.95ms in the PRD's terminal.
80
-
81
- So the pixels are copied twice and blended once, from a buffer the compositor
82
- could have read directly. The waste is not an accident of the surface
83
- presenter — it is what "one bitmap per window" means, and every retained
84
- element pays it in proportion to its area.
85
-
86
- `opaqueRect()` (#497) already removed the other half of this frame — the
87
- window's and the ancestors' background fills under an element that covers
88
- its box. What is left under it is the copying.
89
-
90
- ## 2. What the platform does, spiked
91
-
92
- Five questions, one script (§10), run on an M1 Pro at 2×, macOS 15.2,
93
- `@windowkit/appkit` 0.6.0. Its output, verbatim:
94
-
95
- ```
96
- placement, orientation, scale, colour, clip:
97
- {
98
- bitmap: '128,128,128',
99
- marker: '0,255,0',
100
- top_band: '255,0,0',
101
- grey_in_element: '128,128,128',
102
- rounded_corner: '128,128,128'
103
- }
104
- after drawing into the SHOWN buffer, with no present:
105
- { element: '0,0,255' }
106
- ```
107
-
108
- - **A sublayer composites over the window's bitmap.** The root layer holds
109
- the surface presenter's frame (mid grey); a sublayer showing an IOSurface
110
- covers exactly its own frame and nothing else — `bitmap` outside it is
111
- still the presenter's grey. This is promotion's arrangement, with
112
- contents instead of properties.
113
- - **No flip, no mirror.** `createSurfaceIOSurface` lays a CG bitmap over the
114
- IOSurface with the same top-left CTM `createSurface` uses, so the
115
- surface's device (0,0) lands at the layer's top-left (`marker`) and row
116
- order is preserved (`top_band`). `<glarea>` mirrors its layer
117
- (`transform: { scaleY: -1 }`) because GL renders bottom-up; **a 2d surface
118
- must not**.
119
- - **Device pixels, and the scale carried by the layer.** The layer's frame
120
- is points, its `contentsScale` is the window's scale, and the IOSurface is
121
- device pixels — the units a `Surface` is already sized in
122
- ([scale.md](../scale.md)). A 200×120pt layer showing a 400×240 surface at
123
- 2× is pixel-exact.
124
- - **The same colour space as the bitmap.** Mid grey drawn into the element's
125
- IOSurface composites to the same bytes as mid grey drawn into the window's
126
- backing store (`grey_in_element` = `bitmap`). Both go through
127
- `kCGColorSpaceSRGB`; the Generic-RGB drift that made layer _property_
128
- colours composite paler (fixed in appkit 0.5.1) never applied to contents.
129
- - **A rounded clip clips contents.** `cornerRadius` + `masksToBounds` on the
130
- layer cuts the corner out of the IOSurface too (`rounded_corner` is the
131
- bitmap beneath), so an element with a `borderRadius` keeps it.
132
-
133
- And the finding the whole design turns on:
134
-
135
- > **The render server scans out of our memory.** Filling the IOSurface the
136
- > layer is already showing — no `setLayerContentsIOSurface`, no transaction,
137
- > nothing that tells Core Animation anything happened — put the new colour
138
- > on the screen.
139
-
140
- That is the tear, stated as a measurement: an element drawing into a
141
- presented buffer is drawing into the scanout, and a half-drawn frame is a
142
- half-drawn screen. The window answered this with two IOSurfaces and a flip;
143
- an element-owned surface needs the same, and it is the reason this is a
144
- `Surface` feature rather than "hand core your bitmap".
145
-
146
- ## 3. A presentable `Surface`
147
-
148
- ```js
149
- import { Surface } from 'react-x11/ntk';
150
-
151
- this.surface = new Surface(this.app, { width, height, presentable: true });
152
- ```
153
-
154
- `react-x11/ntk`'s `Surface` already asks the app it is handed for the
155
- implementation (src/ntk.js) — ntk's pixmap on an X connection, a CG bitmap
156
- on a Cocoa app. `presentable` asks that app for a **swapchain** instead of a
157
- single bitmap. Everything else about the object is unchanged: `getContext('2d')`,
158
- `render`, `clear`, `copyWithin`, `destroy`, and `ctx.drawImage(surface, …)`
159
- as a source.
160
-
161
- What the Cocoa implementation (src/cocoa/surface.js) grows:
162
-
163
- - **A pair, not a bitmap.** Two `createSurfaceIOSurface` buffers; the
164
- context draws into the back one. Falls back to the single plain bitmap
165
- when IOSurface creation fails — the window's own fallback, and the element
166
- is none the wiser because it never presents.
167
- - **A generation, so one context spans two bitmaps.** `CocoaContext2D` takes
168
- its handle and a generation as closures (`() => handle`, `() => gen`)
169
- precisely so a flip can swap the bitmap under a cached context: the sticky
170
- graphics state — fill colour, line width, the CTM — is CoreGraphics state
171
- living _in the bitmap_, and the generation is what makes the context
172
- re-sync it. `CocoaWindow` does exactly this on every flip (`_surfaceGen`);
173
- a presentable surface is the same arrangement one level in.
174
- - **`present(damage)`, called by core, not by the element.** Unlock, hand
175
- the back buffer's `iosurfaceId` to the layer, swap, lock the new back, and
176
- memcpy `damage` across (`copySurfaceRegion`) so the new back buffer is not
177
- one frame stale. `damage` is the rects the element claimed since the last
178
- present, translated into surface coordinates; no damage means a full copy.
179
- - **The catch-up happens at the flip, synchronously.** Not lazily at the next
180
- draw, which would be cheaper for an element that repaints everything: the
181
- element may draw into the back buffer at any moment after the frame
182
- returns — that is the whole point of a retained surface — and a copy
183
- arriving after that would overwrite what it drew. The window can afford no
184
- such distinction; here it is a correctness rule, not a preference.
185
- - **_Since #602 (2026-09-17): not into the buffer the flip retired._** The
186
- WindowServer lets go of a buffer a median of about 9ms after a flip takes
187
- it off glass, and a write before then lands in what it may still be
188
- compositing. The
189
- window now takes its next buffer at its next frame's first draw, from
190
- those `surfaceIsInUse` answers free, and grows its chain to four
191
- ([macos.md](../macos.md#measured-the-swapchain-and-the-windowservers-hold)).
192
- A presentable surface still has to catch up at the flip, for the reason
193
- above — so it has to choose there the same way: a free buffer, never the
194
- one just retired.
195
- - **Lock brackets.** `surfaceLock` before a frame's first draw,
196
- `surfaceUnlock` before the flip: 1µs for the pair, measured — free, and
197
- the contract IOSurface asks for.
198
- - **`destroy()` frees both**, on the call, the way `releaseSurface` freed the
199
- window's pair rather than waiting for V8 (the `rss +80MB` a resize drag
200
- used to hold). A resize retires a pair; two window-sized buffers at 2× are
201
- ~18MB, so this is not optional bookkeeping.
202
-
203
- **On X11 the option is accepted and ignored.** ntk's `Surface` answers as it
204
- does today, `present()` returns false, and the element's `drawImage`
205
- composite is the path — an XRender `Composite` the server runs, with no
206
- per-element scanout buffer to hand a compositor and no reason to invent one
207
- (§8). The element's code is identical on both backends, which is the same
208
- bargain `scrollContents` and `opaqueRect()` make.
209
-
210
- ## 4. The node's half: who gets a layer, and when
211
-
212
- ### 4.1 The accessor
213
-
214
- ```js
215
- /** The surface this element's content lives in, and the rect it covers in
216
- * window coordinates — or null. */
217
- presentedSurface() {
218
- return { surface: this.surface, rect: this.contentBox() };
219
- }
220
- ```
221
-
222
- Asked **every frame**, like `opaqueRect()` and like promotion's whole
223
- policy; a `null` answer, or a frame that declines it, changes nothing about
224
- what the element does. The rule promotion states applies here word for word:
225
- _declining is always safe_ — the element's `paint` composites the surface,
226
- and that is the picture.
227
-
228
- The alternative spelling, an imperative `node.presentSurface(surface)`, is
229
- worse for the reason promotion's policy is re-decided every frame: the
230
- answer depends on the scene, and a scene changes without the element
231
- hearing about it.
232
-
233
- ### 4.2 The arrangement
234
-
235
- Three layers, one per thing that is genuinely separate:
236
-
237
- ```
238
- window root layer the surface presenter's bitmap — a hole where the node is
239
- └── node layer the property box: background, border, radius, masksToBounds, z
240
- ├── contents layer frame = the presented rect, contents = the front IOSurface
241
- └── raster layer the children, if it has any (promotion's `_syncContent`)
242
- ```
243
-
244
- The node layer is promotion's `Visual` verbatim (`propBoxProps`): a
245
- promoted node's background, border, corner radius and clip are already
246
- layer properties, which is what makes them animatable. The contents layer is
247
- new and is a sublayer rather than the node layer itself, because the
248
- presented rect is usually the **content box** — inset from the node's box by
249
- padding and border — and a layer's contents fill its bounds. It is also how
250
- `<glarea>` attaches, one level up.
251
-
252
- The children keep promotion's answer: one raster sublayer, painted by the
253
- node's own `_paintChildren` walk, repainted for the claims that reached into
254
- it. A presenting element with drawn children is unusual — a terminal has
255
- none — so the implementing PR may leave the raster out and decline a node
256
- that has drawn children; the machinery is there when a consumer appears.
257
-
258
- ### 4.3 What promotion has to grow
259
-
260
- `CocoaPromotion` is the owner: same candidate set, same z-order test, same
261
- per-frame re-decision, same demote-and-claim-the-hole. Four honest changes:
262
-
263
- - **`promotableNode` gates on `plainBox(node)`** — an identity test on
264
- `node.paint`, "not an element with a paint of its own, whose content
265
- exists nowhere but in that override". A presenting element _is_
266
- that, and is safe for the opposite reason: its content exists nowhere but
267
- in its surface, and the surface is on the layer. So the test grows a
268
- second acceptance path — a node whose ink is defined to be its presented
269
- surface plus the property box.
270
- - **`paintsSomething` / `_reaches`** must count a presenting node's ink as
271
- its presented rect, so a _neighbouring_ candidate is told the truth about
272
- what would be painted over it.
273
- - **The candidate set** gains nodes that answer `presentedSurface()`, not
274
- only nodes that animate. They are not the same set and both are small: a
275
- scene has a handful of each.
276
- - **`IDLE_GRACE_MS` does not apply.** An animation ends; a presented surface
277
- does not. A presenting node keeps its layer until it stops answering, is
278
- overlapped, or is destroyed.
279
-
280
- ### 4.4 The frame
281
-
282
- The seams already exist and are already in the right order (src/nodes/window/flush.js,
283
- `WindowNode.flush`):
284
-
285
- 1. The element draws into its back buffer when its data changes, and claims
286
- the rect it drew — `invalidate(false, rect, 'props')`, exactly as it does
287
- today so the composite happens.
288
- 2. `noteInvalidate` sees the claim on its way to the bitmap. For a presenting
289
- node it answers `true` — the bitmap owes nothing, since the node is a hole
290
- — and records the rect, which is what the catch-up copy will cover. A
291
- frame is still scheduled (`needsPaint`), and the pacer still prices it.
292
- 3. `prepareFrame(root, layoutRan)` runs after layout and before the damage is
293
- taken. It syncs the node layer's properties and, for each presenting node
294
- with claims, calls `surface.present(rects)` — the flip.
295
- 4. The paint pass paints nothing there, because the node is `_promoted`.
296
- With no other claim in the frame the window's backing store is untouched,
297
- `CocoaWindow.present` returns early on `!this._dirty`, and **the window
298
- neither flips nor copies**. That is the second half of the saving, and it
299
- falls out of machinery that already exists rather than being added.
300
-
301
- ### 4.5 What the element must not do
302
-
303
- **Draw inside `paint`.** A presenting node's `paint` is not called — that is
304
- what the hole means — so an element that draws its scene there and
305
- composites in the same pass would simply stop updating. The class this seam
306
- is for already separates the two (draw on data, composite in `paint`), and
307
- the accessor's documentation has to say so in one line. An element that
308
- wants core to drive its drawing has a different answer today and a better
309
- one later (§8).
310
-
311
- ## 5. Scale, orientation, colour, corners
312
-
313
- All four are measured in §2 and none needs new code:
314
-
315
- | question | answer |
316
- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
317
- | flip / mirror | none — the CG bitmap over the IOSurface has `createSurface`'s top-left CTM. `<glarea>`'s `scaleY: -1` is GL's bottom-up rendering, not the IOSurface's |
318
- | scale | layer frame in points, `contentsScale` = the window's scale, surface in device pixels — the units `Surface` already uses |
319
- | colour | both buffers are `kCGColorSpaceSRGB`; an element's grey and the bitmap's grey composite to the same bytes |
320
- | rounded corners | `cornerRadius` + `masksToBounds` on the node layer clips the contents |
321
- | a partly drawn surface | a fresh IOSurface is transparent, not garbage — `Surface` clears on construction anyway (src/cocoa/surface.js) |
322
-
323
- ## 6. What it composes with
324
-
325
- - **The frame pacer** ([frame-pacing.md](frame-pacing.md)). Unchanged and
326
- still the right tool: the claim is held in `_scheduleFrame`, above the
327
- clock, and the frame is priced by what it costs. This design makes that
328
- cost smaller, so a streaming element gets _more_ frames at the same
329
- budget. The two are complementary — pacing decides how often, this decides
330
- how much.
331
- - **`opaqueRect()`.** Moot while presented (`_coverFor` skips `_promoted`
332
- nodes) and correct again the moment the node is handed back. An element
333
- should keep answering it: it is what pays when the scene declines.
334
- - **The scroll blit.** A presenting element's pan is `surface.copyWithin` on
335
- its own surface — the shift it already had, one layer in — and the window's
336
- `scrollRegion` never sees it. `_scrollBlitSafe` already skips `_promoted`
337
- children, so a pan _around_ the element still blits.
338
- - **Promotion.** Same z range, one sort: a promoted animation layer and a
339
- presenting element layer are both sublayers of the root, ordered by paint
340
- order, and each is refused if the other reaches into it.
341
- - **`<glarea>`.** Above both at `zPosition: 1e7`, unchanged.
342
- - **The layers presenter** (`REACT_X11_COCOA_PRESENTER=layers`). Every node
343
- there already has a visual, and a presenting node's raster visual becomes
344
- contents from the element's front IOSurface — strictly less work than the
345
- raster it replaces. Out of scope for v1, which declines and keeps
346
- `drawImage`.
347
- - **Transparent windows and popups.** A layer over a clear window composites
348
- as any layer does; the window's shadow recompute is driven by the bitmap
349
- and unaffected. A presenting element in a `<popup dragPreview>` is
350
- declined for the ordinary reason — nothing is asking for one.
351
-
352
- ## 7. Measured
353
-
354
- M1 Pro, macOS 15.2, 2×, `@windowkit/appkit` 0.6.0. A 1800×1280 element in a
355
- 1800×1400 window — a pane under a strip of chrome, the `stream` scenario's
356
- shape. Per frame, beyond the element's own drawing (2.45ms either way; an
357
- IOSurface-backed CG bitmap and a plain one draw at the same speed, which is
358
- its own finding):
359
-
360
- | the frame paints | today | presented | saving |
361
- | ---------------------------------------- | ---------: | ---------: | ---------: |
362
- | the whole element | | | |
363
- | — composite (`ctxDrawSurface`) | 0.96ms | — | |
364
- | — the window's flip | 0.03ms | — | |
365
- | — the window's catch-up over the element | 0.31ms | — | |
366
- | — the element's flip | — | 0.03ms | |
367
- | — the element's catch-up (whole surface) | — | 0.34ms | |
368
- | **total** | **1.30ms** | **0.37ms** | **0.93ms** |
369
- | a quarter of the element | | | |
370
- | — composite, clipped to the claim | 0.20ms | — | |
371
- | — the window's flip + catch-up | 0.11ms | — | |
372
- | — the element's flip + catch-up | — | 0.10ms | |
373
- | **total** | **0.31ms** | **0.10ms** | **0.21ms** |
374
-
375
- Three things to read out of it. The saving is **about 70% of the copying,
376
- at any damage fraction** — the composite is a blend (read source, read
377
- destination, blend, write) where the catch-up is a memcpy, so the same area
378
- costs roughly three times less; and the flip is constant. It is **largest
379
- exactly where the problem was found**: a full-area frame, which is what a
380
- terminal under a flood produces every time. And it is **not the whole
381
- frame**: the element's own drawing is untouched, which is what the pacer is
382
- for.
383
-
384
- Two caveats, stated rather than buried. These are microbenchmarks — one
385
- operation repeated with warm caches — and the components PRD measured 1.7ms
386
- and 0.95ms for the same two steps inside a live terminal at 2000×1400, where
387
- the source has been evicted by everything else the frame did. **The ratio is
388
- what to trust, not the absolute**; by the PRD's numbers the same arithmetic
389
- removes 2.65ms of a 6ms frame. And the numbers move ±30% between runs with
390
- WindowServer load, as every Cocoa measurement in this repo does.
391
-
392
- ## 8. What is not here, and why
393
-
394
- - **A core-owned raster for any element** — "a node whose content is painted
395
- code gets a raster visual", the Tier L rule in
396
- [macos.md](../macos.md#custom-drawing-on-a-layer-tree). More general (it
397
- needs nothing from the element but its existing `paint`) and a different
398
- design: core owns the buffer, core decides when to redraw, and the
399
- element's incremental drawing — the terminal writing the rows that arrived,
400
- the `copyWithin` pan — has nowhere to live. This seam is for the element
401
- that already owns its pixels; the raster is for the one that does not. They
402
- should both exist and neither is the other's v2.
403
- - **One buffer instead of two.** `IOSurfaceIsInUse` would let an element
404
- skip the pair while the compositor is not reading — the bridge has had the
405
- verb since 0.10 (`surfaceIsInUse`), but it is a race in any case (in-use can
406
- become true between the question and the first fill), and the buffer costs
407
- 9MB.
408
- - **`preserveContents: false`.** An element that redraws every pixel between
409
- presents needs no catch-up copy at all (0.34ms at full size). Real, and
410
- deliberately not in v1: the claim-derived copy is already the right answer
411
- for everyone else, and a flag whose contract is "trust me" should wait
412
- until a measurement asks for it.
413
- - **An X11 equivalent.** A child window with a pixmap of its own would give
414
- the server the same trick, at one X window per element, a second damage
415
- domain, and input plumbing around a window the element does not own — for
416
- a composite the server already runs asynchronously. The X11 answer is
417
- `drawImage`.
418
- - **Video, YUV and GL interop.** An IOSurface can carry a decoder's output
419
- or a GL render target, and a `<video>` element would want exactly this
420
- seam. Out of scope; nothing here forecloses it.
421
-
422
- ## 9. Decisions the implementing PR has to make, and the suggested answer
423
-
424
- | question | suggested answer |
425
- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
426
- | the `Surface` option | `presentable: true`. Not `swapchain: 2` (the count is an implementation detail) and not a separate class (`Surface` is already the backend-neutral name a component reaches for) |
427
- | the node accessor | `presentedSurface()` → `{ surface, rect }` or `null`, asked every frame beside `opaqueRect()` |
428
- | who calls `present()` | core, in `prepareFrame`, so the flip is a frame and the pacer prices it. `Surface.present(damage)` stays public for a consumer outside the renderer |
429
- | what the catch-up copies | the rects the element claimed since the last present, translated into surface coordinates by `rect`'s origin; an unbounded claim is a full copy |
430
- | when it copies | at the flip, synchronously — the element may draw the moment the frame returns |
431
- | a node with drawn children | v1 declines; promotion's raster sublayer is the answer when a consumer needs it |
432
- | an element that draws in `paint` | documented as a contract, not detected: `paint` is not called, so the element would go stale. The accessor's doc line, and a `docs/extending.md` section beside "An element that covers its box" |
433
- | the scene test | promotion's `_clear`, unchanged, plus a presenting node's rect counted as ink in `paintsSomething`/`_reaches` |
434
- | the grace period | none — `IDLE_GRACE_MS` is for an animation that ended; a surface does not end |
435
- | resize | the element reallocates its surface as it does today; the old pair is freed on the call, not by the finalizer |
436
- | two nodes, one surface | refused (the second answers `null` in development with a warning); one buffer cannot be two layers' contents without tearing between them |
437
- | X11 | the option is accepted and ignored, `present()` returns false, `presentedSurface()` is never asked because no window answers `prepareFrame` |
438
- | tracing | a `trace` line for taken/declined with the reason, the way promotion's decisions are traceable — a refusal is invisible in the picture, which is the point and also the debugging problem |
439
- | types | `presentable` on the `Surface` options and `presentedSurface(): { surface, rect } \| null` in `src/node.d.ts`, one line in `test/types/extend.tsx` |
440
- | docs | a section in [extending.md](../extending.md#an-element-that-covers-its-box) after "An element that covers its box"; the ladder in [macos.md](../macos.md#custom-drawing-on-a-layer-tree) gains the presented case; this file becomes the design record it points at |
441
-
442
- **Tests**, in the shape of `test/cocoa-promotion.test.js` and
443
- `test/cocoa-surface.test.js` — the fake bridge for what reaches the natives,
444
- the real one for pixels:
445
-
446
- - a presentable surface allocates a pair and answers one context across a
447
- flip, with the generation bumped so the sticky state re-syncs;
448
- - `present(damage)` unlocks, sets the layer's contents to the back buffer's
449
- id, swaps, locks, and copies exactly `damage` — and copies everything when
450
- told nothing;
451
- - a presenting node gets a layer, leaves a hole, and the bitmap under it is
452
- claimed once, in the frame that takes it;
453
- - a frame in which only the presenting element claimed touches the window's
454
- backing store not at all (the `_dirty` assertion — this is the saving,
455
- stated as a test);
456
- - every promotion refusal applies: a later sibling over it, a clipping
457
- ancestor that does not hold it, a focus ring, a scrollbar strip, hidden —
458
- and each hands the node back in the frame that finds it, with the element
459
- compositing again;
460
- - pixels on a real display: the presented surface lands at its rect, at
461
- scale, unmirrored, and a repaint of the bitmap around it leaves it alone;
462
- - X11: `presentable` is accepted, `present()` is false, and the element
463
- paints identically with and without it.
464
-
465
- **Bench**: a `--presented` column on the `stream` scenario
466
- (`scripts/bench/presenters.js`), whose element already covers its box and
467
- already answers `opaqueRect()`. The gate rule is the one that matters:
468
- frames per second at a fixed budget, and the window's paint count at zero.
469
-
470
- **Sequencing.** Three PRs, each useful alone: the presentable `Surface` and
471
- its tests; `presentedSurface()` and the promotion policy; the bench column,
472
- the numbers and the docs. Nothing needs a bridge release — §2 is the proof
473
- that `@windowkit/appkit` 0.6.0 already has every verb.
474
-
475
- ## 10. The spike
476
-
477
- Reproduced so the numbers can be re-run: save it at the repo root of a
478
- react-x11 checkout and run it with `node` on a Mac with a display. It uses
479
- the bridge alone — no renderer, no React — and honours
480
- `REACT_X11_CALAYERS_PATH` so a bridge checkout can be tested too.
481
-
482
- ```js
483
- import { createRequire } from 'node:module';
484
- const require = createRequire(import.meta.url);
485
- const bridge = require(
486
- process.env.REACT_X11_CALAYERS_PATH ?? '@windowkit/appkit',
487
- );
488
- const N = bridge.native ?? bridge;
489
- const { PNG } = require('pngjs');
490
- const fs = require('node:fs');
491
-
492
- const s = 2; // device pixels per point
493
- const PTS = { width: 400, height: 300 };
494
- N.initApp();
495
- const win = N.createWindow2({ ...PTS, kind: 'borderless', x: 100, y: 100 });
496
- const root = N.windowRootLayer(win);
497
- N.showWindow(win, true);
498
-
499
- // the surface presenter's bitmap: mid grey over the whole window
500
- const bitmap = N.createSurface(PTS.width * s, PTS.height * s, s);
501
- N.ctxSetFillColor(bitmap, 0.5, 0.5, 0.5, 1);
502
- N.ctxFillRect(bitmap, 0, 0, PTS.width * s, PTS.height * s);
503
- N.surfaceToLayer(bitmap, root);
504
-
505
- // an element's own layer, its contents an IOSurface a CG context drew into
506
- const BOX = { x: 50, y: 40, w: 200, h: 120, r: 24 };
507
- const layer = N.createLayer();
508
- N.addSublayer(root, layer);
509
- N.setLayerProps(layer, {
510
- frame: [BOX.x, BOX.y, BOX.w, BOX.h],
511
- contentsScale: s,
512
- cornerRadius: BOX.r,
513
- masksToBounds: true,
514
- zPosition: 1,
515
- });
516
- const ew = BOX.w * s,
517
- eh = BOX.h * s;
518
- const buf = N.createSurfaceIOSurface(ew, eh, s);
519
- N.surfaceLock(buf.handle);
520
- N.ctxSetFillColor(buf.handle, 1, 0, 0, 1); // top half red
521
- N.ctxFillRect(buf.handle, 0, 0, ew, eh / 2);
522
- N.ctxSetFillColor(buf.handle, 0.5, 0.5, 0.5, 1); // bottom half the bitmap's grey
523
- N.ctxFillRect(buf.handle, 0, eh / 2, ew, eh / 2);
524
- N.ctxSetFillColor(buf.handle, 0, 1, 0, 1); // a 60px device marker at 0,0
525
- N.ctxFillRect(buf.handle, 0, 0, 60, 60);
526
- N.surfaceUnlock(buf.handle);
527
- N.setLayerContentsIOSurface(layer, buf.iosurfaceId);
528
-
529
- const shot = (name) => {
530
- N.snapshotWindow(win, name);
531
- const png = PNG.sync.read(fs.readFileSync(name));
532
- const d = png.width / PTS.width;
533
- return (x, y) => {
534
- const i = (png.width * Math.round(y * d) + Math.round(x * d)) * 4;
535
- return [png.data[i], png.data[i + 1], png.data[i + 2]].join(',');
536
- };
537
- };
538
-
539
- const time = (label, fn, n = 60) => {
540
- for (let i = 0; i < 10; i++) fn();
541
- const t0 = performance.now();
542
- for (let i = 0; i < n; i++) fn();
543
- console.log(
544
- ` ${label.padEnd(46)} ${((performance.now() - t0) / n).toFixed(3)} ms`,
545
- );
546
- };
547
-
548
- let tick = 0;
549
- const pump = setInterval(() => {
550
- N.pump2();
551
- if (++tick === 10) {
552
- const at = shot('/tmp/spike-a.png');
553
- console.log('placement, orientation, scale, colour, clip:');
554
- console.log({
555
- bitmap: at(320, 250), // outside the element
556
- marker: at(BOX.x + 26, BOX.y + 26), // inside the surface's own (0,0) corner
557
- top_band: at(BOX.x + 100, BOX.y + 20),
558
- grey_in_element: at(BOX.x + 100, BOX.y + 100),
559
- rounded_corner: at(BOX.x + 2, BOX.y + 2),
560
- });
561
- // draw into the buffer the layer is already showing; present nothing
562
- N.ctxSetFillColor(buf.handle, 0, 0, 1, 1);
563
- N.ctxFillRect(buf.handle, 0, 0, ew, eh);
564
- }
565
- if (tick === 20) {
566
- const at = shot('/tmp/spike-b.png');
567
- console.log('after drawing into the SHOWN buffer, with no present:');
568
- console.log({ element: at(BOX.x + 100, BOX.y + 60) });
569
-
570
- const W = 900 * s,
571
- H = 700 * s,
572
- EH = 640 * s;
573
- const wa = N.createSurfaceIOSurface(W, H, s);
574
- const wb = N.createSurfaceIOSurface(W, H, s);
575
- const el = N.createSurface(W, EH, s);
576
- const ea = N.createSurfaceIOSurface(W, EH, s);
577
- const eb = N.createSurfaceIOSurface(W, EH, s);
578
- console.log(`\ncost, ${W}x${EH} element in a ${W}x${H} window:`);
579
- time('composite ctxDrawSurface(window, element)', () =>
580
- N.ctxDrawSurface(wa.handle, el, 0, 0, W, EH, 0, 0, W, EH),
581
- );
582
- time('window catch-up copySurfaceRegion(element)', () =>
583
- N.copySurfaceRegion(wa.handle, wb.handle, [0, 0, W, EH]),
584
- );
585
- time('flip setLayerContentsIOSurface', () =>
586
- N.setLayerContentsIOSurface(layer, ea.iosurfaceId),
587
- );
588
- time('element catch-up copySurfaceRegion(all)', () =>
589
- N.copySurfaceRegion(ea.handle, eb.handle, null),
590
- );
591
- time('element catch-up copySurfaceRegion(25%)', () =>
592
- N.copySurfaceRegion(ea.handle, eb.handle, [0, 0, W, EH / 4]),
593
- );
594
- clearInterval(pump);
595
- N.destroyWindow2(win);
596
- process.exit(0);
597
- }
598
- }, 8);
599
- ```
600
-
601
- The clipped-composite row in §7 is one more `time()` call with the
602
- destination clipped to a fraction of the element before `ctxDrawSurface`:
603
- 0.96ms unclipped, 0.20ms at a quarter, 0.05ms at a twentieth — the composite
604
- is proportional to the claim, which is why §7 has two halves.
605
-
606
- ## 11. Verdict
607
-
608
- Buildable now, on the bridge as it is, at moderate size: a swapchain and a
609
- `present()` in `src/cocoa/surface.js`, an accessor on `Node`, and a second
610
- kind of candidate in `src/cocoa/promotion.js` — whose hard part, the z-order
611
- rule, is written and tested. It removes about 70% of what a retained-surface
612
- element spends outside its own drawing — 0.93ms of a 3.8ms frame here, and
613
- by the components PRD's numbers 2.65ms of a 6ms one — which is the largest
614
- item left in a streaming element's frame now that the pacer and
615
- `opaqueRect()` have taken theirs.
616
-
617
- The reason to do it after the pacer rather than instead of it is that they
618
- answer different halves: pacing stops the thread painting screens nobody
619
- sees, and this stops each of the screens it does paint from being copied
620
- twice. And the reason to keep `drawImage` working underneath is the reason
621
- promotion keeps the frame clock underneath: the scene decides, every frame,
622
- and the element's pixels must be right when the answer is no.