@mathzetta/qblur 0.0.0-stage → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +58 -0
- package/LICENSE +21 -0
- package/README.md +152 -2
- package/dist/cpu-worker.js +2 -0
- package/dist/qblur-apply.js +2 -0
- package/dist/qblur-element.js +2 -0
- package/dist/qblur.core.js +2 -0
- package/dist/qblur.external.global.min.js +2 -0
- package/dist/qblur.external.js +2 -0
- package/dist/qblur.global.min.js +2 -0
- package/dist/qblur.js +2 -0
- package/docs/filters.md +841 -0
- package/docs/library.md +601 -0
- package/package.json +118 -4
- package/src/apply-filter.js +245 -0
- package/src/apply.d.ts +64 -0
- package/src/apply.js +14 -0
- package/src/auto.js +41 -0
- package/src/cpu-helpers.js +418 -0
- package/src/cpu-runner.js +282 -0
- package/src/cpu-worker.js +58 -0
- package/src/cpu.js +586 -0
- package/src/element.d.ts +16 -0
- package/src/element.js +142 -0
- package/src/format.js +614 -0
- package/src/graphics.js +305 -0
- package/src/index.d.ts +283 -0
- package/src/index.js +13 -0
- package/src/library.js +173 -0
- package/src/png.js +157 -0
- package/src/runtime.js +248 -0
- package/src/save.js +108 -0
- package/src/strokes.js +350 -0
package/docs/filters.md
ADDED
|
@@ -0,0 +1,841 @@
|
|
|
1
|
+
# Writing image filters
|
|
2
|
+
|
|
3
|
+
A filter is a bit of maths that runs on every pixel of a picture. It reads the pixel — its
|
|
4
|
+
colour `r` `g` `b`, its brightness `gray`, where it is (`x` `y`, or `px` `py` in pixels) — and works
|
|
5
|
+
out the new colour. Type one into the editor's *Add a new filter* box and press *Add*:
|
|
6
|
+
|
|
7
|
+
| Write | You get |
|
|
8
|
+
| --- | --- |
|
|
9
|
+
| `1 - rgb` | the negative: every channel turned upside down |
|
|
10
|
+
| `gray` | black and white |
|
|
11
|
+
| `(g, r, b)` | red and green swapped |
|
|
12
|
+
| `if gray > 0.5 then 1 else 0` | a hard black-and-white threshold |
|
|
13
|
+
| `rgb * 1.3` | brighter |
|
|
14
|
+
| `pixel(px + 8 sin(py / 10 + t / 5), py)` | a wave that moves (`t` counts the frames) |
|
|
15
|
+
| `gaussian(3)` | a soft blur |
|
|
16
|
+
|
|
17
|
+
Then give it knobs: with the step's *Edit* on, *+ slider* adds a line like
|
|
18
|
+
`strength := [[0..1 step 0.01 = 0.5]]` — a slider the maths can use as `strength` (*Edit* also shows
|
|
19
|
+
the step's name and author, and lets you write notes). Steps run top to bottom, each on the result of the one
|
|
20
|
+
above, so filters chain; *Copy link* puts the whole thing in a link anyone can open, change and
|
|
21
|
+
keep (see [Sharing filters](#sharing-filters)). Everything runs in your browser — nothing is
|
|
22
|
+
uploaded. The rest of this guide is everything an expression can use; the editor's *Examples* each
|
|
23
|
+
explain one idea, and the *Library* has dozens of filters to open and read.
|
|
24
|
+
|
|
25
|
+
**Saving the result.** *Download* draws the filter at the picture's own size. *Download options*
|
|
26
|
+
choose the format — PNG, JPEG or WebP, or *auto*, which picks the smallest sensible one (a result
|
|
27
|
+
with few colours becomes a small indexed PNG; one with transparency stays lossless; a photo goes to
|
|
28
|
+
whichever of JPEG and WebP is smaller) — the size, a crop you drag on the result, and for moving
|
|
29
|
+
pictures *Save as GIF* (at the preview size; a GIF plays through once with its own delays, anything
|
|
30
|
+
else gets 3 seconds) or *Record* (video). Effects measured in pixels — blur radii, `px` — stay the
|
|
31
|
+
same number of pixels, so they look finer on a big picture.
|
|
32
|
+
|
|
33
|
+
<!-- dev -->
|
|
34
|
+
**For developers.** MathZetta turns the maths into GLSL, and `filterShader()` wraps it in a complete
|
|
35
|
+
fragment shader, so a filter runs on the GPU — fast enough for video:
|
|
36
|
+
|
|
37
|
+
```js
|
|
38
|
+
import { filterShader, filterVertexShader } from 'mathzetta/glsl';
|
|
39
|
+
|
|
40
|
+
filterShader('(1 - r, 1 - g, 1 - b)') // invert
|
|
41
|
+
filterShader('gray') // greyscale
|
|
42
|
+
filterShader({ r: 'g', g: 'r' }) // swap red and green; b and a unchanged
|
|
43
|
+
filterShader('if gray > 0.5 then 1 else 0') // threshold
|
|
44
|
+
filterShader('pixel(px + 8 sin(py / 10 + t / 5), py)') // a wave that moves with t
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
`demo/index.html` is the editor; `LIBRARY` (exported by the package) is the built-in catalogue it
|
|
48
|
+
shows. [qblur on your own pages](library.md) is how to use the package.
|
|
49
|
+
<!-- /dev -->
|
|
50
|
+
|
|
51
|
+
## What an expression can use
|
|
52
|
+
|
|
53
|
+
| Name | Meaning |
|
|
54
|
+
| --- | --- |
|
|
55
|
+
| `r` `g` `b` `a` | the pixel's channels, 0–1 |
|
|
56
|
+
| `rgb` | the same three as a vector; `rgb * 0.5` dims every channel |
|
|
57
|
+
| `gray` | the pixel's brightness, 0.299 r + 0.587 g + 0.114 b |
|
|
58
|
+
| `x` `y` | where the pixel is, 0–1 across and down the image |
|
|
59
|
+
| `px` `py` | the same in pixels (`py` runs downwards, like image rows) |
|
|
60
|
+
| `w` `h` | the image's width and height in pixels |
|
|
61
|
+
| `t` | the frame index — a uniform you set; count it up each frame of a video or GIF |
|
|
62
|
+
| `random()` | a different number 0–1 for every pixel, every call and every frame: `rgb + 0.15 (random() - 0.5)` is film grain |
|
|
63
|
+
| `pixel(px, py)` | another pixel, as a vector (r, g, b, a) |
|
|
64
|
+
| `red(px, py)` `green(…)` `blue(…)` `alpha(…)` `luma(…)` | one channel, or the brightness, of another pixel |
|
|
65
|
+
| `unchanged` | the value this expression replaces (see conditionals); as the missing `else` it takes the other branch's type, so `if c then 1` and `if c then (1, 0, 0)` both work |
|
|
66
|
+
|
|
67
|
+
And everything GLSL has: `clamp(v, 0, 1)`, `mix(a, b, 0.5)`, `smoothstep(0, 1, x)`,
|
|
68
|
+
`step(0.5, x)`, `fract(x)`, `length((x, y))`, `distance(…)`, `floor`, `abs`,
|
|
69
|
+
`min`, `max`, `sin`, `cos`, `exp`, `log`, `sqrt`, `pow`, `atan2(y, x)`, `mod`.
|
|
70
|
+
|
|
71
|
+
## Results: numbers, colours, true/false
|
|
72
|
+
|
|
73
|
+
The shader accepts whatever the expression works out to:
|
|
74
|
+
|
|
75
|
+
- a **number** is grey (or, for a channel, that channel's value);
|
|
76
|
+
- a **tuple** `(r, g, b)` or `(r, g, b, a)` is the colour;
|
|
77
|
+
- a **comparison** is 1 or 0 — `r > 0.5` is a mask you can multiply by.
|
|
78
|
+
|
|
79
|
+
Give one expression for the whole colour, or an object with any of
|
|
80
|
+
`r`, `g`, `b`, `a`, `rgb`, `rgba`. Channels you don't mention keep their
|
|
81
|
+
value. In a per-channel expression, `r g b a` are always the *original*
|
|
82
|
+
pixel, so `{ r: 'g', g: 'r' }` swaps rather than copies.
|
|
83
|
+
|
|
84
|
+
## Conditionals
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
if gray > 0.5 then 1 else 0 threshold
|
|
88
|
+
if y < 0.5 then rgb else (1 - r, 1 - g, 1 - b)
|
|
89
|
+
if g > 0.5 & g > 1.3 r & g > 1.3 b then (0.1, 0.1, 0.4) else rgb chroma key
|
|
90
|
+
if a then … else if b then … else … chains
|
|
91
|
+
IF(gray > 0.5, 1, 0) the spreadsheet form
|
|
92
|
+
gray > 0.5 ? 1 : 0 the C form
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Leave the `else` off and the pixel keeps its value where the condition is
|
|
96
|
+
false: `{ r: 'if x < 0.5 then 1 - r' }` inverts red on the left half
|
|
97
|
+
only. (`unchanged` is the name for that value; you can use it yourself:
|
|
98
|
+
`if y < 0.5 then unchanged * 0.5`.)
|
|
99
|
+
|
|
100
|
+
`&` is "and", `|` is "or", `!` or `not` is "not": `if r > 0.5 & !(g > 0.5)`.
|
|
101
|
+
Comparisons need numbers on both sides and logic needs comparisons —
|
|
102
|
+
GLSL won't take a bare number as true or false.
|
|
103
|
+
|
|
104
|
+
## Animation with `t`
|
|
105
|
+
|
|
106
|
+
`t` is whatever you put in the uniform; counting frames is the natural
|
|
107
|
+
choice for video and GIF, because then a filter means the same thing at
|
|
108
|
+
every frame rate:
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
rgb * (0.75 + 0.25 sin(t / 6)) pulse every 38 frames
|
|
112
|
+
pixel(px + 8 sin(py / 10 + t / 5), py) wave
|
|
113
|
+
if (px + t) mod 8 < 4 then rgb * 0.7 else rgb crawling stripes
|
|
114
|
+
rgb * (1 - 1.6 ((x - 0.5)^2 + (y - 0.5)^2)) vignette (no t)
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
For time in seconds instead, set `t` to `performance.now() / 1000` and
|
|
118
|
+
write `sin(2π t)` for one cycle a second.
|
|
119
|
+
|
|
120
|
+
## Neighbouring pixels
|
|
121
|
+
|
|
122
|
+
`pixel(px, py)` reads any pixel, so blurs, edges and distortions are
|
|
123
|
+
expressions too:
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
(pixel(px-1, py) + pixel(px+1, py) + pixel(px, py-1) + pixel(px, py+1)) / 4 blur
|
|
127
|
+
abs(gray - luma(px + 1, py)) * 8 + abs(gray - luma(px, py + 1)) * 8 edges
|
|
128
|
+
pixel(w - px, py) mirror
|
|
129
|
+
pixel(floor(px / 8) * 8, floor(py / 8) * 8) pixelate
|
|
130
|
+
pixel(w/2 + (px - w/2) cos(0.02 t gray) - (py - h/2) sin(0.02 t gray),
|
|
131
|
+
h/2 + (px - w/2) sin(0.02 t gray) + (py - h/2) cos(0.02 t gray)) swirl
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Coordinates outside the image clamp to the edge (the demo sets
|
|
135
|
+
`CLAMP_TO_EDGE`; set `REPEAT` on the texture for tiling).
|
|
136
|
+
|
|
137
|
+
## Moving the picture: zoom, pan, rotate, crop, tile
|
|
138
|
+
|
|
139
|
+
These read the picture from somewhere else, so with `t` inside them a
|
|
140
|
+
still picture moves:
|
|
141
|
+
|
|
142
|
+
| Function | Result |
|
|
143
|
+
| --- | --- |
|
|
144
|
+
| `zoom(s)` `zoom(s, cx, cy)` | the picture `s` times larger, about its centre or the point (cx, cy) in 0–1 |
|
|
145
|
+
| `shift(dx, dy)` | moved right and down by that many pixels, wrapping around |
|
|
146
|
+
| `rotate(a)` `rotate(a, cx, cy)` | turned clockwise by `a` radians (`rotate(10°)` works) |
|
|
147
|
+
| `crop(x0, y0, x1, y1)` | that part of the picture, in 0–1 fractions, filling the frame |
|
|
148
|
+
| `tile(n)` `tile(nx, ny)` | the picture repeated n × n times |
|
|
149
|
+
| `flipx()` `flipy()` | mirrored |
|
|
150
|
+
|
|
151
|
+
```
|
|
152
|
+
zoom(1 + 0.002 t, 0.5 + 0.0003 t, 0.45) Ken Burns: a slow zoom that drifts
|
|
153
|
+
shift(2 t, 0) scrolling
|
|
154
|
+
rotate(t / 100) spinning
|
|
155
|
+
crop(0.1 + t / 2000, 0.1, 0.6 + t / 2000, 0.6) panning across a crop
|
|
156
|
+
tile(2) four copies
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Each is one transform about the frame; to combine them, chain steps
|
|
160
|
+
(`Zoom` then `Spin`). Where a transform leaves the frame — zoomed out,
|
|
161
|
+
rotated corners — the picture is **transparent** when the transform is
|
|
162
|
+
the whole colour (`zoom(0.5)` alone: the page, or a PNG's transparency,
|
|
163
|
+
shows through; a later step sees `a` = 0 there and can fill it:
|
|
164
|
+
`mix(background, rgb, a)`). Inside a bigger expression it is a colour,
|
|
165
|
+
black there, and `alpha(zoom(0.5))` is 1 inside the picture and 0
|
|
166
|
+
outside, so `mix(rgb, zoom(0.5), alpha(zoom(0.5)))` lays the small copy
|
|
167
|
+
over the original. `shift` and `tile` wrap instead.
|
|
168
|
+
They work on another picture too: `zoom(overlay, 2)`, `shift(overlay,
|
|
169
|
+
2 t, 0)`, `rotate(mask, t / 50)`, and `width(overlay)`, `height(overlay)`
|
|
170
|
+
are that picture's own size in pixels.
|
|
171
|
+
|
|
172
|
+
### Placing a picture of another size
|
|
173
|
+
|
|
174
|
+
A named picture is laid over the frame according to its own size:
|
|
175
|
+
|
|
176
|
+
```json
|
|
177
|
+
"images": { "inset": { "place": "fit", "zoom": 0.35, "x": 0.8, "y": 0.78 } }
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
- `place` — `cover` (fills the frame keeping the aspect ratio, the
|
|
181
|
+
default), `fit` (shows all of it), `stretch` (ignores the ratio), or
|
|
182
|
+
`tile` (repeats it at its own size).
|
|
183
|
+
- `zoom`, `x`, `y`, `angle` — then scaled, centred at (x, y) in 0–1 of
|
|
184
|
+
the frame, and turned. Outside the picture it is transparent, so
|
|
185
|
+
`mix(rgb, inset, alpha(inset))` composites it — a picture in picture.
|
|
186
|
+
|
|
187
|
+
Whoever uses the filter can move it: `"with": { "inset": { "x": 0.2,
|
|
188
|
+
"zoom": 0.5 } }` in a chain step (the demo shows these as knobs beside
|
|
189
|
+
each picture), and `zoom(inset, 1 + t / 500)` in the maths animates on
|
|
190
|
+
top of that. The `source` picture is the frame's size, so it has no
|
|
191
|
+
placement.
|
|
192
|
+
|
|
193
|
+
## Text and shapes
|
|
194
|
+
|
|
195
|
+
A filter can draw text, rectangles and ovals. Each is named in
|
|
196
|
+
`graphics`, drawn by the browser into a transparent picture the size of
|
|
197
|
+
the frame, and read by the maths like any picture:
|
|
198
|
+
|
|
199
|
+
```json
|
|
200
|
+
{ "name": "Caption",
|
|
201
|
+
"graphics": { "title": { "type": "text", "text": "Hello", "y": 0.88, "font": "Georgia, serif" } },
|
|
202
|
+
"color": "mix(rgb, title, alpha(title))" }
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
- `title` is the graphic's colour at this pixel, `alpha(title)` where it
|
|
206
|
+
is — so the maths decides how it lands: `mix(rgb, title, alpha(title))`
|
|
207
|
+
lays it on; `mix(rgb, mix((1, 0.35, 0.3), (1, 0.85, 0.25), x), alpha(title))`
|
|
208
|
+
colours the letters with a gradient; `mix(rgb, gaussian(5) * 0.7, alpha(panel))`
|
|
209
|
+
frosts the picture behind a panel (a shape with its own filter);
|
|
210
|
+
`mix(rgb * 0.3, rgb, alpha(spot))` darkens all but an oval.
|
|
211
|
+
- **Animated** through the same geometry as any picture, with `t`:
|
|
212
|
+
`mix(rgb, shift(title, -3 t, 0), alpha(shift(title, -3 t, 0)))` scrolls
|
|
213
|
+
it, `zoom(title, 1 + 0.06 sin(t / 8))` pulses it,
|
|
214
|
+
`mix(rgb, title, alpha(title) clamp(t / 40, 0, 1))` fades it in.
|
|
215
|
+
|
|
216
|
+
| Setting | For | Meaning |
|
|
217
|
+
| --- | --- | --- |
|
|
218
|
+
| `type` | all | `text` (the default), `rect` or `ellipse` |
|
|
219
|
+
| `x`, `y` | all | the centre, 0–1 across and down the frame |
|
|
220
|
+
| `angle` | all | turned clockwise, radians |
|
|
221
|
+
| `color` | all | any CSS colour: `#ffcc00`, `rgba(0,0,0,0.5)`, `white` |
|
|
222
|
+
| `shadow` | all | `{ "color", "blur", "dx", "dy" }` drawn into the graphic (sizes as fractions of the text size, or the frame height for shapes), or `null` |
|
|
223
|
+
| `shadows` | all | a list of up to 8 shadow layers, back to front, each drawn into a picture of its own (see below) |
|
|
224
|
+
| `outline` | all | `{ "color", "width" }`, or `null` |
|
|
225
|
+
| `text` | text | the words; `\n` starts a new line |
|
|
226
|
+
| `font`, `weight`, `style` | text | a CSS font family (`"Georgia, serif"`), `normal`/`bold`/`100`–`900`, `normal`/`italic` |
|
|
227
|
+
| `fontUrl` | text | an `https://` font file (.woff2, .woff, .ttf, .otf), or a Google Fonts stylesheet address, loaded before drawing; `font` is the fallback while it loads or if it can't |
|
|
228
|
+
| `size`, `align`, `lineHeight` | text | height as a fraction of the frame's, `left`/`center`/`right`, line spacing |
|
|
229
|
+
| `w`, `h`, `radius` | shapes | size as fractions of the frame; a rectangle's corner rounding |
|
|
230
|
+
|
|
231
|
+
### Shadow layers, each with its own filter
|
|
232
|
+
|
|
233
|
+
`shadow` is drawn into the graphic, so whatever the maths does to the
|
|
234
|
+
text it does to that shadow too. `shadows` keeps them apart: each layer
|
|
235
|
+
is drawn into a picture of its own — `title_shadow1`, `title_shadow2`…,
|
|
236
|
+
back to front — and the maths lays each on as it likes:
|
|
237
|
+
|
|
238
|
+
```json
|
|
239
|
+
{ "graphics": { "title": { "text": "LAYERS", "shadow": null,
|
|
240
|
+
"shadows": [ { "color": "#000", "blur": 0.7, "dy": 0.08 }, { "color": "#000", "dx": 0.07, "dy": 0.07 } ] } },
|
|
241
|
+
"color": "mix(mix(mix(rgb, rgb * 0.25, alpha(title_shadow1) (0.7 + 0.3 sin(t / 10))), mix((1, 0.2, 0.6), (0.2, 0.6, 1), x), alpha(title_shadow2)), title, alpha(title))" }
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Here the first layer darkens the picture under a soft glow that breathes
|
|
245
|
+
with `t`, and the second is a hard offset shadow coloured by a gradient
|
|
246
|
+
whatever colour it was cast in ("Layered shadow" in the library). The
|
|
247
|
+
plain way to lay them all on is `mix(mix(mix(rgb, title_shadow1,
|
|
248
|
+
alpha(title_shadow1)), title_shadow2, alpha(title_shadow2)), title,
|
|
249
|
+
alpha(title))`; the editor writes that for you when you add a layer to
|
|
250
|
+
a step whose maths is still the plain one, and points out a layer your
|
|
251
|
+
own maths doesn't use yet.
|
|
252
|
+
|
|
253
|
+
### Fonts
|
|
254
|
+
|
|
255
|
+
`font` is a CSS family list, so anything the viewer's computer has
|
|
256
|
+
works (`"Georgia, serif"`). For a font they may not have, give
|
|
257
|
+
`fontUrl`: a Google Fonts stylesheet (`googleFontUrl('Lobster')`, or the
|
|
258
|
+
editor's "Google Fonts name" box) or the address of a font file. Those
|
|
259
|
+
travel in share links; the viewer's browser fetches them from that
|
|
260
|
+
address. A font file loaded from your own computer works in the editor
|
|
261
|
+
only — a link opened elsewhere shows the fallback `font`.
|
|
262
|
+
|
|
263
|
+
Sizes are fractions of the frame, so at export the graphics are drawn
|
|
264
|
+
again at the export size — crisp, and in the same place. A chain changes
|
|
265
|
+
a graphic with `with`, merging settings: `{ "filter": …Caption…, "with":
|
|
266
|
+
{ "title": { "text": "Bye", "color": "#ffcc00" } } }`. Unlike pictures,
|
|
267
|
+
graphics are not shared between steps: two steps whose text is called
|
|
268
|
+
`title` draw two texts (the compiler binds them as `title`, `title~2`…).
|
|
269
|
+
|
|
270
|
+
In the editor, *+ text*, *+ rectangle* and *+ oval* add a step with an
|
|
271
|
+
editable expression; each graphic gets controls for its words, font,
|
|
272
|
+
size, place, angle, colour, shadow and outline, and can be dragged on
|
|
273
|
+
the picture. A font file loaded from the computer works there but can't
|
|
274
|
+
travel in a link; a font URL can (the browser of whoever opens the link
|
|
275
|
+
fetches it from that address).
|
|
276
|
+
|
|
277
|
+
With the runners directly, draw the graphics yourself after compiling:
|
|
278
|
+
|
|
279
|
+
```js
|
|
280
|
+
const compiled = compileFilter(def);
|
|
281
|
+
for (const [key, canvas] of Object.entries(await renderGraphics(compiled, width, height))) runner.setImage(key, canvas);
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
A graphic not drawn yet is transparent, so nothing covers the picture
|
|
285
|
+
meanwhile.
|
|
286
|
+
|
|
287
|
+
## Brushes
|
|
288
|
+
|
|
289
|
+
A brush is a graphic of type `strokes`: what the user paints on the
|
|
290
|
+
picture, recorded point by point (`[x, y, milliseconds, pressure]`, x and
|
|
291
|
+
y as fractions of the frame) and drawn again for every frame — only the
|
|
292
|
+
paint laid down by frame `t` is there, so the painting animates. Its
|
|
293
|
+
picture carries data rather than colour:
|
|
294
|
+
|
|
295
|
+
| Channel | Meaning |
|
|
296
|
+
| --- | --- |
|
|
297
|
+
| `alpha(brush)` | coverage: how much paint is here so far |
|
|
298
|
+
| `red(brush)` | age: 0 where it was just painted, rising to 1 after `age` frames |
|
|
299
|
+
| `green(brush)` | passes ÷ 4: how many strokes have gone over it (`4 green(brush)` counts them) |
|
|
300
|
+
| `blue(brush)` | progress: how far through the whole drawing it was painted, 0 → 1 |
|
|
301
|
+
|
|
302
|
+
So the maths decides what the paint does, and how it changes:
|
|
303
|
+
|
|
304
|
+
```
|
|
305
|
+
mix(rgb, ink, alpha(brush)) ink in a colour (a knob)
|
|
306
|
+
mix(rgb, gaussian(2 + 3 min(4 green(brush), 3)), alpha(brush)) a blur, stronger with every pass
|
|
307
|
+
rgb (1 - 0.22 min(4 green(brush), 4) alpha(brush)) darker with every pass, like a wash
|
|
308
|
+
mix(rgb, mix((1, 0.9, 0.5), (0.1, 0.07, 0.06), red(brush)), alpha(brush)) glowing as it goes down, then cooling
|
|
309
|
+
mix(rgb, fromhsv(blue(brush), 0.85, 1), alpha(brush)) a rainbow along the drawing
|
|
310
|
+
mix((gray, gray, gray), rgb, alpha(brush)) the picture's colour painted back in
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
| Setting | Meaning |
|
|
314
|
+
| --- | --- |
|
|
315
|
+
| `brush` | `round`, `soft` (airbrush), `marker` (a flat nib at 45°), `chalk` (grain fixed to the paper), `spray` |
|
|
316
|
+
| `size`, `flow`, `spacing` | diameter as a fraction of the frame height; paint per dab; dab spacing in brush widths |
|
|
317
|
+
| `taper` | thinner at both ends, over this fraction of the stroke |
|
|
318
|
+
| `widthFrom` | `pressure` (a pen or touch screen), `speed` (fast is thin, as with ink), or `none` |
|
|
319
|
+
| `buildUp` | coverage adds up where strokes overlap, instead of taking the most |
|
|
320
|
+
| `timing` | `recorded` (the user's own speed), `even` (time follows length), `per-stroke` (each the same time) |
|
|
321
|
+
| `speed`, `gap`, `start`, `fit` | faster or slower; frames between strokes; frames before the first; stretch the whole drawing to this many frames (0: off) |
|
|
322
|
+
| `age` | how many frames `red(brush)` takes to go from 0 to 1 |
|
|
323
|
+
| `clip` | `none`, `inside` or `outside`: paint only inside (or outside) the user's selection |
|
|
324
|
+
|
|
325
|
+
A stroke counts once wherever it goes, however often it crosses itself,
|
|
326
|
+
so `green` counts separate passes. In the editor, "+ brush" adds a step
|
|
327
|
+
and starts drawing (the picture shows the painting finished while you
|
|
328
|
+
draw; "replay" plays it); each brush's panel chooses what the paint does,
|
|
329
|
+
the brush, the timing and where it may paint, and "outline the
|
|
330
|
+
selection" turns the selection's edge into strokes — a brush that draws
|
|
331
|
+
round an object. `traceContours(mask, w, h)` and `rasterizeStrokes(spec,
|
|
332
|
+
w, h, t, { mask })` do the same outside the editor; `renderGraphics`
|
|
333
|
+
takes `{ t, mask }`.
|
|
334
|
+
|
|
335
|
+
## Several pictures: masks and blending
|
|
336
|
+
|
|
337
|
+
A filter can ask for more pictures by name, and the maths treats each
|
|
338
|
+
like the main one:
|
|
339
|
+
|
|
340
|
+
```json
|
|
341
|
+
{ "name": "Through a mask",
|
|
342
|
+
"images": { "mask": { "description": "light where the effect shows" } },
|
|
343
|
+
"color": "mix(source, rgb, luma(mask))" }
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
- `mask` is that picture's colour at this pixel (like `rgb`);
|
|
347
|
+
`mask(px, py)` is any pixel of it, also a colour; `red(mask)` …
|
|
348
|
+
`alpha(mask)` and `luma(mask)` read one channel.
|
|
349
|
+
- `source` is always available: the picture the chain started from,
|
|
350
|
+
before any step. So "Through a mask" as the last step shows everything
|
|
351
|
+
above it only where the mask is light — a blur, a posterize, anything.
|
|
352
|
+
- `images` is a list of names, or an object with a `description` for
|
|
353
|
+
each (the demo shows it beside the picture chooser).
|
|
354
|
+
|
|
355
|
+
```
|
|
356
|
+
mix(rgb, overlay, amount) fade to another picture
|
|
357
|
+
mix(rgb, overlay, luma(mask)) …through a mask
|
|
358
|
+
rgb * overlay multiply
|
|
359
|
+
1 - (1 - rgb)(1 - overlay) screen
|
|
360
|
+
if gray < 0.5 then 2 rgb overlay else 1 - 2 (1 - rgb)(1 - overlay) overlay blend
|
|
361
|
+
abs(rgb - overlay) difference
|
|
362
|
+
mix(rgb, overlay, alpha(overlay)) a transparent PNG laid on top
|
|
363
|
+
if distance(rgb, key) < tolerance then background else rgb chroma key onto a picture
|
|
364
|
+
mix(source, rgb, luma(mask)) keep the chain's result where the mask is light
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### Selecting an object
|
|
368
|
+
|
|
369
|
+
`selected` is 1 inside what the user selected on the picture and 0
|
|
370
|
+
outside — 1 everywhere when nothing is selected, so a filter that uses
|
|
371
|
+
it works unchanged. `mix(source, rgb, selected)` as a last step keeps
|
|
372
|
+
the chain's result only inside the selection ("Only inside the
|
|
373
|
+
selection" in the library); `selection(px, py)` reads the mask elsewhere.
|
|
374
|
+
It is a mask, so a colour space leaves it alone. The runner takes it as
|
|
375
|
+
a picture named `selection` (white = in): `runner.setImage('selection',
|
|
376
|
+
maskCanvas)`.
|
|
377
|
+
|
|
378
|
+
The editor makes one two ways. *Trace the edge* is intelligent scissors
|
|
379
|
+
(livewire): click a few points around the object and the line between
|
|
380
|
+
them follows the edge — every pixel has a cost, low where the brightness
|
|
381
|
+
changes sharply, and the line is the cheapest path (Dijkstra) between
|
|
382
|
+
your points; a click runs the search once from that point, so the wire
|
|
383
|
+
to the cursor moves in real time and only the segments a point touches
|
|
384
|
+
are recomputed when you add or drag one. A clicked point snaps to the
|
|
385
|
+
strongest edge within a few pixels, so it need not be placed exactly.
|
|
386
|
+
With *freeze as I go* (path cooling), the part of the line that has
|
|
387
|
+
stayed the same for a moment as you move on is fixed for you — small
|
|
388
|
+
blue points — so around a clear edge you click once and steer, and click
|
|
389
|
+
only where the line needs help. Click the first point, press Enter or
|
|
390
|
+
double-click to close; Backspace removes the last point. When the
|
|
391
|
+
object runs off the picture, click *outside* it (in the margin round the
|
|
392
|
+
picture): that point (drawn as a square) sits on the picture's edge, and
|
|
393
|
+
between two such points the line runs along the edge, so the shape takes
|
|
394
|
+
in the border without tracing it. *Find objects (AI)* runs **MobileSAM** (Segment Anything with a small image
|
|
395
|
+
encoder) in the browser: click anything — a person, a house, a tree, a
|
|
396
|
+
cloud — and it is outlined. It looks at the picture once (a second or two);
|
|
397
|
+
each click is then about a tenth of a second. One click can be ambiguous
|
|
398
|
+
(the harness or the horse?): click again to add, shift-click where it went
|
|
399
|
+
too far.
|
|
400
|
+
|
|
401
|
+
<!-- dev -->
|
|
402
|
+
In development, `demo/`, a *model* menu also offers the others
|
|
403
|
+
compared under "The object models" below — RF-DETR, which lists and labels
|
|
404
|
+
80 kinds of thing; `?ai=…` in the address chooses one.
|
|
405
|
+
<!-- /dev -->
|
|
406
|
+
|
|
407
|
+
Enter keeps, and the next click starts another
|
|
408
|
+
object. The model and its runtime (ONNX Runtime Web) are
|
|
409
|
+
served by the site itself — no CDN, no Hugging Face — load only when
|
|
410
|
+
asked and are then cached by the browser; the rest of the editor never
|
|
411
|
+
depends on them. Anything not among the 80 kinds is for the edge tracer,
|
|
412
|
+
the colour fill or the refine brush.
|
|
413
|
+
|
|
414
|
+
The *refine* brush fixes any selection, however it was made. Paint with
|
|
415
|
+
*inside* over a part that was missed, or with *outside* over a part that
|
|
416
|
+
leaked; the boundary then settles on the nearest real edge near the
|
|
417
|
+
stroke. Under the hood, the stroke and the rim of a band round it are
|
|
418
|
+
seeds, and each side grows through the band with a cost that is tiny in
|
|
419
|
+
even colour and steep across an edge (measured over three pixels, so a
|
|
420
|
+
soft edge counts in full) — whichever side reaches a pixel first claims it
|
|
421
|
+
(a seeded region competition, Dijkstra again); nothing outside the band
|
|
422
|
+
moves. With nothing selected, an inside stroke makes a selection from
|
|
423
|
+
scratch: the whole picture is in play and its border is "outside", so the
|
|
424
|
+
stroke grows to the object's edges. The size slider sets the brush. *Fill a colour* is a magic wand that
|
|
425
|
+
knows several colours: each click adds that pixel's colour to a set,
|
|
426
|
+
and the fill grows from every click through any pixel within the
|
|
427
|
+
tolerance of *any* colour in the set — a striped shirt is two clicks;
|
|
428
|
+
"connected only" off takes those colours anywhere in the picture;
|
|
429
|
+
click a swatch to drop a colour, and the tolerance slider refills live. Feather softens the edge,
|
|
430
|
+
invert swaps inside and out; the selection is drawn at any export size.
|
|
431
|
+
It belongs to the picture, not the filter, so share links don't carry it.
|
|
432
|
+
|
|
433
|
+
### The object models
|
|
434
|
+
|
|
435
|
+
The site uses **MobileSAM** to find an object from a click or two. It runs in your browser like the
|
|
436
|
+
filters (about 19 MB, downloaded the first time it is needed); the photo never leaves your computer.
|
|
437
|
+
|
|
438
|
+
<!-- dev -->
|
|
439
|
+
The site ships one: **MobileSAM** (`SHIPPED_MODELS` in `demo/ai.js`; the build copies only its files,
|
|
440
|
+
with the LICENSE and its part of the NOTICE). Every file of the site has to stay under 25 MiB to be
|
|
441
|
+
hosted on Cloudflare Pages, which RF-DETR Small (39 MB) isn't; a bigger model can come back split
|
|
442
|
+
into parts and joined when it loads. The others stay in `demo/models/` for comparing in
|
|
443
|
+
development. All are Apache-2.0:
|
|
444
|
+
|
|
445
|
+
| model | file | size | what |
|
|
446
|
+
| --- | --- | --- | --- |
|
|
447
|
+
| RF-DETR Small | rfdetr-seg-small.onnx | 41 MB | exported at 576 × 576 (not its own 384: more found, finer outlines), int8 |
|
|
448
|
+
| RF-DETR Nano | rfdetr-seg-nano.onnx | 37 MB | its own 312 × 312, int8 — the same backbone, so hardly smaller |
|
|
449
|
+
| MobileSAM | mobilesam-encoder/decoder.onnx | 19 MB | int8 (the encoder's MatMuls only: quantizing its convolutions breaks it) |
|
|
450
|
+
| MagicTouch | magic_touch.tflite | 6 MB | as published |
|
|
451
|
+
| MagicTouch 2 | magic_touch_2.task | 31 MB | as published (int8) |
|
|
452
|
+
|
|
453
|
+
`tools/export-model.py` remakes the ONNX files; `node tools/compare-ai.mjs
|
|
454
|
+
out.png` draws every model's outline of the same things in a photo side by
|
|
455
|
+
side, with timings. They run on one thread; ONNX Runtime uses several when
|
|
456
|
+
the page is cross-origin isolated (the host sends
|
|
457
|
+
`Cross-Origin-Opener-Policy: same-origin` and
|
|
458
|
+
`Cross-Origin-Embedder-Policy: require-corp`).
|
|
459
|
+
<!-- /dev -->
|
|
460
|
+
|
|
461
|
+
### Tracking an object through a video or GIF
|
|
462
|
+
|
|
463
|
+
The editor can make a selection follow an object through a clip, so
|
|
464
|
+
`selected` (and brushes that stay inside the selection) follow it frame
|
|
465
|
+
by frame. Pause on a frame, select the object with any tool, "use as
|
|
466
|
+
keyframe", then "track". From each keyframe the outline is carried
|
|
467
|
+
forward and backward a frame at a time:
|
|
468
|
+
|
|
469
|
+
- points round the outline are found again in the next frame by matching
|
|
470
|
+
the patch of picture round each (coarse, then fine); points that
|
|
471
|
+
disagree with the rest are pulled back to the group's movement, and
|
|
472
|
+
each may stray only a few pixels from the group's overall move, scale
|
|
473
|
+
and turn, so the outline can't wander off;
|
|
474
|
+
- the keyframe teaches it the object's colours against a band just
|
|
475
|
+
outside, so each frame has a map of how object-like every pixel is;
|
|
476
|
+
the outline between the points follows the edge of that map (and some
|
|
477
|
+
plain brightness edges) — not every leaf behind the object;
|
|
478
|
+
- each frame's confidence is how much more object-like the inside is than
|
|
479
|
+
the band just outside, against the keyframe, and whether the area
|
|
480
|
+
jumped. Between two keyframes, each frame keeps the direction that
|
|
481
|
+
scored higher.
|
|
482
|
+
|
|
483
|
+
The timeline under the picture shows confidence (green sure, red not):
|
|
484
|
+
click a frame to go there, fix it ("edit this frame" puts the outline into
|
|
485
|
+
the edge tracer to drag its points, or select again), and use it as another
|
|
486
|
+
keyframe. "Fix red frames with AI" runs the chosen model on each unsure frame
|
|
487
|
+
with a box round the tracked outline (RF-DETR takes the object whose box
|
|
488
|
+
matches best; MobileSAM outlines what is in the box). Tracking runs at preview
|
|
489
|
+
size, about 15 s for 240 frames.
|
|
490
|
+
|
|
491
|
+
<!-- dev -->
|
|
492
|
+
Videos in the photo list are held in memory
|
|
493
|
+
so they can be stepped through frame by frame (a server must allow byte
|
|
494
|
+
ranges for that otherwise; `tools/serve.js` does).
|
|
495
|
+
<!-- /dev -->
|
|
496
|
+
|
|
497
|
+
In a chain, pictures are matched by name: every step that asks for
|
|
498
|
+
`mask` gets the same one, and `compiled.images` lists what to supply.
|
|
499
|
+
The runner takes them with `runner.setImage('mask', image)`; a picture
|
|
500
|
+
not given is white, so a missing mask lets everything through. Pictures
|
|
501
|
+
are stretched to the canvas, so size them like the main picture.
|
|
502
|
+
|
|
503
|
+
## Colour spaces
|
|
504
|
+
|
|
505
|
+
The frame holds sRGB — the numbers a screen shows. A filter can work in
|
|
506
|
+
another space instead:
|
|
507
|
+
|
|
508
|
+
```json
|
|
509
|
+
{ "name": "Hue shift", "space": "hsv", "params": { "amount": 0.1 }, "color": "(fract(r + amount), g, b)" }
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
With `"space": "hsv"`, `r g b` (and `rgb`, `pixel()`, the blurs, named
|
|
513
|
+
pictures, `source` — everything read) are hue, saturation and value,
|
|
514
|
+
and the result is taken as HSV and converted back. `gray` is still the
|
|
515
|
+
brightness as seen. `"space": { "in": "lab", "out": "srgb" }` converts
|
|
516
|
+
on the way in only, when the expression writes sRGB itself.
|
|
517
|
+
|
|
518
|
+
| Space | `r`, `g`, `b` are |
|
|
519
|
+
| --- | --- |
|
|
520
|
+
| `srgb` | what the picture holds (the default) |
|
|
521
|
+
| `linear` | light rather than gamma-encoded values: blurs and mixes in it behave like a lens, bright spots stay bright |
|
|
522
|
+
| `hsv`, `hsl` | hue (0–1 around the wheel), saturation, value / lightness |
|
|
523
|
+
| `lab` | L 0–1, then a and b centred on 0.5 (0.5 ± value/256) |
|
|
524
|
+
| `oklab` | L 0–1, then a and b centred on 0.5 (0.5 + value) — perceptually even |
|
|
525
|
+
| `ycbcr` | luma, then Cb and Cr centred on 0.5 |
|
|
526
|
+
|
|
527
|
+
Every space keeps its values in 0–1 so they survive the 8-bit texture
|
|
528
|
+
between passes — but 8 bits of Lab or OKLab is coarse for dark, saturated
|
|
529
|
+
colours (up to ~20/255 off after the round trip), so the editor uses
|
|
530
|
+
half-float textures for a chain with a colour space where the GPU
|
|
531
|
+
allows (`createFilterRunner(gl, c, { precision: 'half' })`,
|
|
532
|
+
`createCpuRunner(c, { precision: 'float' })`). A step's opacity blends
|
|
533
|
+
the result over its input in sRGB. The conversions are also functions you can use anywhere —
|
|
534
|
+
`tohsv(c)`, `fromhsv(c)`, `tolinear`, `fromlinear`, `tohsl`, `tolab`,
|
|
535
|
+
`tooklab`, `toycbcr` and their inverses, sRGB on the other side —
|
|
536
|
+
so a single expression can do it: `fromhsv(fract(red(tohsv(rgb)) + t /
|
|
537
|
+
300), green(tohsv(rgb)), blue(tohsv(rgb)))` cycles the hue. The editor
|
|
538
|
+
has a space chooser beside the expression box.
|
|
539
|
+
|
|
540
|
+
## Blur, edges and other neighbourhood functions
|
|
541
|
+
|
|
542
|
+
The classic image-processing operations are functions you can use in
|
|
543
|
+
any expression, so they combine with everything else and with each
|
|
544
|
+
other. Colour-valued ones return a colour like `rgb` (alpha is left
|
|
545
|
+
alone); the edge detectors work on brightness and return a number.
|
|
546
|
+
|
|
547
|
+
| Function | Result |
|
|
548
|
+
| --- | --- |
|
|
549
|
+
| `blur(radius)` | box blur: the mean of the square ±radius pixels |
|
|
550
|
+
| `gaussian(sigma)` | Gaussian blur, weights falling off with `sigma` |
|
|
551
|
+
| `blurx(r)` `blury(r)` `gaussianx(s)` `gaussiany(s)` | the same along one axis — chain an x then a y step for a wide, fast blur |
|
|
552
|
+
| `sharpen(amount)` | the 3×3 sharpen kernel, scaled by `amount` |
|
|
553
|
+
| `dilate(radius)` `erode(radius)` | the brightest / darkest colour within ±radius: light or dark areas grow |
|
|
554
|
+
| `convolve(k1, …, k9)` | your own 3×3 kernel, row by row |
|
|
555
|
+
| `sobel()` `sobelx()` `sobely()` | Sobel edge strength (and its horizontal / vertical parts) |
|
|
556
|
+
| `laplacian()` | the 4-neighbour Laplacian of brightness |
|
|
557
|
+
| `emboss()` | brightness relief lit from the top-left, around 0.5 |
|
|
558
|
+
|
|
559
|
+
```
|
|
560
|
+
gaussian(3) a blur
|
|
561
|
+
rgb + 1.5 (rgb - gaussian(2)) unsharp mask
|
|
562
|
+
rgb + 0.8 max(gaussian(3) - 0.6, 0) bloom: bright parts glow
|
|
563
|
+
if sobel() > 0.3 then 1 else 0 an edge map
|
|
564
|
+
rgb (1 - 3 sobel()) ink outline over the colour
|
|
565
|
+
convolve(0, -1, 0, -1, 5, -1, 0, -1, 0) sharpen, as a kernel
|
|
566
|
+
dilate(2) - erode(2) morphological gradient: outlines of shapes
|
|
567
|
+
```
|
|
568
|
+
|
|
569
|
+
Every one of them takes expressions, so a parameter can drive it
|
|
570
|
+
(`gaussian(sigma)`), and `t` can too (`gaussian(2 + 2 sin(t / 20))`).
|
|
571
|
+
In a per-channel expression a colour result gives that channel:
|
|
572
|
+
`{ "r": "gaussian(4)" }` blurs only red. The library in the demo has
|
|
573
|
+
each of these as a filter with knobs, and chains built from them
|
|
574
|
+
(Cartoon = Gaussian blur → Posterize → Outline).
|
|
575
|
+
|
|
576
|
+
Two limits, because a shader loop needs fixed bounds: the square
|
|
577
|
+
functions reach ±8 pixels, so `gaussian(sigma)` inside an expression is
|
|
578
|
+
truncated above `sigma ≈ 2.7`; the one-axis functions reach ±24. A step
|
|
579
|
+
whose whole colour is just `gaussian(sigma)` or `blur(radius)` is
|
|
580
|
+
computed separably for you — an x pass then a y pass, 2n taps instead
|
|
581
|
+
of n², reaching ±24 — so the plain blur is the fast, wide one; write
|
|
582
|
+
`gaussianx`/`gaussiany` yourself only when you want one axis.
|
|
583
|
+
Each call samples the neighbourhood again, so `gaussian(3)` used twice
|
|
584
|
+
in one expression costs twice — put it in its own step if that matters.
|
|
585
|
+
"Canny edges" in the library is a chain of these: a Gaussian, then
|
|
586
|
+
gradient strength and direction (`atan2(sobely(), sobelx())` into four
|
|
587
|
+
buckets), non-maximum suppression along the gradient, two thresholds,
|
|
588
|
+
three rounds of hysteresis (`red(dilate(1))` tells a weak pixel whether a
|
|
589
|
+
strong one is next to it), and a clean-up — nine passes, all plain
|
|
590
|
+
expressions, so it runs on the CPU too.
|
|
591
|
+
|
|
592
|
+
## Running it
|
|
593
|
+
|
|
594
|
+
The editor runs a filter on the GPU (WebGL) when the browser has it, and on the CPU otherwise —
|
|
595
|
+
*Advanced* → *runs on* lets you choose. Both give the same pixels; the GPU is much faster, so big
|
|
596
|
+
pictures and video want it.
|
|
597
|
+
|
|
598
|
+
<!-- dev -->
|
|
599
|
+
### Without a GPU
|
|
600
|
+
|
|
601
|
+
Every filter also runs on the CPU, in plain JavaScript, for when WebGL isn't
|
|
602
|
+
there (some browsers, Node, servers). Where it is, the GPU is the one to use:
|
|
603
|
+
a filter's shaders compile once and then serve every picture and frame.
|
|
604
|
+
|
|
605
|
+
```js
|
|
606
|
+
import { compileFilter, createCpuRunner, createRunner } from '@mathzetta/qblur';
|
|
607
|
+
|
|
608
|
+
const runner = createCpuRunner(compileFilter(def)); // same interface as createFilterRunner
|
|
609
|
+
runner.setImage(imageData); // ImageData or { data, width, height }; an <img>/<canvas> in a browser
|
|
610
|
+
const out = runner.draw({ t: 0 }); // { data: Uint8ClampedArray, width, height }
|
|
611
|
+
await runner.drawAsync({ t: 0, canvas }); // rows spread over workers for big pictures
|
|
612
|
+
|
|
613
|
+
const { runner: r, engine, reason } = createRunner(compileFilter(def), { gl }); // the GPU if there is WebGL, else the CPU; and why
|
|
614
|
+
```
|
|
615
|
+
|
|
616
|
+
Each pass is compiled to one JavaScript loop over the pixels — the same
|
|
617
|
+
language, types and sampling as the shader (bilinear `pixel()`, edges
|
|
618
|
+
clamped, 8-bit rounding between passes), so both give the same pixels:
|
|
619
|
+
`npm run check -- --cpu` compares them on every library filter (all within
|
|
620
|
+
1/255 apart from a handful of pixels exactly on an edge; `random()` can't
|
|
621
|
+
match — GPUs compute the hash's `sin` in lower precision). Neighbourhood
|
|
622
|
+
functions with an argument that is the same for every pixel (`blur(radius)`)
|
|
623
|
+
are computed once per pass, separably; one that changes per pixel
|
|
624
|
+
(`blur(x * 8)`) is computed pixel by pixel and is slow. Rough single-thread
|
|
625
|
+
times for 1 megapixel: per-pixel formulas 12–35 ms, `blur`/`sobel` 30–85 ms,
|
|
626
|
+
`gaussian(3)` 115 ms, a σ = 8 Gaussian chain 310 ms; `drawAsync` with 8
|
|
627
|
+
workers is 3–5× faster on the heavy ones.
|
|
628
|
+
|
|
629
|
+
The generated code is compiled only after MathZetta's `checkGeneratedJs`
|
|
630
|
+
passes it (no names but its own, no strings, only its own arrays indexed),
|
|
631
|
+
so a filter from a share link can read the picture and write colours,
|
|
632
|
+
nothing else. Pages whose Content-Security-Policy forbids `'unsafe-eval'`
|
|
633
|
+
can't use the CPU runner.
|
|
634
|
+
|
|
635
|
+
### On the GPU
|
|
636
|
+
|
|
637
|
+
A complete WebGL 1 setup with MathZetta alone (qblur's `createFilterRunner` does all of this for you):
|
|
638
|
+
|
|
639
|
+
```js
|
|
640
|
+
const gl = canvas.getContext('webgl');
|
|
641
|
+
const fragment = MathZetta.filterShader('(1 - r, 1 - g, 1 - b)');
|
|
642
|
+
const program = link(gl, MathZetta.filterVertexShader(), fragment); // compile + link as usual
|
|
643
|
+
|
|
644
|
+
// One triangle covering the canvas.
|
|
645
|
+
gl.bindBuffer(gl.ARRAY_BUFFER, gl.createBuffer());
|
|
646
|
+
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
|
|
647
|
+
const position = gl.getAttribLocation(program, 'position');
|
|
648
|
+
gl.enableVertexAttribArray(position);
|
|
649
|
+
gl.vertexAttribPointer(position, 2, gl.FLOAT, false, 0, 0);
|
|
650
|
+
|
|
651
|
+
// The image, flipped so y = 0 is the top.
|
|
652
|
+
gl.bindTexture(gl.TEXTURE_2D, gl.createTexture());
|
|
653
|
+
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true);
|
|
654
|
+
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, imageOrVideoOrCanvas);
|
|
655
|
+
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
|
|
656
|
+
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
|
|
657
|
+
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
|
|
658
|
+
|
|
659
|
+
// Each frame: upload the new frame (texImage2D again for video), set t, draw.
|
|
660
|
+
gl.useProgram(program);
|
|
661
|
+
gl.uniform1i(gl.getUniformLocation(program, 'image'), 0);
|
|
662
|
+
gl.uniform2f(gl.getUniformLocation(program, 'resolution'), canvas.width, canvas.height);
|
|
663
|
+
gl.uniform1f(gl.getUniformLocation(program, 't'), frameIndex);
|
|
664
|
+
gl.drawArrays(gl.TRIANGLES, 0, 3);
|
|
665
|
+
```
|
|
666
|
+
|
|
667
|
+
For WebGL 2, pass `{ version: 300 }` to both `filterShader` and
|
|
668
|
+
`filterVertexShader`. `npm run check` runs every library filter
|
|
669
|
+
through real WebGL in headless Chrome (`--webgl2` for the ES 3.0 output). To process a GIF or video offline, draw each frame
|
|
670
|
+
to a canvas, upload it, draw, and read the canvas back (`toBlob` or
|
|
671
|
+
`readPixels`).
|
|
672
|
+
<!-- /dev -->
|
|
673
|
+
|
|
674
|
+
## Sharing filters
|
|
675
|
+
|
|
676
|
+
A filter is a small JSON document: the maths, a name and author, and the
|
|
677
|
+
knobs it exposes. Since it is only maths, it is safe to accept from
|
|
678
|
+
anyone — it can read the picture and write colours, nothing else.
|
|
679
|
+
|
|
680
|
+
```json
|
|
681
|
+
{
|
|
682
|
+
"format": "mathzetta-filter/1",
|
|
683
|
+
"name": "Threshold",
|
|
684
|
+
"author": "Ada",
|
|
685
|
+
"description": "Black or white, at a cutoff you choose",
|
|
686
|
+
"params": { "cutoff": { "default": 0.5, "min": 0, "max": 1 } },
|
|
687
|
+
"color": "if gray > cutoff then 1 else 0"
|
|
688
|
+
}
|
|
689
|
+
```
|
|
690
|
+
|
|
691
|
+
- `color` is one expression for the colour, or an object with any of
|
|
692
|
+
`r`, `g`, `b`, `a`, `rgb`, `rgba` — exactly as `filterShader()` takes.
|
|
693
|
+
- `params` are the knobs. Each is a number (`"cutoff": 0.5` means
|
|
694
|
+
0–1; larger defaults get 0–2×) or `{ "default", "min", "max", "step",
|
|
695
|
+
"label" }`, or a colour: `"tint": [1, 0.85, 0.7]` (three numbers, 0–1).
|
|
696
|
+
A parameter is a value, never a function, so `strength (x + 1)` is a
|
|
697
|
+
product.
|
|
698
|
+
A number knob is a slider, unless it has `"options": [2, 4, 8]` (a menu
|
|
699
|
+
of those values; the default is the first unless given) or
|
|
700
|
+
`"input": "box"` (a box to type a value in — any maths that works out
|
|
701
|
+
to a number, such as `1/3`).
|
|
702
|
+
- `notes` is the filter's page, shown in its step in the editor: a list of
|
|
703
|
+
Markdown texts (maths in `$…$` is drawn by Inline Math, and `{{cutoff}}`
|
|
704
|
+
inside it is a knob's value) and `{ "param": "cutoff" }` entries that
|
|
705
|
+
place a knob's input among them; knobs it doesn't place come first.
|
|
706
|
+
Notes never change the picture.
|
|
707
|
+
|
|
708
|
+
```json
|
|
709
|
+
"notes": ["Pixels darker than the cutoff go black.", { "param": "cutoff" }, "Now at ${{cutoff}}$."]
|
|
710
|
+
```
|
|
711
|
+
- `name`, `author`, `description` are free text; `format` says which
|
|
712
|
+
version of this format the document follows.
|
|
713
|
+
|
|
714
|
+
In the editor each knob is written as an Inline Math variable, one line
|
|
715
|
+
you can edit (shown while the step's *Edit* is on): `cutoff := [[0..1 step 0.01 = 0.5]]` (a slider: from, to,
|
|
716
|
+
its step, where it starts), `radius := [[3]]` (a box), `levels := [[2 | 4 | 8]]`
|
|
717
|
+
(a menu) or `tint := #ff8800` (a colour). Renaming a knob there renames it
|
|
718
|
+
in the maths too.
|
|
719
|
+
|
|
720
|
+
### Combining filters
|
|
721
|
+
|
|
722
|
+
A chain lists steps. Each step is a filter — pasted in whole, so the
|
|
723
|
+
chain stays self-contained — or just an expression, with optional
|
|
724
|
+
settings for its parameters, an **opacity** (how much of the step's
|
|
725
|
+
result shows over its input), and `"skip": true` to switch it off
|
|
726
|
+
without removing it (a chain with every step off shows the picture
|
|
727
|
+
as it is):
|
|
728
|
+
|
|
729
|
+
```json
|
|
730
|
+
{
|
|
731
|
+
"format": "mathzetta-filter/1",
|
|
732
|
+
"name": "Old film",
|
|
733
|
+
"author": "Bo",
|
|
734
|
+
"steps": [
|
|
735
|
+
{ "name": "Sepia", "color": "(0.393r + 0.769g + 0.189b, 0.349r + 0.686g + 0.168b, 0.272r + 0.534g + 0.131b)" },
|
|
736
|
+
{ "filter": { "name": "Vignette", "params": { "strength": 1.6 }, "color": "rgb * (1 - strength ((x - 0.5)^2 + (y - 0.5)^2))" },
|
|
737
|
+
"with": { "strength": 1.2 } },
|
|
738
|
+
{ "filter": "if py mod 4 < 2 then rgb else rgb * 0.6", "opacity": 0.5 }
|
|
739
|
+
]
|
|
740
|
+
}
|
|
741
|
+
```
|
|
742
|
+
|
|
743
|
+
A chain is a filter, so Bo's "Old film" can be one step of your chain,
|
|
744
|
+
at half strength and with its vignette turned up:
|
|
745
|
+
|
|
746
|
+
```json
|
|
747
|
+
{ "steps": [ { "filter": { …Old film… }, "opacity": 0.5, "with": { "strength": 2 } }, "rgb * 1.1" ] }
|
|
748
|
+
```
|
|
749
|
+
|
|
750
|
+
`with` on a chain reaches every parameter of that name inside it, and
|
|
751
|
+
the outer setting wins — whoever is combining decides. Compiling lists
|
|
752
|
+
the names and authors of everything used as `credits`.
|
|
753
|
+
|
|
754
|
+
<!-- dev -->
|
|
755
|
+
### The API
|
|
756
|
+
|
|
757
|
+
[qblur on your own pages](library.md) is the full account: the CDN, the builds, running on the GPU, the CPU
|
|
758
|
+
and in Node, knobs, video, other pictures, text, sharing and every export, with examples that are run as tests.
|
|
759
|
+
|
|
760
|
+
```js
|
|
761
|
+
import { compileFilter, createFilterRunner, filterToLink, readFilter } from '@mathzetta/qblur';
|
|
762
|
+
|
|
763
|
+
const compiled = compileFilter(def); // def: JSON, or a string expression
|
|
764
|
+
compiled.passes; // shader passes, in order
|
|
765
|
+
compiled.credits; // [{ name: 'Sepia', author: '' }, …]
|
|
766
|
+
|
|
767
|
+
const runner = createFilterRunner(gl, compiled); // WebGL 1 or 2 (compileFilter(def, { version: 300 }))
|
|
768
|
+
runner.setImage(image); // <img>, <video>, <canvas>, ImageData — again for each video frame
|
|
769
|
+
runner.setImage('mask', maskImage); // each picture in compiled.images
|
|
770
|
+
runner.draw({ t: frameIndex }); // onto the canvas gl belongs to
|
|
771
|
+
runner.passes[0].values.cutoff = 0.3; // knobs change live, no recompiling
|
|
772
|
+
runner.passes[0].placements.inset.x = 0.2; // so does where a picture sits (changing `place` needs a recompile)
|
|
773
|
+
|
|
774
|
+
createFilterRunner(gl, compiled, { precision: 'half' }); // keep values over 1 between passes (bloom, edges); runner.precision says what the GPU gave
|
|
775
|
+
runner.dispose();
|
|
776
|
+
```
|
|
777
|
+
|
|
778
|
+
| Function | Does |
|
|
779
|
+
| --- | --- |
|
|
780
|
+
| `validateFilter(def)` | every problem, as messages naming the filter: `"Threshold": "with" sets "nope", but there is no parameter called that` |
|
|
781
|
+
| `normalizeFilter(def)` | the full form: types and ranges on params, `with` and `opacity` on steps |
|
|
782
|
+
| `filterParams(filter)` | every parameter a filter or chain exposes |
|
|
783
|
+
| `filterImages(filter)` | every picture a filter or chain needs, by name, with how it is placed |
|
|
784
|
+
| `PLACEMENT` | the default placement: `{ place: 'cover', zoom: 1, x: 0.5, y: 0.5, angle: 0 }` |
|
|
785
|
+
| `filterToJSON(def)` / `filterFromJSON(text)` | JSON to publish (defaults left out) and back |
|
|
786
|
+
| `filterToLink(def, pageUrl)` / `filterFromLink(text)` | the filter packed into a URL's `#…` and back |
|
|
787
|
+
| `readFilter(text)` | whatever was pasted: JSON, a link, or a bare expression |
|
|
788
|
+
|
|
789
|
+
A chain compiles to one pass per filter, each drawn into a texture the
|
|
790
|
+
next pass reads (so `pixel(px, py)` in step two sees step one's result,
|
|
791
|
+
while `source` is still the original).
|
|
792
|
+
A chain used at partial opacity keeps its input and ends with a blend
|
|
793
|
+
pass. Parameters become uniforms; `t` and the picture are shared by
|
|
794
|
+
every pass.
|
|
795
|
+
<!-- /dev -->
|
|
796
|
+
|
|
797
|
+
## Format stability
|
|
798
|
+
|
|
799
|
+
Share links carry the whole filter, so a link made today must open
|
|
800
|
+
tomorrow. The rules, from `mathzetta-filter/1` on:
|
|
801
|
+
|
|
802
|
+
- **Additions only.** New keys, new functions in expressions, new
|
|
803
|
+
placement modes may arrive; nothing existing changes meaning or goes
|
|
804
|
+
away. A document that uses only what this guide describes keeps
|
|
805
|
+
working in every later version.
|
|
806
|
+
- **Unknown top-level keys are ignored**, so a document written by a
|
|
807
|
+
newer version still opens in an older one (minus what it doesn't
|
|
808
|
+
know). Unknown *settings* inside `params`, `images` or `with` are
|
|
809
|
+
errors, because they would silently change the result.
|
|
810
|
+
- **A breaking change means a new `format`** (`mathzetta-filter/2`), and
|
|
811
|
+
`readFilter()` will convert `/1` documents to it. An unknown format is
|
|
812
|
+
refused with a clear message rather than guessed at.
|
|
813
|
+
- Expression semantics are pinned by tests: the GPU and the CPU must
|
|
814
|
+
agree, so a change to what `shift` or `gaussian` means would show up there.
|
|
815
|
+
|
|
816
|
+
## What doesn't carry over to a shader
|
|
817
|
+
|
|
818
|
+
`toGLSL()` renders everything the parser understands, but a shader can
|
|
819
|
+
only run what has a numeric meaning per pixel:
|
|
820
|
+
|
|
821
|
+
- **Not available:** derivatives, integrals, limits, sums with an index
|
|
822
|
+
(`\sum_{i=1}^{n}`), set union/intersection, strings. These come out as
|
|
823
|
+
calls to functions you would have to write (`derivative(…)`,
|
|
824
|
+
`summation(…)`).
|
|
825
|
+
- **Needs care:** `x^y` with a negative `x` is undefined in GLSL (small
|
|
826
|
+
whole powers are safe — they're written as products); `log` of a
|
|
827
|
+
non-positive number; `pow`, `exp` overflowing to infinity. `n!`, `gcd`
|
|
828
|
+
and `perm` loop on the GPU and are slow.
|
|
829
|
+
- **Typing:** every number is a float. Logic (`&`, `|`, `!`) needs
|
|
830
|
+
comparisons as operands; chained comparisons like `0 < x < 1` must be
|
|
831
|
+
written `0 < x & x < 1`; `±` gives a two-component vector.
|
|
832
|
+
- **Names:** `x y r g b a t w h px py rgb gray unchanged` are taken, as
|
|
833
|
+
are GLSL's own words (`in`, `out`, `float`, `main`, …). Greek letters are
|
|
834
|
+
spelled out (`θ` → `theta`), primes become `_prime`. `f`, `g` and `h`
|
|
835
|
+
followed by a bracket are read as function calls, as in maths: write
|
|
836
|
+
`g * (1 - r)`, not `g(1 - r)`.
|
|
837
|
+
|
|
838
|
+
MathZetta's [how it works](https://github.com/triplelog/mathzetta/blob/main/packages/mathzetta/docs/how-it-works.md)
|
|
839
|
+
describes the GLSL renderer itself, and its
|
|
840
|
+
[glsl.tsv](https://github.com/triplelog/mathzetta/blob/main/packages/mathzetta/test/cases/glsl.tsv) lists
|
|
841
|
+
filter-style expressions with their GLSL.
|