onejs-sl 0.1.0

package/src/lang/README.md

@@ -0,0 +1,167 @@
+# `.sl`: the shader language as a file
+
+Phase A of `Specs/SL_TEXT.md`. Source text in, the same `Program` the EDSL
+records out.
+
+```hlsl
+// plasma.sl
+uniform float warp = 0.5;
+uniform float hue = 0.5;
+
+float4 main() {
+    float2 p = (uv - 0.5) * (warp * 14 + 2);
+    float v = sin(p.x + time) + sin(p.y - time * 0.8);
+    float n = saturate(v * 0.22 + 0.5);
+    return float4(hsv2rgb(float3(frac(hue + n * 0.18), 0.75, n)), 1);
+}
+```
+
+```ts
+import { parse } from "onejs-sl"
+const plasma = parse(source, { file: "plasma.sl" })
+```
+
+## The one idea
+
+**It is not a second language.** Every construct lowers through the EDSL, so
+`sin(x)` in a file is the same `sl.sin(x)` call a TypeScript author would have
+written, and the two produce the same graph. `parity.test.ts` asserts that as
+hash equality for every GPU fixture and for the example on play.onejs.com: same
+hash means same generated shader and same pixels, established without rendering
+anything. It also means the existing GPU fixtures, the codegen goldens and the
+C# VM tests cover the text form for free, because they run on the IR.
+
+If that property ever breaks, the failure is silent in the worst way: a program
+whose hash does not match its generated shader falls back to the VM and nobody
+is told. So parity is the test to keep green, not the parser's unit tests.
+
+| File | What it does |
+|---|---|
+| `lexer.ts` | Characters to tokens, each carrying line and column |
+| `ast.ts` | The node shapes, all of them, which is not many |
+| `parser.ts` | Pratt parser: shape only, HLSL precedence |
+| `check.ts` | Declarations, names, statement shape, budgets, recursion |
+| `lower.ts` | AST to IR through the EDSL: SSA, unrolling, `select`, inlining |
+| `builtins.ts` | What each name does, keyed by the spelling `ops.ts` gives it |
+| `prelude.ts` | The standard library, written in the language |
+| `onejs-unity`'s `src/esbuild/sl.mjs` | The loader: `import plasma from "./plasma.sl"` |
+| `../index.ts` and `../core.ts` | The package with the parser, and without it. See below |
+
+## The loader
+
+`slPlugin()`, in onejs-unity's `onejs-unity/esbuild`, parses and encodes at BUILD TIME, so an import resolves to a small
+object of numbers and the bundle carries neither the parser nor the source. A
+parse error becomes an esbuild error with the file, line and column, which the
+Play editor surfaces and every terminal editor links.
+
+It writes two things beside the code:
+
+- **`<name>.sl.d.ts`**, the way a USS module gets one, carrying the uniform
+  names in the type. `uniforms={{ wrap: 1 }}` is then a call site error rather
+  than a console warning on a frame nobody is looking at.
+- **`app.sl.json`** beside the bundle, the manifest `SLShaderGenerator` already
+  watches for. This is the thing the file format makes possible and the EDSL
+  cannot: a program in a file is known statically, so **an ejected game is
+  compiled from its first frame** rather than from its second. Runtime
+  recording stays for EDSL programs and for a build older than the loader; the
+  editor reads every `*.sl.json` it can find.
+
+An empty manifest is written only over one that is already there. Deleting the
+last `.sl` file has to stop its shaders being generated, and a project that has
+never had one should not find a new file beside its bundle.
+
+### Why the parser is compiled rather than imported
+
+The parser is TypeScript and the plugin runs under plain Node: an app's
+`esbuild.config.mjs` is executed by `node`, which cannot load a `.ts` file.
+Every plugin in that folder has the same constraint, which is why the Tailwind
+generator beside them is `.mjs`; a parser with two hundred tests is not going
+to be maintained twice.
+
+So the plugin compiles `sl/compiler.ts` once per process, with the esbuild
+already running the build, and imports the result: one source of truth, no
+generated artifact to go stale, about thirty milliseconds once. A Cloudflare
+Worker cannot evaluate code it builds, so PlaySite imports the parser
+statically and hands it in as `compiler`, and that path never runs there.
+
+### Why there are two barrels
+
+`onejs-sl/core` is what a **game** imports (through `onejs-unity/sl`), and the
+Play eject scaffold vendors every file it reaches into the downloaded project.
+`onejs-sl`, which adds this folder, is what a **build** imports (through
+`onejs-unity/sl/compiler`). Re-exporting the parser from the first put two thousand
+lines a played game never executes into every ejected project's source tree,
+where they could only read as clutter. The scaffold vendors what an entry actually
+reaches rather than every file beside it, so the split is enforced by the
+eject's own test.
+
+## Where the types are checked
+
+In `lower.ts`, once, by the EDSL, which computes a width as it records. A
+declaration is then an **assertion** against that width rather than an input to
+inference. `check.ts` deliberately does no type inference: two implementations
+of a type system are two type systems, and the one that would have been written
+here is the one nothing else uses.
+
+## What surprises people
+
+**A scalar on the left of an operator broadcasts through a swizzle.** `uv * 8`
+is `uv.mul(8)`, one constant node of two components. `8 * uv` is
+`sl.float(8).mul(uv)`, a one component constant and a broadcast. Same picture,
+different graph, different hash. Write the vector on the left when it matters.
+
+**Literal arithmetic folds; a builtin call never does.** `2 * 3 + 1` is the
+constant 7 before the IR sees it. `sin(0.5)` is two instructions, because
+folding it would mean a second implementation of `sin` in JavaScript, and a
+second implementation is somewhere for the two backends to disagree.
+
+**A hex default makes a uniform a colour.** `uniform float4 tint = #ff8040;`
+stores the sRGB components as its default, which is what the host sets and what
+the generated shader's Properties block shows, and every read of `tint` goes
+through `toLinear`. Without that rule, `#ff8040` written as a literal and
+`#ff8040` written as a default would be two different colours in one file.
+A default built out of numbers (`float4(1, 0.5, 0.25, 1)`) is not a colour and
+is read unconverted.
+
+**A uniform's default has to be written out.** Numbers, constructors of numbers
+and colours, and nothing else. It is baked into the program before anything
+runs, so it cannot name a const, which could itself name a uniform.
+
+**Shadowing is refused.** A local may not take the name of an input, a uniform,
+a texture, a const or a builtin. Allowing it would make "is this a texture?"
+depend on where the question is asked, for no gain in a language whose
+functions are six lines long. A file function may shadow a *prelude* function,
+which is the sanctioned way to replace one.
+
+## Not yet spellable
+
+Three things the EDSL or the opcode table has and a file cannot say. Each is a
+recorded gap, not an oversight, and each has its own error message rather than
+"unknown identifier".
+
+| | Why |
+|---|---|
+| `rgb2hsv`, `tex2Dlod` | The opcodes are numbered and **neither backend implements them**. Writing one would render as whatever the VM's dispatch falls through to and fail outright in the HLSL emitter. |
+| `fbm`'s simplex base | `sl.fbm(p, octaves, "simplex")` picks the base with a string, and the language has no strings. `fbm(p, octaves)` is the value base, and `turbulence` and `ridged` are the simplex family. |
+
+Related, and worth knowing: **the EDSL lets a program declare 15 textures and
+the VM has 4.** `ir.ts` picked its ceiling from the WebGL2 sampler count rather
+than from `FxProgram.shader`, which declares `_Tex0` to `_Tex3` and samples
+`_Tex3` for every slot past it, so slots 4 and up are silently wrong in the
+browser and correct after an eject. A `.sl` file is held to the real number, 4,
+reported at the declaration. The EDSL's ceiling is untouched here because
+lowering it changes recorded behaviour rather than parser behaviour.
+
+## Errors
+
+Every one carries file, line and column, and an error that starts inside the
+EDSL keeps the EDSL's wording. `"z" is component 3 of a vec2, which has 2` is
+already the right sentence; what it lacked was a place. `errors.test.ts` checks
+the positions as strictly as the words, because a marker under the wrong
+character sends the reader to look at something that is fine.
+
+## See also
+
+- `Specs/SL_TEXT.md`, the whole design, including the four decisions taken
+- `../README.md`, the IR, the EDSL and the hash this lowers to
+- `Specs/SHADER_LANG.md`, the backends

package/src/lang/ast.ts

@@ -0,0 +1,87 @@
+/**
+ * The AST the parser builds and the checker and lowering walk.
+ *
+ * Phase A of `Specs/SL_TEXT.md`. Deliberately small: a `.sl` file is one
+ * fragment function and its declarations, so there is no module system, no
+ * struct, no array and no statement that is not one of six shapes.
+ *
+ * Every node carries the position of the token it started at. That is what lets
+ * an error from the middle of lowering say which character to underline, and it
+ * is why the position lives on the node rather than being recovered by a second
+ * pass over the source.
+ */
+
+import type { Pos } from "./lexer"
+
+/** The four value types, under their HLSL names. `texture2D` is not one: it declares a slot. */
+export type TypeName = "float" | "float2" | "float3" | "float4"
+
+export const TYPE_WIDTH: Record<TypeName, 1 | 2 | 3 | 4> = {
+    float: 1, float2: 2, float3: 3, float4: 4,
+}
+
+export type BinaryOp =
+    | "+" | "-" | "*" | "/" | "%"
+    | "<" | "<=" | ">" | ">=" | "==" | "!="
+    | "&&" | "||"
+
+export type Expr =
+    | { k: "num"; value: number; pos: Pos }
+    | { k: "hex"; hex: string; pos: Pos }
+    | { k: "ident"; name: string; pos: Pos }
+    /** A swizzle, or the shape half of `sdf.circle`. Which one is decided later. */
+    | { k: "member"; obj: Expr; name: string; pos: Pos }
+    | { k: "call"; callee: Expr; args: Expr[]; pos: Pos }
+    | { k: "unary"; op: "-" | "+" | "!"; arg: Expr; pos: Pos }
+    | { k: "binary"; op: BinaryOp; a: Expr; b: Expr; pos: Pos }
+    | { k: "cond"; cond: Expr; then: Expr; else: Expr; pos: Pos }
+
+export type AssignOp = "=" | "+=" | "-=" | "*=" | "/=" | "%="
+
+export type Stmt =
+    | { k: "var"; type: TypeName; name: string; init: Expr; pos: Pos }
+    /** A compile time constant: usable as a value and as a `for` bound. */
+    | { k: "const"; type: TypeName; name: string; init: Expr; pos: Pos }
+    | { k: "assign"; target: Expr; op: AssignOp; value: Expr; pos: Pos }
+    | { k: "if"; cond: Expr; then: Stmt[]; else: Stmt[]; pos: Pos }
+    | { k: "for"; counter: string; from: Expr; to: Expr; inclusive: boolean; step: Expr; body: Stmt[]; pos: Pos }
+    | { k: "return"; value: Expr; pos: Pos }
+
+export interface Param {
+    type: TypeName
+    name: string
+    pos: Pos
+}
+
+export interface FuncDecl {
+    name: string
+    ret: TypeName
+    params: Param[]
+    body: Stmt[]
+    pos: Pos
+    /** Prelude functions are shadowed by a function of the same name in the file. */
+    prelude: boolean
+}
+
+export interface UniformDecl {
+    name: string
+    type: TypeName
+    /** Null when the declaration gave no default; the slot then starts at zero. */
+    init: Expr | null
+    pos: Pos
+}
+
+export interface TextureDecl {
+    name: string
+    pos: Pos
+}
+
+export interface Unit {
+    file: string
+    uniforms: UniformDecl[]
+    textures: TextureDecl[]
+    consts: Extract<Stmt, { k: "const" }>[]
+    funcs: FuncDecl[]
+    /** `float4 main()`. Exactly one per file, and the parser refuses a file without it. */
+    main: FuncDecl | null
+}

package/src/lang/builtins.ts

@@ -0,0 +1,168 @@
+/**
+ * What each builtin name does, keyed by the name `ops.ts` gives it.
+ *
+ * Phase A of `Specs/SL_TEXT.md` 3.4. THE NAMES ARE NOT WRITTEN HERE. Every key
+ * comes from `SL_HLSL`, the one column the EDSL and the parser share, so a new
+ * op cannot arrive with a spelling in one surface and not the other. A contract
+ * test (`lang/surface.test.ts`) walks the same table and refuses an op that has
+ * neither a lowering nor a stated reason to lack one.
+ *
+ * Everything lowers through the EDSL rather than through the Builder. That is
+ * the property the parity test rests on: `sin(x)` in a file and `sl.sin(x)` in
+ * TypeScript are the same call, so they cannot produce different graphs.
+ */
+
+import { SLError } from "../ir"
+import { SLOP, SL_HLSL, SL_UNIMPLEMENTED } from "../ops"
+import * as sl from "../sl"
+import type { Num, Val } from "../sl"
+
+/**
+ * How a builtin consumes its arguments.
+ *
+ * `value` is the ordinary case: every argument is a value and the lowering gets
+ * them as `Num`, so a plain literal stays a literal and broadcasts the way the
+ * EDSL broadcasts it. `special` names the three whose arguments are not values
+ * at all: a texture slot, a shape's constant parameters, a list of colour
+ * stops. Those are lowered in `lower.ts`, where the AST is still in reach.
+ */
+export interface Builtin {
+    op: number
+    kind: "value" | "special"
+    min: number
+    max: number
+    /** Present for `kind: "value"`. */
+    lower?: (args: Num[]) => Val
+}
+
+const name = (op: number): string => {
+    const call = SL_HLSL[op]?.call
+    if (call === undefined) throw new Error(`opcode ${op} has no call spelling in SL_HLSL`)
+    return call
+}
+
+/** A plain number where a value is needed, for the ops whose first operand must be one. */
+const val = (v: Num): Val => (typeof v === "number" ? (sl.float(v) as unknown as Val) : v)
+
+/**
+ * Asserts an operand's width where the two backends would otherwise part ways.
+ *
+ * The VM keeps every value in a float4 register and writes `a.xy` or `a.rgb`
+ * whatever it was handed, so a noise call on a float3 quietly works there. The
+ * generated HLSL passes the real type to `sl_valueNoise(float2)` and either
+ * truncates with a warning or fails to compile. Neither outcome is one an
+ * author should discover after ejecting, so the narrow ops say so here.
+ */
+function widthMustBe(fn: string, v: Num, want: 1 | 2 | 3 | 4, why: string): Val {
+    const got = typeof v === "number" ? 1 : v.width
+    if (got !== want) {
+        throw new SLError(`${fn} takes ${why}, and this is a ${WIDTH_NAME[got]}`)
+    }
+    return val(v)
+}
+
+const WIDTH_NAME: Record<number, string> = { 1: "float", 2: "float2", 3: "float3", 4: "float4" }
+
+/** A colour operand: three components, or four with alpha along for the ride. */
+function colourOperand(fn: string, v: Num): Val {
+    const got = typeof v === "number" ? 1 : v.width
+    if (got < 3) throw new SLError(`${fn} takes a colour, so a float3 or a float4, and this is a ${WIDTH_NAME[got]}`)
+    return val(v)
+}
+
+const table: Record<string, Builtin> = {}
+
+function value(op: number, min: number, max: number, lower: (args: Num[]) => Val): void {
+    table[name(op)] = { op, kind: "value", min, max, lower }
+}
+
+function special(op: number, min: number, max: number): void {
+    table[name(op)] = { op, kind: "special", min, max }
+}
+
+/** Width preserving one argument ops, which the EDSL exposes as free functions. */
+const UNARY: Array<[number, (v: Num) => Val]> = [
+    [SLOP.SIN, (v) => sl.sin(val(v))],
+    [SLOP.COS, (v) => sl.cos(val(v))],
+    [SLOP.TAN, (v) => sl.tan(val(v))],
+    [SLOP.ASIN, (v) => sl.asin(val(v))],
+    [SLOP.ACOS, (v) => sl.acos(val(v))],
+    [SLOP.EXP, (v) => sl.exp(val(v))],
+    [SLOP.LOG, (v) => sl.log(val(v))],
+    [SLOP.SQRT, (v) => sl.sqrt(val(v))],
+    [SLOP.SIGN, (v) => sl.sign(val(v))],
+    [SLOP.CEIL, (v) => sl.ceil(val(v))],
+    [SLOP.ROUND, (v) => sl.round(val(v))],
+    [SLOP.TO_LINEAR, (v) => sl.toLinear(val(v))],
+    [SLOP.ABS, (v) => val(v).abs()],
+    [SLOP.FLOOR, (v) => val(v).floor()],
+    [SLOP.FRACT, (v) => val(v).fract()],
+    [SLOP.SATURATE, (v) => val(v).saturate()],
+    [SLOP.RECIP, (v) => val(v).recip()],
+    [SLOP.NORMALIZE, (v) => val(v).normalize()],
+]
+for (const [op, fn] of UNARY) value(op, 1, 1, (a) => fn(a[0]!))
+
+value(SLOP.POW, 2, 2, (a) => val(a[0]!).pow(a[1]!))
+value(SLOP.MIN, 2, 2, (a) => val(a[0]!).min(a[1]!))
+value(SLOP.MAX, 2, 2, (a) => val(a[0]!).max(a[1]!))
+value(SLOP.CLAMP, 3, 3, (a) => val(a[0]!).clamp(a[1]!, a[2]!))
+value(SLOP.ATAN2, 2, 2, (a) => sl.atan2(a[0]!, a[1]!))
+
+value(SLOP.LENGTH, 1, 1, (a) => val(a[0]!).length())
+value(SLOP.DISTANCE, 2, 2, (a) => val(a[0]!).distance(a[1]!))
+value(SLOP.DOT, 2, 2, (a) => val(a[0]!).dot(a[1]!))
+value(SLOP.CROSS, 2, 2, (a) => sl.cross(val(a[0]!) as never, val(a[1]!) as never))
+value(SLOP.REFLECT, 2, 2, (a) => sl.reflect(val(a[0]!) as never, val(a[1]!) as never))
+
+value(SLOP.MIX, 3, 3, (a) => sl.mix(a[0]!, a[1]!, a[2]!))
+value(SLOP.STEP, 2, 2, (a) => sl.step(a[0]!, a[1]!))
+value(SLOP.SMOOTHSTEP, 3, 3, (a) => sl.smoothstep(a[0]!, a[1]!, a[2]!))
+value(SLOP.REMAP, 5, 5, (a) => sl.remap(a[0]!, a[1]!, a[2]!, a[3]!, a[4]!))
+
+value(SLOP.HSV2RGB, 1, 1, (a) => sl.hsv2rgb(widthMustBe("hsv2rgb", a[0]!, 3, "a float3 of hue, saturation and value") as never))
+value(SLOP.LUMINANCE, 1, 1, (a) => sl.luminance(colourOperand("luminance", a[0]!)))
+
+value(SLOP.NOISE, 1, 1, (a) => sl.noise(point("noise", a[0]!) as never))
+value(SLOP.SIMPLEX, 1, 1, (a) => sl.simplex(point("simplex", a[0]!) as never))
+value(SLOP.VORONOI, 1, 1, (a) => sl.voronoi(point("voronoi", a[0]!) as never))
+// The octave count is an immediate, so it has to be a number at parse time.
+// `lower.ts` folds constants before it gets here and rejects anything else with
+// a message that says why, rather than letting this cast be the error.
+value(SLOP.FBM, 1, 2, (a) => sl.fbm(point("fbm", a[0]!) as never, octaves("fbm", a[1])))
+value(SLOP.TURBULENCE, 1, 2, (a) => sl.turbulence(point("turbulence", a[0]!) as never, octaves("turbulence", a[1])))
+value(SLOP.RIDGED, 1, 2, (a) => sl.ridged(point("ridged", a[0]!) as never, octaves("ridged", a[1])))
+
+/** Every field is sampled at a float2. */
+function point(fn: string, v: Num): Val {
+    return widthMustBe(fn, v, 2, "a float2 to sample at")
+}
+
+special(SLOP.SAMPLE, 2, 2)
+special(SLOP.SDF, 1, 7)
+special(SLOP.RAMP, 3, 64)
+
+function octaves(fn: string, v: Num | undefined): number {
+    if (v === undefined) return 3
+    if (typeof v !== "number") {
+        throw new SLError(
+            `${fn}'s octave count is baked into the instruction, so it has to be a constant. ` +
+            `A uniform or a computed value cannot change how many octaves the shader runs.`,
+        )
+    }
+    return v
+}
+
+export const BUILTINS: Readonly<Record<string, Builtin>> = table
+
+/**
+ * Names that exist in the opcode table and cannot be written yet, with why.
+ *
+ * Separate from "unknown identifier" on purpose: an author who writes
+ * `rgb2hsv` has read the right documentation and deserves the real answer.
+ */
+export const NOT_YET: Readonly<Record<string, string>> = Object.fromEntries(
+    Object.entries(SL_UNIMPLEMENTED)
+        .filter(([op]) => SL_HLSL[Number(op)]?.call !== undefined && table[SL_HLSL[Number(op)]!.call!] === undefined)
+        .map(([op, why]) => [SL_HLSL[Number(op)]!.call!, why]),
+)