volvoxai 0.1.0

package/docs/textbook/06-native-engine-architecture.md

@@ -0,0 +1,338 @@
+# Chapter 6 — The Native Engine (CPU + multi-backend GPU)
+
+*Goal: understand VolvoxAI's **native** side — a freestanding C program that runs the **same
+blueprint** as the browser, on desktops and phones, across CPU and several GPU/NPU backends —
+and the design ideas that make that possible.*
+
+Chapters 1–5 mostly read the JavaScript tiers because they're the clearest teaching text. But
+that's only half of VolvoxAI. The other half is `native/`: a **bare-metal C engine** that takes
+the *identical* `config.json` + `.safetensors` and runs it with no browser, no Node, and — this
+is the striking part — **no statically linked GPU SDK**. This chapter is the architecture and the
+"why" behind it.
+
+---
+
+## 6.1 The core idea: one blueprint, two worlds
+
+The whole project is organized around a single principle:
+
+> **Write the model once as a blueprint; run it anywhere.**
+
+```mermaid
+flowchart TD
+    BP["Blueprint<br/>config.json + model.safetensors"]:::bp
+    BP --> WEB[Browser world · JS/WASM/WGSL]
+    BP --> NAT[Native world · freestanding C]
+    WEB --> W1[WebNN]
+    WEB --> W2[WebGPU]
+    WEB --> W3[WASM SIMD]
+    WEB --> W4[Pure JS]
+    NAT --> N1[CPU · AVX2 / NEON]
+    NAT --> N2[Vulkan]
+    NAT --> N3[OpenGL / GLES]
+    NAT --> N4[Metal]
+    NAT --> N5[Android NNAPI]
+    classDef bp fill:#eef,stroke:#66a;
+```
+
+The browser world was Chapter 5. The native world is a standalone executable (`native/volvoxai`)
+you run from a terminal:
+
+```bash
+./native/volvoxai generate models/tinystories_1m --prompt "Once upon a time, Lily" --max-new 50
+./native/volvoxai detect   models/efficientdet_lite0_int8 --image input0=photo.png \
+                           --image-normalize raw-255 --boxes boxes --scores scores
+```
+
+Same files the browser loads. That symmetry is the design.
+
+---
+
+## 6.2 The freestanding philosophy (why it's unusual)
+
+Two deliberate constraints shape the native engine:
+
+1. **No Emscripten / no heavy runtime.** The WASM module is built with plain
+   `clang --target=wasm32 -msimd128` — a *freestanding* build with `--no-entry`, no libc runtime.
+   The native binary is ordinary `clang -O3 -mavx2 -mfma -pthread`. There is no framework
+   underneath; the engine *is* the code in `native/`.
+2. **No static GPU dependency.** The binary does **not** link `libvulkan` or an OpenGL SDK at
+   build time. Instead it **`dlopen`s** the GPU driver *at runtime* and looks up entry points by
+   name. If the driver is present, you get GPU acceleration; if not, the exact same binary runs on
+   CPU. One artifact, portable across machines with wildly different GPU stacks.
+
+Here is that runtime loading, verbatim (`native/vulkan_engine.c`, `native/opengl_engine.c`):
+
+```c
+// Vulkan: try the platform's loader names, in order, at runtime.
+const char* names[] = { "libvulkan.so.1", "libvulkan.so", "vulkan-1.dll" };
+for (i = 0; i < 3; i++) vulkan_lib = dlopen(names[i], RTLD_NOW | RTLD_LOCAL);
+vkGetInstanceProcAddr = (PFN_vkGetInstanceProcAddr)dlsym(vulkan_lib, "vkGetInstanceProcAddr");
+
+// OpenGL/GLES: likewise dlopen libEGL + libGLESv2/libGL (or *.dll on Windows).
+```
+
+That's the whole trick behind "runs on Vulkan/OpenGL/Metal/NNAPI with no GPU SDK linkage." The
+`-ldl` in the build command is the only price.
+
+---
+
+## 6.3 One kernel source, two machines (the shared ABI)
+
+VolvoxAI avoids maintaining two copies of every kernel. The portable C in `native/kernels/*.c`
+(bundled by `native/kernels.c`) compiles **both** to WebAssembly (Tier 3 in the browser) **and**
+to the native binary. The trick is a **`uintptr_t` heap-pointer ABI**: kernels address memory
+through integer offsets into one flat heap, so the same source works whether that heap is WASM
+linear memory or a native `malloc` arena. The compiler then auto-vectorizes it to **AVX2 on x86**
+and **NEON on arm64**.
+
+```
+                       ┌─────────────────────────────┐
+   native/kernels/*.c  │  portable C, uintptr_t heap │
+                       └───────────┬─────────────────┘
+             clang --target=wasm32 │ clang -O3 -mavx2 (x86) / -march=…(arm64)
+                    ┌──────────────┴───────────────┐
+              volvoxai.wasm                   native/volvoxai
+             (browser Tier 3)              (desktop/Android CPU)
+```
+
+The pure-JS ops (`js/ops/*.js`) remain the reference these are validated against — so there are
+really **three** expressions of each op (JS reference, portable C, and — for hot ops — an
+*optimized* C kernel), all required to agree.
+
+---
+
+## 6.4 The engine lifecycle (`native/engine.c`)
+
+The native engine is a small, explicit state machine. Its public API (`native/engine.h`) is the C
+counterpart of the JS `compile()` / `execute()`:
+
+```c
+int    engine_init(const char* config_path, const char* weights_path);  // load + build ONCE
+float* engine_input_ptr(const char* name, long* numel);                 // poke an input tensor
+int    engine_forward(void);                                            // run the whole graph
+const float* engine_last_logits(int* count);                            // read the output row
+void   engine_free_ctx(void);                                           // tear down
+// autoregressive helpers:
+int    engine_prefill(int n_tokens);   // process a prompt, fill K/V caches
+int    engine_decode(int pos);         // process one new token via the cache
+```
+
+`engine_init` does the one-time heavy lifting (`native/engine.c`):
+
+```c
+int engine_init(const char* config_path, const char* weights_path) {
+    load_weights(weights_path);        // mmap/parse the .safetensors blob
+    build_graph(config_path);          // parse config.json → g_t[] tensors, g_n[] nodes
+    prepack_qconv_weights();           // re-lay-out int8 conv weights for the fast kernel (§5.3)
+    prepack_conv_weights();            // re-lay-out fp32 conv weights (im2col/GEMM order)
+    g_loaded = 1;
+}
+```
+
+Then `engine_forward` is the familiar loop — walk the node list, dispatch each:
+
+```c
+for (int i = 0; i < g_nn; i++)
+    run_node(&g_n[i], i, /*is_last=*/ i == g_nn - 1);
+```
+
+The graph is built **once**; forwards are cheap and repeatable. This is what lets `generate` run
+hundreds of forward passes without reloading weights.
+
+---
+
+## 6.5 Per-node backend selection (`native/engine_runtime.c`)
+
+`run_node` is where "multi-backend" actually happens. For each node it decides *who computes it*,
+and the decision is per-node, not per-model:
+
+```mermaid
+flowchart TD
+    N[node i] --> Q{GPU enabled?<br/>--vulkan / --opengl}
+    Q -- no --> CPU
+    Q -- yes --> AR{autoregressive<br/>decode step?}
+    AR -- yes --> CPU[CPU kernel<br/>conv_f32_opt / quant_cpu_opt / kernels.c]
+    AR -- no --> SUP{op supported on<br/>GPU graph & FP32?}
+    SUP -- yes --> GPU[GPU graph node<br/>vk_graph_* / opengl_*]
+    SUP -- no --> CPU
+```
+
+Key rules baked into the dispatcher:
+
+- **GPU is opt-in** via CLI flags (`--vulkan`, `--opengl`, `--nnapi`); default is CPU. If the
+  driver can't load, it prints e.g. `Backend: CPU (Vulkan unavailable)` and continues.
+- **GPU backends are FP32-only.** `QConv2D` (int8) nodes always stay on the CPU's quantized
+  island (`quant_cpu_opt.c`) even with `--vulkan` — so the int8 detector runs its convs on CPU and
+  only FP32 ops offload.
+- **Generation mostly stays on CPU.** The Vulkan/OpenGL graph path is skipped during
+  `engine_decode`/`engine_prefill`, because token-by-token decode is latency-bound. Large
+  MatMul/Gemm/Linear nodes can still use one-shot Vulkan/OpenGL offload when the work is big
+  enough; small decode-time MatMuls stay on CPU to avoid dispatch overhead.
+- Every node records which backend ran it (`"vulkan-graph"`, `"cpu-qconv"`, …) for the `--debug`
+  profile report.
+
+---
+
+## 6.6 The GPU path is a deferred command graph
+
+The native GPU backends don't execute op-by-op with a CPU round-trip each time. Like the WebGPU
+tier, they **build a command graph and replay it**, keeping data resident on the device. The
+`vk_graph_*` interface (`native/vulkan_engine.h`) shows the shape of it:
+
+```c
+vk_graph_begin_forward();                        // start recording
+vk_graph_conv2d_f32(in, out, w, b, …);           // record a conv
+vk_graph_add_relu_f32(a, b, out, n, relu);       // record a fused add+relu
+vk_graph_maxpool2d_f32(…); vk_graph_resize_nearest_f32(…);
+vk_graph_layernorm_f32(…); vk_graph_gelu_f32(…); vk_graph_softmax_f32(…);
+vk_graph_end_forward();                          // submit + wait once
+```
+
+Two supporting ideas make this correct:
+
+- **Host/device sync tracking.** `vk_graph_mark_host()` / `vk_graph_sync_host()` track which
+  buffers the CPU touched, so data is uploaded/downloaded only when actually needed — not every
+  node. Weights upload once; activations stay on the GPU between nodes.
+- **Fusion carries over.** The dispatcher records fused nodes (`add+relu`, `concat+sigmoid`) as
+  single GPU ops, so the graph-level fusion pass (§6.8) pays off on the GPU too.
+
+The OpenGL/GLES backend mirrors this API (`opengl_graph_*`). Android **NNAPI**
+(`nnapi_engine.c`) has its own selection branch for large dense layers. Metal
+(`metal_engine.m`) has Apple-only graph dispatch for selected F32 ops such as attention,
+Conv1D, elementwise Mul/Sub/Div, Split, DequantizeLinear, NMS, and custom profile ops. The
+exact native GPU op matrix lives in
+[`docs/operation_list.md`](../operation_list.md).
+
+---
+
+## 6.7 The shader pipeline (WGSL is the single source)
+
+You might expect the native GPU backends to need hand-written Vulkan/Metal/GLSL shaders. They
+don't — VolvoxAI keeps **WGSL as the one shader language** and *cross-compiles* it. `make
+compile_shaders` runs `tools/compile_shaders.sh`, which uses Mozilla's **`naga`** to translate
+every `shaders/*.wgsl` into the formats each native backend wants:
+
+```
+shaders/*.wgsl ──naga──▶ native/shaders/spv/   (SPIR-V  → Vulkan)
+                        native/shaders/glsl/  (GLSL    → desktop OpenGL)
+                        native/shaders/gles/  (GLSL ES → Android/embedded)
+                        native/shaders/metal/ (MSL     → Apple Metal)
+```
+
+Write a kernel's shader once in WGSL, then translate it to the native shader formats. Runtime
+support still needs a backend wrapper and dispatcher call: today Vulkan/OpenGL wire selected
+generated shaders, and Metal wires a smaller Apple-only subset through `metal_graph_*`.
+That's the same "one source, many targets" discipline as the C kernels (§6.3), applied to shaders,
+with wiring tracked separately from generation.
+
+---
+
+## 6.8 Compile-time operator fusion (`native/graph_opt_fusion.c`)
+
+Before the first forward, the native engine runs a fusion pass over the parsed graph (the design
+from Chapter 5, here at the C level). It rewrites the node list in place — flagging
+`fuse_relu6`, marking `skip` on elided nodes, tagging `concat_sigmoid_fuse`:
+
+- **Conv + ReLU6** → clamp inside the conv's write (`fuse_relu6`).
+- **Chained `Add`** → collapse sequential residual adds.
+- **Depthwise → Pointwise** → run the MBConv pair without spilling the middle tensor.
+- **Concat + Sigmoid** → fuse the detector's class-head tail.
+- **Alias elision** → drop no-op `Reshape`/copy nodes (`skip`).
+
+Fewer nodes, fewer full passes over big feature maps — the single biggest lever after SIMD.
+
+---
+
+## 6.9 Task runtimes: from tensors to usefulness
+
+`native/main.c` is a CLI that wraps the tensor engine into real tasks. The dispatcher is a plain
+`switch` on `argv[1]`:
+
+| Command | What it does | Extra machinery |
+|---|---|---|
+| `run` | Raw graph runner: feed input tensors/images, dump output tensors | `image_io.c` (stb_image PNG/JPEG → NHWC) |
+| `generate` | Autoregressive text (TinyStories) | `tokenizer.c` (BPE) + prefill/decode + KV-cache |
+| `classify` | Top-K image classification | argmax + `labels.txt` |
+| `detect` | Object detection → ranked boxes | anchor scoring + labels |
+| `ctc` | CTC sequence decoding (e.g. OCR) | CTC collapse |
+| `seq2seq` / `chat` | Encoder–decoder / chat loops | cross-attention runtime |
+
+Two supporting runtimes deserve a mention: **`tokenizer.c`** (a from-scratch byte-level BPE
+tokenizer reading the same `vocab.bin` + `merges.txt` as `js/Tokenizer.js`) and
+**`kie_runtime.c`** (a receipt key-information-extraction task). Everything sits on the one
+`engine_forward` core.
+
+---
+
+## 6.10 KV-cache orchestration (native generation)
+
+Chapter 2 introduced the KV-cache conceptually; the native engine is where it's implemented. Each
+attention node owns a Key and Value cache (`g_kcache[i]`, `g_vcache[i]` in
+`native/engine_internal.h`). Generation splits into two phases:
+
+```
+engine_prefill(n_tokens):   run the graph over the whole prompt once, filling every K/V cache
+loop:
+  engine_last_logits() ─▶ argmax ─▶ next token
+  engine_decode(pos):       run the graph for ONE new position, reading cached K/V,
+                            appending this token's K/V   (O(seq) work, not O(seq²))
+```
+
+This is exactly the prefill/decode split that production LLM servers use — implemented in a few
+hundred lines of C here, which makes it unusually readable.
+
+---
+
+## 6.11 Building the native engine
+
+One `clang` line builds the whole thing (from the `Makefile`):
+
+```bash
+clang -O3 -mavx2 -mfma -pthread -Inative \
+  native/cJSON.c native/safetensors.c native/kernels.c \
+  native/quant_cpu_opt.c native/conv_f32_opt.c native/tensor_f32_opt.c \
+  native/engine_runtime.c native/engine.c native/image_io.c native/kie_runtime.c \
+  native/vulkan_engine.c native/opengl_engine.c native/tokenizer.c native/nnapi_engine.c \
+  native/main.c -o native/volvoxai -lm -ldl
+```
+
+- `make build_native` — desktop build (CPU + Vulkan/OpenGL via runtime `dlopen`).
+- `make build_android` — adds `-DUSE_NNAPI` and links `nnapi_engine.c` for Android arm64.
+- `make compile_shaders` — regenerate SPIR-V/GLSL/GLES/Metal from WGSL.
+
+Notice there is **no** `-lvulkan` / `-lGL`: the only GPU-related flag is `-ldl`. That single
+absence is the whole "no static GPU dependency" promise, made concrete.
+
+---
+
+## 6.12 The design, in one picture
+
+```
+                     ┌───────────────────────── native/volvoxai ─────────────────────────┐
+  config.json ──▶  build_graph ──▶ fusion pass ──▶ prepack weights ──▶ engine_forward loop │
+  .safetensors ─▶  load_weights                                          │                  │
+                                                              per node: run_node()          │
+                                                              ├─ CPU: conv_f32_opt /         │
+                                                              │       quant_cpu_opt /        │
+                                                              │       kernels.c (AVX2/NEON)  │
+                                                              └─ GPU/NPU (dlopen'd):          │
+                                                                 vulkan / opengl / nnapi     │
+                                                                 metal on Apple platforms    │
+                     └────────────────────────────────────────────────────────────────────┘
+                          task wrappers: run · generate · classify · detect · ctc · seq2seq · chat
+```
+
+**The takeaways:**
+
+- VolvoxAI is **dual-target by design**: one blueprint feeds both the browser tiers and a
+  freestanding native binary.
+- The native engine is **portable without being generic**: no Emscripten, no static GPU SDK —
+  GPU drivers are `dlopen`'d at runtime, so one binary spans very different machines.
+- **Reuse is enforced across three axes**: one C kernel source (WASM + native), one shader
+  language (WGSL → generated native shader formats, wired per backend), one blueprint
+  (all backends) — with the pure-JS reference as the correctness oracle.
+- Everything still reduces to the Chapter 1 loop: **walk the graph, dispatch each node to the best
+  available backend.** Native just adds more backends and sharper kernels.
+
+**Next:** [Chapter 7 — Glossary & Next Steps →](07-glossary-and-next-steps.md)

package/docs/textbook/07-glossary-and-next-steps.md

@@ -0,0 +1,258 @@
+# Chapter 7 — Glossary & Next Steps
+
+*Goal: one place for every term, a concrete path through this repo, and exercises that turn
+reading into skill.*
+
+---
+
+## 7.1 Glossary
+
+**Activation** — a tensor of intermediate values flowing between ops (as opposed to a *weight*).
+Kept in fp32 by VolvoxAI (int8 on the native quantized path).
+
+**Anchor** — a fixed reference box (a prior) that a detector adjusts, instead of predicting a box
+from scratch. EfficientDet-Lite0 uses 9 per grid cell → 19,206 total.
+
+**Attention (SDPA)** — the transformer mechanism where each token compares its **Query** to every
+token's **Key** and blends their **Values** by similarity. "Which earlier words matter to me?"
+
+**Autoregressive** — generating a sequence one token at a time, feeding each output back as input.
+
+**Backbone** — the feature-extractor stage of a vision model (here, EfficientNet-Lite0).
+
+**Backend** — a concrete executor for the graph's ops. Browser backends are the four *tiers*;
+native backends are CPU, Vulkan, OpenGL/GLES, Metal, and NNAPI. VolvoxAI picks one per node.
+
+**BiFPN** — Bi-directional Feature Pyramid Network; fuses features across resolutions with
+resize/pool/add so every scale has both detail and meaning.
+
+**BPE (Byte-Pair Encoding)** — the tokenizer algorithm: start from bytes, repeatedly merge the
+most frequent adjacent pair, per a learned merge list.
+
+**Broadcast** — stretching a smaller tensor to match a larger one in an element-wise op (e.g.
+adding a per-channel bias `[C]` to `[N,H,W,C]`).
+
+**Causal mask** — restricting attention so position *q* only sees positions `≤ q`; makes a
+left-to-right generator.
+
+**Channel** — one "feature plane" of a tensor (the `C` in NHWC). Input images have 3 (RGB);
+hidden layers have many.
+
+**Convolution (Conv2D)** — slide a small learned filter over an image, dot-product at each spot,
+to detect a pattern everywhere. **Depthwise** = per-channel spatial filter; **Pointwise** = 1×1
+channel mixer; the two together (**depthwise-separable**) are cheap and power the backbone.
+
+**Dequantize** — convert int8 back to float: `r = (q − zero_point) × scale`.
+
+**dlopen / dlsym** — load a shared library and look up its functions *at runtime* (not link
+time). How VolvoxAI's native binary uses a GPU driver (`libvulkan`, `libGL`) without linking any
+GPU SDK — the "no static GPU dependency" design.
+
+**Embedding** — a learned vector that represents a discrete token; the `Embedding` op is a table
+lookup.
+
+**Forward pass / Inference** — running the graph once, input → output. VolvoxAI does only this
+(no training).
+
+**Fusion** — merging adjacent ops (e.g. Conv+ReLU) so intermediate data is written once.
+
+**GELU / ReLU / ReLU6 / Sigmoid** — nonlinearities; the "decision curves" between linear layers.
+Without them, stacked MatMuls collapse into one.
+
+**GEMM** — GEneral Matrix Multiply; the heavily-optimized routine most fast conv/matmul paths
+reduce to.
+
+**Graph** — the model's op list: nodes (ops) connected by named tensors. Stored as `config.json`.
+
+**Head** — the final task-specific layer(s): the LM head (→ vocabulary logits) or the detector's
+class/box heads.
+
+**im2col** — "image to columns"; unfold conv input patches into a matrix so a conv becomes a GEMM.
+
+**int8 / fp16 / fp32** — 8-bit integer / 16-bit float / 32-bit float number formats (1 / 2 / 4
+bytes). See Chapter 4.
+
+**KV-cache** — caching past tokens' Keys and Values so each generation step only computes the new
+token's attention. In this repo: `engine_prefill` + `engine_decode`.
+
+**LayerNorm / RMSNorm** — normalize a vector (mean 0, variance 1, then learned scale/shift) to
+keep deep-network numbers stable.
+
+**Logits** — raw, un-normalized scores (pre-softmax/sigmoid). The LM head and class head emit
+these.
+
+**MatMul** — matrix multiply; the core feature-mixing op of transformers.
+
+**MBConv** — Mobile inverted BOTTLENECK conv block: expand → depthwise → project, with a residual.
+The backbone's repeating unit.
+
+**NHWC / NCHW** — tensor dimension order (batch, height, width, channels) vs (batch, channels,
+height, width). VolvoxAI vision models use NHWC.
+
+**naga** — the Rust tool VolvoxAI uses to cross-compile one WGSL shader into SPIR-V (Vulkan),
+GLSL (OpenGL), GLSL ES, and MSL (Metal). Generated shader output is not the same as runtime
+support; the native dispatcher must wire a backend wrapper for an op to run there.
+
+**NMS (Non-Max Suppression)** — postprocess that removes overlapping duplicate detections,
+keeping the highest-scoring box per object.
+
+**NNAPI** — Android's Neural Networks API; VolvoxAI's native engine can dispatch to it on Android
+(`native/nnapi_engine.c`, `make build_android`).
+
+**Node** — one entry in the graph: an op plus its input/output tensor names and parameters.
+
+**Op / Operation / Kernel** — a single math routine (Add, Conv2D, SDPA…). "Op" is the graph-level
+name; "kernel" is a specific implementation of it.
+
+**Quantization** — representing weights/activations with fewer bits via `scale` + `zero_point`.
+
+**Residual (skip connection)** — adding a block's input to its output (`out = x + f(x)`) so
+information and gradients survive deep stacks. In both models.
+
+**Safetensors** — the standard binary file format for the weights.
+
+**Scale / Zero-point** — the two numbers of a quantization recipe: tick size, and which integer
+means real 0.
+
+**Softmax** — turns a vector of scores into a probability distribution (positive, sums to 1).
+
+**SPIR-V** — the binary shader format Vulkan consumes; `naga` compiles VolvoxAI's WGSL to it.
+
+**Prefill / Decode** — the two phases of native text generation: *prefill* runs the prompt once
+to fill the KV-cache; *decode* runs one new token at a time using the cache. See `engine_prefill`
+/ `engine_decode`.
+
+**Tensor** — a multi-dimensional array of numbers with a shape; the only data type in the engine.
+
+**Tier** — one of VolvoxAI's four browser backends (WebNN / WebGPU / WASM / Pure-JS), picked by
+capability.
+
+**Token** — a chunk of text (word/sub-word/byte) mapped to an integer id.
+
+**Weight** — a number learned during training, read-only at inference.
+
+---
+
+## 7.2 A path through this repository
+
+Read in this order to go from "I get the concepts" to "I can modify the engine":
+
+1. **The data model** — `js/Tensor.js` (25 lines), `js/Graph.js` (48 lines). Tiny; read fully.
+2. **The executor** — `js/CPUEngine.js`. See the `for (node of graph.nodes)` loop and the
+   `switch` dispatch. This is the whole runtime.
+3. **Four naive kernels** — `js/ops/add.js`, `embedding.js`, `layerNorm.js`, `matMul.js`. Each is
+   a few dozen readable lines.
+4. **The two models' blueprints** — skim `models/tinystories_1m/config.json` and
+   `models/efficientdet_lite0_fp32/config.json`. Match nodes to Chapters 2–3.
+5. **The attention + conv kernels** — `js/ops/sDPA.js`, `js/ops/conv2D.js`. The two "hearts."
+6. **Quantization** — `js/ops/dequantizeLinear.js`, then `native/quant_cpu_opt.c` for the real
+   int8 conv.
+7. **Optimization** — diff `js/ops/conv2D.js` against `native/conv_f32_opt.c` while reading
+   `docs/microkernel_optimization_guide.md` and `docs/xnnpack_optimization_guide.md`.
+8. **The GPU tier** — `shaders/*.wgsl` (e.g. `matmul`), and `js/GraphExecutor.js`.
+9. **The native engine** (Chapter 6) — `native/engine.h` + `native/engine.c` (lifecycle),
+   `native/engine_runtime.c` (`run_node` backend selection), then a device backend such as
+   `native/vulkan_engine.c` (see the `dlopen` at the top). `native/main.c` holds the task CLIs.
+
+`docs/operation_list.md` is the per-op × per-backend support matrix — your reference map.
+
+---
+
+## 7.3 Run the models yourself
+
+```bash
+# Language model — generate text (greedy). Builds native binary first: `make build_native`.
+./native/volvoxai generate models/tinystories_1m \
+  --prompt "Once upon a time, Lily" --max-new 50 [--debug]
+
+# Raw graph runner — dump the logits tensor for a fixed set of tokens.
+./native/volvoxai run models/tinystories_1m \
+  --input tokens=models/tinystories_1m/tokens.i32 \
+  --input positions=models/tinystories_1m/positions.i32 \
+  --output logits=out.f32 --last-token 4
+
+# Object detector — decode an image into ranked boxes.
+./native/volvoxai detect models/efficientdet_lite0_int8 \
+  --image input0=photo.png --image-normalize raw-255 \
+  --boxes boxes --scores scores --max-det 20
+
+# In Node (WASM / pure-JS tiers), smoke-test any blueprint:
+node bin/volvox.js run --model models/tinystories_1m/model.safetensors --backend wasm
+```
+
+Add `--debug` to `generate` to see per-node timing and tokens/sec — a great way to *feel* where
+time goes (and to watch the optimizations from Chapter 5 pay off).
+
+---
+
+## 7.4 Exercises (reading → skill)
+
+1. **Trace by hand.** Take the sequence `[5, 5]` (two identical tokens) and a made-up 2-dim
+   embedding. Walk `Embedding → Add(position) → LayerNorm` with pen and paper. Confirm the shapes
+   match `config.json`.
+2. **Break causality.** In `js/ops/sDPA.js`, change `k <= q` to `k < seq_len`. Predict what
+   happens to generated text and why. (Then revert.)
+3. **Quantize a weight.** Pick `scale = 0.02`, `zero_point = -5`. Quantize `r = 0.31`, then
+   dequantize it back. Report the round-trip error. Now try `scale = 0.002`. What did precision
+   cost you in range?
+4. **Count the FLOPs.** For the first `Conv2D` (stem: 320×320×3 → 160×160×32, 3×3 filter),
+   estimate the multiply-adds. Compare to a 1×1 pointwise conv of the same output size. Why is
+   depthwise-separable cheaper?
+5. **Add an op.** Implement an element-wise `Abs` kernel in `js/ops/`, wire it into
+   `CPUEngine.js`'s `switch`, and confirm it dispatches. (Follow `js/ops/reLU.js` as a template.)
+6. **Find a fusion.** In `models/efficientdet_lite0_fp32/config.json`, find a `Conv2D` whose
+   `relu` param is set — that's a Conv+ReLU fusion already baked in. Explain what two ops it
+   represents.
+
+---
+
+## 7.5 Current gaps in this codebase
+
+VolvoxAI is an inference engine, and this textbook follows that boundary. The codebase can load
+trained weights and run forward passes; it does not yet include the systems needed to create,
+tune, or scientifically validate models.
+
+| Missing area | What would need to be added | Why it matters |
+|---|---|---|
+| **Training** | Reverse-mode autodiff, backward kernels, loss functions, optimizers such as Adam/SGD, learning-rate schedules, initialization, checkpointing, and regularization. | This is how weights are discovered instead of only consumed. |
+| **Math foundations** | Linear algebra derivations, calculus for chain rule and gradients, probability, entropy/cross-entropy, KL divergence, and likelihood. | These are the tools for explaining why training and evaluation behave the way they do. |
+| **Data** | Dataset manifests, streaming/input pipelines, augmentation, cleaning, tokenizer training, train/validation/test splits, and leakage checks. | Model quality is usually bounded by data quality and experimental hygiene. |
+| **Evaluation & experimentation** | Task metrics, validation loops, baselines, ablations, hyperparameter sweeps, overfitting checks, and bias-variance analysis. | This is how you know a model is actually better, not just different. |
+| **Architecture breadth** | Diffusion models, graph neural networks, RNN/LSTM, reinforcement learning, VAE/GAN, retrieval and embedding systems, multimodal models, mixture-of-experts, and state-space models. | The current walkthrough covers one transformer LM and one CNN detector; many domains use different inductive biases. |
+| **Modern LLM training stack** | Pretraining loops, supervised fine-tuning, LoRA/adapters, RLHF/DPO preference training, distributed data/model parallelism, and FlashAttention internals. | Most frontier LLM work happens around training recipes, memory-efficient attention, and large-scale systems. |
+| **Research practice** | Paper reproduction, result derivation, controlled experiments, scaling-law analysis, error analysis, and documentation of assumptions. | This is the difference between running a model and producing reliable new knowledge. |
+
+These are future textbook/code modules, not prerequisites for the inference chapters. Listing
+them makes the scope explicit: this repo is a strong inference/runtime foundation, not a complete
+training-and-research curriculum.
+
+---
+
+## 7.6 Where to go from here
+
+The natural next steps split into two tracks:
+
+- **Deepen this repo's inference path.** Compare `native/conv_f32_opt.c` to Google's **XNNPACK**;
+  `docs/xnnpack_optimization_guide.md` in this repo is a guided tour. Then inspect
+  `shaders/*.wgsl` and the native GPU backends.
+- **Scale the transformer.** GPT-2/3, LLaMA, Mistral, Qwen are Chapter 2's graph, wider/deeper,
+  with tweaks: **RMSNorm** instead of LayerNorm, **RoPE** rotary positions instead of learned
+  `wpe`, **grouped-query attention**, **SwiGLU** MLPs. Each is a small variation on ops you know.
+- **Broaden architectures.** Classification, segmentation, pose, diffusion, retrieval,
+  multimodal, MoE, and SSM systems all reuse the tensor/graph mental model, but add different
+  blocks and training objectives.
+- **Close the training gap deliberately.** A small autograd engine, a cross-entropy loss, Adam,
+  a tiny dataset loader, and a validation loop would be the first concrete additions.
+- **Read the source papers** once the mechanics are concrete: *Attention Is All You Need*
+  (transformers), *EfficientDet* (this detector), *EfficientNet* (the backbone), and a
+  quantization primer (e.g. the "gemmlowp"/TFLite integer-quantization write-ups).
+- **Explore peer runtimes** from the landscape: **wonnx** (WebGPU/ONNX), **ncnn** (no-deps
+  native), **ggml/llama.cpp** (portable C LLM inference).
+
+The mental model you built here — *a model is a graph of small tensor ops with learned weights;
+inference walks the graph; performance is memory layout; precision is a size/accuracy dial* —
+transfers to those future modules, but it is one part of the full stack.
+
+---
+
+*End of the VolvoxAI textbook. Back to the [index](README.md).*