volvoxai 0.1.0

package/docs/microkernel_optimization_guide.md

@@ -0,0 +1,153 @@
+# Microkernel Optimization Guide
+
+This document outlines the architectural roadmap and concrete code-level strategies for
+raising VolvoxAI's native CPU inference performance across INT4, INT8, FP16, and FP32 data
+types.
+
+## 1. Architectural Shift
+
+VolvoxAI should continue moving from generic `Conv2D` and `QConv2D` loops toward a
+microkernel architecture:
+
+1. **NHWC layout**: keep activations in NHWC format so channels are contiguous in memory.
+2. **Indirection buffers**: remove padding and bounds checks from inner loops.
+3. **Weight packing**: arrange weights to match the target SIMD register width and kernel
+   tile shape.
+4. **Register tiling**: hardcode loop unrolling around the target register file, such as
+   computing multiple pixels and output channels per kernel call.
+
+## 2. Indirection Buffers
+
+Branching inside the MAC loop for padding stalls the CPU pipeline. Precompute an array of
+input pointers instead. Padded areas point to a zero buffer, so the microkernel can do
+pointer dereferences and math without bounds checks.
+
+```c
+static const int8_t zero_buffer[MAX_CHANNELS] = {0};
+
+const int8_t** indirection_buffer =
+    malloc(output_height * output_width * kernel_size * sizeof(int8_t*));
+int idx = 0;
+for (int oy = 0; oy < output_height; oy++) {
+    for (int ox = 0; ox < output_width; ox++) {
+        for (int ky = 0; ky < kh; ky++) {
+            for (int kx = 0; kx < kw; kx++) {
+                int iy = oy * stride - pad_top + ky;
+                int ix = ox * stride - pad_left + kx;
+                if (iy >= 0 && iy < input_height && ix >= 0 && ix < input_width) {
+                    indirection_buffer[idx++] =
+                        input_image + (iy * input_width + ix) * channels;
+                } else {
+                    indirection_buffer[idx++] = zero_buffer;
+                }
+            }
+        }
+    }
+}
+```
+
+## 3. Data Type Strategies
+
+### INT8
+
+Use dot-product instructions where available. ARM provides `sdot`, while newer Intel and
+AMD CPUs provide VNNI-family instructions. Pack weights into blocks that align with the
+dot-product width, such as `[OC/8, IC/4, 8, 4]`.
+
+```c
+#include <arm_neon.h>
+
+void microkernel_qconv2d_int8_neon_sdot(
+    const int8_t** input_pointers,
+    const int8_t* packed_weights,
+    int32_t* output_accumulators,
+    int kernel_elements,
+    int channels)
+{
+    int32x4_t acc0 = vld1q_s32(output_accumulators + 0);
+    int32x4_t acc1 = vld1q_s32(output_accumulators + 4);
+
+    for (int k = 0; k < kernel_elements; k++) {
+        const int8_t* in_ptr = input_pointers[k];
+        for (int c = 0; c < channels; c += 4) {
+            int8x8_t in_val = vld1_s8(in_ptr + c);
+            int8x16_t in_dup = vcombine_s8(in_val, in_val);
+            int8x16_t w_val = vld1q_s8(packed_weights);
+            packed_weights += 16;
+            acc0 = vdotq_s32(acc0, in_dup, w_val);
+        }
+    }
+}
+```
+
+### INT4
+
+INT4 cuts memory bandwidth in half compared to INT8, which is useful for bandwidth-bound
+LLM weights or heavily quantized vision models. Most CPUs lack native INT4 MAC
+instructions, so the practical path is to unpack INT4 to INT8 in registers and feed the
+INT8 kernel.
+
+```c
+uint8x16_t w_int4 = vld1q_u8(packed_int4_weights);
+uint8x16_t mask = vdupq_n_u8(0x0F);
+uint8x16_t w_lo = vandq_u8(w_int4, mask);
+uint8x16_t w_hi = vshrq_n_u8(w_int4, 4);
+```
+
+### FP16
+
+Native FP16 execution reduces memory bandwidth and cache footprint on hardware with fast
+half-precision arithmetic, such as ARMv8.2-A FP16 or AVX512-FP16.
+
+```c
+#include <arm_neon.h>
+
+void microkernel_conv2d_fp16_neon(
+    const float16_t* input,
+    const float16_t* packed_weights,
+    float16_t* output)
+{
+    float16x8_t acc = vdupq_n_f16(0.0f);
+    float16x8_t in_val = vdupq_n_f16(input[0]);
+    float16x8_t w_val = vld1q_f16(packed_weights);
+    acc = vfmaq_f16(acc, in_val, w_val);
+    vst1q_f16(output, acc);
+}
+```
+
+### FP32
+
+FP32 should use register tiling to hide latency. On AVX2, one useful pointwise Conv shape is
+six spatial pixels by 16 output channels, using broadcast input values and packed weights.
+
+```c
+#include <immintrin.h>
+
+void microkernel_conv2d_fp32_avx2(
+    const float* in_p0,
+    const float* in_p1,
+    const float* packed_weights,
+    float* out_p0,
+    float* out_p1)
+{
+    __m256 acc0 = _mm256_setzero_ps();
+    __m256 acc1 = _mm256_setzero_ps();
+
+    for (int ic = 0; ic < input_channels; ic++) {
+        __m256 w0 = _mm256_loadu_ps(packed_weights + ic * 8);
+        acc0 = _mm256_fmadd_ps(_mm256_set1_ps(in_p0[ic]), w0, acc0);
+        acc1 = _mm256_fmadd_ps(_mm256_set1_ps(in_p1[ic]), w0, acc1);
+    }
+
+    _mm256_storeu_ps(out_p0, acc0);
+    _mm256_storeu_ps(out_p1, acc1);
+}
+```
+
+## Action Plan
+
+1. Add reusable indirection-buffer builders during graph initialization.
+2. Extend weight-packing caches so each hot op has a layout matched to its microkernel.
+3. Replace generic inner loops in `quant_cpu_opt.c` with CPU-feature-routed microkernels.
+4. Keep the activation arena planner enabled for transient tensors with non-overlapping
+   lifetimes.

package/docs/model-format.md

@@ -0,0 +1,108 @@
+# Model Format
+
+VolvoxAI loads inference packages made of:
+
+```text
+config.json
+model.safetensors
+optional tokenizer/assets, such as vocab.bin, merges.txt, labels.txt
+```
+
+Large `.safetensors` files and generated model directories should stay out of git.
+
+## Graph Model
+
+A model graph contains:
+
+- `Tensor`: name, shape, dtype, and `isWeight`.
+- `Node`: `opType`, named `inputs`, named `outputs`, and `params`.
+- `graph.outputNames`: output tensor names returned by WASM/CPU.
+
+`GraphLoader.js` builds this graph from `config.json` and `model.safetensors`.
+Applications can also construct graphs programmatically with `addInput`,
+`addWeight`, and `addOp`.
+
+## Volvox Blueprint
+
+The primary format is a precomputed Volvox blueprint:
+
+```json
+{
+  "inputs": {
+    "image": { "shape": [1, 320, 320, 3], "dtype": "float32" }
+  },
+  "nodes": [
+    {
+      "opType": "Conv2D",
+      "inputs": { "input": "image", "weight": "stem.weight", "bias": "stem.bias" },
+      "outputs": { "out": "stem.out" },
+      "outputs_shape": { "out": [1, 160, 160, 32] },
+      "params": { "stride": [2, 2], "padding": [1, 1], "weight_layout": "OHWI" }
+    }
+  ]
+}
+```
+
+Exporters map source-model ops to Volvox `opType`s, precompute tensor shapes, and
+write the weights into safetensors. The browser then only maps memory and executes.
+
+## Hugging Face Config Path
+
+If `config.json` has `model_type` but no topological `nodes`, `GraphLoader` checks
+`GraphLoader.ModelBuilders`. A model family can register a builder:
+
+```javascript
+GraphLoader.ModelBuilders.llama = (graph, config, tensors) => {
+  // call graph.addOp(...) to assemble the model
+};
+```
+
+## Safetensors Loading
+
+VolvoxAI reads standard safetensors weights. Tensor metadata provides dtype, shape,
+and byte offsets. Graph topology can also be embedded in safetensors metadata, but
+the normal package layout keeps topology in `config.json`.
+
+## Layout Policy
+
+Runtime image tensors use NHWC:
+
+```text
+[batch, height, width, channels]
+```
+
+Browser Conv2D kernels use HWIO/HWCM internally. Native TFLite exports may keep
+TFLite-native layouts such as `OHWI` for regular conv and `1HWO` for depthwise conv;
+the native engine prepares HWIO/HWCM compute caches at load time.
+
+## Precision Policy
+
+VolvoxAI focuses on FP32 activations/intermediates and INT8-packed or quantized
+weights.
+
+- FP32 is the portable activation format across WebGPU, WASM, CPU, and native paths.
+- INT8 reduces weight bandwidth and storage while avoiding the bit-unpacking cost of
+  INT4 on low-end GPUs.
+- FP16 weights can be kept on disk for exported vision models; runtime paths widen or
+  handle them according to backend support.
+
+INT8 handling:
+
+- MatMul keeps an INT8 plus per-channel-scale fast path in browser shaders and native
+  kernels.
+- Browser Conv2D/Conv1D dequantizes quantized weights to FP32 at load time.
+- Native CPU can keep quantized Conv/Add islands in int8 via `native/quant_cpu_opt.c`.
+- Native Vulkan/OpenGL graph paths are FP32-only, so `QConv2D` stays on CPU.
+
+## LLM Scope
+
+VolvoxAI includes the core ops used by small GPT-style models: `Embedding`,
+`LayerNorm`, `MatMul`, `SDPA`, and `GELU`. The native CLI includes TinyStories
+generation with prefill/decode KV-cache orchestration.
+
+It is not a frontier LLM runtime today:
+
+- No INT4 weight format.
+- Browser text generation currently needs application-side autoregressive loops.
+- Native generation is ahead of browser generation helpers.
+

package/docs/models.md

@@ -0,0 +1,103 @@
+# Models
+
+The `models/` directory is ignored because generated model packages and source
+weights are large. Regenerate examples from public sources when needed.
+
+## Dependencies
+
+Model export is development-only and runs on the host, not in the Docker build image.
+Install exporter dependencies once:
+
+```bash
+make models_deps
+```
+
+This installs `tools/requirements-export.txt`, including numpy, flatbuffers,
+safetensors, torch, and transformers.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `make models` | Download and export EfficientDet Lite0 fp32/fp16/int8 plus TinyStories-1M. |
+| `make models_efficientdet` | Export EfficientDet Lite0 packages from MediaPipe TFLite models. |
+| `make models_tinystories` | Export `roneneldan/TinyStories-1M` plus tokenizer assets. |
+| `make models_clean` | Remove regenerated example model directories. |
+
+`tools/fetch_models.sh` accepts `efficientdet`, `tinystories`, or `all`. For
+EfficientDet, set `ONLY=int8`, `ONLY=float16`, or `ONLY=float32` to fetch one
+precision.
+
+## EfficientDet Lite0
+
+EfficientDet packages contain:
+
+```text
+config.json
+model.safetensors
+labels.txt
+```
+
+The source TFLite models come from Google's MediaPipe model store, for example:
+
+```text
+https://storage.googleapis.com/mediapipe-models/object_detector/efficientdet_lite0/int8/latest/efficientdet_lite0.tflite
+```
+
+The exported package has an NHWC image input and named raw outputs:
+
+```text
+input0: [1, 320, 320, 3] raw RGB 0..255
+scores: [1, 19206, 90]
+boxes:  [1, 19206, 4]
+```
+
+The COCO label file keeps the 90-slot class-id alignment. `???` entries are unused
+placeholders and should not be deleted.
+
+## TinyStories
+
+TinyStories exports:
+
+```text
+config.json
+model.safetensors
+vocab.bin
+merges.txt
+tokens.i32
+positions.i32
+```
+
+The native runtime can use the package directly:
+
+```bash
+./native/volvoxai generate models/tinystories_1m \
+  --prompt "Once upon a time, Lily" \
+  --max-new 50
+```
+
+## TinyReceiptKIE
+
+Custom TinyReceiptKIE checkpoints can be exported into the same package layout:
+
+```bash
+python3 tools/export_kie_safetensors.py \
+  --checkpoint checkpoint.pt \
+  --out models/tiny_receipt_kie
+```
+
+Then run it through the native chat alias:
+
+```bash
+./native/volvoxai chat models/tiny_receipt_kie \
+  --image image=receipt.jpg \
+  --prompt "What is the first number of the store's phone number?" \
+  --family auto \
+  --max-new 128
+```
+
+The package declares the `tiny_receipt_kie` chat interface in `config.json` and keeps
+weights in `model.safetensors`. LoRA is folded into dense linear weights during
+export. If the checkpoint is a base model plus components, add `--lora lora.pt` and
+repeat `--adapter family=adapter_family.pt` for each adapter family.
+

package/docs/native-runtime.md

@@ -0,0 +1,189 @@
+# Native Runtime
+
+`native/` is a freestanding C runtime that runs the same Volvox blueprint package as
+the browser runtime: `config.json` plus `.safetensors`. It has no static GPU SDK
+dependency. Vulkan, OpenGL, Metal, and NNAPI are loaded dynamically when requested.
+
+## Build
+
+```bash
+make build_native
+./native/volvoxai --version
+./native/volvoxai --help
+```
+
+The Docker build compiles the native binary with clang, AVX2/FMA on x86, pthreads,
+and the CPU/GPU backend sources. The binary embeds package version, git commit,
+and UTC build date from the Makefile build.
+
+## Architecture
+
+Key files:
+
+- `engine.c`: parses the blueprint, builds tensors/nodes, owns the global engine context.
+- `engine_runtime.c`: dispatches each node to CPU, Vulkan, OpenGL, Metal, or NNAPI.
+- `graph_opt_fusion.c`: compile-time fusion pass for patterns such as Conv+ReLU6,
+  chained Add, depthwise-to-pointwise, concat+sigmoid, and alias elision.
+- `conv_f32_opt.c`: optimized FP32 Conv2D with weight prepacking and IGEMM.
+- `quant_cpu_opt.c`: INT8 quantized CPU path for `QConv2D`, quantized Add, and
+  quantization helpers.
+- `kernels.c`: shared portable kernels used by both native and wasm32.
+- `image_io.c`: PNG/JPEG decode through `stb_image.h` into NHWC tensors.
+- `tokenizer.c` and `kie_runtime.c`: native BPE tokenizer and receipt KIE task runtime.
+
+## Raw Tensor Runner
+
+```bash
+./native/volvoxai run models/tinystories_1m \
+  --input tokens=models/tinystories_1m/tokens.i32 \
+  --input positions=models/tinystories_1m/positions.i32 \
+  --output logits=out.f32 \
+  --last-token 4 \
+  --debug
+```
+
+Image-shaped inputs can be decoded directly:
+
+```bash
+./native/volvoxai run models/vision_model \
+  --image image=receipt.png \
+  --image-normalize zero-one \
+  --output logits=out.f32
+```
+
+`--image-normalize` supports `zero-one`, `minus-one-one`, and `raw-255`.
+Video, audio, camera, and streaming input should be handled by a frontend that feeds
+decoded tensors to `run`.
+
+## Task Wrappers
+
+Task wrappers sit on top of the tensor runner:
+
+```bash
+./native/volvoxai classify models/classifier \
+  --image image=photo.jpg \
+  --logits logits \
+  --labels labels.txt \
+  --top-k 5
+
+./native/volvoxai detect models/detector \
+  --image image=receipt.jpg \
+  --boxes boxes \
+  --scores scores \
+  --classes classes \
+  --max-det 20
+
+./native/volvoxai ctc models/ocr_line \
+  --image image=line.png \
+  --logits logits \
+  --labels labels.txt \
+  --blank 0
+
+./native/volvoxai seq2seq models/encoder_decoder \
+  --image image=receipt.jpg \
+  --prompt "What is the first number of the store phone?" \
+  --prompt-input q_tokens \
+  --decoder-input y_tokens \
+  --logits logits \
+  --max-new 192
+
+./native/volvoxai chat models/multimodal_chat \
+  --prompt "Summarize this receipt" \
+  --image image=receipt.jpg
+```
+
+`classify` ranks logits, `ctc` performs greedy CTC collapse, and `detect` prints or
+writes named raw tensors because detector output layouts vary by exporter. `seq2seq`
+runs a greedy encoder-decoder loop. `chat` is a user-facing alias for multimodal
+seq2seq packages.
+
+## Generation
+
+TinyStories generation uses the same loaded graph repeatedly:
+
+```bash
+./native/volvoxai generate models/tinystories_1m \
+  --prompt "Once upon a time, Lily" \
+  --max-new 50 \
+  --debug
+```
+
+Weights and graph load once. Generation updates token inputs and reruns the node loop.
+For language models, `--vulkan` uses the MatMul-focused path: large MatMul/Gemm/Linear
+nodes run through the `linearF32` shader, while small decode-time MatMuls stay on CPU
+to avoid dispatch overhead.
+
+## Native GPU and NPU Backends
+
+Driver loading is resolved at runtime:
+
+- Vulkan (`--vulkan`): `libvulkan.so.1`, `libvulkan.so`, or `vulkan-1.dll`.
+- OpenGL (`--opengl`): `libGL.so.1`, `opengl32.dll`, or macOS `OpenGL.framework`.
+- Metal (`--metal`, macOS): default `MTLDevice` through the Objective-C runtime.
+- NNAPI (`--nnapi`, Android build with `-DUSE_NNAPI`): Android Neural Networks API
+  for large MatMul/FullyConnected offload.
+
+If multiple accelerator flags are passed, priority is `--nnapi`, then `--vulkan`,
+then `--metal`, then `--opengl`.
+
+EfficientDet-style vision graphs have graph-resident Vulkan and OpenGL paths for
+fused Conv2D/ReLU6, MaxPool2D, same-size Add, Clip, Sigmoid, 2x nearest upsample,
+Concat, and reshape/identity aliasing. The native GPU graph path is FP32-only;
+quantized `QConv2D` runs on CPU.
+
+On software Vulkan/OpenGL stacks, dispatch synchronization can be slower than the
+AVX2 CPU path. The GPU win is on real mobile or desktop GPUs.
+
+## Benchmark Flags
+
+For TFLite-style timing:
+
+```bash
+./native/volvoxai detect models/efficientdet_lite0_int8 \
+  --image input0=photo.jpg \
+  --image-normalize raw-255 \
+  --boxes boxes \
+  --scores scores \
+  --max-det 5 \
+  --num_threads 4 \
+  --warmup_runs 5 \
+  --num_runs 20
+```
+
+`--num_threads` maps to `VOLVOX_NUM_THREADS`. `--debug` prints graph load, build,
+node backend, and per-op timing logs.
+
+See [efficientdet_tflite_vs_volvoxai.md](efficientdet_tflite_vs_volvoxai.md) for
+the current EfficientDet benchmark methodology and results.
+
+## Android Cross-Compile
+
+```bash
+NDK=$ANDROID_SDK/ndk/<version>
+CC=$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android29-clang
+cd native && "$CC" -O3 -DUSE_NNAPI -pthread -Wno-unknown-attributes -I. \
+  cJSON.c safetensors.c kernels.c quant_cpu_opt.c conv_f32_opt.c tensor_f32_opt.c \
+  engine_runtime.c engine.c image_io.c kie_runtime.c \
+  vulkan_engine.c opengl_engine.c tokenizer.c nnapi_engine.c main.c \
+  -o volvoxai_android -lm -ldl -lneuralnetworks
+```
+
+Then push the binary, model directory, and native shader outputs to the device:
+
+```bash
+adb push volvoxai_android /data/local/tmp/volvoxai
+adb shell "cd /data/local/tmp && VOLVOX_NUM_THREADS=2 ./volvoxai run models/tinystories_1m \
+  --input tokens=models/tinystories_1m/tokens.i32 \
+  --input positions=models/tinystories_1m/positions.i32 \
+  --output logits=out.f32 --vulkan --debug"
+```
+
+ARM/NEON builds default to two CPU threads. Use one thread for stable profiling and
+test two threads for throughput; using all cores is often slower because of big/little
+scheduling and thermal limits.
+
+## Service Runtime
+
+The Rust crate under `runtime/` builds a `libvolvoxai.so` Synurang service wrapper
+around the C engine. See [../runtime/README.md](../runtime/README.md) for the FFI
+service ABI, build commands, and current RPC coverage.