@oriro/orirocli 0.1.0

package/skills/craft/image-generation-engineer/references/architectures.md

@@ -0,0 +1,260 @@
+# Architecture, VAE, and Text Conditioning
+
+Read this when choosing or building a model backbone, a VAE, or a text-conditioning stack.
+
+## Contents
+
+- Architecture evolution (2020→2026) decision table
+- U-Net (original, still useful)
+- DiT (Diffusion Transformer — scalable replacement)
+- MM-DiT (Multimodal DiT — SD3/FLUX)
+- VAE / latent-space compression (why, architecture, perceptual loss)
+- Text conditioning: CLIP & T5, OpenCLIP, dual encoder
+
+---
+
+## Architecture decision table
+
+```
+Year  Architecture         Key Innovation              Use Today?
+2020  DDPM + U-Net         Denoising diffusion         Baseline only
+2022  LDM (Stable Diff)    Latent space + VAE          Yes, still used
+2022  DDIM sampler         50→10 steps                 Always
+2023  DiT                  U-Net → Transformer         Yes (scalable)
+2023  ControlNet           Structural conditioning     Yes
+2024  MM-DiT (SD3)         Text+image streams          Production
+2024  FLUX.1               12B flow matching           Best open 2024
+2025  FLUX.2               Multi-reference, context    Best open 2025
+2026  Z-Image-Turbo        Sub-second, 16GB VRAM       Fastest 2026
+```
+
+## U-Net architecture (original, still useful)
+
+```
+Encoder path: Conv → ResBlock → Attention → Downsample   [64, 128, 256, 512 channels]
+Bottleneck:   ResBlock → Attention → ResBlock
+Decoder path: Upsample → Skip connection → ResBlock → Attention   [512, 256, 128, 64 channels]
+
+Key components:
+  ResNet blocks: handle spatial features at each resolution
+  Self-attention: global coherence at low resolution
+  Cross-attention: where text conditioning enters
+  Time embedding: sinusoidal → MLP → added to every ResBlock
+  Skip connections: preserve spatial info across encoder→decoder
+
+Time embedding:
+  t_emb = sinusoidal_pos_embedding(t, dim=256)
+  t_emb = MLP(t_emb)  # [batch, 512]
+  # Add to each ResBlock output as bias
+
+Text cross-attention:
+  q = linear(spatial_features)  # from image features
+  k,v = linear(text_embeddings) # from CLIP/T5
+  attn = softmax(qkᵀ/√d) @ v
+```
+
+## DiT (Diffusion Transformer — scalable replacement)
+
+```
+Replace U-Net with standard Transformer blocks.
+Patch the image like ViT: image → [N patches] → patch embeddings.
+
+DiT block structure:
+  LayerNorm → Self-attention → LayerNorm → FFN
+  + AdaLN (adaptive layer norm conditioned on timestep+class)
+
+AdaLN conditioning:
+  c = MLP(time_emb + class_emb)
+  scale, shift = chunk(linear(c), 2)
+  x = scale * LayerNorm(x) + shift  # modulate then normalize
+
+Why DiT beats U-Net:
+  - Scales predictably with model size (transformers)
+  - Global attention at all resolutions (not just bottleneck)
+  - Works better with flow matching objectives
+  - Foundation of FLUX, Sora, and all 2024+ SOTA models
+
+DiT model sizes (parameters → FID on ImageNet 256×256):
+  DiT-S: 33M → FID 68.4
+  DiT-B: 130M → FID 43.5
+  DiT-L: 458M → FID 23.3
+  DiT-XL: 675M → FID 2.27 (state of art 2023)
+```
+
+## MM-DiT (Multimodal DiT — SD3, FLUX architecture)
+
+```
+Separate processing streams for image and text tokens.
+Both streams interact through joint attention.
+
+MM-DiT block (double stream):
+  text_tokens: [B, S_text, D] from T5/CLIP
+  image_tokens: [B, S_image, D] from VAE latents
+
+  # Joint attention: text and image tokens attend to each other
+  tokens = concat([text_tokens, image_tokens], dim=1)
+  tokens = self_attention(tokens)
+  text_tokens, image_tokens = split(tokens)
+
+  text_tokens = FFN(text_tokens)
+  image_tokens = FFN(image_tokens)
+
+FLUX hybrid: combines MM-DiT (double stream) + single stream blocks
+  Double stream: process text+image separately, cross-attend
+  Single stream: merge and process together
+  Result: better prompt adherence + visual quality
+
+Scale: FLUX.1 = 12B parameters, 16 latent channels VAE
+```
+
+---
+
+## VAE (latent-space compression)
+
+### Why latent diffusion
+
+```
+Problem: diffusion in pixel space is expensive.
+  512×512 RGB image = 786,432 dimensions
+  1000 denoising steps × 786K dims = prohibitive
+
+Solution: compress to latent space first.
+  VAE encoder: 512×512×3 → 64×64×4 (8× compression)
+  Diffuse in 64×64×4 space (196× cheaper)
+  VAE decoder: 64×64×4 → 512×512×3
+
+FLUX improvement: 16 latent channels (not 4)
+  More information preserved → better quality
+  64×64×16 still much cheaper than pixels
+```
+
+### VAE architecture
+
+```python
+class VAEEncoder(nn.Module):
+    def __init__(self, in_channels=3, latent_dim=4):
+        # Downsample path: 512 → 64 (8× reduction)
+        self.layers = nn.Sequential(
+            # Each block: Conv(stride=2) → ResBlock → ResBlock
+            DownBlock(in_channels, 128),   # 512→256
+            DownBlock(128, 256),            # 256→128
+            DownBlock(256, 512),            # 128→64
+            ResBlock(512),                  # 64
+            AttentionBlock(512),            # global attention
+            ResBlock(512),
+        )
+        self.to_mean = nn.Conv2d(512, latent_dim, 1)
+        self.to_logvar = nn.Conv2d(512, latent_dim, 1)
+
+    def forward(self, x):
+        h = self.layers(x)
+        mu = self.to_mean(h)
+        logvar = self.to_logvar(h)
+        return mu, logvar
+
+class VAEDecoder(nn.Module):
+    def __init__(self, latent_dim=4, out_channels=3):
+        self.layers = nn.Sequential(
+            nn.Conv2d(latent_dim, 512, 1),
+            ResBlock(512),
+            AttentionBlock(512),
+            ResBlock(512),
+            UpBlock(512, 256),   # 64→128
+            UpBlock(256, 128),   # 128→256
+            UpBlock(128, 64),    # 256→512
+            nn.Conv2d(64, out_channels, 3, padding=1),
+            nn.Tanh()
+        )
+
+# Reparameterization trick (enables backprop through sampling)
+def sample_latent(mu, logvar):
+    std = torch.exp(0.5 * logvar)
+    eps = torch.randn_like(std)
+    return mu + eps * std
+
+# VAE Loss
+def vae_loss(x, x_recon, mu, logvar):
+    recon_loss = F.mse_loss(x_recon, x, reduction='sum')
+    kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
+    return recon_loss + kl_loss
+```
+
+### Perceptual loss (required for sharp images)
+
+```python
+# Pure MSE loss produces blurry reconstructions.
+# Add LPIPS (perceptual loss) + adversarial loss for sharpness.
+import lpips
+
+loss_fn_vgg = lpips.LPIPS(net='vgg')
+
+def perceptual_loss(x, x_recon):
+    return loss_fn_vgg(x, x_recon).mean()
+
+# Full VAE training loss:
+total_loss = (
+    recon_loss         # MSE reconstruction
+    + 0.001 * kl_loss  # KL regularization (small weight)
+    + 1.0 * perceptual # LPIPS perceptual
+    + 0.5 * adv_loss   # Discriminator adversarial
+)
+```
+
+---
+
+## Text conditioning (CLIP + T5)
+
+### Text encoder comparison
+
+```
+Encoder       Dim    Seq   Params   Best For              License
+CLIP ViT-L/14  768    77    428M    Speed, compatibility  MIT
+CLIP BigG/14  1280    77    2.5B    Quality               Apache 2.0
+T5-base        512   512    250M    Long text             Apache 2.0
+T5-XXL        4096   512    11B     Nuanced language      Apache 2.0
+E5-Large      1024   512    335M    Multilingual          MIT
+
+Production choice (FLUX/SD3 style):
+  CLIP-L + T5-XXL combined → best quality
+  CLIP-G alone → best speed/quality ratio
+  T5-base alone → good for long prompts
+```
+
+### OpenCLIP implementation
+
+```python
+import open_clip
+
+# Load best quality text encoder
+model, _, preprocess = open_clip.create_model_and_transforms(
+    'ViT-bigG-14',
+    pretrained='laion2b_s39b_b160k'  # Apache 2.0
+)
+tokenizer = open_clip.get_tokenizer('ViT-bigG-14')
+
+def encode_text(prompts):
+    tokens = tokenizer(prompts)  # [B, 77]
+    with torch.no_grad(), torch.cuda.amp.autocast():
+        text_features = model.encode_text(tokens)
+        text_features = F.normalize(text_features, dim=-1)
+    return text_features  # [B, 1280]
+
+# For cross-attention in U-Net/DiT:
+# text_features are used as keys and values, image features as queries
+```
+
+### Dual encoder (FLUX/SD3 style)
+
+```python
+# Encode with both CLIP and T5
+clip_embed = clip_encoder.encode_text(prompt)   # [B, 77, 768]
+t5_embed = t5_encoder.encode(prompt)             # [B, 512, 4096]
+
+# Project to same dimension
+clip_proj = nn.Linear(768, model_dim)(clip_embed)
+t5_proj = nn.Linear(4096, model_dim)(t5_embed)
+
+# Concatenate for conditioning
+text_tokens = torch.cat([clip_proj, t5_proj], dim=1)  # [B, 589, D]
+# Use as key/value in cross-attention within MM-DiT
+```

package/skills/craft/image-generation-engineer/references/foundations.md

@@ -0,0 +1,107 @@
+# Mathematical Foundations
+
+Read this when implementing the training objective, a sampler, or debugging noisy / blurry /
+off-prompt generations.
+
+## Contents
+
+- The core idea (one paragraph)
+- Forward process (noising)
+- Reverse process (denoising)
+- DDIM (faster sampling, same model)
+- Flow matching (FLUX/SD3 state of art)
+- Classifier-free guidance (CFG)
+
+---
+
+## The core idea (one paragraph)
+
+A diffusion model learns to reverse a gradual noise-corruption process. **Forward:** take a clean
+image, add Gaussian noise over T timesteps until it becomes pure noise. **Reverse:** train a neural
+network to predict and remove the noise one step at a time, starting from pure noise. Result:
+sampling from pure noise produces new images that look real.
+
+## Forward process (noising)
+
+```
+Given clean image x₀, add noise at timestep t:
+  q(xₜ|x₀) = N(xₜ; √ᾱₜ x₀, (1-ᾱₜ)I)
+
+Where:
+  βₜ = noise schedule (linear: 0.0001 to 0.02 over T=1000 steps)
+  αₜ = 1 - βₜ
+  ᾱₜ = ∏αₛ for s=1..t (cumulative product)
+
+Key insight: closed-form — can jump directly to any timestep t without computing all
+intermediate steps. This makes training fast.
+
+Implementation:
+  noise = torch.randn_like(x0)
+  sqrt_alphas_cumprod_t = sqrt_alphas_cumprod[t]
+  sqrt_one_minus_alphas_cumprod_t = sqrt_one_minus_alphas_cumprod[t]
+  xt = sqrt_alphas_cumprod_t * x0 + sqrt_one_minus_alphas_cumprod_t * noise
+```
+
+## Reverse process (denoising)
+
+```
+Learn to predict the noise ε that was added at timestep t:
+  Loss = E[||ε - εθ(xₜ, t)||²]
+
+Where εθ is the neural network (U-Net or Transformer).
+At inference: start from pure noise, iteratively remove predicted noise.
+
+Noise schedule implementations:
+  Linear:   β linearly spaced 0.0001 to 0.02
+  Cosine:   better for small images (DDPM paper, improved version)
+  Sigmoid:  best for high-resolution images (2026 standard)
+```
+
+## DDIM (faster sampling, same model)
+
+```
+DDIM replaces stochastic sampling with deterministic:
+  xₜ₋₁ = √ᾱₜ₋₁ * (xₜ - √(1-ᾱₜ)*ε_pred)/√ᾱₜ
+          + √(1-ᾱₜ₋₁)*ε_pred
+
+Key: use the SAME trained DDPM model.
+Change: skip timesteps (e.g., 1000 → 50 steps).
+Result: 20× faster inference, comparable quality.
+Trade-off: η=0 fully deterministic, η=1 same as DDPM.
+```
+
+## Flow matching (FLUX/SD3 — 2024-2026 state of art)
+
+```
+Instead of predicting noise, learn a velocity field v(x,t):
+  dx/dt = v(x,t)  [the ODE that transforms noise → image]
+
+Loss: L = E[||v_θ(xₜ,t) - (x₁-x₀)||²]
+  where x₀ = noise sample, x₁ = real image, xₜ = linear interpolation
+
+Advantages over DDPM:
+  - Straight-line trajectories (faster convergence)
+  - Fewer inference steps needed (4-8 vs 50-1000)
+  - Better scaling with model size
+  - More stable training at large scale
+
+FLUX.1 uses Rectified Flow = special case of flow matching
+  with linear interpolation between noise and data.
+```
+
+## Classifier-free guidance (CFG)
+
+```
+Train model BOTH unconditionally and conditionally:
+  ε_guided = ε_uncond + w * (ε_cond - ε_uncond)
+  where w = guidance scale (7.0 typical, higher = more prompt-adherent)
+
+In code:
+  uncond_embed = text_encoder("")  # empty prompt
+  cond_embed = text_encoder(prompt)
+  both_latents = unet([noisy_latents] * 2, [uncond_embed, cond_embed])
+  uncond_pred, cond_pred = both_latents.chunk(2)
+  guided = uncond_pred + guidance_scale * (cond_pred - uncond_pred)
+
+Dropout during training: randomly drop conditioning with p=0.1
+```

package/skills/craft/image-generation-engineer/references/inference-and-serving.md

@@ -0,0 +1,253 @@
+# Inference Optimization, Evaluation, Serving & Model Reference
+
+Read this when speeding up inference, evaluating image quality, or deploying text-to-image.
+
+## Contents
+
+- Sampler comparison
+- Quantization for production (FP8 / GGUF / INT8)
+- Complete inference pipeline (FLUX schnell)
+- FID / CLIP / quality evaluation
+- Serving stack comparison
+- Modal deployment (ORIRO standard)
+- Cloud Run deployment pattern
+- Open-source model reference (June 2026) + licensed data sources
+- HuggingFace Diffusers patterns
+
+---
+
+## Sampler comparison
+
+```
+Sampler      Steps  Quality  Speed   Notes
+DDPM         1000   Best     Slowest Original paper
+DDIM         50     98%      20×     Deterministic
+PNDM         50     98%      20×     Pseudo-numerical
+DPM-Solver   20     97%      50×     Best quality/speed
+DPM-Solver++ 10     95%      100×    Production standard 2024
+LCM          4-8    90%      250×    Distilled, needs new model
+FLUX schnell 1-4    93%      500×    Adversarial distillation
+```
+
+## Quantization for production
+
+```python
+# FP8 quantization: 2× memory, 33% faster, <1% quality loss
+# Best on NVIDIA H100 (native FP8 support)
+from vllm import LLM
+model = LLM(
+    model="black-forest-labs/FLUX.1-schnell",
+    quantization="fp8",
+    dtype="auto",
+)
+
+# GGUF Q4_K_M: 4× memory reduction, runs on consumer GPU
+# Use llama.cpp / ONNX for deployment. Quality: ~95% of FP16 baseline.
+
+# INT8 dynamic quantization (PyTorch native)
+import torch.quantization
+model_int8 = torch.quantization.quantize_dynamic(
+    model, {nn.Linear}, dtype=torch.qint8
+)
+```
+
+## Complete inference pipeline
+
+```python
+import torch
+from diffusers import FluxPipeline
+
+def generate_image(
+    prompt: str,
+    negative_prompt: str = "",
+    width: int = 1024,
+    height: int = 1024,
+    steps: int = 4,       # FLUX schnell: 1-4
+    guidance: float = 0,  # FLUX schnell: 0 (no CFG needed)
+    seed: int = 42,
+) -> "Image":
+    # Load (cache on first call)
+    pipe = FluxPipeline.from_pretrained(
+        "black-forest-labs/FLUX.1-schnell",
+        torch_dtype=torch.bfloat16
+    ).to("cuda")
+
+    generator = torch.Generator("cuda").manual_seed(seed)
+    image = pipe(
+        prompt=prompt,
+        num_inference_steps=steps,
+        guidance_scale=guidance,
+        width=width,
+        height=height,
+        generator=generator,
+    ).images[0]
+    return image  # PIL Image
+
+# For production serving — batched inference:
+def generate_batch(prompts: list[str]) -> list["Image"]:
+    return pipe(prompt=prompts, num_inference_steps=4, guidance_scale=0).images
+```
+
+## FID / quality evaluation
+
+```
+FID (Frechet Inception Distance): lower = better
+  <10: excellent (matches real images)
+  10-30: good (visually appealing)
+  30-100: acceptable (training artefacts visible)
+  >100: poor
+
+CLIP Score: higher = better text-image alignment
+  >0.3: strong   0.25-0.3: moderate   <0.25: poor
+
+Human Preference: Elo ranking (like chess) via RLHF
+  ImageReward, HPS v2, PickScore: learned preference models
+
+Evaluate FID:
+  from pytorch_fid import fid_score
+  fid = fid_score.calculate_fid_given_paths(
+      [real_path, generated_path], 256, 'cuda', 2048
+  )
+```
+
+---
+
+## Serving stack comparison
+
+```
+Tool         Type           Throughput  Latency  Memory  Cost
+vLLM 0.17.1  Inference eng  High        Medium   Medium  Open
+BentoML      Unified API    Medium      Medium   Low     Open
+Modal        Serverless GPU High        Medium   N/A     Pay/req
+ComfyUI      Visual UI      Medium      High     High    Open
+Replicate    Hosted API     Medium      High     N/A     Pay/req
+```
+
+## Modal deployment (ORIRO standard)
+
+```python
+import modal
+
+app = modal.App("image-gen")
+image = modal.Image.debian_slim().pip_install(
+    "diffusers", "torch", "transformers", "accelerate"
+)
+
+@app.function(gpu="H100", image=image, concurrency_limit=10, timeout=120)
+def generate(prompt: str, steps: int = 4) -> bytes:
+    from diffusers import FluxPipeline
+    import torch, io
+    pipe = FluxPipeline.from_pretrained(
+        "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16
+    ).to("cuda")
+    img = pipe(prompt, num_inference_steps=steps).images[0]
+    buf = io.BytesIO(); img.save(buf, format="PNG")
+    return buf.getvalue()
+
+# Deploy: modal deploy image_gen.py
+# Call via HTTP: POST /generate {"prompt": "..."}
+# Cost: ~$0.001-0.003 per image on H100. Cold start ~5s (cached), warm <1s.
+```
+
+## Cloud Run deployment pattern
+
+```dockerfile
+FROM nvcr.io/nvidia/pytorch:24.01-py3
+WORKDIR /app
+COPY requirements.txt .
+RUN pip install diffusers transformers accelerate fastapi uvicorn
+COPY app.py .
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
+```
+
+```python
+# app.py — FastAPI serving
+from fastapi import FastAPI
+from pydantic import BaseModel
+import torch, io, base64
+from diffusers import FluxPipeline
+
+app = FastAPI()
+pipe = None
+
+@app.on_event("startup")
+async def load_model():
+    global pipe
+    pipe = FluxPipeline.from_pretrained(
+        "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16
+    ).to("cuda")
+
+class GenerateRequest(BaseModel):
+    prompt: str
+    steps: int = 4
+    width: int = 1024
+    height: int = 1024
+    seed: int = 42
+
+@app.post("/generate")
+async def generate(req: GenerateRequest):
+    generator = torch.Generator("cuda").manual_seed(req.seed)
+    img = pipe(req.prompt, num_inference_steps=req.steps,
+               width=req.width, height=req.height, generator=generator).images[0]
+    buf = io.BytesIO(); img.save(buf, format="PNG")
+    return {"image_b64": base64.b64encode(buf.getvalue()).decode()}
+```
+
+---
+
+## Open-source model reference (June 2026)
+
+```
+Model          Params  License     Steps  Best For
+FLUX.1 schnell 12B     Apache 2.0  1-4    Commercial, fastest
+FLUX.1 dev     12B     Non-comm    4-8    Research, quality
+FLUX.2         12B     Commercial  4-8    Multi-ref, consistency
+FLUX.1 Kontext 12B     Non-comm    4-8    Image editing
+SD3.5 Large    8B      Open        20     Text-image alignment
+Z-Image-Turbo  ?       TBD         2-4    Sub-second, 16GB
+Wan 2.2        14B     Apache 2.0  N/A    Video generation
+```
+
+## Training data sources (licensed)
+
+```
+LAION-5B:         5B image-text pairs, open research (CC0/BY)
+LAION-Aesthetics: 12M high-quality subset
+DataComp-1B:      1B curated pairs (better quality, open)
+CC12M:            12M Conceptual Captions (Apache 2.0)
+COCO:             330K images with captions (CC BY 4.0)
+WIT:              Wikipedia images, 37M (CC BY)
+JourneyDB:        Journey-style images + prompts (research)
+
+Quality filtering pipeline:
+  1. CLIP score filter (>0.25) — remove misaligned pairs
+  2. Aesthetic score (LAION aesthetic predictor >5.0)
+  3. NSFW filter (remove explicit content)
+  4. Duplicate detection (perceptual hashing)
+  5. Watermark detection + removal
+```
+
+## HuggingFace Diffusers patterns
+
+```python
+from diffusers import (
+    FluxPipeline,
+    StableDiffusionXLPipeline,
+    ControlNetModel,
+    StableDiffusionControlNetPipeline,
+)
+
+# Load LoRA weights
+pipe.load_lora_weights("path/to/lora", weight_name="model.safetensors")
+pipe.set_adapters(["lora"], adapter_weights=[0.8])
+
+# Memory optimization
+pipe.enable_model_cpu_offload()     # offload to CPU when not used
+pipe.enable_attention_slicing()     # slice attention for less VRAM
+pipe.enable_vae_slicing()           # slice VAE for large batches
+pipe.enable_xformers_memory_efficient_attention()  # xformers
+
+# Attention slicing: 30% less VRAM, 10% slower
+# CPU offload: run on 8GB GPU instead of 24GB
+# xformers: 20% faster on older GPUs, same quality
+```