castkit 0.1.0__tar.gz

castkit-0.1.0/.claude/settings.local.json

@@ -0,0 +1,14 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(codex exec:*)",
+      "Bash(hf whoami:*)",
+      "Bash(hf auth:*)",
+      "Bash(hf:*)",
+      "Bash(uvx ruff:*)",
+      "Bash(uvx pytest tests/test_metadata.py tests/test_cross_format.py -v)",
+      "Bash(uv pip:*)",
+      "Bash(uv build:*)"
+    ]
+  }
+}

castkit-0.1.0/CLAUDE.md

@@ -0,0 +1,72 @@
+# castkit
+
+Universal model quantization and format conversion CLI.
+
+## Quick Reference
+
+- Language: Python 3.12+
+- Package manager: uv
+- CLI framework: typer
+- Config format: TOML
+- Linter/Formatter: ruff
+- Test: pytest (uv run pytest)
+- Build: hatch (via pyproject.toml)
+
+## Commands
+
+```
+uv sync                           # install dependencies
+uv sync --extra gguf              # install with GGUF support
+uv sync --extra mlx               # install with MLX support
+uv sync --extra dev               # install dev tools (ruff, pytest)
+uv sync --all-extras              # install everything
+uv run pytest                     # run tests
+uv run ruff format src/ tests/    # format
+uv run ruff check src/ tests/     # lint
+uv run castkit --help             # run CLI
+```
+
+## Architecture
+
+- src/castkit/cli/ - CLI commands (typer)
+- src/castkit/core/ - shared utilities (config, download, model info, metadata)
+- src/castkit/backends/ - format-specific backends (gguf.py, mlx.py, awq.py, gptq.py)
+- tests/ - pytest tests
+
+Each backend implements the abstract Backend interface (backends/base.py). New format support = new backend file.
+
+## Implementation Plan
+
+See docs/implementation-plan.md for historical reference. All 4 phases are complete:
+
+1. Phase 1: Core + GGUF backend
+2. Phase 2: MLX backend
+3. Phase 3: AWQ + GPTQ backends
+4. Phase 4: Config recipes, perplexity measurement, batch conversion, cross-format conversion
+
+## Project-specific Rules
+
+- Dependencies are split via extras: castkit[gguf], castkit[mlx], castkit[all]
+- Backend imports must be lazy (import inside function) to avoid ImportError when extras are not installed
+- MLX backend: Apple Silicon only. Check platform at import time.
+- AWQ/GPTQ backends: use GPTQModel library. convert requires NVIDIA GPU, decast/info work on CPU/Apple Silicon.
+- GGUF backend: shells out to llama-quantize/convert_hf_to_gguf.py. Detect presence at runtime.
+- Use ruff for formatting and linting (not oxfmt/oxlint - those are for TypeScript)
+- Type hints on all public functions
+- Error messages must be actionable (tell the user what to do next)
+
+## Gotchas
+
+- `uv sync --extra mlx` は dev deps (ruff, pytest) を外す。lint/test が必要なら `uv sync --extra mlx --extra dev` するか、`uvx ruff` / `uvx pytest` を使う
+- mlx-lm の upload 機能は内部関数 (`utils.upload_to_hub`) で公開 API ではない。castkit では `huggingface_hub.HfApi` で直接アップロードしている
+- HF upload は `api.upload_folder` を使用。`api.upload_large_folder` もあるが小〜中規模モデルでは `upload_folder` で十分
+- homebrew の llama.cpp (convert_hf_to_gguf.py) は PyPI の gguf パッケージより新しいバージョンを要求する場合がある。GGUF convert が ImportError で失敗したら `uv pip install 'gguf @ git+https://github.com/ggml-org/llama.cpp#subdirectory=gguf-py'` で更新する
+
+## Reference Docs
+
+- docs/project-overview.md - project vision, CLI design, competitive landscape
+- docs/implementation-plan.md - decisions, package structure, implementation details
+- docs/quantization-formats.md - all quantization format specs
+- docs/conversion-paths.md - format conversion matrix and paths
+- docs/trends-and-benchmarks.md - industry trends, benchmark data
+- docs/cli-architecture.md - existing tool architectures, dependency info

castkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 schroneko
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

castkit-0.1.0/PKG-INFO

@@ -0,0 +1,44 @@
+Metadata-Version: 2.4
+Name: castkit
+Version: 0.1.0
+Summary: Universal model quantization and format conversion CLI
+Project-URL: Repository, https://github.com/schroneko/castkit
+Author: schroneko
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: huggingface-hub>=0.25
+Requires-Dist: rich>=13
+Requires-Dist: typer>=0.15
+Provides-Extra: all
+Requires-Dist: accelerate>=1.0; extra == 'all'
+Requires-Dist: datasets>=3.0; extra == 'all'
+Requires-Dist: gguf>=0.10; extra == 'all'
+Requires-Dist: gptqmodel>=2.0; extra == 'all'
+Requires-Dist: mlx-lm>=0.20; extra == 'all'
+Requires-Dist: mlx>=0.22; extra == 'all'
+Requires-Dist: numpy>=1.26; extra == 'all'
+Requires-Dist: safetensors>=0.4; extra == 'all'
+Requires-Dist: sentencepiece>=0.2; extra == 'all'
+Requires-Dist: torch>=2.4; extra == 'all'
+Requires-Dist: transformers>=4.45; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.9; extra == 'dev'
+Provides-Extra: gguf
+Requires-Dist: gguf>=0.10; extra == 'gguf'
+Requires-Dist: numpy>=1.26; extra == 'gguf'
+Requires-Dist: safetensors>=0.4; extra == 'gguf'
+Requires-Dist: sentencepiece>=0.2; extra == 'gguf'
+Requires-Dist: torch>=2.4; extra == 'gguf'
+Requires-Dist: transformers>=4.45; extra == 'gguf'
+Provides-Extra: gptq
+Requires-Dist: accelerate>=1.0; extra == 'gptq'
+Requires-Dist: datasets>=3.0; extra == 'gptq'
+Requires-Dist: gptqmodel>=2.0; extra == 'gptq'
+Requires-Dist: safetensors>=0.4; extra == 'gptq'
+Requires-Dist: torch>=2.4; extra == 'gptq'
+Requires-Dist: transformers>=4.45; extra == 'gptq'
+Provides-Extra: mlx
+Requires-Dist: mlx-lm>=0.20; extra == 'mlx'
+Requires-Dist: mlx>=0.22; extra == 'mlx'

castkit-0.1.0/README.md

@@ -0,0 +1,47 @@
+# castkit
+
+castkit is a CLI tool for model quantization and format conversion across GGUF, MLX, GPTQ, and AWQ workflows, including cross-format conversion via automatic FP16 decast.
+
+## Installation
+
+### Homebrew (macOS)
+
+```bash
+brew install schroneko/castkit/castkit
+```
+
+### pip / uv
+
+```bash
+uv tool install castkit          # core only
+uv tool install castkit[mlx]     # MLX backend (Apple Silicon)
+uv tool install castkit[gguf]    # GGUF backend (requires torch)
+uv tool install castkit[all]     # all backends
+```
+
+## Quick Start
+
+```bash
+# convert
+castkit convert Qwen/Qwen3-0.6B -f gguf -q q4_k_m -o ./output/Qwen3-0.6B.gguf
+
+# decast (dequantize back to FP16 SafeTensors)
+castkit decast ./output/Qwen3-0.6B.gguf -o ./output/Qwen3-0.6B-fp16
+
+# model info
+castkit info ./output/Qwen3-0.6B.gguf
+```
+
+## Supported Formats
+
+| Format         | Convert            | Decast | Info | Measure           |
+| -------------- | ------------------ | ------ | ---- | ----------------- |
+| GGUF           | Yes                | Yes    | Yes  | Yes               |
+| MLX            | Yes                | Yes    | Yes  | Yes               |
+| GPTQ           | Yes                | Yes    | Yes  | Yes               |
+| AWQ            | Yes                | Yes    | Yes  | Yes               |
+| FP16/BF16/FP32 | Input/Intermediate | N/A    | Yes  | Backend-dependent |
+
+## License
+
+MIT

castkit-0.1.0/castkit.toml.example

@@ -0,0 +1,33 @@
+[default]
+output_dir = "./output"
+
+[recipes.gguf-standard]
+format = "gguf"
+quant = "q4_k_m"
+imatrix = true
+imatrix_data = "calibration.txt"
+
+[recipes.gguf-all]
+format = "gguf"
+quant = ["q8_0", "q6_k", "q5_k_m", "q4_k_m", "q3_k_m", "q2_k"]
+imatrix = true
+
+[recipes.mlx-4bit]
+format = "mlx"
+bits = 4
+group_size = 64
+
+[recipes.mlx-8bit]
+format = "mlx"
+bits = 8
+group_size = 64
+
+[recipes.awq-4bit]
+format = "awq"
+bits = 4
+group_size = 128
+
+[recipes.gptq-4bit]
+format = "gptq"
+bits = 4
+group_size = 128

castkit-0.1.0/docs/cli-architecture.md

@@ -0,0 +1,151 @@
+## Existing Tool Architectures
+
+### llama.cpp Pipeline
+
+Two-step process:
+
+1. convert_hf_to_gguf.py (Python): HF model -> F16 GGUF
+   - Architecture detection via config.json
+   - Model class instantiation via @register decorator (80+ architectures)
+   - Lazy tensor loading (no memory consumption during indexing)
+   - TensorNameMap translates HF names -> GGUF names
+   - Supports SafeTensors (mmap), PyTorch, remote HF streaming
+   - Dependencies: numpy, torch, safetensors, sentencepiece, gguf
+
+2. llama-quantize (C++): F16 GGUF -> quantized GGUF
+   - Sequential tensor processing with mmap
+   - imatrix support via --imatrix flag
+   - Per-tensor type control via --tensor-type REGEX:TYPE
+   - Can process any model size on CPU (no GPU needed for basic quant)
+
+### mlx-lm convert
+
+Single-step Python:
+
+- Downloads HF model via transformers/huggingface_hub
+- Maps weights to MLX arrays
+- Optional quantization: to_quantized(group_size, bits) on Linear/Embedding layers
+- Outputs SafeTensors + config.json
+- --upload-repo for direct HF Hub upload
+- Dependencies: mlx, mlx-lm, transformers, huggingface_hub, safetensors
+- Apple Silicon only
+
+### GPTQModel
+
+Python library + CLI:
+
+- Layer-by-layer Hessian-based optimization
+- Multi-GPU data-parallel quantization
+- Supports GPTQ, AWQ, QQQ, GPTAQ, EoRA, GAR methods
+- Dynamic per-module mixed quantization
+- Inference kernels: Marlin (fastest), ExLlama V2/V1, Triton, Torch, BitBLAS
+
+### AutoAWQ (archived May 2025)
+
+Python library:
+
+- Activation analysis -> salient channel identification -> per-channel scaling
+- 10-30 min for 7B (much faster than GPTQ)
+- Replaced by llm-compressor (AWQModifier) and GPTQModel
+
+### ExLlamaV2 (EXL2)
+
+Two-phase Python:
+
+1. Measurement: model quantized ~12 times with different params, error measured per layer. Saved to measurement.json.
+2. Quantization: optimizer selects per-layer bit allocations to minimize total error at target bpw.
+
+- Memory: only largest transformer layer must fit in VRAM
+- 7B: ~16 GB RAM, ~8 GB VRAM. 70B: ~64 GB RAM, ~24 GB VRAM.
+
+### quantkit (llm-quantkit)
+
+Thin Python CLI wrapper:
+
+```
+quantkit/
+  cli.py           # CLI entry point
+  quantkit.py      # Core orchestration
+  convert.py       # General conversion logic
+  convert_exl2.py  # EXL2-specific
+  convert_hf.py    # HuggingFace download/conversion
+  safetensor.py    # SafeTensor utilities
+```
+
+- Delegates to respective libraries for each format
+- Limitations: no MLX, no ONNX, limited config exposure, Python 3.12+ issues, no imatrix generation, no batch multi-format quantization
+
+## Memory Requirements for Quantization
+
+Rule of thumb: FP16 model requires ~2 GB per billion parameters.
+
+| Model Size | FP16 Size | GPTQ/AWQ VRAM  | GGUF (CPU) | EXL2 VRAM               |
+| ---------- | --------- | -------------- | ---------- | ----------------------- |
+| 7B         | ~14 GB    | ~16-24 GB      | CPU only   | ~8 GB VRAM + 16 GB RAM  |
+| 13B        | ~26 GB    | ~32-48 GB      | CPU only   | ~16 GB VRAM + 32 GB RAM |
+| 34B        | ~68 GB    | Fails on 24 GB | CPU only   | ~24 GB VRAM + 64 GB RAM |
+| 70B        | ~140 GB   | Fails on 24 GB | CPU only   | ~24 GB VRAM + 64 GB RAM |
+
+## Common Dependencies
+
+Core (required by almost all):
+
+- torch/pytorch: tensor operations, model loading
+- transformers: model architectures, tokenizers, from_pretrained
+- safetensors: efficient weight storage (standard on HF)
+- huggingface_hub: model downloading (snapshot_download, hf_hub_download)
+- numpy: numerical operations
+- sentencepiece: tokenizer (LLaMA, Gemma, etc.)
+- accelerate: multi-GPU model loading and sharding
+- datasets: calibration dataset loading
+
+Format-specific:
+
+- gguf (from llama.cpp): GGUF read/write
+- gptqmodel: GPTQ quantization
+- autoawq / llm-compressor: AWQ quantization
+- exllamav2: EXL2 quantization
+- bitsandbytes: NF4/INT8 runtime
+- optimum / optimum-onnx: ONNX export
+- onnxruntime: ONNX inference/quantization
+- hqq: Half-Quadratic Quantization
+- mlx, mlx-lm: MLX format (Apple Silicon only)
+
+## Model Download Patterns
+
+Two approaches:
+
+1. transformers.AutoModelForCausalLM.from_pretrained(): loads into memory. Used by GPTQ/AWQ that need model for calibration.
+2. huggingface_hub.snapshot_download(): downloads to cache without loading. Used by llama.cpp and file-based tools.
+
+HF cache: ~/.cache/huggingface/hub/ with content-addressable storage + symlinks.
+
+## HuggingFace Transformers Quantization Integration
+
+| Format             | Loading                                                   | Backend            | Auto-detection       |
+| ------------------ | --------------------------------------------------------- | ------------------ | -------------------- |
+| GPTQ               | from_pretrained()                                         | GPTQModel          | quantize_config.json |
+| AWQ                | from_pretrained()                                         | AutoAWQ            | config               |
+| BnB                | from_pretrained(quantization_config=BitsAndBytesConfig()) | bitsandbytes       | explicit config      |
+| HQQ                | from_pretrained(quantization_config=HqqConfig())          | HQQ                | explicit config      |
+| GGUF               | from_pretrained(gguf_file="model.gguf")                   | gguf-py            | explicit param       |
+| compressed-tensors | from_pretrained()                                         | compressed-tensors | auto                 |
+| EXL2/EXL3          | NOT supported                                             | ExLlama only       | -                    |
+
+Note: transformers loads GGUF by dequantizing to FP32. Uses same memory as FP16 model. For fine-tuning/conversion only, not inference.
+
+## Design Considerations for castkit
+
+1. Format-specific backends are fundamentally different: GGUF is CPU-based C++ (fast, no GPU), GPTQ/AWQ require GPU + calibration, EXL2 has two-phase flow, ONNX has its own pipeline. Must orchestrate as separate backends.
+
+2. Calibration divides UX: GGUF quantization takes minutes with no calibration. GPTQ/AWQ/EXL2 take hours with calibration data. Surface this difference clearly.
+
+3. Memory constraints vary: GGUF can quantize any model on CPU. GPTQ/AWQ fail for models larger than VRAM. EXL2 only needs largest layer to fit. Surface constraints upfront.
+
+4. quantkit's thin-wrapper approach works but is limited. Room for improvement: MLX support, ONNX support, imatrix generation, batch multi-format output, better config exposure.
+
+5. Ecosystem consolidation: AutoAWQ archived, GPTQModel expanding, llm-compressor for vLLM. Track these shifts.
+
+6. Common infrastructure: HF download, model metadata parsing, output upload. Natural abstraction layer.
+
+7. Intel AutoRound can export to multiple formats from single quantization run - consider similar multi-output capability.

castkit-0.1.0/docs/conversion-paths.md

@@ -0,0 +1,178 @@
+# Conversion Paths Reference
+
+All conversion paths between quantization formats, with tooling and quality implications.
+
+## Terminology
+
+- GGUF: self-contained binary file format (.gguf) by llama.cpp
+- SafeTensors: secure tensor serialization container (.safetensors) by HuggingFace. Standard container for HF-ecosystem models.
+- GPTQ, AWQ, EXL2/EXL3, HQQ, bitsandbytes: quantization algorithms/methods. Output stored in SafeTensors with metadata JSON.
+- MLX: Apple Silicon native format using SafeTensors-style storage.
+
+---
+
+## Direct Paths (FP16/BF16 source)
+
+All quantization methods are designed to take full-precision as input. This is the golden path.
+
+| Target             | Tool                                   | Time (7B)     | Calibration               |
+| ------------------ | -------------------------------------- | ------------- | ------------------------- |
+| GGUF               | convert_hf_to_gguf.py + llama-quantize | Minutes       | Optional (imatrix)        |
+| GPTQ               | GPTQModel                              | 2-4 hours     | 128 samples from C4       |
+| AWQ                | AutoAWQ / llm-compressor               | 10-30 min     | Small dataset             |
+| EXL2               | ExLlamaV2                              | Hours         | Yes (measurement phase)   |
+| EXL3               | ExLlamaV3                              | Hours         | Yes (trellis calibration) |
+| BnB NF4/INT8       | bitsandbytes                           | Runtime       | None                      |
+| HQQ                | HQQ                                    | <5 min (70B)  | None                      |
+| MLX                | mlx_lm.convert                         | Minutes       | None (affine)             |
+| ONNX               | HF Optimum / ONNX Runtime              | Minutes-Hours | Static: Yes               |
+| compressed-tensors | llm-compressor                         | Varies        | Yes                       |
+| FP8                | llm-compressor / TensorRT              | Fast          | Optional                  |
+
+---
+
+## Cross-Format Direct Conversions
+
+| Source -> Target         | Tool                            | Notes                                                                          |
+| ------------------------ | ------------------------------- | ------------------------------------------------------------------------------ |
+| GPTQ -> GGUF             | llama.cpp convert_hf_to_gguf.py | Dequantizes GPTQ internally, then converts. Detects GPTQ config automatically. |
+| AWQ -> GGUF              | llama.cpp convert_hf_to_gguf.py | Requires export_compatible=True when creating AWQ model. Preserves AWQ scales. |
+| GPTQ -> GGUF (optimized) | gptq-gguf-toolkit (IST-DASLab)  | Better quality: applies GPTQ error correction during K-quant quantization.     |
+| GGUF -> MLX              | gguf2mlx (community)            | Dequantizes then re-encodes. Limited quant type support.                       |
+| FP16 SafeTensors -> MLX  | mlx_lm.convert                  | Official. Can quantize during conversion.                                      |
+
+---
+
+## Indirect Paths (via FP16 intermediate)
+
+Most cross-format conversions require dequantizing to FP16 first, then re-quantizing. This causes double precision loss.
+
+AWQ -> GGUF (recommended):
+
+1. Quantize with AutoAWQ using export_compatible=True
+2. Convert with convert_hf_to_gguf.py (preserves AWQ scales)
+3. Quantize with llama-quantize
+
+GPTQ <-> AWQ:
+
+1. Dequantize to FP16
+2. Re-quantize with target tool
+
+- Much better to start from original FP16 if available
+
+GGUF -> GPTQ/AWQ/EXL2:
+
+1. Dequantize GGUF to FP16 SafeTensors (gguf-py dequantize() or gguf_to_safetensors)
+2. Re-quantize with target tool
+
+- Quality depends heavily on source GGUF bit width (Q8_0 OK, Q4 noticeable loss)
+
+BnB NF4 -> any:
+
+1. model.dequantize() in transformers
+2. Save as SafeTensors
+3. Convert to target
+
+Any -> MLX:
+
+1. Get to FP16 SafeTensors
+2. mlx_lm.convert with optional quantization flags
+
+---
+
+## Re-quantization Within Same Format
+
+| Format                     | Tool                              | Notes                                                            |
+| -------------------------- | --------------------------------- | ---------------------------------------------------------------- |
+| GGUF (e.g. Q8_0 -> Q4_K_M) | llama-quantize --allow-requantize | Quality degrades vs quantizing from FP16. Tool warns about this. |
+| GPTQ/AWQ/EXL2/BnB          | None                              | Must dequantize to FP16 first, then re-quantize.                 |
+
+---
+
+## Dequantization (Quantized -> FP16)
+
+All dequantization is lossy - the original precision is permanently lost.
+
+| Format        | Tool                              | Quality (vs original)                                    |
+| ------------- | --------------------------------- | -------------------------------------------------------- |
+| GGUF Q8_0     | gguf-py dequantize()              | Very close (~0.1% PPL diff)                              |
+| GGUF Q6_K     | Same                              | Minor degradation                                        |
+| GGUF Q4_K_M   | Same                              | Noticeable (~1-3% PPL increase)                          |
+| GGUF Q2_K/IQ2 | Same                              | Severe - not useful for re-quantization                  |
+| GPTQ 4-bit    | GPTQModel / transformers          | Moderate - rounding errors visible                       |
+| AWQ 4-bit     | Load + extract weights            | Similar to GPTQ                                          |
+| EXL2/EXL3     | ExLlama loader                    | Depends on bpw (6+ bpw very close)                       |
+| BnB NF4       | model.dequantize()                | Good for 4-bit (NF4 is optimal for normal distributions) |
+| BnB INT8      | Same                              | Very close to original                                   |
+| HQQ           | Native dequant (linear operation) | Depends on bit width                                     |
+
+---
+
+## Impossible / Impractical Conversions
+
+- GPTQ <-> AWQ (direct): fundamentally different algorithms. No mathematical mapping.
+- EXL2 <-> EXL3 (direct): different quantization schemes (GPTQ-based vs QTIP-based).
+- Any quantized -> original FP16: information permanently lost.
+- GGUF Q2_K -> any high-quality format: precision loss too severe.
+- BnB NF4 -> GGUF/GPTQ (direct): BnB is runtime library, must dequantize first.
+- ONNX quantized -> GGUF/GPTQ (direct): completely separate ecosystems.
+- 4-bit -> 8-bit for quality improvement: higher bit width does not recover lost information.
+
+---
+
+## Conversion Matrix
+
+```
+Source ->  | FP16  | GGUF  | GPTQ  | AWQ   | EXL2  | EXL3  | BnB   | HQQ   | MLX   | ONNX
+-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|------
+FP16       |  -    | Yes   | Yes   | Yes   | Yes   | Yes   | Yes   | Yes   | Yes   | Yes
+GGUF       | Deq   | Re*   | Indir | Indir | Indir | Indir | Indir | Indir | Deq   | Indir
+GPTQ       | Deq   | Yes** | -     | Indir | Indir | Indir | Indir | Indir | Indir | Indir
+AWQ        | Deq   | Yes***| Indir | -     | Indir | Indir | Indir | Indir | Indir | Indir
+EXL2       | Deq   | Indir | Indir | Indir | -     | No    | Indir | Indir | Indir | Indir
+EXL3       | Deq   | Indir | Indir | Indir | No    | -     | Indir | Indir | Indir | Indir
+BnB        | Deq   | Indir | Indir | Indir | Indir | Indir | -     | Indir | Indir | Indir
+HQQ        | Deq   | Indir | Indir | Indir | Indir | Indir | Indir | -     | Indir | Indir
+MLX        | Deq   | Indir | Indir | Indir | Indir | Indir | Indir | Indir | -     | Indir
+ONNX       | Deq   | Indir | Indir | Indir | Indir | Indir | Indir | Indir | Indir | -
+```
+
+Legend:
+
+- Yes = direct, well-supported path
+- Deq = dequantize to approximate FP16 (lossy)
+- Re\* = GGUF-to-GGUF requantize with --allow-requantize
+- Yes\*\* = llama.cpp handles GPTQ dequant internally
+- Yes\*\*\* = via AWQ export_compatible=True path
+- Indir = indirect, requires dequantize-to-FP16 then re-quantize (double loss)
+- No = not feasible (incompatible algorithms)
+
+---
+
+## Key Tools Reference
+
+| Tool              | Repository                                | Purpose                                          |
+| ----------------- | ----------------------------------------- | ------------------------------------------------ |
+| llama.cpp         | ggml-org/llama.cpp                        | GGUF conversion and quantization                 |
+| GPTQModel         | ModelCloud/GPTQModel                      | GPTQ/AWQ/GPTAQ quantization                      |
+| AutoAWQ           | casper-hansen/AutoAWQ (archived May 2025) | AWQ quantization                                 |
+| ExLlamaV2         | turboderp-org/exllamav2                   | EXL2 quantization and inference                  |
+| ExLlamaV3         | turboderp-org/exllamav3                   | EXL3 quantization and inference                  |
+| HQQ               | dropbox/hqq                               | Half-Quadratic Quantization                      |
+| quantkit          | xhedit/quantkit                           | Multi-format CLI (GGUF, GPTQ, AWQ, HQQ, EXL2)    |
+| Intel AutoRound   | intel/auto-round                          | Multi-format export (GPTQ, AWQ, GGUF, AutoRound) |
+| llm-compressor    | vllm-project/llm-compressor               | compressed-tensors for vLLM                      |
+| gptq-gguf-toolkit | IST-DASLab/gptq-gguf-toolkit              | GPTQ-optimized GGUF quantization                 |
+| mlx-lm            | ml-explore/mlx-lm                         | MLX format conversion and quantization           |
+| bitsandbytes      | bitsandbytes-foundation/bitsandbytes      | Runtime INT8/NF4 quantization                    |
+| HF Optimum        | huggingface/optimum-onnx                  | ONNX export and quantization                     |
+
+---
+
+## Practical Recommendations
+
+1. Always keep the original FP16/BF16 model. It is the best source for any quantization.
+2. Intel AutoRound can export to GPTQ, AWQ, GGUF, and AutoRound from a single run.
+3. For GGUF, best quality: FP16 -> F16 GGUF -> llama-quantize with imatrix.
+4. For GPTQ-to-GGUF, gptq-gguf-toolkit produces better results than naive conversion.
+5. 8-bit formats (FP8, INT8, Q8_0) are near-lossless and safest for intermediate conversion.

castkit-0.1.0/docs/dgx-spark-testing.md

@@ -0,0 +1,96 @@
+# DGX Spark Testing Guide
+
+## 1. Prerequisites
+
+- Python 3.12+
+- CUDA 12.8+
+- `uv`
+- NVIDIA DGX Spark environment (ARM64)
+
+## 2. Setup
+
+```bash
+git clone https://github.com/schroneko/castkit.git
+cd castkit
+uv sync --extra gptq --extra gguf --extra dev
+```
+
+Optional (for upload-related tests):
+
+```bash
+uv add huggingface-hub
+```
+
+## 3. MLX Support Note
+
+MLX is Apple Silicon focused and is not supported on DGX Spark. Skip MLX runtime quantization/decast tests on this platform.
+
+## 4. Test Phases
+
+### Phase 1: Basic Validation
+
+```bash
+uv run pytest
+uv run ruff check src/ tests/
+uv run castkit --help
+uv run castkit --version
+```
+
+### Phase 2: Decast (CPU)
+
+No GPU is required.
+
+```bash
+uv run castkit decast ./path/to/model-quantized -o ./output/decast-fp16
+uv run castkit info ./output/decast-fp16
+```
+
+### Phase 3: GPU Quantization (GPTQ/AWQ)
+
+```bash
+uv run castkit convert ./path/to/fp16-model -f gptq -b 4 -g 128 -o ./output/model-gptq
+uv run castkit convert ./path/to/fp16-model -f awq -b 4 -g 128 -o ./output/model-awq
+```
+
+### Phase 4: GGUF Conversion
+
+Requires `llama.cpp` build with `llama-quantize`, `llama-perplexity`, and `convert_hf_to_gguf.py`.
+
+```bash
+uv run castkit convert ./path/to/fp16-model -f gguf -q q4_k_m -o ./output/model-q4_k_m.gguf
+uv run castkit info ./output/model-q4_k_m.gguf
+```
+
+### Phase 5: Cross-Format Conversion
+
+```bash
+uv run castkit convert ./output/model-q4_k_m.gguf -f gptq -b 4 -o ./output/model-gptq-from-gguf
+```
+
+### Phase 6: Perplexity Measurement
+
+```bash
+uv run castkit measure ./output/model-q4_k_m.gguf --dataset wikitext-2 --max-samples 128
+```
+
+## 5. Recommended Test Models
+
+- TinyLlama/TinyLlama-1.1B-Chat-v1.0
+- Qwen/Qwen2.5-0.5B-Instruct
+
+Use smaller models first to validate flow, then scale up.
+
+## 6. DGX Spark Specific Notes
+
+- Platform is ARM64; validate wheel/ABI availability for all dependencies.
+- `gptqmodel` may require local compilation steps depending on CUDA/toolchain state.
+- Verify compatibility with Blackwell SM targets in CUDA extension builds.
+- Prefer explicit CUDA environment variables when troubleshooting build/runtime mismatch.
+
+## 7. Troubleshooting
+
+- `Backend '...' is not available`: missing extras or runtime dependencies. Re-run `uv sync --extra ...`.
+- `Command not found: llama-quantize`: install/build `llama.cpp` and add binaries to `PATH`.
+- `CUDA was not detected`: confirm driver/toolkit setup and `torch.cuda.is_available()`.
+- `Could not parse perplexity`: capture full `llama-perplexity` stdout and verify output format.
+- Slow/unstable tests: reduce model size, reduce `--max-samples`, and run phase-by-phase.