mlclaw 0.1.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/mlclaw/SKILL.md +63 -18
- package/Dockerfile +21 -4
- package/README.md +70 -24
- package/assets/hf-logo.svg +8 -0
- package/assets/hf-tooling/manifest.json +50 -0
- package/assets/hf-tooling/skills/hf-cli/SKILL.md +218 -0
- package/assets/hf-tooling/skills/hf-mem/SKILL.md +79 -0
- package/assets/hf-tooling/skills/huggingface-best/SKILL.md +134 -0
- package/assets/hf-tooling/skills/huggingface-datasets/SKILL.md +107 -0
- package/assets/hf-tooling/skills/huggingface-gradio/SKILL.md +298 -0
- package/assets/hf-tooling/skills/huggingface-gradio/examples.md +613 -0
- package/assets/hf-tooling/skills/huggingface-local-models/SKILL.md +113 -0
- package/assets/hf-tooling/skills/huggingface-local-models/references/hardware.md +38 -0
- package/assets/hf-tooling/skills/huggingface-local-models/references/hub-discovery.md +178 -0
- package/assets/hf-tooling/skills/huggingface-local-models/references/quantization.md +256 -0
- package/assets/hf-tooling/skills/huggingface-papers/SKILL.md +239 -0
- package/assets/hf-tooling/skills/huggingface-spaces/SKILL.md +239 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/buckets.md +89 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/debugging.md +236 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/gradio.md +200 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/grants.md +56 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/inference-providers.md +85 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/known-errors.md +232 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/requirements.md +169 -0
- package/assets/hf-tooling/skills/huggingface-spaces/references/zerogpu.md +349 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/SKILL.md +120 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.py +57 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.sh +40 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.tsx +57 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/find_models_by_paper.sh +230 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_enrich_models.sh +96 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_model_card_frontmatter.sh +188 -0
- package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_model_papers_auth.sh +171 -0
- package/assets/hf-tooling/skills/huggingface-zerogpu/SKILL.md +289 -0
- package/assets/hf-tooling/skills/huggingface-zerogpu/references/concurrency.md +79 -0
- package/assets/hf-tooling/skills/huggingface-zerogpu/references/cuda-and-deps.md +66 -0
- package/assets/hf-tooling/skills/huggingface-zerogpu/references/how-quota-works.md +74 -0
- package/assets/hf-tooling/skills/huggingface-zerogpu/references/how-zerogpu-works.md +50 -0
- package/assets/hf-tooling/templates/.agents/mcp/huggingface.json +11 -0
- package/assets/hf-tooling/templates/.env.example +5 -0
- package/assets/hf-tooling/templates/examples/huggingface/README.md +16 -0
- package/assets/hf-tooling/templates/examples/huggingface/bucket-sync.md +19 -0
- package/assets/hf-tooling/templates/examples/huggingface/dataset-upload.py +22 -0
- package/assets/hf-tooling/templates/examples/huggingface/hf-discover.md +13 -0
- package/assets/hf-tooling/templates/examples/huggingface/runtime-inspection.py +9 -0
- package/assets/mlclaw-control-ui/assets/index-D2TFes32.js +9 -0
- package/assets/mlclaw-control-ui/assets/index-DP72PFuv.css +1 -0
- package/assets/mlclaw-control-ui/index.html +13 -0
- package/assets/mlclaw.svg +298 -124
- package/dist/hf-tooling-seed.js +261 -0
- package/dist/mlclaw-space-runtime.js +6634 -3321
- package/dist/mlclaw.mjs +428 -178
- package/entrypoint.sh +4 -0
- package/mlclaw.ps1 +4 -2
- package/mlclaw.sh +5 -3
- package/openclaw.default.json +1 -0
- package/package.json +16 -2
- package/space/README.md +32 -2
- package/tsconfig.json +3 -2
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
## Hardware Acceleration
|
|
2
|
+
|
|
3
|
+
### Apple Silicon (Metal)
|
|
4
|
+
|
|
5
|
+
```bash
|
|
6
|
+
make clean && make GGML_METAL=1
|
|
7
|
+
llama-cli -m model.gguf -ngl 99 -p "Hello"
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
### NVIDIA (CUDA)
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
make clean && make GGML_CUDA=1
|
|
14
|
+
llama-cli -m model.gguf -ngl 35 -p "Hello"
|
|
15
|
+
|
|
16
|
+
# Hybrid for large models
|
|
17
|
+
llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20
|
|
18
|
+
|
|
19
|
+
# Multi-GPU split
|
|
20
|
+
llama-cli -m large-model.gguf --tensor-split 0.5,0.5 -ngl 60
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
### AMD (ROCm)
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
make LLAMA_HIP=1
|
|
27
|
+
llama-cli -m model.gguf -ngl 999
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
### CPU
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
# Match physical cores, not logical threads
|
|
34
|
+
llama-cli -m model.gguf -t 8 -p "Hello"
|
|
35
|
+
|
|
36
|
+
# BLAS acceleration
|
|
37
|
+
make LLAMA_OPENBLAS=1
|
|
38
|
+
```
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
# Hugging Face URL Workflows for llama.cpp
|
|
2
|
+
|
|
3
|
+
Use URL-only workflows first. Do not require `hf` or API clients just to find GGUF files, choose a quant, or build a `llama-server` command.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Core URLs
|
|
8
|
+
- Search for llama.cpp-compatible models
|
|
9
|
+
- Use the local-app page for the recommended quant
|
|
10
|
+
- Confirm exact files from the tree API
|
|
11
|
+
- Build the command
|
|
12
|
+
- Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
|
|
13
|
+
- Notes
|
|
14
|
+
|
|
15
|
+
## Core URLs
|
|
16
|
+
|
|
17
|
+
```text
|
|
18
|
+
Search:
|
|
19
|
+
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
|
20
|
+
|
|
21
|
+
Search with text:
|
|
22
|
+
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
|
23
|
+
|
|
24
|
+
Search with size bounds:
|
|
25
|
+
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
|
26
|
+
|
|
27
|
+
Repo local-app view:
|
|
28
|
+
https://huggingface.co/<repo>?local-app=llama.cpp
|
|
29
|
+
|
|
30
|
+
Repo tree API:
|
|
31
|
+
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
|
32
|
+
|
|
33
|
+
Repo file tree:
|
|
34
|
+
https://huggingface.co/<repo>/tree/main
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## 1. Search for llama.cpp-compatible models
|
|
38
|
+
|
|
39
|
+
Start from the models page with `apps=llama.cpp`.
|
|
40
|
+
|
|
41
|
+
Use:
|
|
42
|
+
|
|
43
|
+
- `search=<term>` for model family names such as `Qwen`, `Gemma`, `Phi`, or `Mistral`
|
|
44
|
+
- `num_parameters=min:0,max:24B` or similar if the user has hardware limits
|
|
45
|
+
- `sort=trending` when the user wants popular repos right now
|
|
46
|
+
|
|
47
|
+
Do not start with random GGUF repos if the user has not chosen a model family yet. Search first, shortlist second.
|
|
48
|
+
|
|
49
|
+
Example: https://huggingface.co/models?search=Qwen&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
|
50
|
+
|
|
51
|
+
## 2. Use the local-app page for the recommended quant
|
|
52
|
+
|
|
53
|
+
Open:
|
|
54
|
+
|
|
55
|
+
```text
|
|
56
|
+
https://huggingface.co/<repo>?local-app=llama.cpp
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Extract, in order:
|
|
60
|
+
|
|
61
|
+
1. The exact `Use this model` snippet, if it is visible as text
|
|
62
|
+
2. The `Hardware compatibility` section from the fetched page text or HTML:
|
|
63
|
+
- quant label
|
|
64
|
+
- file size
|
|
65
|
+
- bit-depth grouping
|
|
66
|
+
3. Any extra launch flags shown in the snippet, such as `--jinja`
|
|
67
|
+
|
|
68
|
+
Treat the HF local-app snippet as the source of truth when it is visible.
|
|
69
|
+
|
|
70
|
+
Do this by reading the URL itself, not by assuming the UI rendered in a browser. If the fetched page source does not expose `Hardware compatibility`, say that the section was not text-visible and fall back to the tree API plus generic guidance from `quantization.md`.
|
|
71
|
+
|
|
72
|
+
## 3. Confirm exact files from the tree API
|
|
73
|
+
|
|
74
|
+
Open:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Treat the JSON response as the source of truth for repo inventory.
|
|
81
|
+
|
|
82
|
+
Keep entries where:
|
|
83
|
+
|
|
84
|
+
- `type` is `file`
|
|
85
|
+
- `path` ends with `.gguf`
|
|
86
|
+
|
|
87
|
+
Use these fields:
|
|
88
|
+
|
|
89
|
+
- `path` for the filename and subdirectory
|
|
90
|
+
- `size` for the byte size
|
|
91
|
+
- optionally `lfs.size` to confirm the LFS payload size
|
|
92
|
+
|
|
93
|
+
Separate files into:
|
|
94
|
+
|
|
95
|
+
- quantized single-file checkpoints, for example `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
|
96
|
+
- projector weights, usually `mmproj-*.gguf`
|
|
97
|
+
- BF16 shard files, usually under `BF16/`
|
|
98
|
+
- everything else
|
|
99
|
+
|
|
100
|
+
Ignore unless the user asks:
|
|
101
|
+
|
|
102
|
+
- `README.md`
|
|
103
|
+
- imatrix or calibration blobs
|
|
104
|
+
|
|
105
|
+
Use `https://huggingface.co/<repo>/tree/main` only as a human fallback if the API endpoint fails or the user wants the web view.
|
|
106
|
+
|
|
107
|
+
## 4. Build the command
|
|
108
|
+
|
|
109
|
+
Preferred order:
|
|
110
|
+
|
|
111
|
+
1. Copy the exact HF snippet from the local-app page
|
|
112
|
+
2. If the page gives a clean quant label, use shorthand selection:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
llama-server -hf <repo>:<QUANT>
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
3. If you need an exact file from the tree API, use the file-specific form:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
4. For CLI usage instead of a server, use:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
llama-cli -hf <repo>:<QUANT>
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Use the exact-file form when the repo uses custom labels or nonstandard naming that could make `:<QUANT>` ambiguous.
|
|
131
|
+
|
|
132
|
+
## 5. Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
|
|
133
|
+
|
|
134
|
+
Use these URLs:
|
|
135
|
+
|
|
136
|
+
```text
|
|
137
|
+
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
|
|
138
|
+
https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
|
|
139
|
+
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
On the local-app page, the hardware compatibility section can expose entries such as:
|
|
143
|
+
|
|
144
|
+
- `UD-IQ4_XS` - 17.7 GB
|
|
145
|
+
- `UD-Q4_K_S` - 20.9 GB
|
|
146
|
+
- `UD-Q4_K_M` - 22.1 GB
|
|
147
|
+
- `UD-Q5_K_M` - 26.5 GB
|
|
148
|
+
- `UD-Q6_K` - 29.3 GB
|
|
149
|
+
- `Q8_0` - 36.9 GB
|
|
150
|
+
|
|
151
|
+
On the tree API, you can confirm exact filenames such as:
|
|
152
|
+
|
|
153
|
+
- `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
|
154
|
+
- `Qwen3.6-35B-A3B-UD-Q5_K_M.gguf`
|
|
155
|
+
- `Qwen3.6-35B-A3B-UD-Q6_K.gguf`
|
|
156
|
+
- `Qwen3.6-35B-A3B-Q8_0.gguf`
|
|
157
|
+
- `mmproj-F16.gguf`
|
|
158
|
+
|
|
159
|
+
Good final output for this repo:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
Repo: unsloth/Qwen3.6-35B-A3B-GGUF
|
|
163
|
+
Recommended quant from HF: UD-Q4_K_M (22.1 GB)
|
|
164
|
+
llama-server: llama-server --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
|
|
165
|
+
Other GGUFs:
|
|
166
|
+
- Qwen3.6-35B-A3B-UD-Q5_K_M.gguf - 26.5 GB
|
|
167
|
+
- Qwen3.6-35B-A3B-UD-Q6_K.gguf - 29.3 GB
|
|
168
|
+
- Qwen3.6-35B-A3B-Q8_0.gguf - 36.9 GB
|
|
169
|
+
Projector:
|
|
170
|
+
- mmproj-F16.gguf - 899 MB
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## Notes
|
|
174
|
+
|
|
175
|
+
- Repo-specific quant labels matter. Do not rewrite `UD-Q4_K_M` to `Q4_K_M` unless the page itself does.
|
|
176
|
+
- `mmproj` files are projector weights for multimodal models, not the main language model checkpoint.
|
|
177
|
+
- If the HF hardware compatibility panel is missing because the user has no hardware profile configured, or because the fetched page source did not expose it, still use the tree API plus generic quant guidance from `quantization.md`.
|
|
178
|
+
- If the repo already has GGUFs, do not jump straight to conversion workflows.
|
|
@@ -0,0 +1,256 @@
|
|
|
1
|
+
# GGUF Quantization Guide
|
|
2
|
+
|
|
3
|
+
Complete guide to GGUF quantization formats and model conversion.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Hub-first quant selection
|
|
8
|
+
- Quantization Formats
|
|
9
|
+
- Converting Models
|
|
10
|
+
- K-Quantization Methods
|
|
11
|
+
- Quality Testing
|
|
12
|
+
- Use Case Guide
|
|
13
|
+
- Model Size Scaling
|
|
14
|
+
- Finding Pre-Quantized Models
|
|
15
|
+
- Importance Matrices (`imatrix`)
|
|
16
|
+
- Troubleshooting
|
|
17
|
+
|
|
18
|
+
## Hub-first quant selection
|
|
19
|
+
|
|
20
|
+
Before using generic tables, open the model repo with:
|
|
21
|
+
|
|
22
|
+
```text
|
|
23
|
+
https://huggingface.co/<repo>?local-app=llama.cpp
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Prefer the exact quant labels and sizes shown in the `Hardware compatibility` section of the fetched `?local-app=llama.cpp` page text or HTML. Then confirm the matching filenames in:
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.
|
|
33
|
+
|
|
34
|
+
## Quantization Formats
|
|
35
|
+
|
|
36
|
+
**GGUF** (GPT-Generated Unified Format) - Standard format for llama.cpp models.
|
|
37
|
+
|
|
38
|
+
### Format Comparison
|
|
39
|
+
|
|
40
|
+
| Format | Perplexity | Size (7B) | Tokens/sec | Notes |
|
|
41
|
+
|--------|------------|-----------|------------|-------|
|
|
42
|
+
| FP16 | 5.9565 (baseline) | 13.0 GB | 15 tok/s | Original quality |
|
|
43
|
+
| Q8_0 | 5.9584 (+0.03%) | 7.0 GB | 25 tok/s | Nearly lossless |
|
|
44
|
+
| **Q6_K** | 5.9642 (+0.13%) | 5.5 GB | 30 tok/s | Best quality/size |
|
|
45
|
+
| **Q5_K_M** | 5.9796 (+0.39%) | 4.8 GB | 35 tok/s | Balanced |
|
|
46
|
+
| **Q4_K_M** | 6.0565 (+1.68%) | 4.1 GB | 40 tok/s | **Recommended** |
|
|
47
|
+
| Q4_K_S | 6.1125 (+2.62%) | 3.9 GB | 42 tok/s | Faster, lower quality |
|
|
48
|
+
| Q3_K_M | 6.3184 (+6.07%) | 3.3 GB | 45 tok/s | Small models only |
|
|
49
|
+
| Q2_K | 6.8673 (+15.3%) | 2.7 GB | 50 tok/s | Not recommended |
|
|
50
|
+
|
|
51
|
+
**Recommendation**: Use **Q4_K_M** for best balance of quality and speed.
|
|
52
|
+
|
|
53
|
+
## Converting Models
|
|
54
|
+
|
|
55
|
+
### Hugging Face to GGUF
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
# 1. Download Hugging Face model
|
|
59
|
+
hf download meta-llama/Llama-2-7b-chat-hf \
|
|
60
|
+
--local-dir models/llama-2-7b-chat/
|
|
61
|
+
|
|
62
|
+
# 2. Convert to FP16 GGUF
|
|
63
|
+
python convert_hf_to_gguf.py \
|
|
64
|
+
models/llama-2-7b-chat/ \
|
|
65
|
+
--outtype f16 \
|
|
66
|
+
--outfile models/llama-2-7b-chat-f16.gguf
|
|
67
|
+
|
|
68
|
+
# 3. Quantize to Q4_K_M
|
|
69
|
+
./llama-quantize \
|
|
70
|
+
models/llama-2-7b-chat-f16.gguf \
|
|
71
|
+
models/llama-2-7b-chat-Q4_K_M.gguf \
|
|
72
|
+
Q4_K_M
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Batch quantization
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
# Quantize to multiple formats
|
|
79
|
+
for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
|
|
80
|
+
./llama-quantize \
|
|
81
|
+
model-f16.gguf \
|
|
82
|
+
model-${quant}.gguf \
|
|
83
|
+
$quant
|
|
84
|
+
done
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## K-Quantization Methods
|
|
88
|
+
|
|
89
|
+
**K-quants** use mixed precision for better quality:
|
|
90
|
+
- Attention weights: Higher precision
|
|
91
|
+
- Feed-forward weights: Lower precision
|
|
92
|
+
|
|
93
|
+
**Variants**:
|
|
94
|
+
- `_S` (Small): Faster, lower quality
|
|
95
|
+
- `_M` (Medium): Balanced (recommended)
|
|
96
|
+
- `_L` (Large): Better quality, larger size
|
|
97
|
+
|
|
98
|
+
**Example**: `Q4_K_M`
|
|
99
|
+
- `Q4`: 4-bit quantization
|
|
100
|
+
- `K`: Mixed precision method
|
|
101
|
+
- `M`: Medium quality
|
|
102
|
+
|
|
103
|
+
## Quality Testing
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# Calculate perplexity (quality metric)
|
|
107
|
+
./llama-perplexity \
|
|
108
|
+
-m model.gguf \
|
|
109
|
+
-f wikitext-2-raw/wiki.test.raw \
|
|
110
|
+
-c 512
|
|
111
|
+
|
|
112
|
+
# Lower perplexity = better quality
|
|
113
|
+
# Baseline (FP16): ~5.96
|
|
114
|
+
# Q4_K_M: ~6.06 (+1.7%)
|
|
115
|
+
# Q2_K: ~6.87 (+15.3% - too much degradation)
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Use Case Guide
|
|
119
|
+
|
|
120
|
+
### General purpose (chatbots, assistants)
|
|
121
|
+
```
|
|
122
|
+
Q4_K_M - Best balance
|
|
123
|
+
Q5_K_M - If you have extra RAM
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
### Code generation
|
|
127
|
+
```
|
|
128
|
+
Q5_K_M or Q6_K - Higher precision helps with code
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### Creative writing
|
|
132
|
+
```
|
|
133
|
+
Q4_K_M - Sufficient quality
|
|
134
|
+
Q3_K_M - Acceptable for draft generation
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Technical/medical
|
|
138
|
+
```
|
|
139
|
+
Q6_K or Q8_0 - Maximum accuracy
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Edge devices (Raspberry Pi)
|
|
143
|
+
```
|
|
144
|
+
Q2_K or Q3_K_S - Fit in limited RAM
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
## Model Size Scaling
|
|
148
|
+
|
|
149
|
+
### 7B parameter models
|
|
150
|
+
|
|
151
|
+
| Format | Size | RAM needed |
|
|
152
|
+
|--------|------|------------|
|
|
153
|
+
| Q2_K | 2.7 GB | 5 GB |
|
|
154
|
+
| Q3_K_M | 3.3 GB | 6 GB |
|
|
155
|
+
| Q4_K_M | 4.1 GB | 7 GB |
|
|
156
|
+
| Q5_K_M | 4.8 GB | 8 GB |
|
|
157
|
+
| Q6_K | 5.5 GB | 9 GB |
|
|
158
|
+
| Q8_0 | 7.0 GB | 11 GB |
|
|
159
|
+
|
|
160
|
+
### 13B parameter models
|
|
161
|
+
|
|
162
|
+
| Format | Size | RAM needed |
|
|
163
|
+
|--------|------|------------|
|
|
164
|
+
| Q2_K | 5.1 GB | 8 GB |
|
|
165
|
+
| Q3_K_M | 6.2 GB | 10 GB |
|
|
166
|
+
| Q4_K_M | 7.9 GB | 12 GB |
|
|
167
|
+
| Q5_K_M | 9.2 GB | 14 GB |
|
|
168
|
+
| Q6_K | 10.7 GB | 16 GB |
|
|
169
|
+
|
|
170
|
+
### 70B parameter models
|
|
171
|
+
|
|
172
|
+
| Format | Size | RAM needed |
|
|
173
|
+
|--------|------|------------|
|
|
174
|
+
| Q2_K | 26 GB | 32 GB |
|
|
175
|
+
| Q3_K_M | 32 GB | 40 GB |
|
|
176
|
+
| Q4_K_M | 41 GB | 48 GB |
|
|
177
|
+
| Q4_K_S | 39 GB | 46 GB |
|
|
178
|
+
| Q5_K_M | 48 GB | 56 GB |
|
|
179
|
+
|
|
180
|
+
**Recommendation for 70B**: Use Q3_K_M or Q4_K_S to fit in consumer hardware.
|
|
181
|
+
|
|
182
|
+
## Finding Pre-Quantized Models
|
|
183
|
+
|
|
184
|
+
Use the Hub search with the llama.cpp app filter:
|
|
185
|
+
|
|
186
|
+
```text
|
|
187
|
+
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
|
188
|
+
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
|
189
|
+
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
For a specific repo, open:
|
|
193
|
+
|
|
194
|
+
```text
|
|
195
|
+
https://huggingface.co/<repo>?local-app=llama.cpp
|
|
196
|
+
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Then launch directly from the Hub without extra Hub tooling:
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
llama-cli -hf <repo>:Q4_K_M
|
|
203
|
+
llama-server -hf <repo>:Q4_K_M
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
If you need the exact file name from the tree API:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
## Importance Matrices (imatrix)
|
|
213
|
+
|
|
214
|
+
**What**: Calibration data to improve quantization quality.
|
|
215
|
+
|
|
216
|
+
**Benefits**:
|
|
217
|
+
- 10-20% perplexity improvement with Q4
|
|
218
|
+
- Essential for Q3 and below
|
|
219
|
+
|
|
220
|
+
**Usage**:
|
|
221
|
+
```bash
|
|
222
|
+
# 1. Generate importance matrix
|
|
223
|
+
./llama-imatrix \
|
|
224
|
+
-m model-f16.gguf \
|
|
225
|
+
-f calibration-data.txt \
|
|
226
|
+
-o model.imatrix
|
|
227
|
+
|
|
228
|
+
# 2. Quantize with imatrix
|
|
229
|
+
./llama-quantize \
|
|
230
|
+
--imatrix model.imatrix \
|
|
231
|
+
model-f16.gguf \
|
|
232
|
+
model-Q4_K_M.gguf \
|
|
233
|
+
Q4_K_M
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
**Calibration data**:
|
|
237
|
+
- Use domain-specific text (e.g., code for code models)
|
|
238
|
+
- ~100MB of representative text
|
|
239
|
+
- Higher quality data = better quantization
|
|
240
|
+
|
|
241
|
+
## Troubleshooting
|
|
242
|
+
|
|
243
|
+
**Model outputs gibberish**:
|
|
244
|
+
- Quantization too aggressive (Q2_K)
|
|
245
|
+
- Try Q4_K_M or Q5_K_M
|
|
246
|
+
- Verify model converted correctly
|
|
247
|
+
|
|
248
|
+
**Out of memory**:
|
|
249
|
+
- Use lower quantization (Q4_K_S instead of Q5_K_M)
|
|
250
|
+
- Offload fewer layers to GPU (`-ngl`)
|
|
251
|
+
- Use smaller context (`-c 2048`)
|
|
252
|
+
|
|
253
|
+
**Slow inference**:
|
|
254
|
+
- Higher quantization uses more compute
|
|
255
|
+
- Q8_0 much slower than Q4_K_M
|
|
256
|
+
- Consider speed vs quality trade-off
|
|
@@ -0,0 +1,239 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: huggingface-papers
|
|
3
|
+
description: Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. Use when the user shares a Hugging Face paper page URL, an arXiv URL or ID, or asks to summarize, explain, or analyze an AI research paper.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Hugging Face Paper Pages
|
|
7
|
+
|
|
8
|
+
Hugging Face Paper pages (hf.co/papers) is a platform built on top of arXiv (arxiv.org), specifically for research papers in the field of artificial intelligence (AI) and computer science. Hugging Face users can submit their paper at hf.co/papers/submit, which features it on the Daily Papers feed (hf.co/papers). Each day, users can upvote papers and comment on papers. Each paper page allows authors to:
|
|
9
|
+
- claim their paper (by clicking their name on the `authors` field). This makes the paper page appear on their Hugging Face profile.
|
|
10
|
+
- link the associated model checkpoints, datasets and Spaces by including the HF paper or arXiv URL in the model card, dataset card or README of the Space
|
|
11
|
+
- link the Github repository and/or project page URLs
|
|
12
|
+
- link the HF organization. This also makes the paper page appear on the Hugging Face organization page.
|
|
13
|
+
|
|
14
|
+
Whenever someone mentions a HF paper or arXiv abstract/PDF URL in a model card, dataset card or README of a Space repository, the paper will be automatically indexed. Note that not all papers indexed on Hugging Face are also submitted to daily papers. The latter is more a manner of promoting a research paper. Papers can only be submitted to daily papers up until 14 days after their publication date on arXiv.
|
|
15
|
+
|
|
16
|
+
The Hugging Face team has built an easy-to-use API to interact with paper pages. Content of the papers can be fetched as markdown, or structured metadata can be returned such as author names, linked models/datasets/spaces, linked Github repo and project page.
|
|
17
|
+
|
|
18
|
+
## When to Use
|
|
19
|
+
|
|
20
|
+
- User shares a Hugging Face paper page URL (e.g. `https://huggingface.co/papers/2602.08025`)
|
|
21
|
+
- User shares a Hugging Face markdown paper page URL (e.g. `https://huggingface.co/papers/2602.08025.md`)
|
|
22
|
+
- User shares an arXiv URL (e.g. `https://arxiv.org/abs/2602.08025` or `https://arxiv.org/pdf/2602.08025`)
|
|
23
|
+
- User mentions a arXiv ID (e.g. `2602.08025`)
|
|
24
|
+
- User asks you to summarize, explain, or analyze an AI research paper
|
|
25
|
+
|
|
26
|
+
## Parsing the paper ID
|
|
27
|
+
|
|
28
|
+
It's recommended to parse the paper ID (arXiv ID) from whatever the user provides:
|
|
29
|
+
|
|
30
|
+
| Input | Paper ID |
|
|
31
|
+
| --- | --- |
|
|
32
|
+
| `https://huggingface.co/papers/2602.08025` | `2602.08025` |
|
|
33
|
+
| `https://huggingface.co/papers/2602.08025.md` | `2602.08025` |
|
|
34
|
+
| `https://arxiv.org/abs/2602.08025` | `2602.08025` |
|
|
35
|
+
| `https://arxiv.org/pdf/2602.08025` | `2602.08025` |
|
|
36
|
+
| `2602.08025v1` | `2602.08025v1` |
|
|
37
|
+
| `2602.08025` | `2602.08025` |
|
|
38
|
+
|
|
39
|
+
This allows you to provide the paper ID into any of the hub API endpoints mentioned below.
|
|
40
|
+
|
|
41
|
+
### Fetch the paper page as markdown
|
|
42
|
+
|
|
43
|
+
The content of a paper can be fetched as markdown like so:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
curl -s "https://huggingface.co/papers/{PAPER_ID}.md"
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
This should return the Hugging Face paper page as markdown. This relies on the HTML version of the paper at https://arxiv.org/html/{PAPER_ID}.
|
|
50
|
+
|
|
51
|
+
There are 2 exceptions:
|
|
52
|
+
- Not all arXiv papers have an HTML version. If the HTML version of the paper does not exist, then the content falls back to the HTML of the Hugging Face paper page.
|
|
53
|
+
- If it results in a 404, it means the paper is not yet indexed on hf.co/papers. See [Error handling](#error-handling) for info.
|
|
54
|
+
|
|
55
|
+
Alternatively, you can request markdown from the normal paper page URL, like so:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
curl -s -H "Accept: text/markdown" "https://huggingface.co/papers/{PAPER_ID}"
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Paper Pages API Endpoints
|
|
62
|
+
|
|
63
|
+
All endpoints use the base URL `https://huggingface.co`.
|
|
64
|
+
|
|
65
|
+
#### Get structured metadata
|
|
66
|
+
|
|
67
|
+
Fetch the paper metadata as JSON using the Hugging Face REST API:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
curl -s "https://huggingface.co/api/papers/{PAPER_ID}"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
This returns structured metadata that can include:
|
|
74
|
+
|
|
75
|
+
- authors (names and Hugging Face usernames, in case they have claimed the paper)
|
|
76
|
+
- media URLs (uploaded when submitting the paper to Daily Papers)
|
|
77
|
+
- summary (abstract) and AI-generated summary
|
|
78
|
+
- project page and GitHub repository
|
|
79
|
+
- organization and engagement metadata (number of upvotes)
|
|
80
|
+
|
|
81
|
+
To find models linked to the paper, use:
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
curl https://huggingface.co/api/models?filter=arxiv:{PAPER_ID}
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
To find datasets linked to the paper, use:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
curl https://huggingface.co/api/datasets?filter=arxiv:{PAPER_ID}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
To find spaces linked to the paper, use:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
curl https://huggingface.co/api/spaces?filter=arxiv:{PAPER_ID}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
#### Claim paper authorship
|
|
100
|
+
|
|
101
|
+
Claim authorship of a paper for a Hugging Face user:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
curl "https://huggingface.co/api/settings/papers/claim" \
|
|
105
|
+
--request POST \
|
|
106
|
+
--header "Content-Type: application/json" \
|
|
107
|
+
--header "Authorization: Bearer $HF_TOKEN" \
|
|
108
|
+
--data '{
|
|
109
|
+
"paperId": "{PAPER_ID}",
|
|
110
|
+
"claimAuthorId": "{AUTHOR_ENTRY_ID}",
|
|
111
|
+
"targetUserId": "{USER_ID}"
|
|
112
|
+
}'
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
- Endpoint: `POST /api/settings/papers/claim`
|
|
116
|
+
- Body:
|
|
117
|
+
- `paperId` (string, required): arXiv paper identifier being claimed
|
|
118
|
+
- `claimAuthorId` (string): author entry on the paper being claimed, 24-char hex ID
|
|
119
|
+
- `targetUserId` (string): HF user who should receive the claim, 24-char hex ID
|
|
120
|
+
- Response: paper authorship claim result, including the claimed paper ID
|
|
121
|
+
|
|
122
|
+
#### Get daily papers
|
|
123
|
+
|
|
124
|
+
Fetch the Daily Papers feed:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
|
128
|
+
"https://huggingface.co/api/daily_papers?p=0&limit=20&date=2017-07-21&sort=publishedAt"
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
- Endpoint: `GET /api/daily_papers`
|
|
132
|
+
- Query parameters:
|
|
133
|
+
- `p` (integer): page number
|
|
134
|
+
- `limit` (integer): number of results, between 1 and 100
|
|
135
|
+
- `date` (string): RFC 3339 full-date, for example `2017-07-21`
|
|
136
|
+
- `week` (string): ISO week, for example `2024-W03`
|
|
137
|
+
- `month` (string): month value, for example `2024-01`
|
|
138
|
+
- `submitter` (string): filter by submitter
|
|
139
|
+
- `sort` (enum): `publishedAt` or `trending`
|
|
140
|
+
- Response: list of daily papers
|
|
141
|
+
|
|
142
|
+
#### List papers
|
|
143
|
+
|
|
144
|
+
List arXiv papers sorted by published date:
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
|
148
|
+
"https://huggingface.co/api/papers?cursor={CURSOR}&limit=20"
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
- Endpoint: `GET /api/papers`
|
|
152
|
+
- Query parameters:
|
|
153
|
+
- `cursor` (string): pagination cursor
|
|
154
|
+
- `limit` (integer): number of results, between 1 and 100
|
|
155
|
+
- Response: list of papers
|
|
156
|
+
|
|
157
|
+
#### Search papers
|
|
158
|
+
|
|
159
|
+
Perform hybrid semantic and full-text search on papers:
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
|
163
|
+
"https://huggingface.co/api/papers/search?q=vision+language&limit=20"
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
This searches over the paper title, authors, and content.
|
|
167
|
+
|
|
168
|
+
- Endpoint: `GET /api/papers/search`
|
|
169
|
+
- Query parameters:
|
|
170
|
+
- `q` (string): search query, max length 250
|
|
171
|
+
- `limit` (integer): number of results, between 1 and 120
|
|
172
|
+
- Response: matching papers
|
|
173
|
+
|
|
174
|
+
#### Index a paper
|
|
175
|
+
|
|
176
|
+
Insert a paper from arXiv by ID. If the paper is already indexed, only its authors can re-index it:
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
curl "https://huggingface.co/api/papers/index" \
|
|
180
|
+
--request POST \
|
|
181
|
+
--header "Content-Type: application/json" \
|
|
182
|
+
--header "Authorization: Bearer $HF_TOKEN" \
|
|
183
|
+
--data '{
|
|
184
|
+
"arxivId": "{ARXIV_ID}"
|
|
185
|
+
}'
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
- Endpoint: `POST /api/papers/index`
|
|
189
|
+
- Body:
|
|
190
|
+
- `arxivId` (string, required): arXiv ID to index, for example `2301.00001`
|
|
191
|
+
- Pattern: `^\d{4}\.\d{4,5}$`
|
|
192
|
+
- Response: empty JSON object on success
|
|
193
|
+
|
|
194
|
+
#### Update paper links
|
|
195
|
+
|
|
196
|
+
Update the project page, GitHub repository, or submitting organization for a paper. The requester must be the paper author, the Daily Papers submitter, or a papers admin:
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
curl "https://huggingface.co/api/papers/{PAPER_OBJECT_ID}/links" \
|
|
200
|
+
--request POST \
|
|
201
|
+
--header "Content-Type: application/json" \
|
|
202
|
+
--header "Authorization: Bearer $HF_TOKEN" \
|
|
203
|
+
--data '{
|
|
204
|
+
"projectPage": "https://example.com",
|
|
205
|
+
"githubRepo": "https://github.com/org/repo",
|
|
206
|
+
"organizationId": "{ORGANIZATION_ID}"
|
|
207
|
+
}'
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
- Endpoint: `POST /api/papers/{paperId}/links`
|
|
211
|
+
- Path parameters:
|
|
212
|
+
- `paperId` (string, required): Hugging Face paper object ID
|
|
213
|
+
- Body:
|
|
214
|
+
- `githubRepo` (string, nullable): GitHub repository URL
|
|
215
|
+
- `organizationId` (string, nullable): organization ID, 24-char hex ID
|
|
216
|
+
- `projectPage` (string, nullable): project page URL
|
|
217
|
+
- Response: empty JSON object on success
|
|
218
|
+
|
|
219
|
+
## Error Handling
|
|
220
|
+
|
|
221
|
+
- **404 on `https://huggingface.co/papers/{PAPER_ID}` or `md` endpoint**: the paper is not indexed on Hugging Face paper pages yet.
|
|
222
|
+
- **404 on `/api/papers/{PAPER_ID}`**: the paper may not be indexed on Hugging Face paper pages yet.
|
|
223
|
+
- **Paper ID not found**: verify the extracted arXiv ID, including any version suffix
|
|
224
|
+
|
|
225
|
+
### Fallbacks
|
|
226
|
+
|
|
227
|
+
If the Hugging Face paper page does not contain enough detail for the user's question:
|
|
228
|
+
|
|
229
|
+
- Check the regular paper page at `https://huggingface.co/papers/{PAPER_ID}`
|
|
230
|
+
- Fall back to the arXiv page or PDF for the original source:
|
|
231
|
+
- `https://arxiv.org/abs/{PAPER_ID}`
|
|
232
|
+
- `https://arxiv.org/pdf/{PAPER_ID}`
|
|
233
|
+
|
|
234
|
+
## Notes
|
|
235
|
+
|
|
236
|
+
- No authentication is required for public paper pages.
|
|
237
|
+
- Write endpoints such as claim authorship, index paper, and update paper links require `Authorization: Bearer $HF_TOKEN`.
|
|
238
|
+
- Prefer the `.md` endpoint for reliable machine-readable output.
|
|
239
|
+
- Prefer `/api/papers/{PAPER_ID}` when you need structured JSON fields instead of page markdown.
|