mlclaw 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. package/.agents/skills/mlclaw/SKILL.md +63 -18
  2. package/Dockerfile +21 -4
  3. package/README.md +70 -24
  4. package/assets/hf-logo.svg +8 -0
  5. package/assets/hf-tooling/manifest.json +50 -0
  6. package/assets/hf-tooling/skills/hf-cli/SKILL.md +218 -0
  7. package/assets/hf-tooling/skills/hf-mem/SKILL.md +79 -0
  8. package/assets/hf-tooling/skills/huggingface-best/SKILL.md +134 -0
  9. package/assets/hf-tooling/skills/huggingface-datasets/SKILL.md +107 -0
  10. package/assets/hf-tooling/skills/huggingface-gradio/SKILL.md +298 -0
  11. package/assets/hf-tooling/skills/huggingface-gradio/examples.md +613 -0
  12. package/assets/hf-tooling/skills/huggingface-local-models/SKILL.md +113 -0
  13. package/assets/hf-tooling/skills/huggingface-local-models/references/hardware.md +38 -0
  14. package/assets/hf-tooling/skills/huggingface-local-models/references/hub-discovery.md +178 -0
  15. package/assets/hf-tooling/skills/huggingface-local-models/references/quantization.md +256 -0
  16. package/assets/hf-tooling/skills/huggingface-papers/SKILL.md +239 -0
  17. package/assets/hf-tooling/skills/huggingface-spaces/SKILL.md +239 -0
  18. package/assets/hf-tooling/skills/huggingface-spaces/references/buckets.md +89 -0
  19. package/assets/hf-tooling/skills/huggingface-spaces/references/debugging.md +236 -0
  20. package/assets/hf-tooling/skills/huggingface-spaces/references/gradio.md +200 -0
  21. package/assets/hf-tooling/skills/huggingface-spaces/references/grants.md +56 -0
  22. package/assets/hf-tooling/skills/huggingface-spaces/references/inference-providers.md +85 -0
  23. package/assets/hf-tooling/skills/huggingface-spaces/references/known-errors.md +232 -0
  24. package/assets/hf-tooling/skills/huggingface-spaces/references/requirements.md +169 -0
  25. package/assets/hf-tooling/skills/huggingface-spaces/references/zerogpu.md +349 -0
  26. package/assets/hf-tooling/skills/huggingface-tool-builder/SKILL.md +120 -0
  27. package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.py +57 -0
  28. package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.sh +40 -0
  29. package/assets/hf-tooling/skills/huggingface-tool-builder/references/baseline_hf_api.tsx +57 -0
  30. package/assets/hf-tooling/skills/huggingface-tool-builder/references/find_models_by_paper.sh +230 -0
  31. package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_enrich_models.sh +96 -0
  32. package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_model_card_frontmatter.sh +188 -0
  33. package/assets/hf-tooling/skills/huggingface-tool-builder/references/hf_model_papers_auth.sh +171 -0
  34. package/assets/hf-tooling/skills/huggingface-zerogpu/SKILL.md +289 -0
  35. package/assets/hf-tooling/skills/huggingface-zerogpu/references/concurrency.md +79 -0
  36. package/assets/hf-tooling/skills/huggingface-zerogpu/references/cuda-and-deps.md +66 -0
  37. package/assets/hf-tooling/skills/huggingface-zerogpu/references/how-quota-works.md +74 -0
  38. package/assets/hf-tooling/skills/huggingface-zerogpu/references/how-zerogpu-works.md +50 -0
  39. package/assets/hf-tooling/templates/.agents/mcp/huggingface.json +11 -0
  40. package/assets/hf-tooling/templates/.env.example +5 -0
  41. package/assets/hf-tooling/templates/examples/huggingface/README.md +16 -0
  42. package/assets/hf-tooling/templates/examples/huggingface/bucket-sync.md +19 -0
  43. package/assets/hf-tooling/templates/examples/huggingface/dataset-upload.py +22 -0
  44. package/assets/hf-tooling/templates/examples/huggingface/hf-discover.md +13 -0
  45. package/assets/hf-tooling/templates/examples/huggingface/runtime-inspection.py +9 -0
  46. package/assets/mlclaw-control-ui/assets/index-D2TFes32.js +9 -0
  47. package/assets/mlclaw-control-ui/assets/index-DP72PFuv.css +1 -0
  48. package/assets/mlclaw-control-ui/index.html +13 -0
  49. package/assets/mlclaw.svg +298 -124
  50. package/dist/hf-tooling-seed.js +261 -0
  51. package/dist/mlclaw-space-runtime.js +6634 -3321
  52. package/dist/mlclaw.mjs +428 -178
  53. package/entrypoint.sh +4 -0
  54. package/mlclaw.ps1 +4 -2
  55. package/mlclaw.sh +5 -3
  56. package/openclaw.default.json +1 -0
  57. package/package.json +16 -2
  58. package/space/README.md +32 -2
  59. package/tsconfig.json +3 -2
@@ -0,0 +1,38 @@
1
+ ## Hardware Acceleration
2
+
3
+ ### Apple Silicon (Metal)
4
+
5
+ ```bash
6
+ make clean && make GGML_METAL=1
7
+ llama-cli -m model.gguf -ngl 99 -p "Hello"
8
+ ```
9
+
10
+ ### NVIDIA (CUDA)
11
+
12
+ ```bash
13
+ make clean && make GGML_CUDA=1
14
+ llama-cli -m model.gguf -ngl 35 -p "Hello"
15
+
16
+ # Hybrid for large models
17
+ llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20
18
+
19
+ # Multi-GPU split
20
+ llama-cli -m large-model.gguf --tensor-split 0.5,0.5 -ngl 60
21
+ ```
22
+
23
+ ### AMD (ROCm)
24
+
25
+ ```bash
26
+ make LLAMA_HIP=1
27
+ llama-cli -m model.gguf -ngl 999
28
+ ```
29
+
30
+ ### CPU
31
+
32
+ ```bash
33
+ # Match physical cores, not logical threads
34
+ llama-cli -m model.gguf -t 8 -p "Hello"
35
+
36
+ # BLAS acceleration
37
+ make LLAMA_OPENBLAS=1
38
+ ```
@@ -0,0 +1,178 @@
1
+ # Hugging Face URL Workflows for llama.cpp
2
+
3
+ Use URL-only workflows first. Do not require `hf` or API clients just to find GGUF files, choose a quant, or build a `llama-server` command.
4
+
5
+ ## Contents
6
+
7
+ - Core URLs
8
+ - Search for llama.cpp-compatible models
9
+ - Use the local-app page for the recommended quant
10
+ - Confirm exact files from the tree API
11
+ - Build the command
12
+ - Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
13
+ - Notes
14
+
15
+ ## Core URLs
16
+
17
+ ```text
18
+ Search:
19
+ https://huggingface.co/models?apps=llama.cpp&sort=trending
20
+
21
+ Search with text:
22
+ https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
23
+
24
+ Search with size bounds:
25
+ https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
26
+
27
+ Repo local-app view:
28
+ https://huggingface.co/<repo>?local-app=llama.cpp
29
+
30
+ Repo tree API:
31
+ https://huggingface.co/api/models/<repo>/tree/main?recursive=true
32
+
33
+ Repo file tree:
34
+ https://huggingface.co/<repo>/tree/main
35
+ ```
36
+
37
+ ## 1. Search for llama.cpp-compatible models
38
+
39
+ Start from the models page with `apps=llama.cpp`.
40
+
41
+ Use:
42
+
43
+ - `search=<term>` for model family names such as `Qwen`, `Gemma`, `Phi`, or `Mistral`
44
+ - `num_parameters=min:0,max:24B` or similar if the user has hardware limits
45
+ - `sort=trending` when the user wants popular repos right now
46
+
47
+ Do not start with random GGUF repos if the user has not chosen a model family yet. Search first, shortlist second.
48
+
49
+ Example: https://huggingface.co/models?search=Qwen&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
50
+
51
+ ## 2. Use the local-app page for the recommended quant
52
+
53
+ Open:
54
+
55
+ ```text
56
+ https://huggingface.co/<repo>?local-app=llama.cpp
57
+ ```
58
+
59
+ Extract, in order:
60
+
61
+ 1. The exact `Use this model` snippet, if it is visible as text
62
+ 2. The `Hardware compatibility` section from the fetched page text or HTML:
63
+ - quant label
64
+ - file size
65
+ - bit-depth grouping
66
+ 3. Any extra launch flags shown in the snippet, such as `--jinja`
67
+
68
+ Treat the HF local-app snippet as the source of truth when it is visible.
69
+
70
+ Do this by reading the URL itself, not by assuming the UI rendered in a browser. If the fetched page source does not expose `Hardware compatibility`, say that the section was not text-visible and fall back to the tree API plus generic guidance from `quantization.md`.
71
+
72
+ ## 3. Confirm exact files from the tree API
73
+
74
+ Open:
75
+
76
+ ```text
77
+ https://huggingface.co/api/models/<repo>/tree/main?recursive=true
78
+ ```
79
+
80
+ Treat the JSON response as the source of truth for repo inventory.
81
+
82
+ Keep entries where:
83
+
84
+ - `type` is `file`
85
+ - `path` ends with `.gguf`
86
+
87
+ Use these fields:
88
+
89
+ - `path` for the filename and subdirectory
90
+ - `size` for the byte size
91
+ - optionally `lfs.size` to confirm the LFS payload size
92
+
93
+ Separate files into:
94
+
95
+ - quantized single-file checkpoints, for example `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
96
+ - projector weights, usually `mmproj-*.gguf`
97
+ - BF16 shard files, usually under `BF16/`
98
+ - everything else
99
+
100
+ Ignore unless the user asks:
101
+
102
+ - `README.md`
103
+ - imatrix or calibration blobs
104
+
105
+ Use `https://huggingface.co/<repo>/tree/main` only as a human fallback if the API endpoint fails or the user wants the web view.
106
+
107
+ ## 4. Build the command
108
+
109
+ Preferred order:
110
+
111
+ 1. Copy the exact HF snippet from the local-app page
112
+ 2. If the page gives a clean quant label, use shorthand selection:
113
+
114
+ ```bash
115
+ llama-server -hf <repo>:<QUANT>
116
+ ```
117
+
118
+ 3. If you need an exact file from the tree API, use the file-specific form:
119
+
120
+ ```bash
121
+ llama-server --hf-repo <repo> --hf-file <filename.gguf>
122
+ ```
123
+
124
+ 4. For CLI usage instead of a server, use:
125
+
126
+ ```bash
127
+ llama-cli -hf <repo>:<QUANT>
128
+ ```
129
+
130
+ Use the exact-file form when the repo uses custom labels or nonstandard naming that could make `:<QUANT>` ambiguous.
131
+
132
+ ## 5. Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
133
+
134
+ Use these URLs:
135
+
136
+ ```text
137
+ https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
138
+ https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
139
+ https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main
140
+ ```
141
+
142
+ On the local-app page, the hardware compatibility section can expose entries such as:
143
+
144
+ - `UD-IQ4_XS` - 17.7 GB
145
+ - `UD-Q4_K_S` - 20.9 GB
146
+ - `UD-Q4_K_M` - 22.1 GB
147
+ - `UD-Q5_K_M` - 26.5 GB
148
+ - `UD-Q6_K` - 29.3 GB
149
+ - `Q8_0` - 36.9 GB
150
+
151
+ On the tree API, you can confirm exact filenames such as:
152
+
153
+ - `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
154
+ - `Qwen3.6-35B-A3B-UD-Q5_K_M.gguf`
155
+ - `Qwen3.6-35B-A3B-UD-Q6_K.gguf`
156
+ - `Qwen3.6-35B-A3B-Q8_0.gguf`
157
+ - `mmproj-F16.gguf`
158
+
159
+ Good final output for this repo:
160
+
161
+ ```text
162
+ Repo: unsloth/Qwen3.6-35B-A3B-GGUF
163
+ Recommended quant from HF: UD-Q4_K_M (22.1 GB)
164
+ llama-server: llama-server --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
165
+ Other GGUFs:
166
+ - Qwen3.6-35B-A3B-UD-Q5_K_M.gguf - 26.5 GB
167
+ - Qwen3.6-35B-A3B-UD-Q6_K.gguf - 29.3 GB
168
+ - Qwen3.6-35B-A3B-Q8_0.gguf - 36.9 GB
169
+ Projector:
170
+ - mmproj-F16.gguf - 899 MB
171
+ ```
172
+
173
+ ## Notes
174
+
175
+ - Repo-specific quant labels matter. Do not rewrite `UD-Q4_K_M` to `Q4_K_M` unless the page itself does.
176
+ - `mmproj` files are projector weights for multimodal models, not the main language model checkpoint.
177
+ - If the HF hardware compatibility panel is missing because the user has no hardware profile configured, or because the fetched page source did not expose it, still use the tree API plus generic quant guidance from `quantization.md`.
178
+ - If the repo already has GGUFs, do not jump straight to conversion workflows.
@@ -0,0 +1,256 @@
1
+ # GGUF Quantization Guide
2
+
3
+ Complete guide to GGUF quantization formats and model conversion.
4
+
5
+ ## Contents
6
+
7
+ - Hub-first quant selection
8
+ - Quantization Formats
9
+ - Converting Models
10
+ - K-Quantization Methods
11
+ - Quality Testing
12
+ - Use Case Guide
13
+ - Model Size Scaling
14
+ - Finding Pre-Quantized Models
15
+ - Importance Matrices (`imatrix`)
16
+ - Troubleshooting
17
+
18
+ ## Hub-first quant selection
19
+
20
+ Before using generic tables, open the model repo with:
21
+
22
+ ```text
23
+ https://huggingface.co/<repo>?local-app=llama.cpp
24
+ ```
25
+
26
+ Prefer the exact quant labels and sizes shown in the `Hardware compatibility` section of the fetched `?local-app=llama.cpp` page text or HTML. Then confirm the matching filenames in:
27
+
28
+ ```text
29
+ https://huggingface.co/api/models/<repo>/tree/main?recursive=true
30
+ ```
31
+
32
+ Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.
33
+
34
+ ## Quantization Formats
35
+
36
+ **GGUF** (GPT-Generated Unified Format) - Standard format for llama.cpp models.
37
+
38
+ ### Format Comparison
39
+
40
+ | Format | Perplexity | Size (7B) | Tokens/sec | Notes |
41
+ |--------|------------|-----------|------------|-------|
42
+ | FP16 | 5.9565 (baseline) | 13.0 GB | 15 tok/s | Original quality |
43
+ | Q8_0 | 5.9584 (+0.03%) | 7.0 GB | 25 tok/s | Nearly lossless |
44
+ | **Q6_K** | 5.9642 (+0.13%) | 5.5 GB | 30 tok/s | Best quality/size |
45
+ | **Q5_K_M** | 5.9796 (+0.39%) | 4.8 GB | 35 tok/s | Balanced |
46
+ | **Q4_K_M** | 6.0565 (+1.68%) | 4.1 GB | 40 tok/s | **Recommended** |
47
+ | Q4_K_S | 6.1125 (+2.62%) | 3.9 GB | 42 tok/s | Faster, lower quality |
48
+ | Q3_K_M | 6.3184 (+6.07%) | 3.3 GB | 45 tok/s | Small models only |
49
+ | Q2_K | 6.8673 (+15.3%) | 2.7 GB | 50 tok/s | Not recommended |
50
+
51
+ **Recommendation**: Use **Q4_K_M** for best balance of quality and speed.
52
+
53
+ ## Converting Models
54
+
55
+ ### Hugging Face to GGUF
56
+
57
+ ```bash
58
+ # 1. Download Hugging Face model
59
+ hf download meta-llama/Llama-2-7b-chat-hf \
60
+ --local-dir models/llama-2-7b-chat/
61
+
62
+ # 2. Convert to FP16 GGUF
63
+ python convert_hf_to_gguf.py \
64
+ models/llama-2-7b-chat/ \
65
+ --outtype f16 \
66
+ --outfile models/llama-2-7b-chat-f16.gguf
67
+
68
+ # 3. Quantize to Q4_K_M
69
+ ./llama-quantize \
70
+ models/llama-2-7b-chat-f16.gguf \
71
+ models/llama-2-7b-chat-Q4_K_M.gguf \
72
+ Q4_K_M
73
+ ```
74
+
75
+ ### Batch quantization
76
+
77
+ ```bash
78
+ # Quantize to multiple formats
79
+ for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
80
+ ./llama-quantize \
81
+ model-f16.gguf \
82
+ model-${quant}.gguf \
83
+ $quant
84
+ done
85
+ ```
86
+
87
+ ## K-Quantization Methods
88
+
89
+ **K-quants** use mixed precision for better quality:
90
+ - Attention weights: Higher precision
91
+ - Feed-forward weights: Lower precision
92
+
93
+ **Variants**:
94
+ - `_S` (Small): Faster, lower quality
95
+ - `_M` (Medium): Balanced (recommended)
96
+ - `_L` (Large): Better quality, larger size
97
+
98
+ **Example**: `Q4_K_M`
99
+ - `Q4`: 4-bit quantization
100
+ - `K`: Mixed precision method
101
+ - `M`: Medium quality
102
+
103
+ ## Quality Testing
104
+
105
+ ```bash
106
+ # Calculate perplexity (quality metric)
107
+ ./llama-perplexity \
108
+ -m model.gguf \
109
+ -f wikitext-2-raw/wiki.test.raw \
110
+ -c 512
111
+
112
+ # Lower perplexity = better quality
113
+ # Baseline (FP16): ~5.96
114
+ # Q4_K_M: ~6.06 (+1.7%)
115
+ # Q2_K: ~6.87 (+15.3% - too much degradation)
116
+ ```
117
+
118
+ ## Use Case Guide
119
+
120
+ ### General purpose (chatbots, assistants)
121
+ ```
122
+ Q4_K_M - Best balance
123
+ Q5_K_M - If you have extra RAM
124
+ ```
125
+
126
+ ### Code generation
127
+ ```
128
+ Q5_K_M or Q6_K - Higher precision helps with code
129
+ ```
130
+
131
+ ### Creative writing
132
+ ```
133
+ Q4_K_M - Sufficient quality
134
+ Q3_K_M - Acceptable for draft generation
135
+ ```
136
+
137
+ ### Technical/medical
138
+ ```
139
+ Q6_K or Q8_0 - Maximum accuracy
140
+ ```
141
+
142
+ ### Edge devices (Raspberry Pi)
143
+ ```
144
+ Q2_K or Q3_K_S - Fit in limited RAM
145
+ ```
146
+
147
+ ## Model Size Scaling
148
+
149
+ ### 7B parameter models
150
+
151
+ | Format | Size | RAM needed |
152
+ |--------|------|------------|
153
+ | Q2_K | 2.7 GB | 5 GB |
154
+ | Q3_K_M | 3.3 GB | 6 GB |
155
+ | Q4_K_M | 4.1 GB | 7 GB |
156
+ | Q5_K_M | 4.8 GB | 8 GB |
157
+ | Q6_K | 5.5 GB | 9 GB |
158
+ | Q8_0 | 7.0 GB | 11 GB |
159
+
160
+ ### 13B parameter models
161
+
162
+ | Format | Size | RAM needed |
163
+ |--------|------|------------|
164
+ | Q2_K | 5.1 GB | 8 GB |
165
+ | Q3_K_M | 6.2 GB | 10 GB |
166
+ | Q4_K_M | 7.9 GB | 12 GB |
167
+ | Q5_K_M | 9.2 GB | 14 GB |
168
+ | Q6_K | 10.7 GB | 16 GB |
169
+
170
+ ### 70B parameter models
171
+
172
+ | Format | Size | RAM needed |
173
+ |--------|------|------------|
174
+ | Q2_K | 26 GB | 32 GB |
175
+ | Q3_K_M | 32 GB | 40 GB |
176
+ | Q4_K_M | 41 GB | 48 GB |
177
+ | Q4_K_S | 39 GB | 46 GB |
178
+ | Q5_K_M | 48 GB | 56 GB |
179
+
180
+ **Recommendation for 70B**: Use Q3_K_M or Q4_K_S to fit in consumer hardware.
181
+
182
+ ## Finding Pre-Quantized Models
183
+
184
+ Use the Hub search with the llama.cpp app filter:
185
+
186
+ ```text
187
+ https://huggingface.co/models?apps=llama.cpp&sort=trending
188
+ https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
189
+ https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
190
+ ```
191
+
192
+ For a specific repo, open:
193
+
194
+ ```text
195
+ https://huggingface.co/<repo>?local-app=llama.cpp
196
+ https://huggingface.co/api/models/<repo>/tree/main?recursive=true
197
+ ```
198
+
199
+ Then launch directly from the Hub without extra Hub tooling:
200
+
201
+ ```bash
202
+ llama-cli -hf <repo>:Q4_K_M
203
+ llama-server -hf <repo>:Q4_K_M
204
+ ```
205
+
206
+ If you need the exact file name from the tree API:
207
+
208
+ ```bash
209
+ llama-server --hf-repo <repo> --hf-file <filename.gguf>
210
+ ```
211
+
212
+ ## Importance Matrices (imatrix)
213
+
214
+ **What**: Calibration data to improve quantization quality.
215
+
216
+ **Benefits**:
217
+ - 10-20% perplexity improvement with Q4
218
+ - Essential for Q3 and below
219
+
220
+ **Usage**:
221
+ ```bash
222
+ # 1. Generate importance matrix
223
+ ./llama-imatrix \
224
+ -m model-f16.gguf \
225
+ -f calibration-data.txt \
226
+ -o model.imatrix
227
+
228
+ # 2. Quantize with imatrix
229
+ ./llama-quantize \
230
+ --imatrix model.imatrix \
231
+ model-f16.gguf \
232
+ model-Q4_K_M.gguf \
233
+ Q4_K_M
234
+ ```
235
+
236
+ **Calibration data**:
237
+ - Use domain-specific text (e.g., code for code models)
238
+ - ~100MB of representative text
239
+ - Higher quality data = better quantization
240
+
241
+ ## Troubleshooting
242
+
243
+ **Model outputs gibberish**:
244
+ - Quantization too aggressive (Q2_K)
245
+ - Try Q4_K_M or Q5_K_M
246
+ - Verify model converted correctly
247
+
248
+ **Out of memory**:
249
+ - Use lower quantization (Q4_K_S instead of Q5_K_M)
250
+ - Offload fewer layers to GPU (`-ngl`)
251
+ - Use smaller context (`-c 2048`)
252
+
253
+ **Slow inference**:
254
+ - Higher quantization uses more compute
255
+ - Q8_0 much slower than Q4_K_M
256
+ - Consider speed vs quality trade-off
@@ -0,0 +1,239 @@
1
+ ---
2
+ name: huggingface-papers
3
+ description: Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. Use when the user shares a Hugging Face paper page URL, an arXiv URL or ID, or asks to summarize, explain, or analyze an AI research paper.
4
+ ---
5
+
6
+ # Hugging Face Paper Pages
7
+
8
+ Hugging Face Paper pages (hf.co/papers) is a platform built on top of arXiv (arxiv.org), specifically for research papers in the field of artificial intelligence (AI) and computer science. Hugging Face users can submit their paper at hf.co/papers/submit, which features it on the Daily Papers feed (hf.co/papers). Each day, users can upvote papers and comment on papers. Each paper page allows authors to:
9
+ - claim their paper (by clicking their name on the `authors` field). This makes the paper page appear on their Hugging Face profile.
10
+ - link the associated model checkpoints, datasets and Spaces by including the HF paper or arXiv URL in the model card, dataset card or README of the Space
11
+ - link the Github repository and/or project page URLs
12
+ - link the HF organization. This also makes the paper page appear on the Hugging Face organization page.
13
+
14
+ Whenever someone mentions a HF paper or arXiv abstract/PDF URL in a model card, dataset card or README of a Space repository, the paper will be automatically indexed. Note that not all papers indexed on Hugging Face are also submitted to daily papers. The latter is more a manner of promoting a research paper. Papers can only be submitted to daily papers up until 14 days after their publication date on arXiv.
15
+
16
+ The Hugging Face team has built an easy-to-use API to interact with paper pages. Content of the papers can be fetched as markdown, or structured metadata can be returned such as author names, linked models/datasets/spaces, linked Github repo and project page.
17
+
18
+ ## When to Use
19
+
20
+ - User shares a Hugging Face paper page URL (e.g. `https://huggingface.co/papers/2602.08025`)
21
+ - User shares a Hugging Face markdown paper page URL (e.g. `https://huggingface.co/papers/2602.08025.md`)
22
+ - User shares an arXiv URL (e.g. `https://arxiv.org/abs/2602.08025` or `https://arxiv.org/pdf/2602.08025`)
23
+ - User mentions a arXiv ID (e.g. `2602.08025`)
24
+ - User asks you to summarize, explain, or analyze an AI research paper
25
+
26
+ ## Parsing the paper ID
27
+
28
+ It's recommended to parse the paper ID (arXiv ID) from whatever the user provides:
29
+
30
+ | Input | Paper ID |
31
+ | --- | --- |
32
+ | `https://huggingface.co/papers/2602.08025` | `2602.08025` |
33
+ | `https://huggingface.co/papers/2602.08025.md` | `2602.08025` |
34
+ | `https://arxiv.org/abs/2602.08025` | `2602.08025` |
35
+ | `https://arxiv.org/pdf/2602.08025` | `2602.08025` |
36
+ | `2602.08025v1` | `2602.08025v1` |
37
+ | `2602.08025` | `2602.08025` |
38
+
39
+ This allows you to provide the paper ID into any of the hub API endpoints mentioned below.
40
+
41
+ ### Fetch the paper page as markdown
42
+
43
+ The content of a paper can be fetched as markdown like so:
44
+
45
+ ```bash
46
+ curl -s "https://huggingface.co/papers/{PAPER_ID}.md"
47
+ ```
48
+
49
+ This should return the Hugging Face paper page as markdown. This relies on the HTML version of the paper at https://arxiv.org/html/{PAPER_ID}.
50
+
51
+ There are 2 exceptions:
52
+ - Not all arXiv papers have an HTML version. If the HTML version of the paper does not exist, then the content falls back to the HTML of the Hugging Face paper page.
53
+ - If it results in a 404, it means the paper is not yet indexed on hf.co/papers. See [Error handling](#error-handling) for info.
54
+
55
+ Alternatively, you can request markdown from the normal paper page URL, like so:
56
+
57
+ ```bash
58
+ curl -s -H "Accept: text/markdown" "https://huggingface.co/papers/{PAPER_ID}"
59
+ ```
60
+
61
+ ### Paper Pages API Endpoints
62
+
63
+ All endpoints use the base URL `https://huggingface.co`.
64
+
65
+ #### Get structured metadata
66
+
67
+ Fetch the paper metadata as JSON using the Hugging Face REST API:
68
+
69
+ ```bash
70
+ curl -s "https://huggingface.co/api/papers/{PAPER_ID}"
71
+ ```
72
+
73
+ This returns structured metadata that can include:
74
+
75
+ - authors (names and Hugging Face usernames, in case they have claimed the paper)
76
+ - media URLs (uploaded when submitting the paper to Daily Papers)
77
+ - summary (abstract) and AI-generated summary
78
+ - project page and GitHub repository
79
+ - organization and engagement metadata (number of upvotes)
80
+
81
+ To find models linked to the paper, use:
82
+
83
+ ```bash
84
+ curl https://huggingface.co/api/models?filter=arxiv:{PAPER_ID}
85
+ ```
86
+
87
+ To find datasets linked to the paper, use:
88
+
89
+ ```bash
90
+ curl https://huggingface.co/api/datasets?filter=arxiv:{PAPER_ID}
91
+ ```
92
+
93
+ To find spaces linked to the paper, use:
94
+
95
+ ```bash
96
+ curl https://huggingface.co/api/spaces?filter=arxiv:{PAPER_ID}
97
+ ```
98
+
99
+ #### Claim paper authorship
100
+
101
+ Claim authorship of a paper for a Hugging Face user:
102
+
103
+ ```bash
104
+ curl "https://huggingface.co/api/settings/papers/claim" \
105
+ --request POST \
106
+ --header "Content-Type: application/json" \
107
+ --header "Authorization: Bearer $HF_TOKEN" \
108
+ --data '{
109
+ "paperId": "{PAPER_ID}",
110
+ "claimAuthorId": "{AUTHOR_ENTRY_ID}",
111
+ "targetUserId": "{USER_ID}"
112
+ }'
113
+ ```
114
+
115
+ - Endpoint: `POST /api/settings/papers/claim`
116
+ - Body:
117
+ - `paperId` (string, required): arXiv paper identifier being claimed
118
+ - `claimAuthorId` (string): author entry on the paper being claimed, 24-char hex ID
119
+ - `targetUserId` (string): HF user who should receive the claim, 24-char hex ID
120
+ - Response: paper authorship claim result, including the claimed paper ID
121
+
122
+ #### Get daily papers
123
+
124
+ Fetch the Daily Papers feed:
125
+
126
+ ```bash
127
+ curl -s -H "Authorization: Bearer $HF_TOKEN" \
128
+ "https://huggingface.co/api/daily_papers?p=0&limit=20&date=2017-07-21&sort=publishedAt"
129
+ ```
130
+
131
+ - Endpoint: `GET /api/daily_papers`
132
+ - Query parameters:
133
+ - `p` (integer): page number
134
+ - `limit` (integer): number of results, between 1 and 100
135
+ - `date` (string): RFC 3339 full-date, for example `2017-07-21`
136
+ - `week` (string): ISO week, for example `2024-W03`
137
+ - `month` (string): month value, for example `2024-01`
138
+ - `submitter` (string): filter by submitter
139
+ - `sort` (enum): `publishedAt` or `trending`
140
+ - Response: list of daily papers
141
+
142
+ #### List papers
143
+
144
+ List arXiv papers sorted by published date:
145
+
146
+ ```bash
147
+ curl -s -H "Authorization: Bearer $HF_TOKEN" \
148
+ "https://huggingface.co/api/papers?cursor={CURSOR}&limit=20"
149
+ ```
150
+
151
+ - Endpoint: `GET /api/papers`
152
+ - Query parameters:
153
+ - `cursor` (string): pagination cursor
154
+ - `limit` (integer): number of results, between 1 and 100
155
+ - Response: list of papers
156
+
157
+ #### Search papers
158
+
159
+ Perform hybrid semantic and full-text search on papers:
160
+
161
+ ```bash
162
+ curl -s -H "Authorization: Bearer $HF_TOKEN" \
163
+ "https://huggingface.co/api/papers/search?q=vision+language&limit=20"
164
+ ```
165
+
166
+ This searches over the paper title, authors, and content.
167
+
168
+ - Endpoint: `GET /api/papers/search`
169
+ - Query parameters:
170
+ - `q` (string): search query, max length 250
171
+ - `limit` (integer): number of results, between 1 and 120
172
+ - Response: matching papers
173
+
174
+ #### Index a paper
175
+
176
+ Insert a paper from arXiv by ID. If the paper is already indexed, only its authors can re-index it:
177
+
178
+ ```bash
179
+ curl "https://huggingface.co/api/papers/index" \
180
+ --request POST \
181
+ --header "Content-Type: application/json" \
182
+ --header "Authorization: Bearer $HF_TOKEN" \
183
+ --data '{
184
+ "arxivId": "{ARXIV_ID}"
185
+ }'
186
+ ```
187
+
188
+ - Endpoint: `POST /api/papers/index`
189
+ - Body:
190
+ - `arxivId` (string, required): arXiv ID to index, for example `2301.00001`
191
+ - Pattern: `^\d{4}\.\d{4,5}$`
192
+ - Response: empty JSON object on success
193
+
194
+ #### Update paper links
195
+
196
+ Update the project page, GitHub repository, or submitting organization for a paper. The requester must be the paper author, the Daily Papers submitter, or a papers admin:
197
+
198
+ ```bash
199
+ curl "https://huggingface.co/api/papers/{PAPER_OBJECT_ID}/links" \
200
+ --request POST \
201
+ --header "Content-Type: application/json" \
202
+ --header "Authorization: Bearer $HF_TOKEN" \
203
+ --data '{
204
+ "projectPage": "https://example.com",
205
+ "githubRepo": "https://github.com/org/repo",
206
+ "organizationId": "{ORGANIZATION_ID}"
207
+ }'
208
+ ```
209
+
210
+ - Endpoint: `POST /api/papers/{paperId}/links`
211
+ - Path parameters:
212
+ - `paperId` (string, required): Hugging Face paper object ID
213
+ - Body:
214
+ - `githubRepo` (string, nullable): GitHub repository URL
215
+ - `organizationId` (string, nullable): organization ID, 24-char hex ID
216
+ - `projectPage` (string, nullable): project page URL
217
+ - Response: empty JSON object on success
218
+
219
+ ## Error Handling
220
+
221
+ - **404 on `https://huggingface.co/papers/{PAPER_ID}` or `md` endpoint**: the paper is not indexed on Hugging Face paper pages yet.
222
+ - **404 on `/api/papers/{PAPER_ID}`**: the paper may not be indexed on Hugging Face paper pages yet.
223
+ - **Paper ID not found**: verify the extracted arXiv ID, including any version suffix
224
+
225
+ ### Fallbacks
226
+
227
+ If the Hugging Face paper page does not contain enough detail for the user's question:
228
+
229
+ - Check the regular paper page at `https://huggingface.co/papers/{PAPER_ID}`
230
+ - Fall back to the arXiv page or PDF for the original source:
231
+ - `https://arxiv.org/abs/{PAPER_ID}`
232
+ - `https://arxiv.org/pdf/{PAPER_ID}`
233
+
234
+ ## Notes
235
+
236
+ - No authentication is required for public paper pages.
237
+ - Write endpoints such as claim authorship, index paper, and update paper links require `Authorization: Bearer $HF_TOKEN`.
238
+ - Prefer the `.md` endpoint for reliable machine-readable output.
239
+ - Prefer `/api/papers/{PAPER_ID}` when you need structured JSON fields instead of page markdown.