bitsandbytes-cpu-fork 0.50.2.dev6__py3-none-win_amd64.whl → 0.50.2.dev9__py3-none-win_amd64.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,452 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: bitsandbytes-cpu-fork
3
- Version: 0.50.2.dev6
4
- Summary: bitsandbytes with a pure-CPU training backend (AVX2 / NEON): 8-bit optimizers, fused GDN, gemm_8bit, disk balancer. No GPU required.
5
- License-Expression: MIT
6
- Project-URL: repository, https://github.com/Ganyvnanoaa09-0906/bitsandbytes-CPU
7
- Project-URL: Upstream bitsandbytes, https://github.com/bitsandbytes-foundation/bitsandbytes
8
- Keywords: cpu,optimizers,optimization,8-bit,quantization,compression,avx2,neon
9
- Classifier: Development Status :: 4 - Beta
10
- Classifier: Intended Audience :: Developers
11
- Classifier: Intended Audience :: Science/Research
12
- Classifier: Operating System :: Microsoft :: Windows
13
- Classifier: Operating System :: POSIX :: Linux
14
- Classifier: Programming Language :: C++
15
- Classifier: Programming Language :: Python :: Implementation :: CPython
16
- Classifier: Programming Language :: Python :: 3.10
17
- Classifier: Programming Language :: Python :: 3.11
18
- Classifier: Programming Language :: Python :: 3.12
19
- Classifier: Programming Language :: Python :: 3.13
20
- Classifier: Programming Language :: Python :: 3.14
21
- Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
22
- Requires-Python: >=3.10
23
- Description-Content-Type: text/markdown
24
- License-File: LICENSE
25
- Requires-Dist: numpy>=1.17
26
- Requires-Dist: packaging>=20.9
27
- Requires-Dist: psutil>=5.9
28
- Provides-Extra: test
29
- Requires-Dist: pytest~=8.3; extra == "test"
30
- Dynamic: license-file
31
-
32
- > **This is the CPU fork** — `bitsandbytes-cpu-fork`, not upstream `bitsandbytes`.
33
- > It replaces the CUDA kernels with a pure-CPU backend (AVX2 on x86-64, NEON on ARM64):
34
- > 8-bit optimizers, fused Gated DeltaNet, `gemm_8bit`, block-wise 4/8-bit quantisation.
35
- > No GPU, no CUDA toolkit, no NVIDIA driver. Full guide:
36
- > [`docs_cpu/QUICKSTART.md`](docs_cpu/QUICKSTART.md) (中文) ·
37
- > [`docs_cpu/QUICKSTART_EN.md`](docs_cpu/QUICKSTART_EN.md).
38
-
39
- ## Install
40
-
41
- **`torch` is deliberately not a dependency.** On PyPI the name `torch` resolves to the
42
- CUDA build, so declaring it made this CPU-only library download roughly 2.5 GB of
43
- `nvidia-*` wheels — cublas, cudnn, nccl, nvshmem and thirteen more — onto machines with
44
- no NVIDIA device at all. Measured on a clean Ubuntu install. `install_requires` cannot
45
- express "the CPU build", because that lives on a different index, so install it first:
46
-
47
- ```bash
48
- pip install torch --index-url https://download.pytorch.org/whl/cpu
49
- pip install bitsandbytes-cpu-fork
50
- ```
51
-
52
- **Linux wheels.** This release ships two Linux x86_64 wheels: a native one
53
- (`manylinux_2_39`) with the OpenMP-parallel kernels, and a portable one
54
- (`manylinux_2_17`, single-threaded) for older distributions. pip picks the best one
55
- for the host automatically. If neither matches your glibc, install from the sdist
56
- (it compiles `csrc` with the local toolchain via `build_linux.sh`).
57
-
58
-
59
- ### Windows
60
-
61
- A wheel is published with the CPU library already built in, so no compiler and no
62
- Visual C++ Redistributable are needed:
63
-
64
- ```bat
65
- pip install bitsandbytes-cpu-fork
66
- ```
67
-
68
- ### Linux
69
-
70
- Only the source distribution is published. `pip` builds it, and `setup.py` runs
71
- `build_linux.sh` for you, so the library is compiled against your own glibc rather than
72
- shipped prebuilt. That needs a C++ compiler with OpenMP:
73
-
74
- ```bash
75
- sudo apt install g++ libomp-dev # Debian / Ubuntu
76
- sudo dnf install gcc-c++ libomp-devel # Fedora / RHEL
77
- sudo pacman -S gcc openmp # Arch
78
- ```
79
-
80
- If the toolchain is missing the build **stops and names that command**, rather than
81
- producing a wheel that installs cleanly and then fails at `import bitsandbytes`.
82
-
83
- To build by hand instead, or to check the kernels without torch:
84
-
85
- ```bash
86
- bash build_linux.sh --selftest
87
- ```
88
-
89
- ### Verify the install
90
-
91
- ```bash
92
- python -m bitsandbytes
93
- ```
94
-
95
- ### Then ask it what it can do
96
-
97
- `pip` installs a `bitsandbytes-cpu` command (also available as `bnb-cpu`). It is the
98
- reference for this fork: what the hardware is, whether the kernels load, and **how to
99
- call every feature it adds** — each entry carries the exact call, and every Python
100
- example in it is executed before release.
101
-
102
- ```bash
103
- bitsandbytes-cpu detect # CPU model, cores, SIMD, RAM, recommended thread count
104
- bitsandbytes-cpu selftest # runs a 4-bit layer, an 8-bit optimizer step, the GDN kernel
105
- bitsandbytes-cpu help # the whole reference
106
- bitsandbytes-cpu help 8bitopt # one section: the 8-bit optimizer
107
- bitsandbytes-cpu help 4bit # one section: 4-bit layers on a CPU
108
- bitsandbytes-cpu help kernels # one section: the fused kernels, called directly
109
- bitsandbytes-cpu doctor # which .dll/.so will load, and which symbols it has
110
- ```
111
-
112
- Section keys: `intro start 4bit qlora 8bitopt kernels toolkit threads memory gdn disks
113
- layers optim functional notes`. The same tables are published as
114
- [`docs_cpu/API_REFERENCE.md`](docs_cpu/API_REFERENCE.md)
115
- ([English](docs_cpu/API_REFERENCE_EN.md)).
116
-
117
- ---
118
-
119
- <p align="center"><img src="https://avatars.githubusercontent.com/u/175231607?s=200&v=4" alt=""></p>
120
- <h1 align="center">bitsandbytes</h1>
121
- <p align="center">
122
- <a href="https://github.com/bitsandbytes-foundation/bitsandbytes/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/bitsandbytes-foundation/bitsandbytes.svg?color=blue"></a>
123
- <a href="https://pepy.tech/project/bitsandbytes"><img alt="Downloads" src="https://static.pepy.tech/badge/bitsandbytes/month"></a>
124
- <a href="https://github.com/bitsandbytes-foundation/bitsandbytes/actions/workflows/tests-nightly.yml"><img alt="Nightly Unit Tests" src="https://img.shields.io/github/actions/workflow/status/bitsandbytes-foundation/bitsandbytes/tests-nightly.yml?logo=github&label=Nightly%20Tests"></a>
125
- <a href="https://github.com/bitsandbytes-foundation/bitsandbytes/releases"><img alt="GitHub Release" src="https://img.shields.io/github/v/release/bitsandbytes-foundation/bitsandbytes"></a>
126
- <a href="https://pypi.org/project/bitsandbytes/"><img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/bitsandbytes"></a>
127
- </p>
128
-
129
- `bitsandbytes` enables accessible large language models via k-bit quantization for PyTorch. We provide three main features for dramatically reducing memory consumption for inference and training:
130
-
131
- * 8-bit optimizers uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
132
- * LLM.int8() or 8-bit quantization enables large language model inference with only half the required memory and without any performance degradation. This method is based on vector-wise quantization to quantize most features to 8-bits and separately treating outliers with 16-bit matrix multiplication.
133
- * QLoRA or 4-bit quantization enables large language model training with several memory-saving techniques that don't compromise performance. This method quantizes a model to 4-bits and inserts a small set of trainable low-rank adaptation (LoRA) weights to allow training.
134
-
135
- The library includes quantization primitives for 8-bit & 4-bit operations, through `bitsandbytes.nn.Linear8bitLt` and `bitsandbytes.nn.Linear4bit` and 8-bit optimizers through `bitsandbytes.optim` module.
136
-
137
- ## System Requirements
138
- bitsandbytes has the following minimum requirements for all platforms:
139
-
140
- * Python 3.10+
141
- * [PyTorch](https://pytorch.org/get-started/locally/) 2.4+
142
- * _Note: While we aim to provide wide backwards compatibility, we recommend using the latest version of PyTorch for the best experience._
143
-
144
- #### Accelerator support:
145
-
146
- <small>Note: this table reflects the status of the current development branch. For the latest stable release, see the
147
- [document in the 0.50.0 tag](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.50.0/README.md#accelerator-support).
148
- </small>
149
-
150
- ##### Legend:
151
- 🚧 = Planned |
152
- 〰️ = Partially Supported |
153
- ✅ = Supported |
154
- ❌ = Not Supported
155
-
156
- <table>
157
- <thead>
158
- <tr>
159
- <th>Platform</th>
160
- <th>Accelerator</th>
161
- <th>Hardware Requirements</th>
162
- <th>LLM.int8()</th>
163
- <th>QLoRA 4-bit</th>
164
- <th>8-bit Optimizers</th>
165
- </tr>
166
- </thead>
167
- <tbody>
168
- <tr>
169
- <td colspan="6">🐧 <strong>Linux, glibc >= 2.24</strong></td>
170
- </tr>
171
- <tr>
172
- <td align="right">x86-64</td>
173
- <td>◻️ CPU</td>
174
- <td>Minimum: AVX2<br>Optimized: AVX512F, AVX512BF16</td>
175
- <td>✅</td>
176
- <td>✅</td>
177
- <td>✅</td>
178
- </tr>
179
- <tr>
180
- <td></td>
181
- <td>🟩 NVIDIA GPU <br><code>cuda</code></td>
182
- <td>SM60+ minimum<br>SM75+ recommended</td>
183
- <td>✅</td>
184
- <td>✅</td>
185
- <td>✅</td>
186
- </tr>
187
- <tr>
188
- <td></td>
189
- <td>🟥 AMD GPU <br><code>cuda</code></td>
190
- <td>
191
- CDNA: gfx908, gfx90a, gfx942, gfx950, gfx1250<br>
192
- RDNA: gfx101X, gfx103X, gfx110X, gfx115X, gfx120X
193
- </td>
194
- <td>✅</td>
195
- <td>✅</td>
196
- <td>✅</td>
197
- </tr>
198
- <tr>
199
- <td></td>
200
- <td>🟦 Intel GPU <br><code>xpu</code></td>
201
- <td>
202
- Data Center GPU Max Series<br>
203
- Arc A-Series (Alchemist)<br>
204
- Arc B-Series (Battlemage)
205
- </td>
206
- <td>✅</td>
207
- <td>✅</td>
208
- <td>✅</td>
209
- </tr>
210
- <tr>
211
- <td></td>
212
- <td>🟪 Intel Gaudi <br><code>hpu</code></td>
213
- <td>Gaudi2, Gaudi3</td>
214
- <td>✅</td>
215
- <td>〰️</td>
216
- <td>❌</td>
217
- </tr>
218
- <tr>
219
- <td align="right">aarch64</td>
220
- <td>◻️ CPU</td>
221
- <td></td>
222
- <td>✅ *</td>
223
- <td>✅</td>
224
- <td>✅</td>
225
- </tr>
226
- <tr>
227
- <td></td>
228
- <td>🟩 NVIDIA GPU <br><code>cuda</code></td>
229
- <td>SM75+</td>
230
- <td>✅</td>
231
- <td>✅</td>
232
- <td>✅</td>
233
- </tr>
234
- <tr>
235
- <td colspan="6">🪟 <strong>Windows 11 / Windows Server 2022+</strong></td>
236
- </tr>
237
- <tr>
238
- <td align="right">x86-64</td>
239
- <td>◻️ CPU</td>
240
- <td>AVX2</td>
241
- <td>✅</td>
242
- <td>✅</td>
243
- <td>✅</td>
244
- </tr>
245
- <tr>
246
- <td></td>
247
- <td>🟩 NVIDIA GPU <br><code>cuda</code></td>
248
- <td>SM60+ minimum<br>SM75+ recommended</td>
249
- <td>✅</td>
250
- <td>✅</td>
251
- <td>✅</td>
252
- </tr>
253
- <tr>
254
- <td></td>
255
- <td>🟥 AMD GPU <br><code>cuda</code></td>
256
- <td>
257
- CDNA: gfx908, gfx90a<br>
258
- RDNA: gfx101X, gfx103X, gfx110X, gfx115X, gfx120X
259
- </td>
260
- <td>✅</td>
261
- <td>✅</td>
262
- <td>✅</td>
263
- </tr>
264
- <tr>
265
- <td></td>
266
- <td>🟦 Intel GPU <br><code>xpu</code></td>
267
- <td>
268
- Arc A-Series (Alchemist) <br>
269
- Arc B-Series (Battlemage)
270
- </td>
271
- <td>✅</td>
272
- <td>✅</td>
273
- <td>✅</td>
274
- </tr>
275
- <tr>
276
- <td align="right">arm64</td>
277
- <td>◻️ CPU</td>
278
- <td></td>
279
- <td>✅</td>
280
- <td>✅</td>
281
- <td>✅</td>
282
- </tr>
283
- <tr>
284
- <td></td>
285
- <td>🟩 NVIDIA GPU <br><code>cuda</code></td>
286
- <td>SM121</td>
287
- <td>✅</td>
288
- <td>✅</td>
289
- <td>✅</td>
290
- </tr>
291
- <tr>
292
- <td colspan="6">🍎 <strong>macOS 14+</strong></td>
293
- </tr>
294
- <tr>
295
- <td align="right">arm64</td>
296
- <td>◻️ CPU</td>
297
- <td>Apple M1+</td>
298
- <td>✅ *</td>
299
- <td>✅</td>
300
- <td>✅</td>
301
- </tr>
302
- <tr>
303
- <td></td>
304
- <td>⬜ Metal <br><code>mps</code></td>
305
- <td>Apple M1+</td>
306
- <td>✅ *</td>
307
- <td>✅</td>
308
- <td>🚧</td>
309
- </tbody>
310
- </table>
311
- <sup>* While supported, these marked features may lack in performance optimizations.</sup>
312
-
313
- ## :book: Documentation
314
- * [Official Documentation](https://huggingface.co/docs/bitsandbytes/main)
315
- * 🤗 [Transformers](https://huggingface.co/docs/transformers/quantization/bitsandbytes)
316
- * 🤗 [Diffusers](https://huggingface.co/docs/diffusers/quantization/bitsandbytes)
317
- * 🤗 [PEFT](https://huggingface.co/docs/peft/developer_guides/quantization#quantize-a-model)
318
-
319
- ## :computer: CPU Backend Extensions (this fork)
320
-
321
- This fork adds **training-grade CPU kernels** to the bitsandbytes CPU backend, for
322
- machines **without NVIDIA GPUs** (AVX2 only / ARM64 NEON, e.g. consumer laptops,
323
- Mini-PCs). The upstream CPU backend is inference-oriented; these
324
- extensions make pure-CPU LoRA / quantization / diffusion training practical.
325
-
326
- ### Install (Windows, no compiler needed)
327
-
328
- ```bash
329
- pip install bitsandbytes-cpu-fork
330
- ```
331
-
332
- The wheel bundles the prebuilt CPU kernel and its OpenMP runtime, so nothing has to
333
- be built after installing; `pip` pulls `torch` and `numpy` itself. One wheel covers
334
- every Python 3 version on 64-bit Windows. The import name is unchanged, so existing
335
- code and the Transformers / PEFT / Diffusers integrations keep working as-is:
336
-
337
- ```python
338
- import bitsandbytes as bnb # the fork, not upstream
339
- print(bnb.__version__) # 0.50.2.dev6
340
- ```
341
-
342
- Building from source instead (any platform) is described in
343
- [`docs_cpu/QUICKSTART.md`](docs_cpu/QUICKSTART.md) §2 and §2A.
344
-
345
- > 📚 **Docs** (`docs_cpu/`):
346
- > - `TECHNICAL_GUIDE.md` — **技术文档**:改了什么、每个内核的架构与细节(面向技术人员)
347
- > - `QUICKSTART.md` — **快速入门手册**:命令、用途与参数(面向首次使用者)
348
- > - `TECH_REPORT.md` — **技术报告**:完整方法论、实验数据与结论(含负面结论)
349
- > - 灾难恢复指南已并入 `QUICKSTART.md` §9。
350
- > - 🌐 **English**: 英文为**独立文件** —— `QUICKSTART_EN.md`、`TECHNICAL_GUIDE_EN.md`、
351
- > `TECH_REPORT_EN.md`(内容与中文版对应,以中文为准;灾难恢复同样在 `QUICKSTART_EN.md` §9)。
352
- >
353
- > 🌐 **English README**: see [`README_EN.md`](README_EN.md).
354
- >
355
- > > **关于硬盘均衡负载**:它能在内存紧张时帮助维持训练,但**不能替代 Windows 虚拟内存**。
356
- > > 训练期间仍**建议保留至少 1GB 虚拟内存**以应对突发内存峰值。
357
-
358
- ### What's added
359
-
360
- | Kernel | Entry point | Purpose |
361
- |---|---|---|
362
- | **Gated DeltaNet fwd/bwd** | `gdn_fwd_cpu` / `gdn_bwd_cpu` (csrc/cpu_gdn.cpp) | Fused Gated DeltaNet (Qwen3-Next/3.5 linear attention) — ~35× vs per-step Python loop; chunk-checkpointed backward |
363
- | **Fused 8-bit blockwise dequant GEMM** | `cgemm_8bit_inference_cpu_fp32` (`bitsandbytes::gemm_8bit`, `bnb.functional.fused_dequant_linear_8bit`) | `out = A @ dequant8(B)` with weights kept uint8 (¼ DRAM traffic), no fp32 temporary |
364
- | **Fused 8-bit optimizer** | `coptimizer_update_8bit_blockwise_cpu` (`bnb.optim.AdamW8bit` etc.) | Single-pass dequant→update→requant; optimizer state memory ≈ 1/3.8 of fp32 |
365
- | **Blockwise 8/4-bit quant & dequant** | `cquantize_blockwise_cpu_*` / `cdequantize_blockwise_cpu_*` | AVX2 LUT quantize + dequantize; NF4/FP4 kernels with NEON/AVX2 paths |
366
- | **4-bit inference GEMV** | `cgemv_4bit_inference_cpu_*` | Fused 4-bit dequant GEMV/GEMM for AVX2 machines (symbol alias fixed) |
367
- | **GDN runtime patch** | `bitsandbytes/gdn_cpu.py` | `patch_transformers()` / `patch_fla()` — route transformers GDN slow paths to the fused kernel |
368
- | **iGPU (DirectML) block-level executor** | `gpu_scheduler.py` (`IgpuExecutor`, `train.py --igpu`) | Experimental: frozen weights resident on the DirectML iGPU, forward/backward on-device, **one sync per step**; **off by default** — measured **+30~35%** for GEMM-dense seq≤256 training, **negative** at long sequences (driver limits) |
369
-
370
- ### Build
371
-
372
- ```bash
373
- # Linux (x86_64 AVX2 or aarch64 NEON, g++/clang + OpenMP)
374
- # deps: sudo apt install g++ libomp-dev (Debian/Ubuntu)
375
- bash build_linux.sh # -> bitsandbytes/libbitsandbytes_cpu.so
376
- bash build_linux.sh --selftest # also build+run torch-free C self-test
377
-
378
- # Windows (MSVC, manual cl build, no CMake)
379
- build_manual\build_manual.bat amd # (in VS x64 Native Tools prompt)
380
- ```
381
-
382
- > `.gitattributes` 强制 C/C++ 源码与构建脚本用 LF 换行,clone 到 Windows/Linux 均可
383
- > 直接编译(CRLF 会让 MSVC 吞行、Linux `g++` 报 `bad interpreter`)。
384
- > `disk_balancer`(`--flash`)跨平台:Windows 用 ctypes;Linux 用 `/proc/mounts` + psutil + `/sys/block/*/queue/rotational`;**WSL 已支持**——缓存只落在 WSL 自己的 ext4(默认 `~/.cache/bnb_flash_cache`),9P 宿主盘 `/mnt/[盘符]`、网络盘、只读挂载、tmpfs 一律拒绝(显式 `--flash_paths` 指过去会直接报错)。零拷贝写盘队列有在飞字节上限(`max_inflight_bytes`),卸载不会把源张量内存偷偷留住。
385
-
386
- ### Prebuilt wheels (optional)
387
-
388
- 预编译的 wheel(`pip install bitsandbytes-<ver>-py3-none-<platform>.whl`)可从 **Release**
389
- 获取,无需自己编译(Windows `win_amd64` / Linux `linux_x86_64` / aarch64
390
- `linux_aarch64`)。wheel 内含 `bitsandbytes/` 包 + 编译好的 native 库
391
- (`.dll`/`.so`),安装即用。
392
-
393
- ### Self-test (no torch required)
394
-
395
- ```bash
396
- # C-level kernel self-test: quantize roundtrip / gemm_8bit / 4bit GEMV / 8bit optimizer
397
- bash build_linux.sh --selftest
398
- ```
399
-
400
- ### Notes
401
-
402
- - This fork is based on bitsandbytes v0.45.1 (MIT, Facebook) and keeps the upstream
403
- MIT license. See `NOTICE.md` for contributions.
404
- - The CPU backend keeps the upstream NEON/AVX paths for plain quant/dequant;
405
- the **fused training kernels** (GDN, gemm_8bit, 8-bit optimizer) are the
406
- addition of this fork.
407
- - Not recommended for use without the compiled native library — build first
408
- (Windows: `build_manual\build_manual.bat amd`, Linux: `build_linux.sh`).
409
-
410
- ## :heart: Sponsors
411
- The continued maintenance and development of `bitsandbytes` is made possible thanks to the generous support of our sponsors. Their contributions help ensure that we can keep improving the project and delivering valuable updates to the community.
412
-
413
- <kbd><a href="https://hf.co" target="_blank"><img width="100" src="https://huggingface.co/datasets/huggingface/brand-assets/resolve/main/hf-logo.svg" alt="Hugging Face"></a></kbd>
414
-
415
- ## License
416
- `bitsandbytes` is MIT licensed.
417
-
418
- ## How to cite us
419
- If you found this library useful, please consider citing our work:
420
-
421
- ### QLoRA
422
-
423
- ```bibtex
424
- @article{dettmers2023qlora,
425
- title={Qlora: Efficient finetuning of quantized llms},
426
- author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
427
- journal={arXiv preprint arXiv:2305.14314},
428
- year={2023}
429
- }
430
- ```
431
-
432
- ### LLM.int8()
433
-
434
- ```bibtex
435
- @article{dettmers2022llmint8,
436
- title={LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale},
437
- author={Dettmers, Tim and Lewis, Mike and Belkada, Younes and Zettlemoyer, Luke},
438
- journal={arXiv preprint arXiv:2208.07339},
439
- year={2022}
440
- }
441
- ```
442
-
443
- ### 8-bit Optimizers
444
-
445
- ```bibtex
446
- @article{dettmers2022optimizers,
447
- title={8-bit Optimizers via Block-wise Quantization},
448
- author={Dettmers, Tim and Lewis, Mike and Shleifer, Sam and Zettlemoyer, Luke},
449
- journal={9th International Conference on Learning Representations, ICLR},
450
- year={2022}
451
- }
452
- ```