@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/unsloth/docs/glm-4.7-flash.md

@@ -1,997 +0,0 @@
-# GLM-4.7-Flash: How To Run Locally
-
-GLM-4.7-Flash is Z.ai’s new 30B MoE reasoning model built for local deployment, delivering best-in-class performance for coding, agentic workflows, and chat. It uses \~3.6B parameters, supports 200K context, and leads SWE-Bench, GPQA, and reasoning/chat benchmarks.
-
-GLM-4.7-Flash runs on **24GB RAM**/VRAM/unified memory (32GB for full precision), and you can now fine-tune with Unsloth. To run GLM 4.7 Flash with vLLM, see [#glm-4.7-flash-in-vllm](#glm-4.7-flash-in-vllm "mention")
-
-{% hint style="success" %}
-Jan 21 update: `llama.cpp` fixed a bug specifying the wrong `scoring_func`: `"softmax"` (should be `"sigmoid"`). This caused looping and poor outputs. We updated the GGUFs - please re-download the model for much better outputs.
-
-You can now use Z.ai’s recommended parameters and get great results:
-
-* **For general use-case:** `--temp 1.0 --top-p 0.95`
-* **For tool-calling:** `--temp 0.7 --top-p 1.0`
-* **Repeat penalty:** Disable it, or set `--repeat-penalty 1.Jan 22: Faster inference is here as the FA fix for CUDA is now merged.
-{% endhint %}
-
-<a href="#run-glm-4.7-flash" class="button primary">Running Tutorial</a><a href="#fine-tuning-glm-4.7-flash" class="button secondary">Fine-tuning</a>
-
-GLM-4.7-Flash GGUF to run: [unsloth/GLM-4.7-Flash-GGUF](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF)
-
-### ⚙️ Usage Guide
-
-After speaking with the Z.ai's team, they recommend using their GLM-4.7 sampling parameters:
-
-| Default Settings (Most Tasks)                                      | Terminal Bench, SWE Bench Verified                                 |
-| ------------------------------------------------------------------ | ------------------------------------------------------------------ |
-| <mark style="background-color:green;">**temperature = 1.0**</mark> | <mark style="background-color:green;">**temperature = 0.7**</mark> |
-| <mark style="background-color:green;">**top\_p = 0.95**</mark>     | <mark style="background-color:green;">**top\_p = 1.0**</mark>      |epeat penalty = disabled or 1.0                                   | repeat penalty = disabled or 1.0                                   |
-
-* For general use-case:  `--temp 1.0 --top-p 0.95`
-* For tool-calling:  `--temp 0.7 --top-p 1.0`
-* If using llama.cpp, set `--min-p 0.01` as llama.cpp's default is 0.05
-* Sometimes you'll need to experiment what numbers work best for your use-case.
-
-{% hint style="warning" %}
-For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).
-
-**Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`
-{% endhint %}
-
-* **Maximum context window:** `202,752`
-* Use `--jinja` for llama.cpp variants
-
-### 🖥️ Run GLM-4.7-Flash
-
-Depending on your use-case you will need to use different settings. Some GGUFs end up similar in size because the model architecture (like [gpt-oss](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune)) has dins not divisible by 128, so parts can’t be quantized to lower bits.
-
-Because this guide uses 4-bit, you will need around 18GB RAM/unified memory. We recommend using at least 4-bit precision for best performance.
-
-{% hint style="warning" %}
-For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).
-
-**Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`
-{% endhint %}
-
-#### Llama.cpp Tutorial (GGUF):
-
-Instructions to run in llama.cpp (note we will be using 4-bit to fit most devices):
-
-{% stepper %}
-{% step %}
-Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.
-
-{% code overflow="wrap" %}
-
-```bash
-apt-get update
-apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev it clone https://github.com/ggml-org/llama.cpp
-cmake llama.cpp -B llama.cpp/build \
-    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
-cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
-cp llama.cpp/build/bin/llama-* llama.cpp
-```
-
-{% endcode %}
-{% endstep %}
-
-{% step %}
-You can directly pull from Hugging Face. You can increase the context to 200K as your RAM/VRAM allows.
-
-You can also try Z.ai's recommended GLM-4.7 sampling parameters:
-
-* For general use-case:  `--temp 1.0 --top-p 0.95`
-* For tool-calling:  `--temp 0.7 --top-p 1.0`
-* **Remember to disable repeat penalty!**
-
-Follow this for **general instruction** use-cases:
-
-```bash
-./llama.cpp/llama-cli \
-    -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \
-    --jinja --ctx-size 16384 \
-    --temp 1.0 --top-p 0.95 --min-p 0.01 --fit on
-```
-
-Follow this for **tool-calling** use-cases:
-
-```bash
-./llama.cpp/llama-cli \
-    -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \
-    --jinja --ctx-size 16384 \
-    --temp 0.7 --top-p 1.0 --min-p 0.01 --fit on
-```
-
-{% endstep %}
-
-{% step %}
-Download the model via (after installing `pip install huggingface_hub`). You can choose `UD-Q4_K_XL` or other quantized versions.
-
-{% code overflow="wrap" %}
-
-```bash
-pip install -U huggingface_hub
-hf download unsloth/GLM-4.7-Flash-GGUF \
-    --local-dir unsloth/GLM-4.7-Flash-GGUF \
-    --include "*UD-Q2_K_XL*"
-```
-
-{% endcode %}
-{% endstep %}
-
-{% step %}
-Then run the model in conversation mode:
-
-{% code overflow="wrap" %}
-
-```bash
-./llama.cpp/llama-cli \
-    --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf \
-    --ctx-size 16384 \
-    --fit on \
-    --seed 3407 \
-    --temp 1.0 \
-    --top-p 0.95 \
-    --min-p 0.01 \
-    --jinja
-```
-
-{% endcode %}
-
-Also, adjust **context window** as required, up to `202752`
-{% endstep %}
-{% endstepper %}
-
-### :loop:Reducing repetition and looping
-
-{% hint style="success" %}
-**JAN 21 UPDATE: llama.cpp fixed a bug specifying the wrong** `"scoring_func": "softmax"` **that caused looping and poor outputs (should be sigmoid) We updated the GGUFs. Please re-download the model for much better outputs.**
-{% endhint %}
-
-This means you can now use Z.ai's recommended parameters and get great results:
-
-* For general use-case:  `--temp 1.0 --top-p 0.95`
-* For tool-calling:  `--temp 0.7 --top-p 1.0`
-* If using llama.cpp, set `--min-p 0.01` as llama.cpp's default is 0.05
-* **Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`
-
-We added `"scoring_func": "sigmoid"` to `config.json` for the main model - [see](https://huggingface.co/unsloth/GLM-4.7-Flash/commit/3fd53b491e04f707f307aef2f70f8a7520511e6d).
-
-{% hint style="warning" %}
-For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).
-{% endhint %}
-
-### :bird:Flappy Bird Example with UD-Q4\_K\_XL
-
-As an example, we did the following long conversation by using UD-Q4\_K\_XL via `./llama.cpp/llama-cli model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf --fit on --temp 1.0 --top-p 0.95 --min-p 0.01 --jinja` :
-
-```
-Hi
-What is 2+2
-Create a Python Flappy Bird game
-Create a totally different game in Rust
-Find bugs in both
-Make the 1st game I mentioned but in a standalone HTML file
-Find bugs and show the fixed game
-```
-
-which rendered the following Flappy Bird game in HTML form:
-
-<details>
-
-<summary>Flappy Bird Game in HTML (Expandable)</summary>
-
-```html
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
-    <title>Flappy Bird Fixed</title>
-    <style>
-        body {
-            margin: 0;
-            display: flex;
-            justify-content: center;
-            align-items: center;
-            height: 100vh;
-            background-color: #222;
-            font-family: 'Arial', sans-serif;
-            overflow: hidden;
-            user-select: none;
-            -webkit-user-select: none;
-            touch-action: none; /* Prevents zoom on mobile */
-        }
-
-        #game-container {
-            position: relative;
-            box-shadow: 0 0 20px rgba(0,0,0,0.5);
-        }
-
-        canvas {
-            background-color: #87CEEB;
-            display: block;
-            border-radius: 4px;
-        }
-
-        /* UI Overlays */
-        #ui-layer {
-            position: absolute;
-            top: 0;
-            left: 0;
-            width: 100%;
-            height: 100%;
-            pointer-events: none;
-            display: flex;
-            flex-direction: column;
-            justify-content: center;
-            align-items: center;
-            text-align: center;
-        }
-
-        #score-display {
-            position: absolute;
-            top: 40px;
-            left: 50%;
-            transform: translateX(-50%);
-            font-size: 48px;
-            font-weight: bold;
-            color: white;
-            text-shadow: 3px 3px 0 #000;
-            z-index: 10;
-            font-family: 'Courier New', Courier, monospace;
-        }
-
-        #start-screen, #game-over-screen {
-            background: rgba(0, 0, 0, 0.7);
-            width: 100%;
-            height: 100%;
-            display: flex;
-            flex-direction: column;
-            justify-content: center;
-            align-items: center;
-            color: white;
-            pointer-events: auto; /* Allow clicks */
-            cursor: pointer;
-        }
-
-        h1 { margin: 0 0 10px 0; font-size: 60px; text-shadow: 4px 4px 0 #000; line-height: 1; }
-        p { font-size: 22px; margin: 10px 0; color: #ddd; }
-        
-        .btn {
-            background: linear-gradient(to bottom, #ffeb3b, #fbc02d);
-            border: 3px solid #fff;
-            color: #333;
-            padding: 15px 40px;
-            font-size: 28px;
-            font-weight: bold;
-            cursor: pointer;
-            border-radius: 8px;
-            box-shadow: 0 6px 0 #c49000, 0 10px 10px rgba(0,0,0,0.3);
-            text-transform: uppercase;
-            transition: all 0.1s;
-            margin-top: 10px;
-        }
-
-        .btn:active {
-            transform: translateY(4px);
-            box-shadow: 0 2px 0 #c49000, 0 4px 4px rgba(0,0,0,0.3);
-        }
-
-        .score-board {
-            background: #ded895;
-            border: 2px solid #543847;
-            padding: 20px 40px;
-            border-radius: 10px;
-            box-shadow: 4px 4px 0 #543847;
-            margin-bottom: 30px;
-            display: none;
-            border: 4px solid #543847;
-        }
-        
-        .score-board h2 { margin: 0 0 5px 0; color: #e86101; font-size: 40px; }
-        .score-board span { font-size: 20px; color: #543847; display: block; text-align: center; }
-
-    </style>
-</head>
-<body>
-
-    <div id="game-container">
-        <canvas id="gameCanvas" width="400" height="600"></canvas>
-        
-        <div id="score-display">0</div>
-
-        <div id="ui-layer">
-            <div id="start-screen">
-                <h1>FLAPPY<br>BIRD</h1>
-                <p>Tap or Press Space to Start</p>
-                <button class="btn" style="display:none;" id="touch-instruction">Click to Start</button>
-            </div>
-
-            <div id="game-over-screen">
-                <h1>GAME OVER</h1>
-                <div class="score-board" id="score-board">
-                    <h2>Score: <span id="final-score">0</span></h2>
-                </div>
-                <button class="btn" id="restart-btn">Try Again</button>
-            </div>
-        </div>
-    </div>
-
-<script>
-    const canvas = document.getElementById('gameCanvas');
-    const ctx = canvas.getContext('2d');
-
-    // --- Constants ---
-    const GRAVITY = 0.35; // Slightly harder gravity for better feel
-    const JUMP_STRENGTH = -6.5;
-    const PIPE_GAP = 180;
-    const PIPE_WIDTH = 60;
-    const PIPE_SPEED = 2.5;
-    const PIPE_SPAWN_RATE = 100;
-
-    // --- State ---
-    let frames = 0;
-    let score = 0;
-    let isGameOver = false;
-    let isPlaying = false;
-    let gameLoopId;
-
-    const ui = {
-        startScreen: document.getElementById('start-screen'),
-        gameOverScreen: document.getElementById('game-over-screen'),
-        scoreDisplay: document.getElementById('score-display'),
-        scoreBoard: document.getElementById('score-board'),
-        finalScore: document.getElementById('final-score'),
-        restartBtn: document.getElementById('restart-btn')
-    };
-
-    const bird = {
-        x: 80,
-        y: 150,
-        radius: 12, // Fixed radius
-        velocity: 0,
-        
-        draw: function() {
-            // Rotate bird based on velocity for visual flair
-            let angle = Math.min(Math.PI / 4, Math.max(-Math.PI / 4, (this.velocity * 0.1)));
-            
-            ctx.save();
-            ctx.translate(this.x, this.y);
-            ctx.rotate(angle);
-            
-            // Draw Body
-            ctx.fillStyle = '#FFD700';
-            ctx.beginPath();
-            ctx.arc(0, 0, this.radius, 0, Math.PI * 2);
-            ctx.fill();
-            
-            // Eye
-            ctx.fillStyle = 'white';
-            ctx.beginPath();
-            ctx.arc(4, -4, 4, 0, Math.PI * 2);
-            ctx.fill();
-            ctx.fillStyle = 'black';
-            ctx.beginPath();
-            ctx.arc(6, -4, 2, 0, Math.PI * 2);
-            ctx.fill();
-            
-            // Wing
-            ctx.fillStyle = '#FFA500';
-            ctx.beginPath();
-            ctx.arc(-4, 4, 5, 0, Math.PI * 2);
-            ctx.fill();
-
-            ctx.restore();
-        },
-
-        update: function() {
-            this.velocity += GRAVITY;
-            this.y += this.velocity;
-        },
-
-        jump: function() {
-            this.velocity = JUMP_STRENGTH;
-        },
-
-        reset: function() {
-            this.y = 150;
-            this.velocity = 0;
-        }
-    };
-
-    let pipes = [];
-
-    function createPipe() {
-        const minHeight = 50;
-        const maxPos = canvas.height - PIPE_GAP - minHeight;
-        const topHeight = Math.floor(Math.random() * (maxPos - minHeight + 1)) + minHeight;
-        
-        pipes.push({
-            x: canvas.width,
-            topHeight: topHeight,
-            bottomY: topHeight + PIPE_GAP,
-            width: PIPE_WIDTH,
-            passed: false
-        });
-    }
-
-    function drawPipes() {
-        ctx.fillStyle = '#2ecc71';
-        ctx.strokeStyle = '#27ae60';
-        ctx.lineWidth = 2;
-        
-        pipes.forEach(pipe => {
-            // Top Pipe
-            ctx.fillRect(pipe.x, 0, pipe.width, pipe.topHeight);
-            ctx.strokeRect(pipe.x, 0, pipe.width, pipe.topHeight);
-            
-            // Bottom Pipe
-            ctx.fillRect(pipe.x, pipe.bottomY, pipe.width, canvas.height - pipe.bottomY);
-            ctx.strokeRect(pipe.x, pipe.bottomY, pipe.width, canvas.height - pipe.bottomY);
-
-            // Cap
-            const capH = 20;
-            ctx.fillStyle = '#27ae60'; 
-            ctx.fillRect(pipe.x - 2, pipe.topHeight - capH, pipe.width + 4, capH);
-            ctx.fillRect(pipe.x - 2, pipe.bottomY, pipe.width + 4, capH);
-        });
-    }
-
-    function updatePipes() {
-        if (frames % PIPE_SPAWN_RATE === 0) createPipe();
-
-        for (let i = 0; i < pipes.length; i++) {
-            let p = pipes[i];
-            p.x -= PIPE_SPEED;
-
-            // --- FIXED COLLISION DETECTION ---
-            // Treat bird as a circle of radius 'bird.radius'
-            // Pipe is a rect: x, x+w, y_top, y_bottom
-            let birdLeft = bird.x - bird.radius;
-            let birdRight = bird.x + bird.radius;
-            let birdTop = bird.y - bird.radius;
-            let birdBottom = bird.y + bird.radius;
-
-            // Horizontal Overlap
-            if (birdRight > p.x && birdLeft < p.x + p.width) {
-                // Vertical Overlap (Hit Top Pipe OR Hit Bottom Pipe)
-                if (birdTop < p.topHeight || birdBottom > p.bottomY) {
-                    gameOver();
-                }
-            }
-
-            // --- FIXED SCORING ---
-            // If pipe is off screen to the left, and hasn't been scored
-            if (p.x + p.width < 0 && !p.passed) {
-                score++;
-                p.passed = true;
-                ui.scoreDisplay.innerText = score;
-            }
-
-            if (p.x < -60) {
-                pipes.shift();
-                i--;
-            }
-        }
-    }
-
-    function checkCollisions() {
-        // Floor
-        if (bird.y + bird.radius >= canvas.height) {
-            gameOver();
-        }
-        // Ceiling
-        if (bird.y - bird.radius <= 0) {
-            bird.y = bird.radius;
-            bird.velocity = 0;
-        }
-    }
-
-    function drawBackground() {
-        // Clear
-        ctx.clearRect(0, 0, canvas.width, canvas.height);
-        
-        // Floor
-        ctx.fillStyle = '#654321';
-        ctx.fillRect(0, canvas.height - 10, canvas.width, 10);
-        
-        // Clouds
-        ctx.fillStyle = "rgba(255, 255, 255, 0.6)";
-        for(let i=0; i<4; i++) {
-            let x = (frames * 0.5 + i * 150) % (canvas.width + 100) - 50;
-            let y = (i * 40) + 20;
-            let scale = 1 + (Math.sin(frames * 0.02 + i) * 0.1);
-            let size = 30 * scale;
-            ctx.beginPath();
-            ctx.arc(x, y, size, 0, Math.PI * 2);
-            ctx.arc(x + 20*scale, y - 10*scale, size * 1.2, 0, Math.PI * 2);
-            ctx.arc(x + 40*scale, y, size, 0, Math.PI * 2);
-            ctx.fill();
-        }
-    }
-
-    function update() {
-        if (!isPlaying) return;
-        bird.update();
-        updatePipes();
-        checkCollisions();
-        frames++;
-    }
-
-    function draw() {
-        drawBackground();
-        drawPipes();
-        bird.draw();
-    }
-
-    function loop() {
-        update();
-        draw();
-        if (isPlaying || !isGameOver) {
-            gameLoopId = requestAnimationFrame(loop);
-        }
-    }
-
-    function startGame() {
-        isPlaying = true;
-        isGameOver = false;
-        
-        // UI
-        ui.startScreen.style.display = 'none';
-        ui.gameOverScreen.style.display = 'none';
-        ui.scoreBoard.style.display = 'none';
-        
-        // Logic
-        bird.reset();
-        pipes = [];
-        score = 0;
-        frames = 0;
-        ui.scoreDisplay.innerText = '0';
-        
-        loop();
-    }
-
-    function gameOver() {
-        isPlaying = false;
-        isGameOver = true;
-        cancelAnimationFrame(gameLoopId);
-        
-        ui.finalScore.innerText = score;
-        ui.gameOverScreen.style.display = 'flex';
-        ui.scoreBoard.style.display = 'block';
-    }
-
-    // --- Input Handling ---
-
-    function handleInput(e) {
-        if (e.type === 'keydown' && e.code === 'Space') e.preventDefault();
-
-        if (isPlaying) {
-            bird.jump();
-        } else if (!isGameOver) {
-            // Click on start screen (or any click if game hasn't started)
-            startGame();
-        }
-    }
-
-    // Keyboard
-    window.addEventListener('keydown', (e) => {
-        if (e.code === 'Space') handleInput(e);
-    });
-
-    // Mouse / Touch
-    window.addEventListener('mousedown', handleInput);
-    window.addEventListener('touchstart', (e) => {
-        // Prevent zoom/scroll
-        // e.preventDefault(); 
-        handleInput(e);
-    }, {passive: false});
-
-    // UI Interactions
-    ui.restartBtn.addEventListener('click', (e) => {
-        e.stopPropagation();
-        startGame();
-    });
-    
-    // Allow clicking the Game Over overlay to restart
-    ui.gameOverScreen.addEventListener('mousedown', (e) => {
-        if(e.target === ui.gameOverScreen) startGame();
-    });
-    ui.gameOverScreen.addEventListener('touchstart', (e) => {
-        if(e.target === ui.gameOverScreen) {
-            e.preventDefault();
-            startGame();
-        }
-    });
-
-    // Initial Draw
-    drawBackground();
-    bird.reset();
-    bird.draw();
-
-</script>
-</body>
-</html>
-```
-
-</details>
-
-And we took some screenshots (4bit works):
-
-<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJ85uzHPDWinaXPe6kWyU%2Fimage.png?alt=media&#x26;token=6547f49d-2544-4c48-a7d5-5c1c67d34a87" alt="" width="188"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHAc2SjNLo1OsyAC4dArm%2Fimage.png?alt=media&#x26;token=87d4bfea-4ac9-41ef-be1c-1e51664d30b1" alt="" width="188"><figcaption></figcaption></figure></div>
-
-### 🦥 Fine-tuning GLM-4.7-Flash
-
-Unsloth now supports fine-tuning of GLM-4.7-Flash, however you will need to use `transformers v5`. The 30B model does not fit on a free Colab GPU; however, you can use our notebook. 16-bit LoRA fine-tuning of GLM-4.7-Flash will use around **60GB VRAM**:
-
-* [GLM-4.7-Flash SFT LoRA notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100\(80GB\).ipynb)
-
-{% hint style="warning" %}
-You may encounter out of memory sometimes when using A100 40GB VRAM. You will need to use H100/A100 80GB VRAM for smoother runs.
-{% endhint %}
-
-{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100(80GB).ipynb>" %}
-
-On fine-tung MoE's, it's probably not a good idea to fine-tune the router layer so we disabled it by default. If you want to maintain its reasoning capabilities (optional), you can use a mix of direct answers and chain-of-thought examples. Use at least <mark style="background-color:green;">75% reasoning</mark> and <mark style="background-color:green;">25% non-reasoning</mark> in your dataset to make the model retain its reasoning capabilities.
-
-### 🦙Llama-server serving & deployment
-
-To deploy GLM-4.7-Flash for production, we use `llama-server` In a new terminal say via tmux, deploy the model via:
-
-{% code overflow="wrap" %}
-
-```bash
-./llama.cpp/llama-server \
-    --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf \
-    --alias "unsloth/GLM-4.7-Flash" \
-    --fit on \
-    --seed 3407 \
-    --temp 1.0 \
-    --top-p 0.95 \
-    --min-p 0.01 \
-    --ctx-size 16384 \
-    --port 8001 \
-    --jinja
-```
-
-{% endcode %}
-
-Then in a new terminal, after doing `pip install openai`, do:
-
-{% code overflow="wrap" %}
-
-`python
-from openai import OpenAI
-import json
-openai_client = OpenAI(
-    base_url = "http://127.0.0.1:8001/v1",
-    api_key = "sk-no-key-required",
-)
-completion = openai_client.chat.completions.create(
-    model = "unsloth/GLM-4.7-Flash",
-    messages = [{"role": "user", "content": "What is 2+2?"},],
-)
-print(completion.choices[0].message.content)
-```
-
-{% endcode %}
-
-Which will print
-
-{% code overflow="wrap" %}
-
-```
-User asks a simple question: "What is 2+2?" The answer is 4. Provide answer.
-
-2 + 2 = 4.
-```
-
-{% endcode %}
-
-### :computer: GLM-4.7-Flash in vLLM
-
-You can now use our new [FP8 Dynamic quant](https://huggingface.co/unsloth/GLM-4.7-Flash-FP8-Dynamic) of the model for premium and fast inference. First install vLLM from nightly:
-
-{% code overflow="wrap" %}
-
-```bash
-uv pip install --upgrade --force-reinstall vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly/cu130
-uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
-uv pip instrce-reinstall numba
-```
-
-{% endcode %}
-
-Then serve [Unsloth's dynamic FP8 version](https://huggingface.co/unsloth/GLM-4.7-Flash-FP8-Dynamic) of the model. We enabled FP8 to reduce KV cache memory usage by 50%, and on 4 GPUs. If you have 1 GPU, use `CUDA_VISIBLE_DEVICES='0'` and set `--tensor-parallel-size 1` or remove this argument. To disable FP8, remove `--quantization fp8 --kv-cache-dtype fp8`
-
-```bash
-export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
-CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/GLM-4.7-Flash-FP8-Dynamic \
-    --served-model-name unsloth/GLM-4.7-Flash \
-    --tensor-parallel-size 4 \
-    --tool-call-parser glm47 \
-    --reasoning-parser glm45 \
-    --enable-auto-tool-choice \
-    --dtype bfloat16 \
-    --seed 3407 \
-    --max-model-len 200000 \
-    --gpu-memory-utilization 0.95 \
-    --max_num_batched_tokens 16384 \
-    --port 8001 \
-    --kv-cache-dtype fp8
-```
-
-You can then call the served model via the OpenAI API:
-
-```python
-from openai import AsyncOpenAI, OpenAI
-openai_api_key = "EMPTY"
-openai_api_base = "http://localhost:8001/v1"
-client = OpenAI( # or AsyncOpenAI
-    api_key=openai_api_key,
-    base_url=openai_api_base,
-)
-```
-
-#### :star: vLLM GLM-4.7-Flash Speculative Decoding
-
-We found using the MTP (multi token prediction) module from GLM 4.7 Flash makes generation throughput drop from 13,000 tokens on 1 B200 to 1,300 tokens! (10x slower) On Hopper, it should be fine hopefully.
-
-```bash
-    --speculative-config.method mtp \
-    --speculative-config.num_speculative_tokens 1
-```
-
-Only 1,300 tokens / s throughput on 1xB200 (130 tokens / s decoding per user)
-
-<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCJPJYh1uIS1yL8vskiOF%2Fimage.png?alt=media&#x26;token=f07aaad9-93bd-4507-836f-967a3d39b0e5" alt=""><figcaption></figcaption></figure>
-
-And 13,000 tokens / s throughput  on 1xB200 (still 130 token /s decoding per user)
-
-<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrXaHEknUb1QW1v0arO0q%2Fimage.png?alt=media&#x26;token=dd81b731-90bb-4d1b-a647-a64618f5952a" alt=""><figcaption></figcaption></figure>
-
-### :hammer:Tool Calling with GLM-4.7-Flash
-
-See [tool-calling-guide-for-local-llms](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms "mention") for more details on how to do tool calling. In a new terminal (if using tmux, use CTRL+B+D), we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:
-
-{% code expandable="true" %}
-
-```python
-import json, subprocess, random
-from typing import Any
-def add_number(a: float | str, b: float | str) -> float:
-    return float(a) + float(b)
-def multiply_number(a: float | str, b: float | str) -> float:
-    return float(a) * float(b)
-def substract_number(a: float | str, b: float | str) -> float:
-    return float(a) - float(b)
-def write_a_story() -> str:
-    return random.choice([
-        "A long time ago in a galaxy far far away...",
-        "There were 2 friends who loved sloths and code...",
-        "The world was ending because every sloth evolved to have superhuman intelligence...",
-        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
-    ])
-def terminal(command: str) -> str:
-    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
-        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
-        print(msg); return msg
-    print(f"Executing terminal command `{command}`")
-    try:
-        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
-    except subprocess.CalledProcessError as e:
-        return f"Command failed: {e.stderr}"
-def python(code: str) -> str:
-    data = {}
-    exec(code, data)
-    del data["__builtins__"]
-    return str(data)
-MAP_FN = {
-    "add_number": add_number,
-    "multiply_number": multiply_number,
-    "substract_number": substract_number,
-    "write_a_story": write_a_story,
-    "terminal": terminal,
-    "python": python,
-}
-tools = [
-    {
-        "type": "function",
-        "function": {
-            "name": "add_number",
-            "description": "Add two numbers.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "a": {
-                        "type": "string",
-                        "description": "The first number.",
-                    },
-                    "b": {
-                        "type": "string",
-                        "description": "The second number.",
-                    },
-                },
-                "required": ["a", "b"],
-            },
-        },
-    },
-    {
-        "type": "function",
-        "function": {
-            "name": "multiply_number",
-            "description": "Multiply two numbers.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "a": {
-                        "type": "string",
-                        "description": "The first number.",
-                    },
-                    "b": {
-                        "type": "string",
-                        "description": "The second number.",
-                    },
-                },
-                "required": ["a", "b"],
-            },
-        },
-    },
-    {
-        "type": "function",
-        "function": {
-            "name": "substract_number",
-            "description": "Substract two numbers.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "a": {
-                        "type": "string",
-                        "description": "The first number.",
-                    },
-                    "b": {
-                        "type": "string",
-                        "description": "The second number.",
-                    },
-                },
-                "required": ["a", "b"],
-            },
-        },
-    },
-    {
-        "type": "function",
-        "function": {
-            "name": "write_a_story",
-            "description": "Writes a random story.",
-            "parameters": {
-                "type": "object",
-                "properties": {},
-                "required": [],
-            },
-        },
-    },
-    {
-        "type": "function",
-        "function": {
-            "name": "terminal",
-            "description": "Perform operations from the terminal.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "command": {
-                        "type": "string",
-                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
-                    },
-                },
-                "required": ["command"],
-            },
-        },
-    },
-    {
-        "type": "function",
-        "function": {
-            "name": "python",
-            "description": "Call a Python interpreter with some Python code that will be ran.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "code": {
-                        "type": "string",
-                        "description": "The Python code to run",
-                    },
-                },
-                "required": ["code"],
-            },
-        },
-    },
-]
-```
-
-{% endcode %}
-
-We then use the below functions (copy and paste and execute) which will parse the function calls automatically and call the OpenAI endpoint for any model:
-
-{% code overflow="wrap" expandable="true" %}
-
-```python
-from openai import OpenAI
-def unsloth_inference(
-    messages,
-    temperature = 0.7,
-    top_p = 1.0,
-    top_k = -1,
-    repetition_penalty = 0.0,
-):
-    messages = messages.copy()
-    openai_client = OpenAI(
-        base_url = "http://127.0.0.1:8001/v1",
-        api_key = "sk-no-key-required",
-    )
-    model_name = next(iter(openai_client.models.list())).id
-    print(f"Using model = {model_name}")
-    has_tool_calls = True
-    original_messages_len = len(messages)
-    while has_tool_calls:
-        print(f"Current messages = {messages}")
-        response = openai_client.chat.completions.create(
-            model = model_name,
-            messages = messages,
-            temperature = temperature,
-            top_p = top_p,
-            tools = tools if tools else None,
-            tool_choice = "auto" if tools else None,
-            extra_body = {"top_k": top_k, "min_p": min_p, "dry_multiplier" :repetition_penalty,}
-        )
-        tool_calls = response.choices[0].message.tool_calls or []
-        content = response.choices[0].message.content or ""
-        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
-        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
-        for tool_call in tool_calls:
-            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
-            out = MAP_FN[fx](**json.loads(args))
-            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
-        else:
-            has_tool_calls = False
-    return messages
-```
-
-{% endcode %}
-
-After launching GLM-4.7-Flash via `llama-server` like in [#deploy-with-llama-server-and-openais-completion-library](#deploy-with-llama-server-and-openais-completion-library "mention") or see [tool-calling-guide-for-local-llms](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms "mention") for more details, we then can do some tool calls:
-
-**Tool Call for mathematical operations for GLM 4.7**
-
-{% code overflow="wrap" %}
-
-```python
-messages = [{
-    "role": "user",
-    "content": [{"type": "text", "text": "What is today's date plus 3 days?"}],
-}]
-unsloth_inference(messages, temperature = 1.0, top_p = 0.95, top_k = -1, min_p = 0.01)
-```
-
-{% endcode %}
-
-<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoFkZ20QOSGdzT4iz2SOB%2Fimage.png?alt=media&#x26;token=e4ca30b0-dcec-4a26-b019-dd33f0600949" alt=""><figcaption></figcaption></figure>
-
-**Tool Call to execute generated Python code for GLM-4.7-Flash**
-
-{% code overflow="wrap" %}
-
-```python
-messages = [{
-    "role": "user",
-    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],
-}]
-unsloth_inference(messages, temperature = 1.0, top_p = 0.95, top_k = -1, min_p = 0.01)
-```
-
-{% endcode %}
-
-<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhS8sWtZwjwerElezCc2C%2Fimage.png?alt=media&#x26;token=39032ef8-386e-4837-8dd2-c552c80a3ee3" alt="" width="563"><figcaption></figcaption></figure>
-
-### Benchmarks
-
-GLM-4.7-Flash is the best performing 30B model across all benchmarks except AIME 25.