codefinetuner 0.1.0__tar.gz → 0.2.0__tar.gz

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/.github/workflows/release.yaml

@@ -1,4 +1,4 @@
-name: Release new version and upload to PyPI
+name: Release
 
 on:
   push:

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/.github/workflows/release_test.yaml

@@ -1,9 +1,8 @@
-name: Release new version and upload to PyPI
+name: Test Reslease
 
 on:
   push:
     tags:
-      - '[0-9]*.[0-9]*.[0-9]*'
       - '[0-9]*.[0-9]*.[0-9]*a[0-9]*'
       - '[0-9]*.[0-9]*.[0-9]*b[0-9]*'
       - '[0-9]*.[0-9]*.[0-9]*rc[0-9]*'

codefinetuner-0.2.0/.github/workflows/tests.yaml

@@ -0,0 +1,24 @@
+name: Tests
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    name: Pytest
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          submodules: recursive
+      - name: Install uv and Set up Python
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+          enable-cache: true
+      - name: Install Dependencies
+        run: uv sync --group dev
+      - name: Run Pytest
+        run: uv run pytest

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codefinetuner
-Version: 0.1.0
+Version: 0.2.0
 Summary: Add your description here
 Project-URL: Homepage, https://github.com/cuolm/codefinetuner
 Project-URL: Repository, https://github.com/cuolm/codefinetuner.git
@@ -32,15 +32,13 @@ Requires-Dist: tree-sitter<0.23.0,>=0.22.0
 Description-Content-Type: text/markdown
 
 # CodeFinetuner
-[![Build](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yml/badge.svg)](https://github.com/cuolm/codefinetuner/actions)
 [![PyPI](https://img.shields.io/pypi/v/codefinetuner.svg)](https://pypi.org/project/codefinetuner/)
 [![License](https://img.shields.io/github/license/cuolm/codefinetuner.svg)](LICENSE.txt)
-[![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/cuolm/gist_id/raw/coverage.json)](https://github.com/cuolm/codefinetuner/actions)
+[![Release](https://github.com/cuolm/codefinetuner/actions/workflows/release.yaml/badge.svg)](https://github.com/cuolm/codefinetuner/actions/workflows/release.yaml)
+[![Tests](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yaml/badge.svg)](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yaml)
 
-
-Create your own local code autocomplete model, fine-tuned on your custom code repository, for use in editors like VS Code or Vim/Neovim.
-
-Fine-tuning is achieved by training a Low-Rank Adapter ([LoRA](https://arxiv.org/abs/2106.09685)) to perform Fill-In-the-Middle ([FIM](https://arxiv.org/abs/2207.14255)) completion. 
+CodeFinetuner fine-tunes a local code autocomplete model on your own repository for use in editors like VS Code or Vim/Neovim.
+It trains a Low-Rank Adapter ([LoRA](https://arxiv.org/abs/2106.09685)) on Fill-In-the-Middle ([FIM](https://arxiv.org/abs/2207.14255)) examples so the model learns the structure and patterns of your codebase.
 
 ## Table of Contents
 - [Architecture](#architecture)
@@ -50,23 +48,25 @@ Fine-tuning is achieved by training a Low-Rank Adapter ([LoRA](https://arxiv.org
 - [Quick Start](#quick-start)
 - [Configuration](#configuration)
 - [Usage](#usage)
-- [Deployment](#deployment)
-- [Docker](#docker)
-- [Tree-sitter Setup](#tree-sitter-setup) <!-- Link to new docs -->
+- [Finetuned Model Usage](#finetuned-model-usage)
+- [Docker Image](#docker-image)
+- [Tree-sitter Customization](#tree-sitter-customization)
 - [Tests](#tests)
 - [Resources](#resources)
 - [License](#license)
 
 
 ## Architecture
+CodeFinetuner follows a simple pipeline. First, raw code is parsed and turned into FIM examples. These examples are then used to train a LoRA adapter and evaluate the fine‑tuned model using multiple metrics. Finally, the model is converted into GGUF format for deployment.
+
 ```text
 Raw Code Files
      |
      v
-[Preprocess]  -- tree-sitter parsing -> FIM examples -> tokenized JSONL
+[Preprocess]  -- tree-sitter parsing -> FIM examples -> tokenized jsonl datasets
      |
      v
-[Finetune]    -- LoRA adapter training -> merged model
+[Finetune]    -- LoRA adapter training -> merged safetensors model
      |
      v
 [Evaluate]    -- CodeBLEU, SentenceBLEU, exact match, line match, perplexity
@@ -86,16 +86,16 @@ Raw Code Files
 │       └── convert/             
 ├── config/                      # User configuration
 │   └── codefinetuner_config.yaml
-├── data/                        # Default data directory (Workspace root)
-├── outputs/                     # Pipeline artifacts (Workspace root)
+├── data/                        # Default data directory 
+├── outputs/                     # Pipeline outputs
 ├── scripts/                     # Utility scripts
-├── tests/                       # Unit tests 
-├── third_party/                 # External submodules (e.g., custom parsers)
-└── docs/                        # Documentation and assets
+├── tests/                       # Tests 
+├── third_party/                 # External submodules 
+└── docs/                        # Documentation 
 ```
 
 ## How Training Examples Are Created
-To generate high-quality FIM examples, high-level structural code blocks are extracted (e.g., functions, classes). From these blocks, logical sub-blocks (e.g., statements, expressions) are masked to serve as the "middle" section for the model to predict.
+CodeFinetuner builds FIM examples from real code structure, not random text chunks. It first extracts blocks such as functions or classes, then masks smaller sub-blocks like statements or expressions for the model to predict. This approach helps the model learn the logical structure of your codebase instead of unrelated fragments.
 
 Here is an example illustrating how a single FIM example is created:
 <table>
@@ -121,8 +121,6 @@ Here is an example illustrating how a single FIM example is created:
 <|fim_middle|>count = count + (value & 1);\n    value = (value >> 1);
 ```
 
-Using this technique, rather than randomly splitting code into unrelated text chunks, helps the model learn the logical patterns and structure of your specific codebase.
-
 ## Installation
 
 ### From PyPI
@@ -131,7 +129,7 @@ uv add codefinetuner
 # or
 pip install codefinetuner
 ```
-### From Source (Development)
+### From Source
 ```bash
 git clone --recurse-submodules https://github.com/cuolm/codefinetuner
 cd codefinetuner
@@ -151,17 +149,15 @@ Create a configuration file according to the [Configuration](#configuration) sec
 ```python
 import codefinetuner
 
-# Run the complete pipeline
 codefinetuner.run_pipeline("codefinetuner_config.yaml")
 ```
 
 ## Configuration
-
-The pipeline uses a single-source-of-truth YAML configuration file. It utilizes YAML anchors (`&globals`) to share core parameters across all stages (`preprocess`, `finetune`, `evaluate`), ensuring consistency and reducing redundancy.
+The pipeline uses a single-source-of-truth YAML configuration file. It utilizes YAML anchors (`&globals`) to share core parameters across all stages (`preprocess`, `finetune`, `evaluate`, `convert`), ensuring consistency and reducing redundancy.
 
 ### Configuration Structure
 
-Create a codefinetuner_config.yaml using the template below. For a full list of all available parameters and their effects, see the [Configuration Reference Guide](/docs/config-file.md).
+Create `codefinetuner_config.yaml` using the template below. For the full parameter list, see the [Configuration Reference Guide](/docs/config-file.md).
 
 ```yaml
 # globals contain all the mandatory parameters.
@@ -194,8 +190,11 @@ evaluate:
   benchmark_sample_size: 4
   # ... (evaluate specific settings)
 
+convert:
+  <<: *globals
+  # ... (convert specific settings)
 ```
-> **Note:** For a complete, production-ready example, see [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml).
+> **Note:** See [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml) for a full production example.
 
 ### Data Preparation
 Place source files in your `raw_data_path` (default: `workspace_path/data`).
@@ -210,9 +209,12 @@ Run the pipeline using the unified CLI:
 uv run codefinetuner --config="config/codefinetuner_config.yaml"
 ```
 
-**Pipeline Flags:**
-* `--config`: Specify path to a different config file.
-* `--skip-preprocess`, `--skip-finetune`, `--skip-evaluate`, `--skip-convert`: Skip specific stages.
+**Pipeline flags**
+- `--config`: Use a different config file.
+- `--skip-preprocess`: Skip preprocessing.
+- `--skip-finetune`: Skip fine-tuning.
+- `--skip-evaluate`: Skip evaluation.
+- `--skip-convert`: Skip conversion.
 
 ### Python Module Usage
 ```python
@@ -227,18 +229,14 @@ codefinetuner.run_pipeline(
     skip_preprocess=True,
     skip_convert=True
 )
-
-# Individual stages
-from codefinetuner import preprocess, finetune
-preprocess.run("config.yaml")
 ```
 
-## Deployment: Using the Model
-The `convert` stage converts the model to GGUF format. The final GGUF file is located under `outputs/convert/results/lora_model.gguf`.  
-For a detailed guide on how to use the gguf model with the VS Code extension [llama.vscode](https://github.com/ggml-org/llama-vscode), check out the [inference-vscode](/docs/inference-vscode.md) guide.
+## Finetuned Model Usage
+The `convert` stage exports the final model to [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) format for local inference. The resulting file is saved at `outputs/convert/results/lora_model.gguf`. 
+For setup instructions with the VS Code extension [llama.vscode](https://github.com/ggml-org/llama.vscode), see the [inference-vscode](/docs/inference-vscode.md) guide.
 
 
-## Create And Run Docker Image
+## Docker Image
 
 #### 1. Build the Docker Image
 Build the image from the `Dockerfile`, tagging it as codefinetuner-image.
@@ -246,10 +244,8 @@ Build the image from the `Dockerfile`, tagging it as codefinetuner-image.
 docker build -t codefinetuner-image .
 ```
 #### 2. Prepare Data and Run the Container
-To allow the container to access your data for fine-tuning, use a bind mount to link your host machine's `data` directory to the container.
-- On your host machine (where you run Docker), create a folder named `data` if it doesn't already exist.
-- Put all files you want to use for fine-tuning inside the `data` directory. For `manual` mode, include `train`, `eval`, and `test` subdirectories containing your manually splitted files.
-- Start the container with the bind mount, and open a Bash shell depending on your host machine hardware:
+To allow the container to access your data for fine-tuning, use a bind mount to link your host machine's `data` directory to the container.  
+On your host machine (where you run Docker), create a folder named `data` if it does not already exist. Put all files you want to use for fine-tuning inside the `data` directory. For manual mode, include `train`, `eval`, and `test` subfolders with the split you want to use.  
 
 #### NVIDIA GPU (Recommended)
 Use this command to enable CUDA support for `torch` and `bitsandbytes`. Requires the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed on the host machine.
@@ -260,8 +256,7 @@ docker run --gpus all -it --rm \
   codefinetuner-image /bin/bash
 ```
 #### CPU Only
-
-Use this command if no compatible GPU is available. Note that fine-tuning will be significantly slower.
+Use this if you do not have a compatible GPU. Fine-tuning will be much slower.
 ```bash
 docker run -it --rm \
   -v $(pwd)/data:/app/data \
@@ -269,21 +264,22 @@ docker run -it --rm \
 ```
 
 ## Tree-sitter Customization
-Tree-sitter parses code into structural blocks for generating FIM training examples. Customize for new languages or build missing parsers.
-
-- [Add Language Definitions](docs/tree-sitter-customization.md#add-new-language-block-definitions) - Define `block_types`/`subblock_types` in JSON.
-- [Build Custom Parser](docs/tree-sitter-customization.md#build-custom-parser) - Compile from source (e.g., Mojo).
+Tree-sitter turns source code into structural blocks used to generate FIM examples. Use this section to add new languages or build missing parsers.
 
+- [Add Language Definitions](docs/tree-sitter-customization.md#add-new-language-block-definitions): define `block_types` and `subblock_types` in JSON.
+- [Build Custom Parser](docs/tree-sitter-customization.md#build-custom-parser): compile a parser from source, for example for Mojo.
 
 ## Tests
+Run the test suite with:
+
 ```bash
 pytest
 ```
 
-## Useful Resources
+## Resources
 - [Qwen2.5-Coder Technical Report](https://arxiv.org/pdf/2409.12186)
 - [Structure-Aware Fill-in-the-Middle Pretraining for Code](https://arxiv.org/pdf/2506.00204)
-- [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685)
+- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/pdf/2106.09685)
 - [Efficient Training of Language Models to Fill in the Middle](https://arxiv.org/pdf/2207.14255)
 - [From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?](https://arxiv.org/pdf/2505.18789)
 - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://arxiv.org/pdf/2009.10297)

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/README.md

@@ -1,13 +1,11 @@
 # CodeFinetuner
-[![Build](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yml/badge.svg)](https://github.com/cuolm/codefinetuner/actions)
 [![PyPI](https://img.shields.io/pypi/v/codefinetuner.svg)](https://pypi.org/project/codefinetuner/)
 [![License](https://img.shields.io/github/license/cuolm/codefinetuner.svg)](LICENSE.txt)
-[![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/cuolm/gist_id/raw/coverage.json)](https://github.com/cuolm/codefinetuner/actions)
+[![Release](https://github.com/cuolm/codefinetuner/actions/workflows/release.yaml/badge.svg)](https://github.com/cuolm/codefinetuner/actions/workflows/release.yaml)
+[![Tests](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yaml/badge.svg)](https://github.com/cuolm/codefinetuner/actions/workflows/tests.yaml)
 
-
-Create your own local code autocomplete model, fine-tuned on your custom code repository, for use in editors like VS Code or Vim/Neovim.
-
-Fine-tuning is achieved by training a Low-Rank Adapter ([LoRA](https://arxiv.org/abs/2106.09685)) to perform Fill-In-the-Middle ([FIM](https://arxiv.org/abs/2207.14255)) completion. 
+CodeFinetuner fine-tunes a local code autocomplete model on your own repository for use in editors like VS Code or Vim/Neovim.
+It trains a Low-Rank Adapter ([LoRA](https://arxiv.org/abs/2106.09685)) on Fill-In-the-Middle ([FIM](https://arxiv.org/abs/2207.14255)) examples so the model learns the structure and patterns of your codebase.
 
 ## Table of Contents
 - [Architecture](#architecture)
@@ -17,23 +15,25 @@ Fine-tuning is achieved by training a Low-Rank Adapter ([LoRA](https://arxiv.org
 - [Quick Start](#quick-start)
 - [Configuration](#configuration)
 - [Usage](#usage)
-- [Deployment](#deployment)
-- [Docker](#docker)
-- [Tree-sitter Setup](#tree-sitter-setup) <!-- Link to new docs -->
+- [Finetuned Model Usage](#finetuned-model-usage)
+- [Docker Image](#docker-image)
+- [Tree-sitter Customization](#tree-sitter-customization)
 - [Tests](#tests)
 - [Resources](#resources)
 - [License](#license)
 
 
 ## Architecture
+CodeFinetuner follows a simple pipeline. First, raw code is parsed and turned into FIM examples. These examples are then used to train a LoRA adapter and evaluate the fine‑tuned model using multiple metrics. Finally, the model is converted into GGUF format for deployment.
+
 ```text
 Raw Code Files
      |
      v
-[Preprocess]  -- tree-sitter parsing -> FIM examples -> tokenized JSONL
+[Preprocess]  -- tree-sitter parsing -> FIM examples -> tokenized jsonl datasets
      |
      v
-[Finetune]    -- LoRA adapter training -> merged model
+[Finetune]    -- LoRA adapter training -> merged safetensors model
      |
      v
 [Evaluate]    -- CodeBLEU, SentenceBLEU, exact match, line match, perplexity
@@ -53,16 +53,16 @@ Raw Code Files
 │       └── convert/             
 ├── config/                      # User configuration
 │   └── codefinetuner_config.yaml
-├── data/                        # Default data directory (Workspace root)
-├── outputs/                     # Pipeline artifacts (Workspace root)
+├── data/                        # Default data directory 
+├── outputs/                     # Pipeline outputs
 ├── scripts/                     # Utility scripts
-├── tests/                       # Unit tests 
-├── third_party/                 # External submodules (e.g., custom parsers)
-└── docs/                        # Documentation and assets
+├── tests/                       # Tests 
+├── third_party/                 # External submodules 
+└── docs/                        # Documentation 
 ```
 
 ## How Training Examples Are Created
-To generate high-quality FIM examples, high-level structural code blocks are extracted (e.g., functions, classes). From these blocks, logical sub-blocks (e.g., statements, expressions) are masked to serve as the "middle" section for the model to predict.
+CodeFinetuner builds FIM examples from real code structure, not random text chunks. It first extracts blocks such as functions or classes, then masks smaller sub-blocks like statements or expressions for the model to predict. This approach helps the model learn the logical structure of your codebase instead of unrelated fragments.
 
 Here is an example illustrating how a single FIM example is created:
 <table>
@@ -88,8 +88,6 @@ Here is an example illustrating how a single FIM example is created:
 <|fim_middle|>count = count + (value & 1);\n    value = (value >> 1);
 ```
 
-Using this technique, rather than randomly splitting code into unrelated text chunks, helps the model learn the logical patterns and structure of your specific codebase.
-
 ## Installation
 
 ### From PyPI
@@ -98,7 +96,7 @@ uv add codefinetuner
 # or
 pip install codefinetuner
 ```
-### From Source (Development)
+### From Source
 ```bash
 git clone --recurse-submodules https://github.com/cuolm/codefinetuner
 cd codefinetuner
@@ -118,17 +116,15 @@ Create a configuration file according to the [Configuration](#configuration) sec
 ```python
 import codefinetuner
 
-# Run the complete pipeline
 codefinetuner.run_pipeline("codefinetuner_config.yaml")
 ```
 
 ## Configuration
-
-The pipeline uses a single-source-of-truth YAML configuration file. It utilizes YAML anchors (`&globals`) to share core parameters across all stages (`preprocess`, `finetune`, `evaluate`), ensuring consistency and reducing redundancy.
+The pipeline uses a single-source-of-truth YAML configuration file. It utilizes YAML anchors (`&globals`) to share core parameters across all stages (`preprocess`, `finetune`, `evaluate`, `convert`), ensuring consistency and reducing redundancy.
 
 ### Configuration Structure
 
-Create a codefinetuner_config.yaml using the template below. For a full list of all available parameters and their effects, see the [Configuration Reference Guide](/docs/config-file.md).
+Create `codefinetuner_config.yaml` using the template below. For the full parameter list, see the [Configuration Reference Guide](/docs/config-file.md).
 
 ```yaml
 # globals contain all the mandatory parameters.
@@ -161,8 +157,11 @@ evaluate:
   benchmark_sample_size: 4
   # ... (evaluate specific settings)
 
+convert:
+  <<: *globals
+  # ... (convert specific settings)
 ```
-> **Note:** For a complete, production-ready example, see [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml).
+> **Note:** See [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml) for a full production example.
 
 ### Data Preparation
 Place source files in your `raw_data_path` (default: `workspace_path/data`).
@@ -177,9 +176,12 @@ Run the pipeline using the unified CLI:
 uv run codefinetuner --config="config/codefinetuner_config.yaml"
 ```
 
-**Pipeline Flags:**
-* `--config`: Specify path to a different config file.
-* `--skip-preprocess`, `--skip-finetune`, `--skip-evaluate`, `--skip-convert`: Skip specific stages.
+**Pipeline flags**
+- `--config`: Use a different config file.
+- `--skip-preprocess`: Skip preprocessing.
+- `--skip-finetune`: Skip fine-tuning.
+- `--skip-evaluate`: Skip evaluation.
+- `--skip-convert`: Skip conversion.
 
 ### Python Module Usage
 ```python
@@ -194,18 +196,14 @@ codefinetuner.run_pipeline(
     skip_preprocess=True,
     skip_convert=True
 )
-
-# Individual stages
-from codefinetuner import preprocess, finetune
-preprocess.run("config.yaml")
 ```
 
-## Deployment: Using the Model
-The `convert` stage converts the model to GGUF format. The final GGUF file is located under `outputs/convert/results/lora_model.gguf`.  
-For a detailed guide on how to use the gguf model with the VS Code extension [llama.vscode](https://github.com/ggml-org/llama-vscode), check out the [inference-vscode](/docs/inference-vscode.md) guide.
+## Finetuned Model Usage
+The `convert` stage exports the final model to [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) format for local inference. The resulting file is saved at `outputs/convert/results/lora_model.gguf`. 
+For setup instructions with the VS Code extension [llama.vscode](https://github.com/ggml-org/llama.vscode), see the [inference-vscode](/docs/inference-vscode.md) guide.
 
 
-## Create And Run Docker Image
+## Docker Image
 
 #### 1. Build the Docker Image
 Build the image from the `Dockerfile`, tagging it as codefinetuner-image.
@@ -213,10 +211,8 @@ Build the image from the `Dockerfile`, tagging it as codefinetuner-image.
 docker build -t codefinetuner-image .
 ```
 #### 2. Prepare Data and Run the Container
-To allow the container to access your data for fine-tuning, use a bind mount to link your host machine's `data` directory to the container.
-- On your host machine (where you run Docker), create a folder named `data` if it doesn't already exist.
-- Put all files you want to use for fine-tuning inside the `data` directory. For `manual` mode, include `train`, `eval`, and `test` subdirectories containing your manually splitted files.
-- Start the container with the bind mount, and open a Bash shell depending on your host machine hardware:
+To allow the container to access your data for fine-tuning, use a bind mount to link your host machine's `data` directory to the container.  
+On your host machine (where you run Docker), create a folder named `data` if it does not already exist. Put all files you want to use for fine-tuning inside the `data` directory. For manual mode, include `train`, `eval`, and `test` subfolders with the split you want to use.  
 
 #### NVIDIA GPU (Recommended)
 Use this command to enable CUDA support for `torch` and `bitsandbytes`. Requires the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed on the host machine.
@@ -227,8 +223,7 @@ docker run --gpus all -it --rm \
   codefinetuner-image /bin/bash
 ```
 #### CPU Only
-
-Use this command if no compatible GPU is available. Note that fine-tuning will be significantly slower.
+Use this if you do not have a compatible GPU. Fine-tuning will be much slower.
 ```bash
 docker run -it --rm \
   -v $(pwd)/data:/app/data \
@@ -236,21 +231,22 @@ docker run -it --rm \
 ```
 
 ## Tree-sitter Customization
-Tree-sitter parses code into structural blocks for generating FIM training examples. Customize for new languages or build missing parsers.
-
-- [Add Language Definitions](docs/tree-sitter-customization.md#add-new-language-block-definitions) - Define `block_types`/`subblock_types` in JSON.
-- [Build Custom Parser](docs/tree-sitter-customization.md#build-custom-parser) - Compile from source (e.g., Mojo).
+Tree-sitter turns source code into structural blocks used to generate FIM examples. Use this section to add new languages or build missing parsers.
 
+- [Add Language Definitions](docs/tree-sitter-customization.md#add-new-language-block-definitions): define `block_types` and `subblock_types` in JSON.
+- [Build Custom Parser](docs/tree-sitter-customization.md#build-custom-parser): compile a parser from source, for example for Mojo.
 
 ## Tests
+Run the test suite with:
+
 ```bash
 pytest
 ```
 
-## Useful Resources
+## Resources
 - [Qwen2.5-Coder Technical Report](https://arxiv.org/pdf/2409.12186)
 - [Structure-Aware Fill-in-the-Middle Pretraining for Code](https://arxiv.org/pdf/2506.00204)
-- [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685)
+- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/pdf/2106.09685)
 - [Efficient Training of Language Models to Fill in the Middle](https://arxiv.org/pdf/2207.14255)
 - [From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?](https://arxiv.org/pdf/2505.18789)
 - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://arxiv.org/pdf/2009.10297)

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/docs/config-file.md

@@ -1,6 +1,6 @@
 # Configuration Reference
 
-This document defines all mandatory and optional parameters in the YAML cofig file. The pipeline uses YAML anchors (`&globals`) and aliases (`<<: *globals`) to propagate shared settings across the `preprocess`, `finetune`, and `evaluate` stages. See [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml) for an example file.
+This document defines all mandatory and optional parameters in the YAML config file. The pipeline uses YAML anchors (`&globals`) and aliases (`<<: *globals`) to propagate shared settings across the `preprocess`, `finetune`, `evaluate` and `convert` stages. See [`config/codefinetuner_config.yaml`](/config/codefinetuner_config.yaml) for an example file.
 
 ## Global Parameters (Mandatory)
 *Shared across all stages.* 
@@ -27,10 +27,10 @@ This document defines all mandatory and optional parameters in the YAML cofig fi
 | `eval_ratio` | float | `0.1` | Validation set ratio (used in `auto` split). |
 | `test_ratio` | float | `0.1` | Test set ratio (used in `auto` split). |
 | `max_token_sequence_length` | int | `1024` | Maximum tokens per training example. |
-| `max_code_blocks_ast_depth` | int | `2` | Tree Sitter AST depth limit for block extraction. Depth 1 is root, 2 includes child nodes (e.g. functions) |
+| `max_code_blocks_ast_depth` | int | `2` | Tree-Sitter AST depth limit for block extraction. Depth 1 is root, 2 includes child nodes (e.g. functions). |
 | `min_middle_tokens_length` | int | `20` | Minimum tokens required in the FIM "middle" section of an example. |
 | `max_middle_tokens_length` | int | `200` | Maximum tokens allowed in the FIM "middle" section of an example. |
-| `fim_examples_per_subblock_ratio` | float | `1.0` | Number of FIM examples generated per subblock.  1.0 = all fim examples of a subblock are extracted, 0.5 = onls 50% of fim examples of a subblock are extracted |
+| `fim_examples_per_subblock_ratio` | float | `1.0` | Number of FIM examples generated per subblock.  1.0 = all FIM examples of a subblock are extracted, 0.5 = only 50% are extracted |
 | `tokenizer_batch_size` | int | `32` | Batch size for the tokenizer. The number of examples processed simultaneously by the tokenizer to improve throughput. |
 | `raw_data_path` | str/null | `"data"` | Location of source code files used to generate the datasets. `null` defaults to `<workspace>/data`. |
 | `tree_sitter_parser_path` | str/null | `null` | Path to custom `.so`/`.dylib` parser file. |
@@ -44,7 +44,7 @@ This document defines all mandatory and optional parameters in the YAML cofig fi
 |-----------|------|---------|-------------|
 | `model_attn_implementation` | str | `"sdpa"` | Attention backend. `flash_attention_2` is fastest (requires Ampere+ GPUs); `sdpa` is the efficient PyTorch default. |
 | `lora_r` | int | `32` | Rank of adapter matrices. Controls parameter count and the complexity of learned patterns. |
-| `lora_alpha` | int | `64` | Scaling factor for LoRA updates. Usually set to $2 \times$ `lora_r` to maintain numerical stability. |
+| `lora_alpha` | int | `64` | Scaling factor for LoRA updates. Usually set to 2*`lora_r` to maintain numerical stability. |
 | `lora_dropout` | float | `0.1` | Dropout probability for LoRA layers to prevent overfitting on specific code snippets. |
 | `lora_bias` | str | `"none"` | Specifies if bias parameters are trained (`"none"`, `"all"`, `"lora_only"`). |
 | `lora_target_modules` | list | `["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "down_proj", "up_proj"]` | Model layers targeted for adaptation. Increasing the list improves performance but consumes more VRAM. |
@@ -66,7 +66,7 @@ This document defines all mandatory and optional parameters in the YAML cofig fi
 | `trainer_save_strategy` | str | `"steps"` | Trigger for saving model checkpoints (`"steps"`, `"epoch"`, or `"no"`). |
 | `trainer_save_steps` | int | `100` | Interval of training steps between model checkpoint saves. Only active if `trainer_save_strategy` is set to `"steps"` |
 | `trainer_logging_strategy` | str | `"steps"` | Trigger for logging metrics (`"steps"`, `"epoch"`, or `"no"`). |
-| `dataset_shuffle_buffer_size` | int | `50000` | Number of training examples to load into memory for randomly shuffling datasets since steramable datasets are used. |
+| `dataset_shuffle_buffer_size` | int | `50000` | Number of training examples to load into memory for randomly shuffling datasets since streamable datasets are used. |
 | `dataset_shuffle_seed` | int | `0` | Seed for shuffling datasets. |
 
 ## Evaluate Parameters (Optional)
@@ -80,9 +80,9 @@ This document defines all mandatory and optional parameters in the YAML cofig fi
 | `benchmark_shuffle_seed` | int | `42` | Random seed ensuring that dataset shuffling is deterministic and reproducible. |
 | `benchmark_use_existing_dataset` | bool | `false` | If `true`, the pipeline reuses a previously generated benchmark dataset file instead of creating a new one. |
 | `generation_max_new_tokens` | int | `128` | Upper limit on the number of tokens the models are permitted to generate for each code completion. |
-| `generation_do_sample` | bool | `false` | Enables probabilistic sampling. If `false`, the model uses greedy decoding (picking only the top token) and `generation_temperature` and `generation_top_p parameters` are ignored.|
-| `generation_temperature` | float | `0.7` | Probability smoothing factor. Values < 1.0 make the model more confident; > 1.0 make it more random. Only active when `generation_do_sample` is set to `true`.|
-| `generation_top_p` | float | `0.95` | Cumulative probability threshold for nucleus sampling. Limits choices to the most likely tokens totaling 95% probability. Only active when `generation_do_sample` is set to `true`.|
+| `generation_do_sample` | bool | `false` | Enables probabilistic sampling. If `false`, the model uses greedy decoding (picking only the top token) and `generation_temperature` and `generation_top_p` are ignored.|
+| `generation_temperature` | float | `0.7` | Probability smoothing factor. Values < 1.0 make the model more confident; > 1.0 make it more random. Only active when `generation_do_sample` is `true`.|
+| `generation_top_p` | float | `0.95` | Cumulative probability threshold for nucleus sampling. Limits choices to the most likely tokens totaling 95% probability. Only active when `generation_do_sample` is `true`.|
 | `codebleu_language` | str | `"c"` | Forces the CodeBLEU parser to use a specific language's grammar rules. |
 | `codebleu_ngram_weight` | float | `0.25` | Standard token match. Measures how many exact words or symbols match the ground truth, treating every character (like `;` or `sum`) with equal importance. |
 | `codebleu_weighted_ngram_weight` | float | `0.25` | Keyword-based scoring. Gives higher points for correctly predicting programming keywords (like `if`, `while`, `return`, `int`) than for standard symbols, ensuring the score reflects the model's grasp of the code's logic. |
@@ -92,6 +92,9 @@ This document defines all mandatory and optional parameters in the YAML cofig fi
 | `sentencebleu_ngram_weight_2` | float | `0.25` | Short phrase accuracy. Weight of 2-token pairs (2-grams) in the final BLEU score calculation. |
 | `sentencebleu_ngram_weight_3` | float | `0.25` | Medium phrase accuracy. Weight of 3-token chains (3-grams) in the final BLEU score calculation. |
 | `sentencebleu_ngram_weight_4` | float | `0.25` | Long phrase accuracy. Weight of 4-token blocks (4-grams) in the final BLEU score calculation. |
-| `line_match_number_of_lines` | int | `2` | Number of identical consecutive lines required between prediction and ground truth to score a line match. |
+| `line_match_number_of_lines` | int | `2` | Number of identical consecutive lines required between prediction and ground truth to count as a line match. |
 | `trainer_checkpoint` | str | `"last"` | Specifies the checkpoint folder to load. Use `last` for the latest or provide a specific directory name. |
-| `plot_only` | bool | `false` | If `true`, skips the heavy inference and scoring steps to only generate charts from existing results. |
+| `plot_only` | bool | `false` | If `true`, skips the heavy inference and scoring steps to only generate charts from existing results. |
+
+## Convert Parameters (Optional)
+*This stage currently has no configurable parameters.*

{codefinetuner-0.1.0 → codefinetuner-0.2.0}/docs/inference-vscode.md

@@ -3,34 +3,10 @@
 ## How to Use the LoRA Fine-tuned Model
 You can use the fine-tuned model with any code autocompletion tool that supports FIM (Fill-In-the-Middle) models. This section explains how to use the VS Code extension [llama.vscode](https://github.com/ggml-org/llama.vscode) to run the model locally on your machine.
 
-### 1. Convert to GGUF Format
-Convert the fine-tuned model to GGUF format and save it to the `outputs/export/results` directory.
-
-Clone the [llama.cpp](https://github.com/ggml-org/llama.cpp) repository. Adjust the `codefinetuner_ws_path` variable to the absolute path of your `codefinetuner` workspace (see YAML config file -> `workspace_path` ) in the script below.
-
-```bash
-codefinetuner_ws_path="/path/to/your/codefinetuner_workspace" 
-
-git clone https://github.com/ggml-org/llama.cpp
-cd llama.cpp
-python -m venv .venv
-source .venv/bin/activate
-pip install -r requirements.txt
-
-# Define and create the results directory
-export_results_path="$codefinetuner_ws_path/outputs/export/results"
-mkdir -p "$export_results_path"
-
-python convert_hf_to_gguf.py \
-    "$codefinetuner_ws_path/outputs/finetune/lora_model" \
-    --outfile "$export_results_path/lora_model.gguf" \
-    --outtype bf16
-```
-
-### 2. Install llama.vscode
+### 1. Install llama.vscode
 Install the extension from the VS Code Marketplace: [Click here](https://marketplace.visualstudio.com/items?itemName=ggml-org.llama-vscode).
 
-### 3. Ensure llama.cpp is Installed
+### 2. Ensure llama.cpp is Installed
 Open the **llama-vscode** menu by clicking it in the status bar or pressing `Ctrl+Shift+M`, then select **Install/Upgrade llama.cpp**.
 
 *Note: On macOS, you can also install via Homebrew:*
@@ -38,14 +14,14 @@ Open the **llama-vscode** menu by clicking it in the status bar or pressing `Ctr
 brew install llama.cpp
 ```
 
-### 4. Select the Fine-tuned Model in llama-vscode
+### 3. Select the Fine-tuned Model in llama-vscode
 In the llama-vscode menu:  
 -> "Completion models..."  
 -> "Add local completion model..."  
 -> Enter model name lora_model_gguf (or any name you prefer).  
 -> Enter the following command to start the model locally:  
 ```bash
-    llama-server -m /path/to/your/codefinetuner_workspace/outputs/export/results/lora_model.gguf --port 8012
+    llama-server -m /path/to/your/codefinetuner_workspace/outputs/convert/results/lora_model.gguf --port 8012
 ```  
 -> Confirm Endpoint ```http://127.0.0.1:8012```  
 -> No API Key required