raft-ft 2.2.0__tar.gz

raft_ft-2.2.0/.gitignore

@@ -0,0 +1,39 @@
+# python / pipenv gitignore
+
+*.pyc
+__pycache__/
+*.pyo
+*.pyd
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+*.so
+*.o
+*.dylib
+.Python
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+.env
+.venv
+.spyderproject
+.spyproject
+.ropeproject
+.idea/
+.vscode/
+*.swp
+*.swo
+.DS_Store
+secret/
+data/garymarcus/**/*.*
+data/garymarcus/*
+data/garymarcus/
+data/garymarcus/chroma.sqlite3
+data/garymarcus/9878a8dc-7749-4f04-897e-ce480fc4f0f0/*.bin
+.ruff_cache/

raft_ft-2.2.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Claudio Brandolino
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

raft_ft-2.2.0/PKG-INFO

@@ -0,0 +1,215 @@
+Metadata-Version: 2.5
+Name: raft-ft
+Version: 2.2.0
+Summary: RAFT: Retrieval-Augmented Fine-Tuning — build persona datasets and finetune models on them.
+Project-URL: Homepage, https://github.com/lumpenspace/raft
+Project-URL: Issues, https://github.com/lumpenspace/raft/issues
+Author: lumpenspace
+License-Expression: MIT
+License-File: LICENSE
+Keywords: finetuning,huggingface,llm,openai,persona,rag
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.11
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: chromadb<2,>=1.0
+Requires-Dist: openai>=1.14
+Requires-Dist: pandas>=2.2
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: requests>=2.31
+Requires-Dist: rich>=13.7
+Requires-Dist: tiktoken>=0.6
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.8; extra == 'dev'
+Requires-Dist: twine>=5.0; extra == 'dev'
+Provides-Extra: hf
+Requires-Dist: opbdh>=1.3; extra == 'hf'
+Description-Content-Type: text/markdown
+
+# RAFT: Retrieval-Augmented Fine-Tuning
+
+Note from [@lumpenspace](http://x.com/lumpenspace):
+
+> This technique is something ive been working last summer/fall, originally planning to get a paper out of it. Then it seemed obvious so i didn't, and instead used
+> pieces of this repo for other projects and abandoned this repo.
+>
+> I [discovered not without horror](https://x.com/lumpenspace/status/1769809977030426831?s=20) that some of the tech is still cutting edge, so i might as well share it.
+>
+> In this old version, the main simulee was Gary Marcus; the idea was to make a model that could pass as him in a conversation and demonstrate how stochastic
+> parrots are still plenty capable to mimic the deterministic ones, but there's a couple interesting tidbits that i've moved to more decent repos but, given my 
+> pretty annoying habit of not sharing subpar code, you might as well start here.
+>
+> Scroll to [usage and functionality](#usage-and-functionality) for cli options, what you can do (apart from what's described below) is automagically fetch, chunk, embed, story, and query a db starting from a substack url.
+> 
+> Not guaranteeing anything works, but it's a good starting point for a lot of things and includes a couple of new ideas.
+
+### OH GOSH
+
+ok then, a friend asked so now it is more lenient with the version number and uses poetry for the dependencies. It's still a mess, but it's a more runnable mess.
+
+(2.1: poetry is gone — it's uv + hatchling now, like the other repos in this constellation.)
+
+## 2.0
+
+New major version. Substack is no longer the only way in:
+
+- **`raft interactive`** — guided end-to-end session. Asks who the target is, collects
+  text sources (substack / tweets / local files) and conversation examples. Structured
+  inputs (raft transcripts, chat-message JSON, grounding jsonl) are recognised and
+  imported as-is; unstructured ones (raw chat logs, podcast transcripts, whatever) are
+  converted into transcript datasets with an LLM (`RAFT_LLM_MODEL`, default `gpt-4o`).
+  Then chunk/embed/ft:gen/ft:run, each step optional.
+- **`raft tweets`** — tweet mode. First asks which network(s) to draw from — **X /
+  Twitter**, **Bluesky**, or **both, merged into one dataset** — then calls
+  [ariadne](https://github.com/lumpenspace/ariadne)'s Python API to reconstruct reply
+  branches and imports them: thread texts become grounding documents, reply branches
+  become q/a transcripts. The target's own posts become the answers, whoever they were
+  replying to becomes the questioner. For X you choose the source (archive export,
+  CSV/JSON dump, or a public handle) and can add the **Community Archive**
+  (community-archive.org — no key, and it completes reply threads whose parents were
+  authored by other people) and/or a **twitterapi.io** key. Bluesky needs nothing but a
+  handle. Needs ariadne (≥ 0.4), which is not on PyPI (the name is taken by the GraphQL
+  library): `pip install git+https://github.com/lumpenspace/ariadne`.
+- **`raft ft:run <name> --model <model>`** — model routing. OpenAI-finetunable ids
+  (gpt-4o-mini and friends) go through the OpenAI finetuning API as before. Any other
+  model — i.e. a huggingface `org/name` id — is trained on a rented GPU pod via
+  [opbdh](https://github.com/lumpenspace/opbdh): raft generates a self-contained LoRA SFT
+  run directory (script + requirements + dataset) and hands it to `opbdh launch`.
+  Interactively it helps you pick the model (`opbdh models search`) and size the pod;
+  non-interactively, opbdh settings pass straight through:
+
+  ```bash
+  raft ft:run garymarcus --model Qwen/Qwen2.5-7B-Instruct --vram-gb 48 --max-spend 5
+  ```
+
+  Anything opbdh accepts (`--provider`, `--max-dollars-per-hour`, ...) can be appended
+  and is forwarded to its Python API, and an `opbdh.json` in the project root works too.
+  The trained adapter lands in `runpod_results/<run_id>/results/adapter`. Install with
+  `pip install 'raft-ft[hf]'` (python ≥ 3.11) plus a one-time `opbdh config wizard`.
+
+Both integrations go through the two tools' Python APIs rather than shelling out, so
+raft gets the reconstructed threads and the run result as data — and surfaces their
+errors (spend guard tripped, remote job failed) directly.
+
+# RAFT / RATF
+
+- [RAFT: Retrieval-Augmented Fine-Tuning](#raft-retrieval-augmented-fine-tuning)
+    - [OH GOSH](#oh-gosh)
+- [RAFT / RATF](#raft--ratf)
+  - [Abstract](#abstract)
+  - [Process](#process)
+    - [Retrieval-Augmented Fine-Tuning](#retrieval-augmented-fine-tuning)
+    - [Generation](#generation)
+  - [Usage and Functionality](#usage-and-functionality)
+    - [Installation](#installation)
+    - [Usage](#usage)
+  - [Licence](#licence)
+
+RAFT, or Retrieval-Augmented Fine-Tuning, is a method comprising of a fine-tuning and a RAG-based retrieval phase. It is particularly suited for the creation of agents that realistically emulate a specific human target.
+
+RATF, or Replica Agent Testing Framework, is a framework for evaluating the performance of dialogue agents emulating real-world targets.
+
+## Abstract
+
+The emulation of specific humans in conversational agents presents unique challenges and opportunities for contextual understanding, theory of mind and personalization. In this paper, we introduce the Retrieval-Augmented Fine-Tuning (RAFT) methodology, designed explicitly for simulating individual humans.
+
+RAFT employs a dual-phase process:
+
+In the **Retrieval-Augmented Fine-Tuning phase** proper, combines interview transcripts featuring the human target with appropriately selected, rephrased and evaluated "memories" from the author's past output to give the model a sense of the way the target human combines past writings with the current context to generate responses.
+
+In the **generation phase**, these memories augment the language model's responses to create a nuanced and personalized dialogue.
+
+We demonstrate the efficacy of RAFT through a unique evaluation metric, RATF (Replica Agent Testing Framework) that compares model-generated responses with original human responses in an interview setting. Our findings highlight RAFT's potential to significantly advance the field of personalized, context-sensitive conversational agents.
+
+## Process
+
+### Retrieval-Augmented Fine-Tuning
+
+Two datasets are required for the fine-tuning phase:
+
+- A dataset of **interview transcripts** featuring the target human
+- A dataset of the **target's past written output** (tweets, essays, etc.)
+
+The interview transcripts used within a RAG-inspired process retreiving "memories" from the target's written output for each of the interviewer's questions. These memories are then rephrased and evaluated in the context of the target user's answer and, if found useful, they are interpolated between question and answer for the fine-tuning phase.
+
+The steps to reproduce this process are as follows:
+
+1. Create a dataset of interview transcripts featuring the target human. Each interview is a separate `data/{name}_transcript_{i}.json` file holding `{"participants": {"q": ..., "a": ...}, "date": ..., "url": ..., "exchanges": [[question, answer], ...]}`. As of 2.0 you don't have to write these by hand: `raft interactive` takes chat-message JSON, ariadne output or plain unstructured transcripts and produces them for you.
+2. Create a dataset of the author's past written output — `data/{name}.jsonl`, one `{"title", "link", "date", "content"}` object per line. `raft fetch` builds this from a substack; `raft tweets` from a tweet archive; `raft interactive` from arbitrary local files.
+3. Split the past output dataset in chunks of a size suitable for the chosen embedding model (8192 tokens for Openai's text-embedding-ada-002), and collect metadata and embeddings for each chunk.
+4. Store the resulting metadata and embeddings in a vector database (we use ChromaDB).
+
+Then, in order to generate a fine-tuning dataset:
+
+1. For each interview, run the RAG process to retrieve memories from the author's past output for each of the interviewer's questions.
+2. Ask the model to rephrase each memory in the context of the interviewer's question. The same model and prompt will be used in the generation phase.
+3. Evaluate the resulting memory by the question only first, and discard it if it is not considered useful by the model. We apply this first pass separately because, at inference time, we will not have access to the target human's answer.
+4. Save the resulting context including question, memory and as many of the previous [question, memory and answers] tuples as possible, up to the maximum context size the finetune allows, as a new finetune sample.
+
+#### Before/after pics (interview/ ft dataset)
+
+![](https://github.com/lumpenspace/raft/blob/main/Screenshot%202024-03-18%20at%2021.05.39.png?raw=true)
+
+### Generation
+
+The fine-tuned model is then used to generate responses to the interviewer's questions. The model is prompted with the question and the rephrased memories, and the resulting response is evaluated using the RATF framework.
+
+
+## Usage and Functionality
+
+### Installation
+
+The distribution is named `raft-ft` (`raft` was taken on PyPI); the import and the
+CLI are still `raft`. Until the first `raft-ft` release lands on PyPI, install from
+git:
+
+```bash
+pip install git+https://github.com/lumpenspace/raft
+```
+
+Once released:
+
+```bash
+pip install raft-ft
+```
+
+For development, [uv](https://docs.astral.sh/uv/) manages the environment:
+
+```bash
+uv sync --extra dev --extra hf
+uv run pytest
+uv run ruff check .
+```
+
+### Usage
+
+```bash
+raft -h
+```
+
+```
+
+The following actions are available:
+
+- interactive: Guided end-to-end session: sources, conversations, finetune.
+- tweets: Build a dataset from tweets via ariadne interactive.
+- fetch: Fetch the blog from Substack and store it in the data directory.
+- chunk: Chunk the blog into 4096 token pieces and store them in the data directory.
+- embed: Create embeddings for the chunks and store them.
+- ft:gen: Generate finetune files for the blog.
+- ft:run: Run the finetune job (OpenAI, or huggingface via opbdh).
+- bench:setup: Setup the benchmark for the blog.
+- ask: Ask a question about the blog content.
+```
+
+## Licence
+
+MIT

raft_ft-2.2.0/README.md

@@ -0,0 +1,179 @@
+# RAFT: Retrieval-Augmented Fine-Tuning
+
+Note from [@lumpenspace](http://x.com/lumpenspace):
+
+> This technique is something ive been working last summer/fall, originally planning to get a paper out of it. Then it seemed obvious so i didn't, and instead used
+> pieces of this repo for other projects and abandoned this repo.
+>
+> I [discovered not without horror](https://x.com/lumpenspace/status/1769809977030426831?s=20) that some of the tech is still cutting edge, so i might as well share it.
+>
+> In this old version, the main simulee was Gary Marcus; the idea was to make a model that could pass as him in a conversation and demonstrate how stochastic
+> parrots are still plenty capable to mimic the deterministic ones, but there's a couple interesting tidbits that i've moved to more decent repos but, given my 
+> pretty annoying habit of not sharing subpar code, you might as well start here.
+>
+> Scroll to [usage and functionality](#usage-and-functionality) for cli options, what you can do (apart from what's described below) is automagically fetch, chunk, embed, story, and query a db starting from a substack url.
+> 
+> Not guaranteeing anything works, but it's a good starting point for a lot of things and includes a couple of new ideas.
+
+### OH GOSH
+
+ok then, a friend asked so now it is more lenient with the version number and uses poetry for the dependencies. It's still a mess, but it's a more runnable mess.
+
+(2.1: poetry is gone — it's uv + hatchling now, like the other repos in this constellation.)
+
+## 2.0
+
+New major version. Substack is no longer the only way in:
+
+- **`raft interactive`** — guided end-to-end session. Asks who the target is, collects
+  text sources (substack / tweets / local files) and conversation examples. Structured
+  inputs (raft transcripts, chat-message JSON, grounding jsonl) are recognised and
+  imported as-is; unstructured ones (raw chat logs, podcast transcripts, whatever) are
+  converted into transcript datasets with an LLM (`RAFT_LLM_MODEL`, default `gpt-4o`).
+  Then chunk/embed/ft:gen/ft:run, each step optional.
+- **`raft tweets`** — tweet mode. First asks which network(s) to draw from — **X /
+  Twitter**, **Bluesky**, or **both, merged into one dataset** — then calls
+  [ariadne](https://github.com/lumpenspace/ariadne)'s Python API to reconstruct reply
+  branches and imports them: thread texts become grounding documents, reply branches
+  become q/a transcripts. The target's own posts become the answers, whoever they were
+  replying to becomes the questioner. For X you choose the source (archive export,
+  CSV/JSON dump, or a public handle) and can add the **Community Archive**
+  (community-archive.org — no key, and it completes reply threads whose parents were
+  authored by other people) and/or a **twitterapi.io** key. Bluesky needs nothing but a
+  handle. Needs ariadne (≥ 0.4), which is not on PyPI (the name is taken by the GraphQL
+  library): `pip install git+https://github.com/lumpenspace/ariadne`.
+- **`raft ft:run <name> --model <model>`** — model routing. OpenAI-finetunable ids
+  (gpt-4o-mini and friends) go through the OpenAI finetuning API as before. Any other
+  model — i.e. a huggingface `org/name` id — is trained on a rented GPU pod via
+  [opbdh](https://github.com/lumpenspace/opbdh): raft generates a self-contained LoRA SFT
+  run directory (script + requirements + dataset) and hands it to `opbdh launch`.
+  Interactively it helps you pick the model (`opbdh models search`) and size the pod;
+  non-interactively, opbdh settings pass straight through:
+
+  ```bash
+  raft ft:run garymarcus --model Qwen/Qwen2.5-7B-Instruct --vram-gb 48 --max-spend 5
+  ```
+
+  Anything opbdh accepts (`--provider`, `--max-dollars-per-hour`, ...) can be appended
+  and is forwarded to its Python API, and an `opbdh.json` in the project root works too.
+  The trained adapter lands in `runpod_results/<run_id>/results/adapter`. Install with
+  `pip install 'raft-ft[hf]'` (python ≥ 3.11) plus a one-time `opbdh config wizard`.
+
+Both integrations go through the two tools' Python APIs rather than shelling out, so
+raft gets the reconstructed threads and the run result as data — and surfaces their
+errors (spend guard tripped, remote job failed) directly.
+
+# RAFT / RATF
+
+- [RAFT: Retrieval-Augmented Fine-Tuning](#raft-retrieval-augmented-fine-tuning)
+    - [OH GOSH](#oh-gosh)
+- [RAFT / RATF](#raft--ratf)
+  - [Abstract](#abstract)
+  - [Process](#process)
+    - [Retrieval-Augmented Fine-Tuning](#retrieval-augmented-fine-tuning)
+    - [Generation](#generation)
+  - [Usage and Functionality](#usage-and-functionality)
+    - [Installation](#installation)
+    - [Usage](#usage)
+  - [Licence](#licence)
+
+RAFT, or Retrieval-Augmented Fine-Tuning, is a method comprising of a fine-tuning and a RAG-based retrieval phase. It is particularly suited for the creation of agents that realistically emulate a specific human target.
+
+RATF, or Replica Agent Testing Framework, is a framework for evaluating the performance of dialogue agents emulating real-world targets.
+
+## Abstract
+
+The emulation of specific humans in conversational agents presents unique challenges and opportunities for contextual understanding, theory of mind and personalization. In this paper, we introduce the Retrieval-Augmented Fine-Tuning (RAFT) methodology, designed explicitly for simulating individual humans.
+
+RAFT employs a dual-phase process:
+
+In the **Retrieval-Augmented Fine-Tuning phase** proper, combines interview transcripts featuring the human target with appropriately selected, rephrased and evaluated "memories" from the author's past output to give the model a sense of the way the target human combines past writings with the current context to generate responses.
+
+In the **generation phase**, these memories augment the language model's responses to create a nuanced and personalized dialogue.
+
+We demonstrate the efficacy of RAFT through a unique evaluation metric, RATF (Replica Agent Testing Framework) that compares model-generated responses with original human responses in an interview setting. Our findings highlight RAFT's potential to significantly advance the field of personalized, context-sensitive conversational agents.
+
+## Process
+
+### Retrieval-Augmented Fine-Tuning
+
+Two datasets are required for the fine-tuning phase:
+
+- A dataset of **interview transcripts** featuring the target human
+- A dataset of the **target's past written output** (tweets, essays, etc.)
+
+The interview transcripts used within a RAG-inspired process retreiving "memories" from the target's written output for each of the interviewer's questions. These memories are then rephrased and evaluated in the context of the target user's answer and, if found useful, they are interpolated between question and answer for the fine-tuning phase.
+
+The steps to reproduce this process are as follows:
+
+1. Create a dataset of interview transcripts featuring the target human. Each interview is a separate `data/{name}_transcript_{i}.json` file holding `{"participants": {"q": ..., "a": ...}, "date": ..., "url": ..., "exchanges": [[question, answer], ...]}`. As of 2.0 you don't have to write these by hand: `raft interactive` takes chat-message JSON, ariadne output or plain unstructured transcripts and produces them for you.
+2. Create a dataset of the author's past written output — `data/{name}.jsonl`, one `{"title", "link", "date", "content"}` object per line. `raft fetch` builds this from a substack; `raft tweets` from a tweet archive; `raft interactive` from arbitrary local files.
+3. Split the past output dataset in chunks of a size suitable for the chosen embedding model (8192 tokens for Openai's text-embedding-ada-002), and collect metadata and embeddings for each chunk.
+4. Store the resulting metadata and embeddings in a vector database (we use ChromaDB).
+
+Then, in order to generate a fine-tuning dataset:
+
+1. For each interview, run the RAG process to retrieve memories from the author's past output for each of the interviewer's questions.
+2. Ask the model to rephrase each memory in the context of the interviewer's question. The same model and prompt will be used in the generation phase.
+3. Evaluate the resulting memory by the question only first, and discard it if it is not considered useful by the model. We apply this first pass separately because, at inference time, we will not have access to the target human's answer.
+4. Save the resulting context including question, memory and as many of the previous [question, memory and answers] tuples as possible, up to the maximum context size the finetune allows, as a new finetune sample.
+
+#### Before/after pics (interview/ ft dataset)
+
+![](https://github.com/lumpenspace/raft/blob/main/Screenshot%202024-03-18%20at%2021.05.39.png?raw=true)
+
+### Generation
+
+The fine-tuned model is then used to generate responses to the interviewer's questions. The model is prompted with the question and the rephrased memories, and the resulting response is evaluated using the RATF framework.
+
+
+## Usage and Functionality
+
+### Installation
+
+The distribution is named `raft-ft` (`raft` was taken on PyPI); the import and the
+CLI are still `raft`. Until the first `raft-ft` release lands on PyPI, install from
+git:
+
+```bash
+pip install git+https://github.com/lumpenspace/raft
+```
+
+Once released:
+
+```bash
+pip install raft-ft
+```
+
+For development, [uv](https://docs.astral.sh/uv/) manages the environment:
+
+```bash
+uv sync --extra dev --extra hf
+uv run pytest
+uv run ruff check .
+```
+
+### Usage
+
+```bash
+raft -h
+```
+
+```
+
+The following actions are available:
+
+- interactive: Guided end-to-end session: sources, conversations, finetune.
+- tweets: Build a dataset from tweets via ariadne interactive.
+- fetch: Fetch the blog from Substack and store it in the data directory.
+- chunk: Chunk the blog into 4096 token pieces and store them in the data directory.
+- embed: Create embeddings for the chunks and store them.
+- ft:gen: Generate finetune files for the blog.
+- ft:run: Run the finetune job (OpenAI, or huggingface via opbdh).
+- bench:setup: Setup the benchmark for the blog.
+- ask: Ask a question about the blog content.
+```
+
+## Licence
+
+MIT

raft_ft-2.2.0/pyproject.toml

@@ -0,0 +1,75 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+# The distribution is `raft-ft` because `raft` is taken on PyPI; the import
+# and the CLI are still `raft`.
+name = "raft-ft"
+dynamic = ["version"]
+description = "RAFT: Retrieval-Augmented Fine-Tuning — build persona datasets and finetune models on them."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+  { name = "lumpenspace" }
+]
+keywords = ["finetuning", "rag", "llm", "persona", "openai", "huggingface"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+]
+dependencies = [
+  "beautifulsoup4>=4.12",
+  "chromadb>=1.0,<2",
+  "openai>=1.14",
+  "pandas>=2.2",
+  "python-dotenv>=1.0",
+  "requests>=2.31",
+  "rich>=13.7",
+  "tiktoken>=0.6",
+]
+
+[project.urls]
+Homepage = "https://github.com/lumpenspace/raft"
+Issues = "https://github.com/lumpenspace/raft/issues"
+
+[project.scripts]
+raft = "raft.cli:main"
+
+[project.optional-dependencies]
+# Huggingface finetuning on a GPU pod (`raft ft:run` with a non-OpenAI model).
+hf = ["opbdh>=1.3"]
+# Tweet mode also needs ariadne, which is not on PyPI (the name is taken by
+# the GraphQL library): pip install git+https://github.com/lumpenspace/ariadne
+dev = ["build>=1.2", "pytest>=8.0", "ruff>=0.8", "twine>=5.0"]
+
+[tool.hatch.version]
+path = "src/raft/__init__.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/raft"]
+
+# Without this, hatchling's sdist default packs the repo root — including the
+# 1.9 MB of sample corpora and chroma databases under data/.
+[tool.hatch.build.targets.sdist]
+include = ["src/raft", "tests", "README.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+
+# Pinned explicitly rather than inherited, mirroring opbdh: ruff's default
+# rule set changes between releases, and `ruff>=0.8` means CI can install any
+# of them. These are ruff's pre-0.16 defaults.
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F"]

raft_ft-2.2.0/src/raft/__init__.py

@@ -0,0 +1,9 @@
+"""RAFT: Retrieval-Augmented Fine-Tuning."""
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+__all__ = ["__version__"]
+
+__version__ = "2.2.0"

raft_ft-2.2.0/src/raft/chunker.py

@@ -0,0 +1,138 @@
+"""
+This module contains the chunker function, which splits blog posts into chunks
+of a maximum length.
+"""
+
+import json
+from typing import List, Dict, Tuple, Generator
+import tiktoken
+import pandas as pd
+
+MAX_EMBEDDING_LENGTH = 2048
+encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
+
+
+def map_line(line: str) -> Tuple[str, str, int]:
+    """
+    Encode and potentially truncate a line of text.
+
+    Args:
+        line (str): The input line of text.
+
+    Returns:
+        Tuple[str, str, int]: A tuple containing:
+            - the original or truncated line,
+            - the decoded version, and
+            - the length of the encoded line.
+    """
+    encoded = encoding.encode(line)
+    if len(encoded) < MAX_EMBEDDING_LENGTH:
+        return (line, encoding.decode(encoded), len(encoded))
+    else:
+        truncated_encoded = encoded[:MAX_EMBEDDING_LENGTH]
+        return (
+            line[:MAX_EMBEDDING_LENGTH],
+            encoding.decode(truncated_encoded),
+            MAX_EMBEDDING_LENGTH,
+        )
+
+
+def split_into_chunks(
+    blog_posts: pd.DataFrame,
+) -> Generator[Tuple[Dict[str, str], str], None, None]:
+    """
+    Split blog posts into chunks of a maximum length.
+
+    Args:
+        blog_posts (pd.DataFrame): DataFrame containing blog post data.
+
+    Yields:
+        Tuple[Dict[str, str], str]: A tuple containing metadata
+            and the chunked content.
+    """
+    for _, post in blog_posts.iterrows():
+        print(f"Splitting {post['title']}")
+        lines = list(map(map_line, post["content"].split("\n")))
+
+        # Calculate total parts
+        total_parts = 0
+        temp_chunk_length = 0
+        for _, _, length in lines:
+            if temp_chunk_length + length > MAX_EMBEDDING_LENGTH:
+                total_parts += 1
+                temp_chunk_length = length
+            else:
+                temp_chunk_length += length
+        total_parts += 1  # for the last chunk
+
+        chunk: List[str] = []
+        chunk_length = 0
+        part = 1
+
+        for line, _, length in lines:
+            if chunk_length + length > MAX_EMBEDDING_LENGTH:
+                metadata = {
+                    "title": post["title"],
+                    "url": post["link"],
+                    "date": (
+                        post["date"].isoformat()
+                        if isinstance(post["date"], pd.Timestamp)
+                        else post["date"]
+                    ),
+                    "total_parts": str(total_parts),
+                    "part": str(part),
+                }
+                yield metadata, "\n".join(chunk)
+                chunk = []
+                chunk_length = 0
+                part += 1
+            chunk.append(line)
+            chunk_length += length
+
+        if chunk:
+            metadata = {
+                "title": post["title"],
+                "url": post["link"],
+                "date": (
+                    post["date"].isoformat()
+                    if isinstance(post["date"], pd.Timestamp)
+                    else post["date"]
+                ),
+                "total_parts": str(total_parts),
+                "part": str(part),
+            }
+            yield metadata, "\n".join(chunk)
+
+
+def chunker(name: str) -> None:
+    """
+    Process a JSONL file of blog posts, split them into chunks,
+    and save the results.
+
+    Args:
+        name (str): The name of the file to process (without extension).
+    """
+    sourcefile = f"data/{name}.jsonl"
+    outputfile = f"data/{name}_chunked.jsonl"
+
+    try:
+        # Read the JSONL file line by line
+        with open(sourcefile, "r") as f:
+            data = [json.loads(line) for line in f]
+
+        blog_posts: pd.DataFrame = pd.DataFrame(data)
+        print(f"Splitting {len(blog_posts)} blog posts into chunks")
+
+        with open(outputfile, "w") as f:
+            for item in split_into_chunks(blog_posts):
+                f.write(json.dumps(item) + "\n")
+
+    except json.JSONDecodeError as e:
+        print(f"Error decoding JSON: {e}")
+        print(f"Error occurred at line {e.lineno}")
+        with open(sourcefile, "r") as f:
+            problematic_line = f.readlines()[e.lineno - 1]
+        print(f"Problematic line: {problematic_line}")
+
+    except Exception as e:
+        print(f"An error occurred: {e}")