chatan 0.1.0__tar.gz → 0.1.3__tar.gz

{chatan-0.1.0 → chatan-0.1.3}/.gitignore

@@ -8,3 +8,5 @@ wheels/
 
 # Virtual environments
 .venv
+
+.DS_Store

chatan-0.1.3/.readthedocs.yml

@@ -0,0 +1,22 @@
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+  jobs:
+    pre_create_environment:
+      - asdf plugin add uv
+      - asdf install uv latest
+      - asdf global uv latest
+    create_environment:
+      - uv venv "${READTHEDOCS_VIRTUALENV_PATH}"
+    install:
+      - UV_PROJECT_ENVIRONMENT="${READTHEDOCS_VIRTUALENV_PATH}" uv sync --frozen --group docs
+
+sphinx:
+  configuration: docs/source/conf.py
+
+formats:
+  - pdf
+  - epub

chatan-0.1.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Christian Reetz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chatan-0.1.3/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.4
+Name: chatan
+Version: 0.1.3
+Summary: Create synthetic datasets with LLM generators and samplers
+Project-URL: Documentation, https://github.com/cdreetz/chatan#readme
+Project-URL: Issues, https://github.com/cdreetz/chatan/issues
+Project-URL: Source, https://github.com/cdreetz/chatan
+Author-email: Christian Reetz <cdreetz@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dataset generation,llm,machine learning,synthetic data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Requires-Dist: anthropic>=0.7.0
+Requires-Dist: datasets>=2.0.0
+Requires-Dist: nltk>=3.9.1
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: scikit-learn>=1.3.2
+Requires-Dist: sentence-transformers>=3.2.1
+Requires-Dist: torch>=2.5.1
+Requires-Dist: tqdm>=4.0.0
+Requires-Dist: transformers>=4.0.0
+Description-Content-Type: text/markdown
+
+# Chatan
+
+Create diverse, synthetic datasets. Start from scratch or augment an existing dataset. Simple define your dataset schema as a set of generators, typically being LLMs with a prompt describing what kind of examples you want.
+
+## Installation
+
+```
+pip install chatan
+```
+
+## Getting Started
+
+```
+import chatan
+
+# Create a generator
+gen = chatan.generator("openai", "YOUR_API_KEY")
+
+# Define a dataset schema
+ds = chatan.dataset({
+    "topic": chatan.sample.choice(["Python", "JavaScript", "Rust"]),
+    "prompt": gen("write a programming question about {topic}"),
+    "response": gen("answer this question: {prompt}")
+})
+
+# Generate the data with a progress bar
+df = ds.generate(n=10)
+```
+
+## Examples
+
+Create Data Mixes
+
+```
+from chatan import dataset, generator, sample
+import uuid
+
+gen = generator("openai", "YOUR_API_KEY")
+
+mix = [
+    "san antonio, tx",
+    "marfa, tx",
+    "paris, fr"
+]
+
+ds = dataset({
+    "id": sample.uuid(),
+    "topic": sample.choice(mix),
+    "prompt": gen("write an example question about the history of {topic}"),
+    "response": gen("respond to: {prompt}"),
+})
+```
+
+Augment datasets
+
+```
+from chatan import generator, dataset, sample
+from datasets import load_dataset
+
+gen = generator("openai", "YOUR_API_KEY")
+hf_data = load_dataset("some/dataset")
+
+ds = dataset({
+    "original_prompt": sample.from_dataset(hf_data, "prompt"),
+    "variation": gen("rewrite this prompt: {original_prompt}"),
+    "response": gen("respond to: {variation}")
+})
+
+```
+
+## Citation
+
+If you use this code in your research, please cite:
+
+```
+@software{reetz2025chatan,
+  author = {Reetz, Christian},
+  title = {chatan: Create synthetic datasets with LLM generators.},
+  url = {https://github.com/cdreetz/chatan},
+  year = {2025}
+}
+```
+
+## Contributing
+
+Community contributions are more than welcome, bug reports, bug fixes, feature requests, feature additions, please refer to the Issues tab.

chatan-0.1.3/README.md

@@ -0,0 +1,86 @@
+# Chatan
+
+Create diverse, synthetic datasets. Start from scratch or augment an existing dataset. Simple define your dataset schema as a set of generators, typically being LLMs with a prompt describing what kind of examples you want.
+
+## Installation
+
+```
+pip install chatan
+```
+
+## Getting Started
+
+```
+import chatan
+
+# Create a generator
+gen = chatan.generator("openai", "YOUR_API_KEY")
+
+# Define a dataset schema
+ds = chatan.dataset({
+    "topic": chatan.sample.choice(["Python", "JavaScript", "Rust"]),
+    "prompt": gen("write a programming question about {topic}"),
+    "response": gen("answer this question: {prompt}")
+})
+
+# Generate the data with a progress bar
+df = ds.generate(n=10)
+```
+
+## Examples
+
+Create Data Mixes
+
+```
+from chatan import dataset, generator, sample
+import uuid
+
+gen = generator("openai", "YOUR_API_KEY")
+
+mix = [
+    "san antonio, tx",
+    "marfa, tx",
+    "paris, fr"
+]
+
+ds = dataset({
+    "id": sample.uuid(),
+    "topic": sample.choice(mix),
+    "prompt": gen("write an example question about the history of {topic}"),
+    "response": gen("respond to: {prompt}"),
+})
+```
+
+Augment datasets
+
+```
+from chatan import generator, dataset, sample
+from datasets import load_dataset
+
+gen = generator("openai", "YOUR_API_KEY")
+hf_data = load_dataset("some/dataset")
+
+ds = dataset({
+    "original_prompt": sample.from_dataset(hf_data, "prompt"),
+    "variation": gen("rewrite this prompt: {original_prompt}"),
+    "response": gen("respond to: {variation}")
+})
+
+```
+
+## Citation
+
+If you use this code in your research, please cite:
+
+```
+@software{reetz2025chatan,
+  author = {Reetz, Christian},
+  title = {chatan: Create synthetic datasets with LLM generators.},
+  url = {https://github.com/cdreetz/chatan},
+  year = {2025}
+}
+```
+
+## Contributing
+
+Community contributions are more than welcome, bug reports, bug fixes, feature requests, feature additions, please refer to the Issues tab.

chatan-0.1.3/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

chatan-0.1.3/docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

chatan-0.1.3/docs/source/api.rst

@@ -0,0 +1,17 @@
+API Reference
+=====================================
+
+Dataset
+-------
+.. automodule:: chatan.dataset
+   :members:
+
+Generator
+---------
+.. automodule:: chatan.generator
+   :members:
+
+Sampler
+-------
+.. automodule:: chatan.sampler
+   :members:

chatan-0.1.3/docs/source/conf.py

@@ -0,0 +1,46 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../src'))
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = 'chatan'
+copyright = '2025, Christian Reetz'
+author = 'Christian Reetz'
+release = '0.1.1'
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.viewcode',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.autosummary',
+    'sphinx_autodoc_typehints',
+]
+
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = 'sphinx_rtd_theme'
+html_theme_options = {
+    'navigation_depth': 5,
+}
+html_static_path = ['_static']
+
+# -- Extension configuration -------------------------------------------------
+
+
+

chatan-0.1.3/docs/source/examples.rst

@@ -0,0 +1,72 @@
+Examples
+================================================
+
+
+
+Basic QA Dataset
+----------------------------
+
+.. code-block:: python
+
+   import chatan
+
+   gen = chatan.generator("openai", "YOUR_API_KEY")
+   ds = chatan.dataset({
+       "question": gen("write a example question from a 5th grade math test"),
+       "answer": gen("answer: {question}")
+   })
+
+   df = ds.generate(100)
+
+Creating Data Mixes
+----------------------------
+
+.. code-block:: python
+
+   import uuid
+   from chatan import dataset, generator, sample
+
+   gen = generator("openai", "YOUR_API_KEY")
+
+   mix = [
+       "san antonio, tx",
+       "marfa, tx", 
+       "paris, fr"
+   ]
+
+   ds = dataset({
+       "id": sample.uuid(),
+       "topic": sample.choice(mix),
+       "prompt": gen("write an example question about the history of {topic}"),
+       "response": gen("respond to: {prompt}"),
+   })
+
+Dataset Augmentation
+-------------------------------
+
+.. code-block:: python
+
+   from datasets import load_dataset
+   import chatan
+
+   gen = chatan.generator("openai", "YOUR_API_KEY")
+   hf_data = load_dataset("some/dataset")
+
+   ds = chatan.dataset({
+       "original_prompt": chatan.sample.from_dataset(hf_data, "prompt"),
+       "variation": gen("rewrite this prompt: {original_prompt}"),
+       "response": gen("respond to: {variation}")
+   })
+
+Saving Datasets
+---------------
+
+.. code-block:: python
+
+   # Generate and save
+   df = ds.generate(1000)
+   ds.save("my_dataset.parquet")
+   ds.save("my_dataset.csv", format="csv")
+
+   # Convert to HuggingFace format
+   hf_dataset = ds.to_huggingface()

chatan-0.1.3/docs/source/index.rst

@@ -0,0 +1,53 @@
+.. chatan documentation master file, created by
+   sphinx-quickstart on Mon Jun  9 13:53:32 2025.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+chatan documentation
+===========================================
+
+Create synthetic datasets with LLM generators and samplers.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+   :hidden:
+
+   quickstart
+   examples
+   other_examples
+   api
+
+Installation
+------------
+
+.. code-block:: bash
+
+   pip install chatan
+
+Quick Start
+-----------
+
+.. code-block:: python
+
+   import chatan
+
+   # Create a generator
+   gen = chatan.generator("openai", "YOUR_API_KEY")
+
+   # Define a dataset schema
+   ds = chatan.dataset({
+       "topic": chatan.sample.choice(["Python", "JavaScript", "Rust"]),
+       "prompt": gen("write a programming question about {topic}"),
+       "response": gen("answer this question: {prompt}")
+   })
+
+   # Generate the data with a progress bar
+   df = ds.generate(n=10)
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

chatan-0.1.3/docs/source/other_examples.rst

@@ -0,0 +1,91 @@
+Examples
+================================================
+
+
+
+
+Dataset Triton
+----------------------------------------------
+
+.. code-block:: python
+
+   from datasets import load_dataset
+   from chatan import generator, dataset, sample
+   import chatan
+
+   gen = generator("openai", "YOUR_API_KEY")
+   kernelbook = load_dataset("GPUMODE/KernelBook")
+   kernelbench = load_dataset("ScalingIntelligence/KernelBench")
+
+   ds_1 = dataset({
+       "operation": sample.from_dataset(kernelbench, "id"),
+       "prompt": gen("write a prompt asking for a Triton kernel for: {operation}"),
+       "response": gen("{prompt}")
+   })
+
+   ds_2 = dataset({
+       "original_prompt": sample.from_dataset(kernelbook, "python_code"),
+       "prompt": gen("write a question asking for this code to be written as a Triton kernel"),
+       "response": gen("{prompt}")
+   })
+
+   df_1 = ds_1(n=500)
+   df_2 = ds_2(n=500)
+   combined_df = pd.concat([df_1, df_2], ignore_index=True)
+
+
+~~WIP~~ Ways to Create Complex Mixes
+------------------------------------------------------------
+
+.. code-block:: python
+
+   # Method 2: Create a mixed dataset with sampling
+    mixed_ds = dataset({
+        "dataset_type": sample.choice(["kernelbench", "kernelbook"]),
+        "operation": sample.from_dataset(kernelbench, "id"),
+        "original_code": sample.from_dataset(kernelbook, "python_code"),
+        "prompt": gen("""
+        {%- if dataset_type == "kernelbench" -%}
+        write a prompt asking for a Triton kernel for: {operation}
+        {%- else -%}
+        write a question asking for this code to be written as a Triton kernel: {original_code}
+        {%- endif -%}
+        """),
+        "response": gen("{prompt}")
+    })
+
+    # Method 3: Use sample.choice to pick between schemas
+    schema_choice = sample.choice([
+        {"source": "kernelbench", "operation": sample.from_dataset(kernelbench, "id")},
+        {"source": "kernelbook", "code": sample.from_dataset(kernelbook, "python_code")}
+    ])
+
+    final_ds = dataset({
+        "source": schema_choice,
+        "prompt": gen("create a Triton kernel prompt based on {source}"),
+        "response": gen("{prompt}")
+    })
+
+    # Generate the final mixed dataset
+    final_df = final_ds.generate(1000)
+    final_ds.save("triton_kernel_dataset.parquet")
+
+Transformers Local Generation
+----------------------------------------------
+
+.. code-block:: python
+
+   from chatan import generator, dataset, sample
+
+   # Use a local HuggingFace model
+   gen = generator("transformers", model="gpt2")
+
+   ds = dataset({
+       "topic": sample.choice(["space", "history", "science"]),
+       "prompt": gen("Ask a short question about {topic}"),
+       "response": gen("{prompt}")
+   })
+
+   df = ds.generate(5)
+
+

chatan-0.1.3/docs/source/quickstart.rst

@@ -0,0 +1,65 @@
+Quick Start
+===================================
+
+Installation
+------------
+
+Install chatan from PyPI:
+
+.. code-block:: bash
+
+   pip install chatan
+
+Basic Usage
+-----------
+
+1. **Create a generator**
+
+   .. code-block:: python
+
+      import chatan
+      
+      gen = chatan.generator("openai", "YOUR_OPENAI_API_KEY")
+      # or for Anthropic
+      # gen = chatan.generator("anthropic", "YOUR_ANTHROPIC_API_KEY")
+
+2. **Define your dataset schema**
+
+   .. code-block:: python
+
+      ds = chatan.dataset({
+          "prompt": gen("write a coding question about {language}"),
+          "language": chatan.sample.choice(["Python", "JavaScript", "Rust"]),
+          "response": gen("answer this question: {prompt}")
+      })
+
+3. **Generate data**
+
+   .. code-block:: python
+
+      # Generate 100 samples with a progress bar
+      df = ds.generate(100)
+      
+      # Save to file
+      ds.save("my_dataset.parquet")
+
+Core Concepts
+-------------
+
+**Generators**
+   Use LLMs to create text content. Support OpenAI and Anthropic APIs.
+
+**Samplers** 
+   Create structured data like UUIDs, choices, ranges, dates.
+
+**Schemas**
+   Define relationships between columns using generators and samplers.
+
+**Dependencies**
+   Columns can reference other columns using ``{column_name}`` syntax.
+
+Next Steps
+----------
+
+- Check out :doc:`examples` for more complex use cases
+- Browse the :doc:`api` reference for all available functions