liquidrandom 0.1.0__tar.gz

liquidrandom-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.venv/
+seed_generation/output/
+.pytest_cache/

liquidrandom-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

liquidrandom-0.1.0/CLAUDE.md

@@ -0,0 +1,183 @@
+# Liquidrandom
+
+This is a python package for pseudo random data generation. It is made for machine learning to seed the data generation process.
+ie. a typical use-case is using an LLM to generate data for training another model, which has the problem of a lack of randomness in the generated data. This package is designed to solve that problem by providing a way to add randomness to the data generation process by for instance adding random personas or jobs into the prompt.
+
+## Package
+
+The package is called `liquidrandom` and can be installed via pip.
+We are using typed python.
+We are using ty and uv for type checking and dependency management respectively.
+Create a pyproject.toml file for the package.
+
+The package should be usable as following:
+
+```python3
+import liquidrandom
+
+persona = liquidrandom.persona()
+job = liquidrandom.job()
+coding_task = liquidrandom.coding_task()
+math_category = liquidrandom.math_category()
+writing_style = liquidrandom.writing_style()
+scenario = liquidrandom.scenario()
+domain = liquidrandom.domain()
+science_topic = liquidrandom.science_topic()
+language = liquidrandom.language()
+reasoning_pattern = liquidrandom.reasoning_pattern()
+emotional_state = liquidrandom.emotional_state()
+instruction_complexity = liquidrandom.instruction_complexity()
+```
+
+Use types and objects for the seed data, for instance a Persona object with name, age, etc. and a Job object with name and description.
+Make sure to overwrite the __str__ method of the objects to return a string representation of the object that can be used in the prompt for the LLM.
+
+Create a clear and concise README.md file that explains the package, how to install it, and how to use it. Include examples of how to use the package in the README.md file.
+
+## Seed data
+
+The seed data we want to host on huggingface. Make sure only the needed data is fetched and that the package is not too heavy. 
+The seed data should be cached locally after the first fetch.
+
+## Seed data generation
+
+Let's create a seperate folder where we put the "run once" scripts that generate the seed data and upload it to huggingface.
+We want to use openrouter.ai for the generation of the seed data.
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+  base_url="https://openrouter.ai/api/v1",
+  api_key="<OPENROUTER_API_KEY>",
+)
+
+# First API call with reasoning
+response = client.chat.completions.create(
+  model="qwen/qwen3.5-397b-a17b",
+  messages=[
+          {
+            "role": "user",
+            "content": "How many r's are in the word 'strawberry'?"
+          }
+        ],
+  extra_body={"reasoning": {"enabled": True}}
+)
+
+# Extract the assistant message with reasoning_details
+response = response.choices[0].message
+
+# Preserve the assistant message with reasoning_details
+messages = [
+  {"role": "user", "content": "How many r's are in the word 'strawberry'?"},
+  {
+    "role": "assistant",
+    "content": response.content,
+    "reasoning_details": response.reasoning_details  # Pass back unmodified
+  },
+  {"role": "user", "content": "Are you sure? Think carefully."}
+]
+
+# Second API call - model continues reasoning from where it left off
+response2 = client.chat.completions.create(
+  model="qwen/qwen3.5-397b-a17b",
+  messages=messages,
+  extra_body={"reasoning": {"enabled": True}}
+)
+```
+
+You can assume the envvar OPENROUTER_API_KEY is set.
+For uploading to huggingface, either let me know how to do it, or you the envvar HF_TOKEN.
+
+We want to accelerate the process, thus using either async or ThreadPoolExecutor for the generation of the seed data in parallel.
+The batch size should be controlable via a cli arguemnt --batch-size
+
+Per LLM call we want to generate at least k samples, where n is controlable via a cli argument --k. Make sure to include in the prompting and the parsing of the response that there are multiple samples generated per call. 
+
+After each generation call, we want to call the same LLM again to check if the generated data is good enough. This process should check if there are collapsed or degenerate samples, e.g. halluications or repetitive samples. If the generated data is not good enough, we discard the current call and generate new data until we have good enough data. This is to ensure that the generated data is of high quality and not too similar to each other.
+
+In total we want to generate at least n samples per category, where n is controlable via a cli argument --n.
+
+## Diversity strategy: Hierarchical Taxonomy Tree
+
+To generate 10k-100k samples without collapsing into repetitive subsets, we use a two-phase hierarchical approach:
+
+### Phase 1: Taxonomy generation
+
+Before generating any seed data samples, first use the LLM to generate a deep taxonomy tree for each category. The tree should be broad at the top and increasingly specific at the leaves.
+
+Example for Science topics:
+```
+Science
+├── Physics
+│   ├── Quantum Mechanics
+│   │   ├── Entanglement phenomena
+│   │   │   ├── Bell state preparation in ion traps
+│   │   │   ├── Quantum teleportation protocols
+│   │   │   └── Loophole-free Bell tests
+│   │   ├── Decoherence
+│   │   │   ├── Environment-induced superselection
+│   │   │   └── ...
+│   ├── Thermodynamics
+│   │   └── ...
+├── Biology
+│   └── ...
+```
+
+The taxonomy depth should auto-scale based on the target sample count:
+- `required_leaf_nodes = target_samples / samples_per_leaf`
+- More samples → deeper tree → more leaf nodes
+- The taxonomy itself should be generated in stages: first top-level branches, then expand each branch deeper, to avoid context window limitations.
+
+The taxonomy generation should also be controllable via a cli argument `--taxonomy-depth` to control the depth of the taxonomy tree and `--samples-per-leaf` to control how many samples to generate per leaf node.
+
+The taxonomy should be saved to disk as JSON so it can be inspected, reused, and resumed.
+
+### Phase 2: Round-robin sample generation
+
+Generate samples by cycling through all leaf nodes in round-robin fashion:
+
+```
+for leaf in cycle(all_leaf_nodes):
+    generate k samples for this specific leaf
+    validate & dedup
+    if leaf.count >= target_per_leaf:
+        mark leaf as done
+```
+
+This ensures that no single subtopic gets over-represented. Each generation prompt should include the leaf node's full path in the taxonomy (e.g. "Science > Physics > Quantum Mechanics > Entanglement phenomena > Bell state preparation in ion traps") to anchor the LLM to that specific subtopic.
+
+The previously generated samples for the current leaf should be included in the prompt to avoid repetition within the same leaf.
+
+### Deduplication: Fuzzy string matching
+
+Use token-level fuzzy string matching to detect and reject near-duplicate samples. No ML dependency needed.
+
+Approach:
+- Use Jaccard similarity on token sets (word-level) to compare new samples against all existing samples in the same category.
+- Reject samples with a Jaccard similarity above a configurable threshold (default: 0.7).
+- Additionally, normalize samples before comparison (lowercase, strip punctuation, collapse whitespace) to catch trivial reformulations.
+- The dedup check should run on the `__str__` representation of each sample.
+- The threshold should be controllable via a cli argument `--dedup-threshold`.
+
+## Categories of seed data
+
+- Persona: random personas with name, age etc.
+- Professions or jobs: Random professions or jobs with a name and description of the job.
+- Coding tasks: Specific programming challenges and tasks (e.g. "implement a trie-based autocomplete with fuzzy matching", "write a rate limiter using the token bucket algorithm"). Should include the programming context, constraints, and expected behavior.
+- Math categories: Random math categories like algebra, geometry, etc. with a description of the category.
+- Writing styles / tones: Diverse writing styles and tones (e.g. "sardonic academic critique", "enthusiastic technical blogger", "dry legal prose"). Includes the style name and a description of its characteristics.
+- Scenarios / situations: Real-world scenarios or situations (e.g. "debugging a production outage at 2am", "negotiating a contract with a difficult vendor"). Includes the scenario description and relevant context.
+- Domains / topics: Specific knowledge domains and subtopics (e.g. "supply chain logistics for perishable goods", "comparative analysis of NoSQL databases"). Includes the domain name and a detailed description.
+- Science topics: Specific scientific topics and phenomena (e.g. "quantum entanglement in photon pairs", "CRISPR-Cas9 off-target effects in gene therapy", "tidal locking in exoplanetary systems"). Includes the topic name, scientific field, and a description.
+- Languages / locales: Random languages, dialects, or cultural contexts (e.g. "Brazilian Portuguese, informal register", "Kansai Japanese dialect"). Includes the language, region, register, and cultural notes.
+- Reasoning patterns: Types of reasoning or problem-solving approaches (e.g. "proof by contradiction", "cost-benefit analysis with uncertainty", "analogical reasoning from biology to engineering"). Includes the pattern name and description of how it works.
+- Emotional states: Specific emotional states or moods (e.g. "frustrated but trying to stay polite", "cautiously optimistic after a setback"). Includes the emotional state and behavioral description.
+- Instruction complexity: Varying levels of instruction complexity and ambiguity (e.g. "vague one-liner request", "detailed multi-step specification with constraints", "contradictory requirements"). Includes the complexity level, a description, and an example.
+
+Make sure the seed data is very specific, ie. "algebra" is not a good seed data, but "different ways to solve linear equations" is a good seed data. This is to ensure that the generated data for training is more diverse and not too similar to each other.
+
+For the generation process, use the rich python library to show progress as well as the expected time of arrival (ETA) for the generation process.
+
+dependencies and README for the run once seed generation scripts should be in the same folder as the scripts and different from the main package. The main package should not have any dependencies that are not needed for the generation of the seed data.
+

liquidrandom-0.1.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: liquidrandom
+Version: 0.1.0
+Summary: Pseudo-random seed data generation for ML/LLM training diversity
+License-Expression: MIT
+Requires-Python: >=3.11
+Requires-Dist: huggingface-hub>=0.20.0
+Requires-Dist: pyarrow>=14.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ty; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# liquidrandom
+
+Pseudo-random seed data for ML/LLM training diversity.
+
+When using LLMs to generate training data, outputs tend to be repetitive and lack variety. `liquidrandom` solves this by providing a large pool of diverse, pre-generated seed data (personas, jobs, scenarios, etc.) that you can inject into your prompts to steer generation toward more varied outputs.
+
+## Installation
+
+```bash
+pip install liquidrandom
+# or
+uv add liquidrandom
+```
+
+## Quick Start
+
+```python
+import liquidrandom
+
+# Get a random persona to inject into your LLM prompt
+persona = liquidrandom.persona()
+print(persona)
+# Alice is a 30-year-old female from Canada. They work as an engineer. ...
+
+# Get a random coding task
+task = liquidrandom.coding_task()
+print(task)
+# [Python, medium] Implement a trie: Build a trie data structure ...
+```
+
+## Available Categories
+
+| Function | Returns | Description |
+|---|---|---|
+| `liquidrandom.persona()` | `Persona` | Random personas with name, age, gender, occupation, nationality, personality traits, background |
+| `liquidrandom.job()` | `Job` | Professions with title, industry, description, required skills, experience level |
+| `liquidrandom.coding_task()` | `CodingTask` | Programming challenges with title, language, difficulty, description, constraints, expected behavior |
+| `liquidrandom.math_category()` | `MathCategory` | Math categories with name, field, description, example problems |
+| `liquidrandom.writing_style()` | `WritingStyle` | Writing styles with name, tone, characteristics, description |
+| `liquidrandom.scenario()` | `Scenario` | Real-world scenarios with title, context, setting, stakes, description |
+| `liquidrandom.domain()` | `Domain` | Knowledge domains with name, parent field, description, key concepts |
+| `liquidrandom.science_topic()` | `ScienceTopic` | Scientific topics with name, field, subfield, description |
+| `liquidrandom.language()` | `Language` | Languages/locales with name, region, register, script, cultural notes |
+| `liquidrandom.reasoning_pattern()` | `ReasoningPattern` | Reasoning approaches with name, category, description, when to use |
+| `liquidrandom.emotional_state()` | `EmotionalState` | Emotional states with name, intensity, valence, behavioral description |
+| `liquidrandom.instruction_complexity()` | `InstructionComplexity` | Instruction complexity levels with level, ambiguity, description, example |
+
+## Usage Example
+
+Use `liquidrandom` to add diversity to your LLM data generation pipeline:
+
+```python
+import liquidrandom
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="https://openrouter.ai/api/v1",
+    api_key="<OPENROUTER_API_KEY>",
+)
+
+persona = liquidrandom.persona()
+style = liquidrandom.writing_style()
+topic = liquidrandom.science_topic()
+
+prompt = f"""You are {persona}
+Write in the following style: {style}
+Explain the following topic: {topic}"""
+
+response = client.chat.completions.create(
+    model="liquid/lfm-2-24b-a2b",
+    messages=[{"role": "user", "content": prompt}],
+)
+```
+
+Each call to a `liquidrandom` function returns a typed dataclass. You can use them directly in f-strings (via `__str__`) or access their individual fields:
+
+```python
+persona = liquidrandom.persona()
+print(persona.name)               # "Alice"
+print(persona.age)                 # 30
+print(persona.personality_traits)  # ["curious", "patient"]
+```
+
+## How It Works
+
+The dataset contains 340,000+ samples across 12 categories, generated using hierarchical taxonomy trees with LLM-based quality validation and fuzzy deduplication.
+
+Seed data is hosted on HuggingFace ([mlech26l/liquidrandom-data](https://huggingface.co/datasets/mlech26l/liquidrandom-data)) as zstd-compressed Parquet files. On first use, only the requested category file is downloaded and cached locally. Subsequent calls use the cached data.
+
+## License
+
+MIT

liquidrandom-0.1.0/README.md

@@ -0,0 +1,92 @@
+# liquidrandom
+
+Pseudo-random seed data for ML/LLM training diversity.
+
+When using LLMs to generate training data, outputs tend to be repetitive and lack variety. `liquidrandom` solves this by providing a large pool of diverse, pre-generated seed data (personas, jobs, scenarios, etc.) that you can inject into your prompts to steer generation toward more varied outputs.
+
+## Installation
+
+```bash
+pip install liquidrandom
+# or
+uv add liquidrandom
+```
+
+## Quick Start
+
+```python
+import liquidrandom
+
+# Get a random persona to inject into your LLM prompt
+persona = liquidrandom.persona()
+print(persona)
+# Alice is a 30-year-old female from Canada. They work as an engineer. ...
+
+# Get a random coding task
+task = liquidrandom.coding_task()
+print(task)
+# [Python, medium] Implement a trie: Build a trie data structure ...
+```
+
+## Available Categories
+
+| Function | Returns | Description |
+|---|---|---|
+| `liquidrandom.persona()` | `Persona` | Random personas with name, age, gender, occupation, nationality, personality traits, background |
+| `liquidrandom.job()` | `Job` | Professions with title, industry, description, required skills, experience level |
+| `liquidrandom.coding_task()` | `CodingTask` | Programming challenges with title, language, difficulty, description, constraints, expected behavior |
+| `liquidrandom.math_category()` | `MathCategory` | Math categories with name, field, description, example problems |
+| `liquidrandom.writing_style()` | `WritingStyle` | Writing styles with name, tone, characteristics, description |
+| `liquidrandom.scenario()` | `Scenario` | Real-world scenarios with title, context, setting, stakes, description |
+| `liquidrandom.domain()` | `Domain` | Knowledge domains with name, parent field, description, key concepts |
+| `liquidrandom.science_topic()` | `ScienceTopic` | Scientific topics with name, field, subfield, description |
+| `liquidrandom.language()` | `Language` | Languages/locales with name, region, register, script, cultural notes |
+| `liquidrandom.reasoning_pattern()` | `ReasoningPattern` | Reasoning approaches with name, category, description, when to use |
+| `liquidrandom.emotional_state()` | `EmotionalState` | Emotional states with name, intensity, valence, behavioral description |
+| `liquidrandom.instruction_complexity()` | `InstructionComplexity` | Instruction complexity levels with level, ambiguity, description, example |
+
+## Usage Example
+
+Use `liquidrandom` to add diversity to your LLM data generation pipeline:
+
+```python
+import liquidrandom
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="https://openrouter.ai/api/v1",
+    api_key="<OPENROUTER_API_KEY>",
+)
+
+persona = liquidrandom.persona()
+style = liquidrandom.writing_style()
+topic = liquidrandom.science_topic()
+
+prompt = f"""You are {persona}
+Write in the following style: {style}
+Explain the following topic: {topic}"""
+
+response = client.chat.completions.create(
+    model="liquid/lfm-2-24b-a2b",
+    messages=[{"role": "user", "content": prompt}],
+)
+```
+
+Each call to a `liquidrandom` function returns a typed dataclass. You can use them directly in f-strings (via `__str__`) or access their individual fields:
+
+```python
+persona = liquidrandom.persona()
+print(persona.name)               # "Alice"
+print(persona.age)                 # 30
+print(persona.personality_traits)  # ["curious", "patient"]
+```
+
+## How It Works
+
+The dataset contains 340,000+ samples across 12 categories, generated using hierarchical taxonomy trees with LLM-based quality validation and fuzzy deduplication.
+
+Seed data is hosted on HuggingFace ([mlech26l/liquidrandom-data](https://huggingface.co/datasets/mlech26l/liquidrandom-data)) as zstd-compressed Parquet files. On first use, only the requested category file is downloaded and cached locally. Subsequent calls use the cached data.
+
+## License
+
+MIT

liquidrandom-0.1.0/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "liquidrandom"
+version = "0.1.0"
+description = "Pseudo-random seed data generation for ML/LLM training diversity"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+dependencies = [
+    "huggingface-hub>=0.20.0",
+    "pyarrow>=14.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "ty",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/liquidrandom"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+

liquidrandom-0.1.0/seed_generation/README.md

@@ -0,0 +1,113 @@
+# Seed Data Generation
+
+Scripts for generating diverse seed data for the `liquidrandom` package.
+
+## Setup
+
+```bash
+cd seed_generation
+uv sync
+```
+
+## Environment Variables
+
+| Variable | Required | Description |
+|---|---|---|
+| `OPENROUTER_API_KEY` | Yes (for generation) | OpenRouter API key |
+| `HF_TOKEN` | Yes (for upload) | HuggingFace write token |
+
+## Usage
+
+### Generate seed data
+
+```bash
+# Generate 1000 samples per category for all categories
+python generate.py generate --n 1000 --k 10
+
+# Generate for specific categories only
+python generate.py generate --n 1000 --k 10 --categories persona --categories job
+
+# Full control over all parameters
+python generate.py generate \
+    --n 10000 \
+    --k 10 \
+    --batch-size 5 \
+    --taxonomy-depth 4 \
+    --samples-per-leaf 10 \
+    --dedup-threshold 0.7 \
+    --output-dir ./output
+```
+
+### Resume interrupted generation
+
+```bash
+python generate.py generate --resume --output-dir ./output
+```
+
+### Upload to HuggingFace
+
+```bash
+python generate.py upload-only --output-dir ./output --repo-id mlech26l/liquidrandom-data
+```
+
+## CLI Arguments
+
+| Argument | Default | Description |
+|---|---|---|
+| `--n` | 1000 | Target samples per category |
+| `--k` | 10 | Samples generated per LLM call |
+| `--batch-size` | 5 | Number of concurrent LLM calls |
+| `--taxonomy-depth` | 4 | Maximum depth of the taxonomy tree |
+| `--samples-per-leaf` | 10 | Target samples per leaf node |
+| `--dedup-threshold` | 0.7 | Jaccard similarity threshold for dedup |
+| `--categories` | all | Which categories to generate |
+| `--resume` | false | Resume from checkpoint |
+| `--output-dir` | output | Output directory |
+
+## How It Works
+
+### Phase 1: Taxonomy Generation
+
+For each category, the LLM generates a deep taxonomy tree to ensure diversity. The tree is expanded breadth-first, level by level, until there are enough leaf nodes.
+
+Taxonomies are saved as JSON in `output/taxonomies/` and can be inspected or reused.
+
+### Phase 2: Round-Robin Sample Generation
+
+Samples are generated by cycling through all leaf nodes in round-robin fashion. Each generation prompt includes:
+- The leaf node's full taxonomy path (to anchor specificity)
+- Previously generated samples for that leaf (to avoid repetition)
+
+### Quality Validation
+
+Each batch is validated by a second LLM call that checks for:
+- Empty or placeholder content
+- Hallucinations or factual impossibility
+- Intra-batch repetitiveness
+- Off-topic samples
+- Insufficient specificity
+
+If >50% of a batch is rejected, the entire batch is discarded and regenerated.
+
+### Deduplication
+
+Token-level Jaccard similarity is used to detect near-duplicates:
+- Text is normalized (lowercase, strip punctuation, collapse whitespace)
+- Word-level token sets are compared
+- Samples above the similarity threshold are rejected
+
+## Output Structure
+
+```
+output/
+├── taxonomies/           # JSON taxonomy trees per category
+│   ├── persona.json
+│   ├── job.json
+│   └── ...
+├── samples/              # Per-leaf JSONL files
+│   ├── persona/
+│   │   ├── Personas_Young-Adults_...jsonl
+│   │   └── ...
+│   └── ...
+└── state.json            # Checkpoint for resumability
+```