sdg-hub 0.1.0a2.dev0__py3-none-any.whl → 0.1.0a4__py3-none-any.whl

sdg_hub-0.1.0a2.dev0.dist-info/METADATA

@@ -1,154 +0,0 @@
-Metadata-Version: 2.4
-Name: sdg_hub
-Version: 0.1.0a2.dev0
-Summary: Synthetic Data Generation
-Author-email: Red Hat AI Innovation <abhandwa@redhat.com>
-License: Apache-2.0
-Project-URL: homepage, https://ai-innovation.team/
-Project-URL: source, https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub
-Project-URL: issues, https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/issues
-Classifier: Development Status :: 3 - Alpha
-Classifier: Environment :: Console
-Classifier: License :: OSI Approved :: Apache Software License
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: MacOS :: MacOS X
-Classifier: Operating System :: POSIX :: Linux
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: Implementation :: CPython
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: click<9.0.0,>=8.1.7
-Requires-Dist: datasets<4.0.0,>=2.18.0
-Requires-Dist: httpx<1.0.0,>=0.25.0
-Requires-Dist: jinja2
-Requires-Dist: langchain-text-splitters
-Requires-Dist: openai<2.0.0,>=1.13.3
-Requires-Dist: rich
-Requires-Dist: tenacity!=8.4.0,>=8.3.0
-Requires-Dist: tqdm<5.0.0,>=4.66.2
-Dynamic: license-file
-
-# Synthetic Data Generation for LLMs
-
-The SDG Framework is a modular, scalable, and efficient solution for creating synthetic data generation workflows in a "no-code" manner. At its core, this framework is designed to simplify data creation for LLMs, allowing users to chain computational units and build powerful pipelines for generating data and processing tasks.
-
-
-
-## Core Design Principles
-
-The framework is built around the following principles:
-
-1. **Modular Design**: Highly composable blocks form the building units of the framework, allowing users to build workflows effortlessly.
-2. **No-Code Workflow Creation**: Specify workflows using simple YAML configuration files.
-3. **Scalability and Performance**: Optimized for handling large-scale workflows with millions of records.
-
----
-
-## Framework Architecture
-
-![overview](assets/imgs/overview.png)
-
-### Blocks: The Fundamental Unit
-
-At the heart of the framework is the **Block**. Each block is a self-contained computational unit that performs specific tasks, such as:
-
-- Making LLM calls
-- Performing data transformations
-- Applying filters
-
-Blocks are designed to be:
-- **Modular**: Reusable across multiple pipelines.
-- **Composable**: Easily chained together to create workflows.
-
-These blocks are implemented in the [src/sdg_hub/blocks](src/sdg_hub/blocks) directory.
-
-### Pipelines: Higher-Level Abstraction
-
-Blocks can be chained together to form a **Pipeline**. Pipelines enable:
-- Linear or recursive chaining of blocks.
-- Execution of complex workflows by chaining multiple pipelines together.
-
-### SDG Workflow: Full Workflow Automation
-
-Pipelines are further orchestrated into **SDG Workflows**, enabling seamless end-to-end processing. When invoking `sdg_hub.generate`, it triggers a pipeline/ or multiple pipelines that processes data through all the configured blocks.
-
----
-
-### YAML-Based Workflow: The Flow
-
-The YAML configuration file, known as the **Flow**, is central to defining data generation workflows in the SDG Framework. A Flow describes how blocks and pipelines are orchestrated to process and generate data efficiently. By leveraging YAML, users can create highly customizable and modular workflows without writing any code.
-
-#### Key Features of a Flow
-
-1. **Modular Design**:
-   - Flows are composed of blocks, which can be chained together into pipelines.
-   - Each block performs a specific task, such as generating, filtering, or transforming data.
-
-2. **Reusability**:
-   - Blocks and configurations defined in a Flow can be reused across different workflows.
-   - YAML makes it easy to tweak or extend workflows without significant changes.
-
-3. **Ease of Configuration**:
-   - Users can specify block types, configurations, and data processing details in a simple and intuitive manner.
-
----
-
-### Sample Flow
-
-Here is an example of a Flow configuration:
-
-```yaml
-- block_type: LLMBlock
-  block_config:
-    block_name: gen_questions
-    config_path: configs/skills/freeform_questions.yaml
-    model_id: mistralai/Mixtral-8x7B-Instruct-v0.1
-    output_cols:
-      - question
-    batch_kwargs:
-      num_samples: 30
-  drop_duplicates:
-    - question
-- block_type: FilterByValueBlock
-  block_config:
-    block_name: filter_questions
-    filter_column: score
-    filter_value: 1.0
-    operation: operator.eq
-    convert_dtype: float
-    batch_kwargs:
-      num_procs: 8
-  drop_columns:
-    - evaluation
-    - score
-    - num_samples
-- block_type: LLMBlock
-  block_config:
-    block_name: gen_responses
-    config_path: configs/skills/freeform_responses.yaml
-    model_id: mistralai/Mixtral-8x7B-Instruct-v0.1
-    output_cols:
-      - response
-```
-
-### Dataflow and Storage
-
-- **Data Representation**: Dataflow between blocks and pipelines is handled using **Hugging Face Datasets**, which are based on Arrow tables. This provides:
-  - Native parallelization capabilities (e.g., maps, filters).
-  - Support for efficient data transformations.
-
-- **Data Checkpoints**: Intermediate caches of generated data. Checkpoints allow users to:
-  - Resume workflows from the last successful state if interrupted.
-  - Improve reliability for long-running workflows.
-
----
-
-## Examples
-
-For sample use cases and implementation examples, please refer to the [examples](examples) directory. This directory contains various examples demonstrating different workflows and use cases of the SDG Framework.