sdg-hub 0.1.4__py3-none-any.whl → 0.2.1__py3-none-any.whl

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/detailed_summary.yaml

@@ -0,0 +1,13 @@
+- role: system
+  content: You are an AI assistant that is expert at summarizing text.
+
+- role: user
+  content: |
+    Give me detailed summary for below document, making sure all key points are covered.
+
+    Do not add any new information.
+    Do not miss any key points from the provided document
+
+    Document:
+    {{document_outline}}
+    {{document}}

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/evaluate_faithfulness.yaml

@@ -0,0 +1,64 @@
+- role: system
+  content: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+
+- role: user
+  content: |
+    Determine if the provided information is corroborated by the given context. Respond with YES if the context substantiates the information, even partially. Answer NO if the context does not support the information.
+
+    Guidelines:
+    - Answer YES when the context provides either direct or indirect evidence supporting the information. Indirect evidence may include contextual implications or inferred connections that reasonably support the information.
+    - Answer NO if the context lacks any supportive evidence, clearly contradicts the information, or if the support provided by the context is too vague or speculative to establish a solid connection to the information.
+    - Avoid using "partially" in your response. If the context provides any reasonable support (direct or indirect) for the information, consider it as a YES.
+
+    Strictly answer in this format:
+    [Start of Context]
+    ...
+    [End of Context]
+    [Start of Response]
+    ... 
+    [End of Response]
+    [Start of Explanation]
+    ...
+    [End of Explanation]
+    [Start of Answer]
+    ...
+    [End of Answer]
+
+    Example 1:
+    [Start of Context]
+    An apple pie is a fruit pie with apples as the main filling. It's often served with whipped cream, ice cream, custard, or cheddar cheese. Typically, it has a double crust, with pastry above and below the filling. The upper crust can be solid or latticed.
+    [End of Context]
+    [Start of Response]
+    Apple pie is generally double-crusted.
+    [End of Response]
+    [Start of Explanation]
+    The context directly supports the information by stating that apple pie is "generally double-crusted," which matches the information provided.
+    [End of Explanation]
+    [Start of Answer]
+    YES
+    [End of Answer]
+
+    Example 2:
+    [Start of Context]
+    An apple pie is a fruit pie with apples as the main filling. It's often served with whipped cream, ice cream, custard, or cheddar cheese. Typically, it has a double crust, with pastry above and below the filling. The upper crust can be solid or latticed.
+    [End of Context]
+    [Start of Response]
+    Apple pies taste bad.
+    [End of Response]
+    [Start of Explanation]
+    The context does not provide any information about the taste of apple pies. The statement "Apple pies taste bad" is a subjective opinion and is not supported or mentioned in the given context.
+    [End of Explanation]
+    [Start of Answer]
+    NO
+    [End of Answer]
+
+    Now, based on the above examples and guidelines, determine if the following information is supported by the context provided. Answer YES or NO.
+    * Return the explanation within the [Start of Explanation] and [End of Explanation] tags.
+    * Return the answer between [Start of Answer] and [End of Answer] tags.
+
+    [Start of Context]
+    {{document}}
+    [End of Context]
+    [Start of Response]
+    {{response}}
+    [End of Response]

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/evaluate_question.yaml

@@ -0,0 +1,29 @@
+- role: system
+  content: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+
+- role: user
+  content: |
+    Given below question can you verify if it meets below requirements and based on them give a rating of 1 if it meets all of them or 0 otherwise.
+
+    Here are the requirements:
+
+    Non-Referential Clarity and Contextual Independence: Ensure that the question is self-explanatory and does not rely on specific, unprovided external content, such as particular documents, specific tables, or detailed datasets. The question should be structured to be understandable and clear without requiring direct access to or knowledge of these specific external sources.
+
+    Subject-Aware Completeness: The question should be crafted to be answerable on its own, given a reasonable level of specialized knowledge in the relevant subject area. It is acceptable and encouraged for the question to require specialized understanding pertinent to the topic; however, it should not depend on unique, external information not provided in the question itself. This distinction allows for questions that necessitate a deep understanding of a subject while ensuring they are not tied to specific external content like a particular dataset or a line in a document.
+
+    Please give your answer as short explanation followed by rating of either 0 or 1 as below.
+
+    * Return a short explanation within the [Start of Explanation] and [End of Explanation] tags.
+    * Return the rating on a binary 0/1 scale between [Start of Rating] and [End of Rating] tags.
+
+    [Start of Question]  
+    {{question}}
+    [End of Question]
+
+    [Start of Explanation]
+    ...  
+    [End of Explanation]
+
+    [Start of Rating]
+    ...
+    [End of Rating]

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/evaluate_relevancy.yaml

@@ -0,0 +1,81 @@
+- role: system
+  content: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+
+- role: user
+  content: |
+    Your task is to assess the relevance of a given response to a specific query. This evaluation should be conducted methodically by answering two key questions:
+
+    1. Subject Matter Relevance: Does the provided response accurately match the subject matter of the user's query? This question aims to determine if the response is directly related to the main topic or issue presented in the query.
+    2. Focus and Perspective Addressing: Does the provided response effectively address the focus or perspective on the subject matter as outlined in the user's query? This question seeks to evaluate whether the response not only matches the subject matter but also aligns with the specific angle or concern raised by the user.
+
+    For each question, assign a score of 1 point if the response meets the criteria, and 0 points if it does not. After evaluating each question, provide detailed feedback explaining your reasoning behind the scores awarded.
+
+    Conclude your evaluation with a total score as a final result. The total score should represent the sum of points assigned for each question, with a maximum possible score of 2 points.
+    Only evaluate the response based on the above criteria, do not create new questions.
+
+    Example 1:
+    [Start of Question]
+    What is the impact of global warming on polar bears?
+    [End of Question]
+
+    [Start of Response]
+    Global warming leads to melting ice caps, reducing the habitat of polar bears and negatively impacting their hunting grounds.
+    [End of Response]
+
+    [Start of Feedback]
+    - Subject Matter Relevance Score: 1 (The response is directly related to the impact of global warming on polar bears.)
+    - Alignment with Query's Focus Score: 1 (The response specifically addresses how global warming affects polar bears' habitat and hunting grounds.)
+    [End of Feedback]
+
+    [Start of Score]
+    2
+    [End of Score]
+
+    Example 2:
+    [Start of Question]
+    How does photosynthesis work?
+    [End of Question]
+
+    [Start of Response]
+    Plants require sunlight and water to grow.
+    [End of Response]
+
+    [Start of Feedback]
+    - Subject Matter Relevance Score: 0 (The response is related to plant growth, but does not specifically address the process of photosynthesis.)
+    - Alignment with Query's Focus Score: 0 (The response fails to detail the photosynthesis process, missing the specific focus of the query.)
+    [End of Feedback]
+
+    [Start of Score]
+    0
+    [End of Score]
+
+    Example 3:
+    [Start of Question]
+    What are the benefits of electric vehicles?
+    [End of Question]
+
+    [Start of Response]
+    Electric vehicles reduce dependency on fossil fuels and decrease greenhouse gas emissions.
+    [End of Response]
+
+    [Start of Feedback]
+    - Subject Matter Relevance Score: 1 (The response matches the query's subject on the benefits of electric vehicles.)
+    - Alignment with Query's Focus Score: 1 (The response effectively addresses the environmental benefits of electric vehicles, aligning with the query's focus.)
+    [End of Feedback]
+
+    [Start of Score]
+    2
+    [End of Score]
+
+    Begin your response by providing the feedback followed by the score. Be as objective as possible.
+
+    [Start of Question]
+    {{question}}
+    [End of Question]
+
+    [Start of Response]
+    {{response}}
+    [End of Response]
+
+    * Return the feedback within the [Start of Feedback] and [End of Feedback] tags.
+    * Return the final score between [Start of Score] and [End of Score] tags.

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/extractive_summary.yaml

@@ -0,0 +1,13 @@
+- role: system
+  content: You are an AI assistant that is expert at summarizing text.
+
+- role: user
+  content: |
+    Give me detailed extractive summary for below document, making sure all key points are covered.
+
+    Do not add any new information.
+    Do not miss any key points from the provided document
+
+    Document:
+    {{document_outline}}
+    {{document}}

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/flow.yaml

@@ -0,0 +1,192 @@
+metadata:
+  id: small-rock-799
+  name: "Advanced Document Grounded Question-Answer Generation Flow for Knowledge Tuning"
+  description: "A comprehensive flow that generates high-quality question-answer pairs from input documents using multiple LLM blocks for question generation, answer synthesis, and quality evaluation."
+  version: "1.0.0"
+  author: "SDG Hub Contributors"
+  
+  recommended_models:
+    default: "meta-llama/Llama-3.3-70B-Instruct"
+    compatible: ["microsoft/phi-4", "mistralai/Mixtral-8x7B-Instruct-v0.1"]
+    experimental: []
+  
+  tags:
+    - "question-generation"
+    - "knowledge-extraction"
+    - "qa-pairs"
+    - "document-processing"
+    - "educational"
+  
+  license: "Apache-2.0"
+  min_sdg_hub_version: "0.2.0"
+  
+  dataset_requirements:
+    required_columns:
+      - "document"
+      - "document_outline"
+      - "domain"
+      - "icl_document"
+      - "icl_query_1"
+      - "icl_response_1"
+      - "icl_query_2"
+      - "icl_response_2"
+      - "icl_query_3"
+      - "icl_response_3"
+    description: "Input dataset should contain documents with text content and domain classification. Each document should be substantial enough for meaningful question generation (minimum 100 words recommended)."
+
+blocks:
+  - block_type: DuplicateColumnsBlock
+    block_config:
+      block_name: duplicate_document_col
+      input_cols: {document: base_document}
+
+  - block_type: PromptBuilderBlock
+    block_config:
+      block_name: detailed_summary_prompt
+      input_cols: [document, document_outline]
+      output_cols: summary_prompt
+      prompt_config_path: detailed_summary.yaml
+      format_as_messages: true
+
+  - block_type: LLMChatBlock
+    block_config:
+      block_name: gen_detailed_summary
+      input_cols: summary_prompt
+      output_cols: raw_summary_detailed
+      max_tokens: 2048
+      async_mode: true
+
+  - block_type: TextParserBlock
+    block_config:
+      block_name: parse_detailed_summary
+      input_cols: raw_summary_detailed
+      output_cols: summary_detailed
+      start_tags: [""]
+      end_tags: [""]
+
+  - block_type: PromptBuilderBlock
+    block_config:
+      block_name: atomic_facts_prompt
+      input_cols: [document, document_outline, domain]
+      output_cols: atomic_facts_prompt
+      prompt_config_path: atomic_facts.yaml
+      format_as_messages: true
+
+  - block_type: LLMChatBlock
+    block_config:
+      block_name: gen_atomic_facts
+      input_cols: atomic_facts_prompt
+      output_cols: raw_atomic_facts
+      max_tokens: 2048
+      async_mode: true
+
+  - block_type: TextParserBlock
+    block_config:
+      block_name: parse_atomic_facts
+      input_cols: raw_atomic_facts
+      output_cols: summary_atomic_facts
+      start_tags: [""]
+      end_tags: [""]
+
+  - block_type: PromptBuilderBlock
+    block_config:
+      block_name: extractive_summary_prompt
+      input_cols: [document, document_outline]
+      output_cols: extractive_summary_prompt
+      prompt_config_path: extractive_summary.yaml
+      format_as_messages: true
+
+  - block_type: LLMChatBlock
+    block_config:
+      block_name: gen_extractive_summary
+      input_cols: extractive_summary_prompt
+      output_cols: raw_summary_extractive
+      max_tokens: 2048
+      async_mode: true
+
+  - block_type: TextParserBlock
+    block_config:
+      block_name: parse_extractive_summary
+      input_cols: raw_summary_extractive
+      output_cols: summary_extractive
+      start_tags: [""]
+      end_tags: [""]
+
+  - block_type: MeltColumnsBlock
+    block_config:
+      block_name: melt_summary_columns
+      input_cols: [summary_detailed, summary_extractive, summary_atomic_facts, base_document]
+      output_cols: [summary, dataset_type]
+
+  - block_type: RenameColumnsBlock
+    block_config:
+      block_name: rename_to_document_column
+      input_cols: {document: raw_document, summary: document}
+
+  - block_type: PromptBuilderBlock
+    block_config:
+      block_name: knowledge_generation_prompt
+      input_cols: [domain, document, document_outline, icl_document, icl_query_1, icl_response_1, icl_query_2, icl_response_2, icl_query_3, icl_response_3]
+      output_cols: knowledge_generation_prompt
+      prompt_config_path: generate_questions_responses.yaml
+      format_as_messages: true
+
+  - block_type: LLMChatBlock
+    block_config:
+      block_name: knowledge_generation
+      input_cols: knowledge_generation_prompt
+      output_cols: raw_knowledge_generation
+      temperature: 0.0
+      max_tokens: 2048
+      async_mode: true
+
+  - block_type: TextParserBlock
+    block_config:
+      block_name: parse_knowledge_generation
+      input_cols: raw_knowledge_generation
+      output_cols: [question, response]
+      parsing_pattern: "\\[(?:Question|QUESTION)\\]\\s*(.*?)\\s*\\[(?:Answer|ANSWER)\\]\\s*(.*?)\\s*(?=\\[(?:Question|QUESTION)\\]|$)"
+      parser_cleanup_tags: ["[END]"]
+
+  - block_type: EvaluateFaithfulnessBlock
+    block_config:
+      block_name: eval_faithfulness
+      input_cols: [document, response]
+      output_cols: [faithfulness_explanation, faithfulness_judgment]
+      prompt_config_path: evaluate_faithfulness.yaml
+      filter_value: "YES"
+      operation: eq
+      async_mode: true
+      format_as_messages: true
+      start_tags: ["[Start of Explanation]", "[Start of Answer]"]
+      end_tags: ["[End of Explanation]", "[End of Answer]"]
+
+  - block_type: EvaluateRelevancyBlock
+    block_config:
+      block_name: eval_relevancy
+      input_cols: [question, response]
+      output_cols: [relevancy_explanation, relevancy_score]
+      prompt_config_path: evaluate_relevancy.yaml
+      filter_value: 2.0
+      operation: eq
+      convert_dtype: float
+      max_tokens: 2048
+      async_mode: true
+      format_as_messages: true
+      start_tags: ["[Start of Feedback]", "[Start of Score]"]
+      end_tags: ["[End of Feedback]", "[End of Score]"]
+
+  - block_type: VerifyQuestionBlock
+    block_config:
+      block_name: verify_question
+      input_cols: [question]
+      output_cols: [verification_explanation, verification_rating]
+      prompt_config_path: evaluate_question.yaml
+      filter_value: 1.0
+      operation: ge
+      convert_dtype: float
+      max_tokens: 2048
+      async_mode: true
+      format_as_messages: true
+      start_tags: ["[Start of Explanation]", "[Start of Rating]"]
+      end_tags: ["[End of Explanation]", "[End of Rating]"]

sdg_hub/flows/qa_generation/document_grounded_qa/multi_summary_qa/instructlab/generate_questions_responses.yaml

@@ -0,0 +1,54 @@
+- role: system
+  content: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+
+- role: user
+  content: |
+    Develop a series of educational question and answer pairs from a chapter in a {{domain}} textbook.
+
+    The questions should:
+    * Be self-contained, not requiring references to tables, figures, or specific sections in the text for understanding.
+    * Focus on teaching and reinforcing the key knowledge and concepts presented in the chapter.
+    * Avoid sections with minimal educational content like index pages or prefaces. In such cases, respond with [UNANSWERABLE].
+    * Be directly relevant to the textbook's domain. For instance, in a science textbook, questions should revolve around scientific terms, definitions, and practical applications, while in a legal textbook, they should cover legal principles, case law, and precedents.
+    * Be formulated to allow for independent answers, avoiding direct references to specific theorems or text sections. For example, rather than asking 'Under what conditions is the fixed point of a function unique according to Theorem 3.1.5?', ask 'How does the Fixed Point Iteration method contribute to understanding function uniqueness?'
+    * Span a range of difficulty levels to accommodate a diverse student audience, from basic understanding to advanced comprehension.
+    * Include a variety of question types such as multiple-choice for basic recall, short answer for deeper understanding, and essay or problem-solving questions to test application and analysis skills.
+    * Align closely with the learning objectives of the textbook or the specific chapter, ensuring that the questions test the fundamental concepts and skills that the chapter aims to impart.
+
+    Strictly follow this format for each question answer pair your generate while responding:
+
+    [QUESTION]
+    <Insert question here>
+    [ANSWER]
+    <Insert answer here>
+    [END]
+
+    Each question and answer pair should stand alone as a mini-lesson, encapsulating a key concept or idea from the chapter in a way that is accessible and informative without requiring the reader to refer back to the textbook.
+
+    Here are some examples of questions:
+
+    [Document]
+    {{icl_document}}
+
+    [QUESTION]
+    {{icl_query_1}}
+    [ANSWER]
+    {{icl_response_1}}
+    [END]
+
+    [QUESTION]
+    {{icl_query_2}}
+    [ANSWER]
+    {{icl_response_2}}
+    [END]
+
+    [QUESTION]
+    {{icl_query_3}}
+    [ANSWER]
+    {{icl_response_3}}
+    [END]
+
+    Now, here is the document:
+    [DOCUMENT]
+    {{document_outline}}
+    {{document}}

sdg_hub-0.2.1.dist-info/METADATA

@@ -0,0 +1,221 @@
+Metadata-Version: 2.4
+Name: sdg_hub
+Version: 0.2.1
+Summary: Synthetic Data Generation
+Author-email: Red Hat AI Innovation <abhandwa@redhat.com>
+License: Apache-2.0
+Project-URL: homepage, https://ai-innovation.team/
+Project-URL: source, https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub
+Project-URL: issues, https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/issues
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click<9.0.0,>=8.1.7
+Requires-Dist: datasets<4.0.0,>=2.18.0
+Requires-Dist: httpx<1.0.0,>=0.25.0
+Requires-Dist: jinja2
+Requires-Dist: litellm<1.75.0,>=1.73.0
+Requires-Dist: openai<2.0.0,>=1.13.3
+Requires-Dist: rich
+Requires-Dist: pydantic<3.0.0,>=2.0.0
+Requires-Dist: python-dotenv<2.0.0,>=1.0.0
+Requires-Dist: tenacity!=8.4.0,>=8.3.0
+Requires-Dist: tqdm<5.0.0,>=4.66.2
+Provides-Extra: vllm
+Requires-Dist: vllm>=0.9.1; extra == "vllm"
+Requires-Dist: torch>=2.0.0; extra == "vllm"
+Requires-Dist: transformers>=4.37.0; extra == "vllm"
+Requires-Dist: accelerate>=0.21.0; extra == "vllm"
+Requires-Dist: xformers>=0.0.22.post7; extra == "vllm"
+Provides-Extra: examples
+Requires-Dist: tabulate>=0.9.0; extra == "examples"
+Requires-Dist: transformers>=4.37.0; extra == "examples"
+Requires-Dist: langchain-text-splitters; extra == "examples"
+Requires-Dist: docling>=2.3.0; extra == "examples"
+Requires-Dist: scikit-learn; extra == "examples"
+Requires-Dist: pandas; extra == "examples"
+Requires-Dist: polars; extra == "examples"
+Requires-Dist: matplotlib; extra == "examples"
+Requires-Dist: spacy; extra == "examples"
+Requires-Dist: nltk; extra == "examples"
+Requires-Dist: sentence-transformers; extra == "examples"
+Requires-Dist: instructor; extra == "examples"
+Requires-Dist: fastapi; extra == "examples"
+Requires-Dist: nest-asyncio; extra == "examples"
+Provides-Extra: dev
+Requires-Dist: pre-commit<4.0,>=3.0.4; extra == "dev"
+Requires-Dist: pylint<4.0,>=2.16.2; extra == "dev"
+Requires-Dist: pylint-pydantic; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-html; extra == "dev"
+Requires-Dist: tox<5,>=4.4.2; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Dynamic: license-file
+
+# `sdg_hub`: Synthetic Data Generation Toolkit
+
+[![Build](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/actions/workflows/pypi.yaml/badge.svg?branch=main)](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/actions/workflows/pypi.yaml)
+[![Release](https://img.shields.io/github/v/release/Red-Hat-AI-Innovation-Team/sdg_hub)](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/releases)
+[![License](https://img.shields.io/github/license/Red-Hat-AI-Innovation-Team/sdg_hub)](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/blob/main/LICENSE)
+[![Tests](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/actions/workflows/test.yml/badge.svg)](https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub/actions/workflows/test.yml)
+[![codecov](https://codecov.io/gh/Red-Hat-AI-Innovation-Team/sdg_hub/graph/badge.svg?token=SP75BCXWO2)](https://codecov.io/gh/Red-Hat-AI-Innovation-Team/sdg_hub)
+
+
+
+A modular Python framework for building synthetic data generation pipelines using composable blocks and flows. Transform datasets through **building-block composition** - mix and match LLM-powered and traditional processing blocks to create sophisticated data generation workflows.
+
+**📖 Full documentation available at: [https://ai-innovation.team/sdg_hub](https://ai-innovation.team/sdg_hub)**
+
+## ✨ Key Features
+
+**🔧 Modular Composability** - Mix and match blocks like Lego pieces. Build simple transformations or complex multi-stage pipelines with YAML-configured flows.
+
+**⚡ Async Performance** - High-throughput LLM processing with built-in error handling.
+
+**🛡️ Built-in Validation** - Pydantic-based type safety ensures your configurations and data are correct before execution.
+
+**🔍 Auto-Discovery** - Automatic block and flow registration. No manual imports or complex setup.
+
+**📊 Rich Monitoring** - Detailed logging with progress bars and execution summaries.
+
+**🧩 Easily Extensible** - Create custom blocks with simple inheritance. Rich logging and monitoring built-in.
+
+
+## 📦 Installation
+
+Recommended: Install uv  — see https://docs.astral.sh/uv/getting-started/installation/
+
+```bash
+# Production
+uv pip install sdg-hub
+
+# Development
+git clone https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub.git
+cd sdg_hub
+uv pip install .[dev]
+# or: uv sync --extra dev
+```
+
+### Optional Dependencies
+```bash
+# For vLLM support
+uv pip install sdg-hub[vllm]
+
+# For examples
+uv pip install sdg-hub[examples]
+```
+
+## 🚀 Quick Start
+
+### Core Concepts
+
+**Blocks** are composable units that transform datasets - think of them as data processing Lego pieces. Each block performs a specific task: LLM chat, text parsing, evaluation, or transformation.
+
+**Flows** orchestrate multiple blocks into complete pipelines defined in YAML. Chain blocks together to create complex data generation workflows with validation and parameter management.
+
+```python
+# Simple concept: Blocks transform data, Flows chain blocks together
+dataset → Block₁ → Block₂ → Block₃ → enriched_dataset
+```
+
+### Try it out!
+
+#### Flow Discovery
+```python
+from sdg_hub import FlowRegistry, Flow
+
+# Auto-discover all available flows (no setup needed!)
+FlowRegistry.discover_flows()
+
+# List available flows
+flows = FlowRegistry.list_flows()
+print(f"Available flows: {flows}")
+
+# Search for specific types
+qa_flows = FlowRegistry.search_flows(tag="question-generation")
+print(f"QA flows: {qa_flows}")
+```
+
+Each flow has a **unique, human-readable ID** automatically generated from its name. These IDs provide a convenient shorthand for referencing flows:
+
+```python
+# Every flow gets a deterministic ID 
+# Same flow name always generates the same ID
+flow_id = "small-rock-799" 
+
+# Use ID to reference the flow
+flow_path = FlowRegistry.get_flow_path(flow_id)
+flow = Flow.from_yaml(flow_path)
+```
+
+#### Discovering Models and Configuring them
+```python
+# Discover recommended models
+default_model = flow.get_default_model()
+recommendations = flow.get_model_recommendations()
+
+# Configure model settings at runtime
+# This assumes you have a hosted vLLM instance of meta-llama/Llama-3.3-70B-Instruct running at http://localhost:8000/v1
+flow.set_model_config(
+    model=f"hosted_vllm/{default_model}",
+    api_base="http://localhost:8000/v1",
+    api_key="your_key",
+)
+```
+#### Load your dataset and run the flow
+```python
+# Create your dataset with required columns
+dataset = Dataset.from_dict({
+    'document': ['Your document text here...'],
+    'document_outline': ['1. Topic A; 2. Topic B; 3. Topic C'],
+    'domain': ['Computer Science'],
+    'icl_document': ['Example document for in-context learning...'],
+    'icl_query_1': ['Example question 1?'],
+    'icl_response_1': ['Example answer 1'],
+    'icl_query_2': ['Example question 2?'], 
+    'icl_response_2': ['Example answer 2'],
+    'icl_query_3': ['Example question 3?'],
+    'icl_response_3': ['Example answer 3']
+})
+
+# Quick Testing with Dry Run
+dry_result = flow.dry_run(dataset, sample_size=1)
+print(f"Dry run completed in {dry_result['execution_time_seconds']:.2f}s")
+print(f"Output columns: {dry_result['final_dataset']['columns']}")
+
+# Generate high-quality QA pairs
+result = flow.generate(dataset)
+
+# Access generated content
+questions = result['question']
+answers = result['response']
+faithfulness_scores = result['faithfulness_judgment']
+relevancy_scores = result['relevancy_score']
+```
+
+
+## 📄 License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
+
+## 🤝 Contributing
+
+We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to this project.
+
+---
+
+Built with ❤️ by the Red Hat AI Innovation Team