sdg-hub 0.3.0__tar.gz → 0.4.0__tar.gz

sdg_hub-0.4.0/.github/workflows/packer.yml

@@ -0,0 +1,15 @@
+name: Build AMI with Packer
+
+on:
+  workflow_dispatch:
+
+jobs:
+  build-ami:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write # This is required for OIDC
+      contents: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4

{sdg_hub-0.3.0/src/sdg_hub.egg-info → sdg_hub-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sdg_hub
-Version: 0.3.0
+Version: 0.4.0
 Summary: Synthetic Data Generation
 Author-email: Red Hat AI Innovation <abhandwa@redhat.com>
 License: Apache-2.0

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/README.md

@@ -49,7 +49,6 @@ Learn about the modular block architecture that powers SDG Hub:
 - **[LLM Blocks](blocks/llm-blocks.md)** - Chat, prompt building, and text parsing
 - **[Transform Blocks](blocks/transform-blocks.md)** - Data transformation and manipulation
 - **[Filtering Blocks](blocks/filtering-blocks.md)** - Quality filtering and data validation
-- **[Evaluation Blocks](blocks/evaluation-blocks.md)** - Faithfulness and relevancy assessment
 - **[Custom Blocks](blocks/custom-blocks.md)** - Building your own processing blocks
 
 ### Flow System

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/_sidebar.md

@@ -9,7 +9,6 @@
   * [LLM Blocks](blocks/llm-blocks.md)
   * [Transform Blocks](blocks/transform-blocks.md)
   * [Filtering Blocks](blocks/filtering-blocks.md)
-  * [Evaluation Blocks](blocks/evaluation-blocks.md)
   * [Custom Blocks](blocks/custom-blocks.md)
 
 * **Flow System**

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/blocks/filtering-blocks.md

@@ -10,7 +10,6 @@ Filters dataset rows based on column values using flexible comparison operators
 
 ## 🚀 Next Steps
 
-- **[Evaluation Blocks](evaluation-blocks.md)** - Quality assessment and scoring
 - **[LLM Blocks](llm-blocks.md)** - AI-powered text generation
 - **[Transform Blocks](transform-blocks.md)** - Data manipulation and reshaping
 - **[Flow Integration](../flows/overview.md)** - Combine filtering into complete pipelines

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/blocks/llm-blocks.md

@@ -239,5 +239,4 @@ Extracts structured data from LLM responses using patterns, schemas, or custom p
 
 - **[Transform Blocks](transform-blocks.md)** - Data manipulation and reshaping
 - **[Filtering Blocks](filtering-blocks.md)** - Quality control and validation
-- **[Evaluation Blocks](evaluation-blocks.md)** - Quality assessment and scoring
 - **[Flow Integration](../flows/overview.md)** - Combine LLM blocks into complete pipelines

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/blocks/overview.md

@@ -65,11 +65,6 @@ Data manipulation and transformation:
 Quality control and data validation:
 - **ColumnValueFilterBlock** - Filter rows based on column values
 
-### 📊 Evaluation Blocks (`evaluation/`)
-Quality assessment and scoring:
-- **EvaluateFaithfulnessBlock** - Assess factual accuracy
-- **EvaluateRelevancyBlock** - Measure relevance scores
-- **VerifyQuestionBlock** - Validate question quality
 
 ## 🔧 Block Lifecycle
 
@@ -149,5 +144,4 @@ Ready to dive deeper? Explore specific block categories:
 - **[LLM Blocks](llm-blocks.md)** - AI-powered language model operations
 - **[Transform Blocks](transform-blocks.md)** - Data manipulation and reshaping
 - **[Filtering Blocks](filtering-blocks.md)** - Quality control and validation
-- **[Evaluation Blocks](evaluation-blocks.md)** - Quality assessment and scoring
 - **[Custom Blocks](custom-blocks.md)** - Build your own processing blocks

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/blocks/transform-blocks.md

@@ -26,6 +26,5 @@ Sets uniform values across specified columns, useful for adding metadata or defa
 ## 🚀 Next Steps
 
 - **[Filtering Blocks](filtering-blocks.md)** - Quality control and data validation
-- **[Evaluation Blocks](evaluation-blocks.md)** - Quality assessment and scoring
 - **[LLM Blocks](llm-blocks.md)** - AI-powered text generation
 - **[Flow Integration](../flows/overview.md)** - Combine transform blocks into complete pipelines

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/concepts.md

@@ -152,7 +152,7 @@ Every block validates data at runtime:
 - Validate your pipeline before scaling up
 
 ### 2. Layer Validation
-- Use evaluation blocks to assess quality
+- Use basic block composition (PromptBuilder → LLMChat → Parser → Filter) to assess quality
 - Implement filtering to maintain data standards
 
 ### 3. Monitor Performance

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/development.md

@@ -206,13 +206,6 @@ class TestMyNewBlock:
   - Comprehensive operator support
   - Good performance on large datasets
 
-#### Evaluation Blocks (`src/sdg_hub/core/blocks/evaluation/`)
-- **Purpose**: Quality assessment and scoring
-- **Examples**: Faithfulness evaluation, relevancy scoring
-- **Requirements**:
-  - Consistent scoring methodology
-  - Support for different evaluation criteria
-  - Clear documentation of scoring rubrics
 
 ## 🌊 Contributing Flows
 

{sdg_hub-0.3.0 → sdg_hub-0.4.0}/docs/flows/overview.md

@@ -169,13 +169,41 @@ blocks:
       max_tokens: 300
       async_mode: true
   
-  # Quality evaluation
-  - block_type: "EvaluateFaithfulnessBlock"
+  # Quality evaluation using basic blocks
+  - block_type: "PromptBuilderBlock"
     block_config:
-      block_name: "check_faithfulness"
+      block_name: "faithfulness_prompt"
       input_cols: ["document", "answer"]
-      output_cols: ["faithfulness_score"]
+      output_cols: ["eval_prompt"]
+      prompt_template: "Evaluate if this answer is faithful to the document..."
+
+  - block_type: "LLMChatBlock"
+    block_config:
+      block_name: "eval_faithfulness_llm"
+      input_cols: ["eval_prompt"]
+      output_cols: ["eval_response"]
       async_mode: true
+
+  - block_type: "LLMParserBlock"
+    block_config:
+      block_name: "extract_eval_content"
+      input_cols: ["eval_response"]
+      extract_content: true
+
+  - block_type: "TextParserBlock"
+    block_config:
+      block_name: "parse_evaluation"
+      input_cols: ["extract_eval_content_content"]
+      output_cols: ["explanation", "judgment"]
+      start_tags: ["[Start of Explanation]", "[Start of Answer]"]
+      end_tags: ["[End of Explanation]", "[End of Answer]"]
+
+  - block_type: "ColumnValueFilterBlock"
+    block_config:
+      block_name: "filter_faithful"
+      input_cols: ["judgment"]
+      filter_value: "YES"
+      operation: "eq"
   
   # Quality filtering
   - block_type: "ColumnValueFilterBlock"

sdg_hub-0.4.0/examples/annotation/annotation_classification.ipynb

@@ -0,0 +1,486 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Third Party\n",
+    "from datasets import load_dataset\n",
+    "from openai import OpenAI\n",
+    "from rich import print\n",
+    "from rich.panel import Panel\n",
+    "from sklearn.metrics import classification_report\n",
+    "\n",
+    "# First Party\n",
+    "from sdg_hub import Flow, FlowMetadata, BlockRegistry\n",
+    "\n",
+    "import nest_asyncio\n",
+    "nest_asyncio.apply()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Classifying news articles\n",
+    "\n",
+    "\n",
+    "In this tutorial, you’ll learn how to create your own custom data generation flow using SDG Hub. This notebook will walk you through all the essential pieces to make your own flow using `sdg_hub` for any use-case using the fundamental components of sdg_hub: `Blocks` and `Flows`\n",
+    "\n",
+    "As an example use-case, we will pick news classification. Classification is a fundamental task in machine learning, where the goal is to assign predefined categories to input data. To address the classic machine learning use-case of news or text classification, we will use sdg_hub and leverage a language model to **classify news articles** with topic labels — specifically using the [AG News dataset](https://huggingface.co/datasets/fancyzhx/ag_news) from Hugging Face.\n",
+    "\n",
+    "We’ll go step by step through a progressively improving flow. Each stage builds on the previous one, giving you a practical sense of how you can evolve your flow from using simple heuristics to highly customized and reliable data generation, using different inference paradigms such as self assessment.\n",
+    "\n",
+    "### 🔍 Understand the Task\n",
+    "Before we write any prompts or code, we’ll take time to understand what we want the model to do. For this exercise, the task is **text classification** — assigning one of 4 possible categories (e.g., \"World\", \"Sports\", \"Sci/Tech\", \"Business\") to a given news article\n",
+    "\n",
+    "### 🛠️ Build a Basic Annotation Flow and learn the `sdg_hub` way\n",
+    "We’ll start by creating a minimal flow that simply prompts the model to generate topic labels on the unlabeled data. This will use default prompts, simply populating the prompt with the text and asking the model to generate one of the 4 possible labels, with no examples.\n",
+    "\n",
+    "### 🎯 Improve with Assessment and Iteration\n",
+    "Next, we’ll refine the flow by adding an assessment step. Iterations and self verification on a task often lead to better performance\n",
+    "\n",
+    "Let’s get started by loading a sample of the dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset = load_dataset(\"fancyzhx/ag_news\")\n",
+    "\n",
+    "train_data = dataset[\"train\"].shuffle(seed=42).select(range(500))\n",
+    "test_data = dataset[\"test\"].shuffle(seed=42).select(range(100))\n",
+    "\n",
+    "# map the labels to the category names\n",
+    "label_map = train_data.features['label'].names\n",
+    "\n",
+    "train_data = train_data.map(lambda x: {\"category\": label_map[x[\"label\"]]})\n",
+    "test_data = test_data.map(lambda x: {\"category\": label_map[x[\"label\"]]})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Group examples by category\n",
+    "examples_by_category = {}\n",
+    "for item in train_data:\n",
+    "    category = item['category']\n",
+    "    if category not in examples_by_category:\n",
+    "        examples_by_category[category] = []\n",
+    "    examples_by_category[category].append(item['text'])\n",
+    "\n",
+    "# Print one example from each category in a panel\n",
+    "for category, examples in examples_by_category.items():\n",
+    "    print(Panel(examples[0], title=f\"Category: {category}\", expand=False))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Simple Data Annotation Pipeline\n",
+    "\n",
+    "In this section, we’ll create our **first working flow** to perform classification using a language model. The goal is to understand the building blocks of `sdg_hub` and how we can employ them to get a language model to classify a given text.\n",
+    "\n",
+    "### Recap: How `sdg_hub` Works\n",
+    "\n",
+    "```mermaid\n",
+    "flowchart LR\n",
+    "    A[Flow] --> B[Blocks] --> C[Prompts]\n",
+    "    C --> D[Generated Data]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Building a Simple Classification Flow\n",
+    "\n",
+    "### Discover Blocks for us to use\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "BlockRegistry.discover_blocks()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It seems all the functionality we are interested in, such as building a prompt, chatting with an llm and parsing its output are under the `llm` category in sdg_hub. Lets start there."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sdg_hub.core.blocks.llm import PromptBuilderBlock, LLMChatBlock, TextParserBlock, LLMParserBlock"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Creating the required blocks\n",
+    "\n",
+    "To get started, we'll construct the simplest possible flow for text classification using SDG Hub. We will focus on 3 main blocks that will often appear as a triplet while using `sdg_hub`\n",
+    "\n",
+    "1. **Prompt Builder Block**: Converts each input text into a prompt formatted for the LLM. The important input argument to keep in mind for  `PromptBuilderblock` is the `prompt_config_path` which is where the prompt template is saved. Any prompt engineering we would want to do would be done in such a prompt template.\n",
+    "2. **LLM Chat Block**: Sends the prompt to the language model and receives its response (the predicted label).\n",
+    "3. **Text Parser Block**: Extracts the final label from the LLM's output.\n",
+    "\n",
+    "This setup results in a single LLM interaction per sample, forming a minimal classification pipeline.\n",
+    "\n",
+    "We are going to be using the simple prompt that can be found in `news_articles_classification_prompt.yaml`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "promptbuilderblock_1 = PromptBuilderBlock(block_name='annotation_prompt_builder', input_cols=['text'], output_cols=['annotation_prompt'], prompt_config_path=\"news_classification_prompt.yaml\", format_as_messages=True)\n",
+    "llmchatblock_1 = LLMChatBlock(block_name='annotation_llm_chat_block', input_cols=['annotation_prompt'], output_cols=['raw_output'], temperature=0.0, max_tokens=5, extra_body={'guided_choice': ['World', 'Sports', 'Business', 'Sci/Tech']}, async_mode=True)\n",
+    "llmparserblock_1 = LLMParserBlock(block_name='annotation_llm_parser_block', input_cols=['raw_output'], extract_content=True, expand_lists=True)\n",
+    "textparserblock_1 = TextParserBlock(block_name='annotation_text_parser_block', input_cols=['annotation_llm_parser_block_content'], output_cols=['output'], start_tags=[''], end_tags=[''])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Designing the `Flow`\n",
+    "\n",
+    "The `Flow` class is at the heart of SDG Hub. Simply put, a `Flow` is a chain of `Blocks` that get executed sequentially. Here, we will simply chain our PromptBuilder -> LLMChatBlock -> TextParser, in that order:\n",
+    "\n",
+    "```mermaid\n",
+    "flowchart LR\n",
+    "    subgraph Flow\n",
+    "        direction LR\n",
+    "        A[PromptBuilderBlock] --> B[LLMChatBlock] --> C[TextParserBlock]\n",
+    "    end\n",
+    "```\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = Flow(blocks=[promptbuilderblock_1, llmchatblock_1, llmparserblock_1, textparserblock_1], metadata=FlowMetadata(name=\"annotation_flow\", description=\"A flow for news article classification\", author=\"sdg_hub\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set the model configs for the `Flow`\n",
+    "\n",
+    "In SDG Hub, model details such as the API base URL, the API Key (if any) and the model name are set at a Flow level using the `set_model_config` method as shown. The `model` parameter accepts a string in the format of \"`provider`/`model_name`\". Here our `provider` is 'hosted_vllm' as we are using a locally hosted model through vllm, and the model name is \"meta-llama/Llama-3.3-70B-Instruct\"\n",
+    "\n",
+    "We must set the `api_base` parameter and point it to where the model endpoint can be found, in this case, `http://localhost:8000/v1`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# flow.set_model_config(model=\"hosted_vllm/meta-llama/Llama-3.3-70B-Instruct\", api_base=\"http://localhost:8000/v1\", api_key=\"\")\n",
+    "\n",
+    "flow.set_model_config(model=\"hosted_vllm/qwen3-8b\", api_base=\"http://localhost:8101/v1\", api_key=\"empty\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Time to generate!\n",
+    "\n",
+    "In sdg_hub, the way to generate data is very simple. we simply use the `generate` method from `Flow`. At its simplest form, all the `generate` method needs is the input dataset to operate on. Additionally, we can pass runtime parameters for each block as well, if we wish to override any of the block specific model configs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generated_data = flow.generate(test_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Evaluation\n",
+    "\n",
+    "Now that we’ve generated synthetic labels using our simple classification flow, it’s time to evaluate how well the model performed. The goal of this section is to compare the predicted labels against the **true labels** from the dataset using standard classification metrics (precision, recall, f-1 score and classification accuracy)\n",
+    "\n",
+    "We’ll use `sklearn.metrics.classification_report`, which provides precision, recall, F1-score, and support for each class.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(classification_report(generated_data[\"category\"], generated_data[\"output\"]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introducing an Assessment step\n",
+    "\n",
+    "Our initial flow used a one step approach — the model was given the task, a fixed label set, and some input text. While this baseline gives us a useful starting point, it has clear limitations:\n",
+    "\n",
+    "- The model may rely on generic heuristics or surface patterns that don’t generalize well.\n",
+    "- It can confuse similar categories (e.g., \"World\" vs. \"Business\") without knowing how they're typically used.\n",
+    "- Without guidance, the model may underperform on edge cases or ambiguous queries.\n",
+    "\n",
+    "\n",
+    "### What is Assessment\n",
+    "\n",
+    "With an assessment step, we will call to the same LLM, but this time, we provide the LLM with its own previous categorization label, and the original text. We will prompt the LLM to think about the original prediction, and give it context about challening cases\n",
+    "In this manner, we can elicit critical judgement from the model about its own prior classification decision. This type of additional context can be useful in the next iteration.\n",
+    "\n",
+    "\n",
+    "### What We’ll Do Next\n",
+    "\n",
+    "We’ll now enhance our flow by introducing another chain of `PromptBuilder` -> `LLMChatBlock` -> `TextParserBlock` whose purpose is to pass the (original text +  prediction) to the LLM and obtain a verification or assessment of the prediction.\n",
+    "\n",
+    "\n",
+    "```mermaid\n",
+    "flowchart LR\n",
+    " subgraph Flow1[Initial Classification]\n",
+    " direction LR\n",
+    " A[PromptBuilderBlock] --> B[LLMChatBlock] --> C[TextParserBlock]\n",
+    " end\n",
+    " subgraph Flow2[Assessment]\n",
+    " direction LR\n",
+    " D[PromptBuilderBlock_Assessment] --> E[LLMChatBlock_Assessment] --> F[TextParserBlock_Assessment]\n",
+    " end\n",
+    " \n",
+    " C --> D\n",
+    "```\n",
+    "\n",
+    "\n",
+    "We will investigate if this catches any of the mis-classifications, and get an idea of how well our verification prompting works!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "promptbuilderblock_assessment = PromptBuilderBlock(block_name='verifier_prompt_builder', input_cols=['text', 'output'], output_cols=['assessment_prompt'], prompt_config_path=\"news_classification_assessment_prompt.yaml\", format_as_messages=True)\n",
+    "llmchatblock_assessment = LLMChatBlock(block_name='verifier_llm_chat_block', input_cols=['assessment_prompt'], output_cols=['raw_assessment_output'], async_mode=True)\n",
+    "llmparserblock_assessment = LLMParserBlock(block_name='verifier_llm_parser_block', input_cols=['raw_assessment_output'], extract_content=True, expand_lists=True)\n",
+    "textparserblock_assessment = TextParserBlock(block_name='verifier_text_parser_block', input_cols=['verifier_llm_parser_block_content'], output_cols=['assessment_output'], start_tags=[''], end_tags=[''])\n",
+    "\n",
+    "flow = Flow(blocks=[promptbuilderblock_1, llmchatblock_1, llmparserblock_1, textparserblock_1, promptbuilderblock_assessment, llmchatblock_assessment, llmparserblock_assessment, textparserblock_assessment], metadata=FlowMetadata(name=\"annotation_flow\", description=\"A flow for news article classification\", author=\"sdg_hub\"))\n",
+    "# flow.set_model_config(model=\"hosted_vllm/meta-llama/Llama-3.3-70B-Instruct\", api_base=\"http://localhost:8000/v1\", api_key=\"\")\n",
+    "flow.set_model_config(model=\"hosted_vllm/qwen3-8b\", api_base=\"http://localhost:8101/v1\", api_key=\"empty\")\n",
+    "\n",
+    "\n",
+    "\n",
+    "generated_data = flow.generate(test_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generated_data_pd = generated_data.to_pandas()\n",
+    "mislabeled_samples = generated_data_pd[generated_data_pd[\"category\"] != generated_data_pd[\"output\"]]\n",
+    "\n",
+    "print(Panel(mislabeled_samples.iloc[0]['assessment_output'], title=\"Assessment\"))\n",
+    "print(Panel(str(mislabeled_samples.iloc[0]['category']), title=\"Ground truth label\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Great! Now we can see that the assessment step is working good, especially on the misclassified samples as shown above. The above is a hard example which has slipped past our original classification flow, but was caught by our assessment step's critical judgement."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Revising the Classifications\n",
+    "\n",
+    "We will now create our final revision step, which will take the results of the initial prediction and the assessment steps and pass it onto the LLM once again for a revised attempt at classifying the same input text. The flow can be imagined like so:\n",
+    "\n",
+    "```mermaid\n",
+    "flowchart LR\n",
+    " subgraph Flow1[Initial Classification]\n",
+    " direction LR\n",
+    " A[PromptBuilderBlock] --> B[LLMChatBlock] --> C[TextParserBlock]\n",
+    " end\n",
+    " subgraph Flow2[Assessment]\n",
+    " direction LR\n",
+    " D[PromptBuilderBlock_Assessment] --> E[LLMChatBlock_Assessment] --> F[TextParserBlock_Assessment]\n",
+    " end\n",
+    " subgraph Flow3[Revised Classification]\n",
+    " direction LR\n",
+    " G[PromptBuilderBlock_Revision] --> H[LLMChatBlock_Revision] --> I[TextParserBlock_Revision]\n",
+    " end\n",
+    " \n",
+    " C --> D\n",
+    " F --> G\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "promptbuilderblock_revision = PromptBuilderBlock(block_name='revised_prompt_builder', input_cols=['text', 'output', 'assessment_output'], output_cols=['revised_prompt'], prompt_config_path=\"revise_news_classification_prompt.yaml\", format_as_messages=True)\n",
+    "llmchatblock_revision = LLMChatBlock(block_name='revised_llm_chat_block', input_cols=['revised_prompt'], output_cols=['raw_revised_output'], temperature=0.0, max_tokens=5, extra_body={'guided_choice': ['World', 'Sports', 'Business', 'Sci/Tech']}, async_mode=True)\n",
+    "llmparserblock_revision = LLMParserBlock(block_name='revised_llm_parser_block', input_cols=['raw_revised_output'], extract_content=True, expand_lists=True)\n",
+    "textparserblock_revision = TextParserBlock(block_name='revised_text_parser_block', input_cols=['revised_llm_parser_block_content'], output_cols=['revised_output'], start_tags=[''], end_tags=[''])\n",
+    "\n",
+    "flow = Flow(blocks=[promptbuilderblock_1, llmchatblock_1, llmparserblock_1, textparserblock_1, promptbuilderblock_assessment, llmchatblock_assessment, llmparserblock_assessment, textparserblock_assessment, promptbuilderblock_revision, llmchatblock_revision, llmparserblock_revision, textparserblock_revision], metadata=FlowMetadata(name=\"news_classification_flow\", description=\"A flow for news article classification with assessment and revision\", author=\"sdg_hub\"))\n",
+    "# flow.set_model_config(model=\"hosted_vllm/meta-llama/Llama-3.3-70B-Instruct\", api_base=\"http://localhost:8000/v1\", api_key=\"\")\n",
+    "flow.set_model_config(model=\"hosted_vllm/qwen3-8b\", api_base=\"http://localhost:8101/v1\", api_key=\"empty\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generated_data = flow.generate(test_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(classification_report(generated_data[\"category\"], generated_data[\"revised_output\"]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "🔥 We improved the results drastically! Let us take a look at the number of mislabeled samples before and after the assessment + revision steps\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generated_data_pd = generated_data.to_pandas()\n",
+    "num_mislabeled_output = (generated_data_pd[\"category\"] != generated_data_pd[\"output\"]).sum()\n",
+    "num_mislabeled_revised = (generated_data_pd[\"category\"] != generated_data_pd[\"revised_output\"]).sum()\n",
+    "print(f\"Number of mislabeled samples (original output): {num_mislabeled_output}\")\n",
+    "print(f\"Number of mislabeled samples (revised output): {num_mislabeled_revised}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Great, we whave now improved the classification accuracy of our system by augmenting our naive classification flow by adding an assessment followed by a revision step\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Export the flow to yaml form\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow.to_yaml(\"news_classification_flow.yaml\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## ✅ Summary: What You’ve Learned\n",
+    "\n",
+    "In this tutorial, you learned how to create your own flow for a custom use-case using `sdg_hub`, using the fundamental components: `Flow` and `Block`. You also learned how to create and structure the prompts. You learned how to design an assessment or a judgement step in order to improve the performance of the overall system. You started from scratch and evolved it into a robust, high-accuracy system.\n",
+    "\n",
+    "## 🚀 What’s Next?\n",
+    "\n",
+    "* Prompt Engineer! -  You can add examples for classifications directly in the classification steps and see how this improves the performance. In-context examples are extremely effective at aligning the model's outputs to the task at hand\n",
+    "* Try it out on your own data!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "test_nb",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}