sdg-hub 0.1.0a4__py3-none-any.whl → 0.1.1__py3-none-any.whl

sdg_hub/configs/skills/evaluate_grounded_pair.yaml

@@ -1,39 +1,104 @@
-system: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+system: You are a highly knowledgeable and impartial AI Assistant tasked with evaluating the quality of responses to user questions.
 
 introduction: |
-  Please act as an impartial judge and evaluate the quality of the answer provided by an AI assistant to the questions displayed below. Evaluate whether or not the answer is a good example of how AI Assistant should respond to the user's instruction. Please assign a score using the following 3-point scale.
+  Please act as an objective evaluator and assess the quality of the AI Assistant's answer to the user's question. Your job is to determine whether the answer is complete, correct, relevant, safe, and well-written according to the task. Use the scoring rubric below and assign a score from 1 to 3.
+
 principles: |
-  1: It means the answer is incorrect, irrelevant, unsafe or provides incomplete and garbage information. For instance, the answer may be factually wrong, off-topic, or filled with irrelevant content that doesn't address the user's question or it could be incomplete and hanging. It may also include any harmful, unethical, racist, sexist, explicit, offensive, toxic, dangerous, or illegal content. If the answer is not supported by the context given.
+  Use the following 3-point scale to score the answer:
+
+  **1 — Poor Quality**
+  - The answer is factually incorrect, off-topic, incomplete, or nonsensical.
+  - It may contain hallucinations, hanging statements, or irrelevant content.
+  - It does not align with the task description or question intent.
 
-  2: It means the answer provides the correct answer, but it is brief and to the point without explanations. While it directly answers the user's question, it lacks additional context or in-depth explanations.
+  **2 — Acceptable but Minimal**
+  - The answer is correct and safe, but short and lacks depth or clarity.
+  - It may be technically correct but overly terse or missing explanation.
+  - It satisfies the user's question at a surface level without context.
 
-  3: It means the answer is a perfect answer from an AI Assistant. It intentionally addresses the user's question with a comprehensive and detailed explanation. It demonstrates expert knowledge in the area, is very well written, logical, easy to follow, engaging, and insightful. And the answer is safe and does not include any harmful content.
+  **3 — Excellent Answer**
+  - The answer is complete, correct, relevant, and insightful.
+  - It provides clear reasoning, well-structured writing, and deep understanding.
+  - It directly addresses the question with expert-level knowledge.
 
 examples: |
-   [Start of Context]
-   In recent years, electric vehicles (EVs) have gained popularity due to their environmental benefits and advancements in technology. EVs produce zero emissions while driving, which helps reduce air pollution and combat climate change. Additionally, with the development of more efficient batteries, the range and performance of EVs have significantly improved, making them a more viable option for consumers.
-   [End of Context]
-
-   [Start of Question]
-   How do electric vehicles help in reducing air pollution?
-   [End of Question]
-   
-   [Start of Answer]
-   Electric vehicles (EVs) help in reducing air pollution because they produce zero emissions while driving.
-   [End of Answer]
-
-   [Start of Evaluation]
-   This answer is correct, directly addresses the user's question, and is supported by the provided context. It is brief and to the point but lacks additional context or in-depth explanations.
-   [End of Evaluation]
-   
-   [Start of Score]
-   2
-   [End of Score]
+  Example 1 — Score: 1 (Poor Quality)
+
+  Task Description: Summarize the key point of a news paragraph in one sentence.
+
+  [Start of Context]
+  The mayor of Springfield unveiled a new plan to reduce urban pollution, which includes a ban on gas-powered scooters, expansion of bike lanes, and free public transit passes for residents.
+  [End of Context]
+
+  [Start of Question]
+  What is the main point of the paragraph?
+  [End of Question]
+
+  [Start of Answer]
+  Scooters are fun but kind of dangerous if you go too fast lol.
+  [End of Answer]
+
+  [Start of Evaluation]
+  The answer is irrelevant and nonsensical in context. It does not reflect the content of the paragraph and includes casual language and commentary that breaks task intent. It is an example of a hallucinated, off-topic response.
+  [End of Evaluation]
+
+  [Start of Score]
+  1
+  [End of Score]
+
+  Example 2 — Score: 2 (Acceptable but Minimal)
+
+  Task Description: Summarize the key point of a news paragraph in one sentence.
+
+  [Start of Context]
+  The mayor of Springfield unveiled a new plan to reduce urban pollution, which includes a ban on gas-powered scooters, expansion of bike lanes, and free public transit passes for residents.
+  [End of Context]
+
+  [Start of Question]
+  What is the main point of the paragraph?
+  [End of Question]
+
+  [Start of Answer]
+  The city is taking steps to reduce pollution.
+  [End of Answer]
+
+  [Start of Evaluation]
+  The answer is factually correct and safe, but vague. It lacks depth and omits the key details about *how* the city plans to reduce pollution. It does not demonstrate full understanding of the context or question.
+  [End of Evaluation]
+
+  [Start of Score]
+  2
+  [End of Score]
+
+  Example 3 — Score: 3 (Excellent Answer)
+
+  Task Description: Summarize the key point of a news paragraph in one sentence.
+
+  [Start of Context]
+  The mayor of Springfield unveiled a new plan to reduce urban pollution, which includes a ban on gas-powered scooters, expansion of bike lanes, and free public transit passes for residents.
+  [End of Context]
+
+  [Start of Question]
+  What is the main point of the paragraph?
+  [End of Question]
+
+  [Start of Answer]
+  The mayor of Springfield introduced a pollution-reduction plan that bans gas-powered scooters, expands bike lanes, and provides free transit passes to residents.
+  [End of Answer]
+
+  [Start of Evaluation]
+  The answer is complete, relevant, and clearly written. It captures the full intent and detail of the paragraph in a single well-structured sentence, demonstrating both factual understanding and clarity. It aligns perfectly with the task instruction.
+  [End of Evaluation]
+
+  [Start of Score]
+  3
+  [End of Score]
 
 generation: |
-  Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, you must rate the answer on a scale of 1 to 3 as mentioned above. 
-  Here's the context, question and the answer you need to evaluate:
-  
+  Now begin your evaluation of the following QA pair. Use the rubric above and be objective and concise in your reasoning.
+
+  Task Description: {{task_description}}
+
   [Start of Context]
   {{ context }}
   [End of Context]
@@ -46,9 +111,9 @@ generation: |
   {{ response }}
   [End of Answer]
 
-  * Return the evaluation between [Start of Evaluation] and [End of Evaluation] tags.
-  * Return the score between [Start of Score] and [End of Score] tags.
-
+  * Provide your evaluation between [Start of Evaluation] and [End of Evaluation] tags.
+  * Provide the score between [Start of Score] and [End of Score] tags.
+  * Do not include any content outside these tags.
 
 start_tags: ["[Start of Evaluation]", "[Start of Score]"]
 end_tags: ["[End of Evaluation]", "[End of Score]"]

sdg_hub/configs/skills/freeform_questions.yaml

@@ -1,29 +1,34 @@
-system: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+system: You are a highly intelligent and helpful AI Assistant specializing in generating well-crafted questions tailored to specific tasks.
 
 introduction: |
-  You are asked to come up with a set of {{ num_samples }} diverse questions - {{ task_description }}.
+  Your task is to generate {{num_samples}} diverse and well-structured questions for the following task:
+  "{{task_description}}"
 
 principles: |
-  Please follow these guiding principles when generating responses:
-  * Use proper grammar and punctuation.
-  * Always generate safe and respectful content. Do not generate content that is harmful, abusive, or offensive.
-  * Always generate content that is factually accurate and relevant to the prompt.
-  * The questions should be clear and human-like.
-  * The questions should be diverse and cover a wide range of topics.
-  * The questions should not be template-based or generic, it should be very diverse.
-  * Simply return the questions, do not return any answers or explanations.
-  * Strictly adhere to the prompt and generate responses in the same style and format as the example.
-  * Return each question between [Start of Question] and [End of Question] tags.
+  Please follow these strict guidelines when generating each question:
+  * Each question must be directly related to the task description.
+  * Use correct grammar, spelling, and punctuation.
+  * Questions must be clear, natural-sounding, and human-like.
+  * Do **not** include answers, explanations, or commentary—only the question.
+  * Ensure **maximum diversity** in wording and perspective—no repetitive or template-based phrasing.
+  * Each question must strictly follow the **exact same format and style** as shown in the example.
+  * Do **not deviate** from the example structure in any way.
+  * Wrap each question between `[Start of Question]` and `[End of Question]` tags.
 
 examples: |
-  To better assist you with this task, here is an example:
-  
+  To guide you, here is an example of a correctly formatted question:
+
   [Start of Question]
-  {{ seed_question }}
+  {{seed_question}}
   [End of Question]
 
 generation: |
-  Now generate {{ num_samples }} such questions, remember to follow the principles mentioned above and use the same format as the examples. Remember to use the same style and format as the example above. Return each question between [Start of Question] and [End of Question] tags.
+  Now generate {{num_samples}} such questions.
+  Ensure that each one is:
+  - Fully relevant to the task description.
+  - Consistent with the example format.
+  - Clearly enclosed between [Start of Question] and [End of Question] tags.
+  Do not include any content outside these tags.
 
 start_tags: ["[Start of Question]"]
 end_tags: ["[End of Question]"]

sdg_hub/configs/skills/freeform_responses.yaml

@@ -1,45 +1,39 @@
-system: You are a very knowledgeable AI Assistant that will faithfully assist the user with their task.
+system: You are a highly knowledgeable and precise AI Assistant tasked with generating high-quality responses based strictly on user prompts.
 
-introduction: Your task is to faithfully follow the user's prompt and generate a response.
+introduction: |
+  Your task is to carefully read and understand the user's prompt, and generate a well-structured response that directly addresses it.
 
 principles: |
-  Please follow these guiding principles when generating responses:
-  * Use proper grammar and punctuation.
-  * Always generate safe and respectful content. Do not generate content that is harmful, abusive, or offensive.
-  * Always generate content that is factually accurate and relevant to the prompt.
-  * Strictly adhere to the prompt and generate responses in the same style and format as the example.
-  * Return the response between [Start of Response] and [End of Response] tags.
+  Please follow these strict guidelines when generating your response:
+  * The response must be directly relevant to the given question and accurately fulfill its intent.
+  * The response should be clear, natural, and human-like in tone.
+  * You must strictly follow the **exact structure and formatting** shown in the example.
+  * Do **not** add any extra commentary, explanations, greetings, or metadata.
+  * The response must be enclosed between `[Start of Response]` and `[End of Response]` tags.
+  * Do **not** deviate from the formatting under any circumstance.
 
 examples: |
-  To better assist you with this task, here is an example:
-  {% if seed_samples is defined %}
-  {% for sample in seed_samples %}
-  [Start of Question]
-  {{ sample.seed_question }}
-  [End of Question]
+  Below is an example of a properly formatted question and response:
 
-  [Start of Response]
-  {{ sample.seed_response }}
-  [End of Response]
-  {% endfor %}
-  {% else %}
   [Start of Question]
-  {{ seed_question }}
+  {{seed_question}}
   [End of Question]
 
   [Start of Response]
-  {{ seed_response }}
+  {{seed_response}}
   [End of Response]
-  {% endif %}
 
 generation: |
-  Now generate a response to the following prompt. Remember to use the same style and format as the example above. 
+  Now generate a response to the following prompt. Make sure the response:
+  - Accurately addresses the question.
+  - Strictly follows the example format and style.
+  - Appears **only** between [Start of Response] and [End of Response] tags.
 
   [Start of Question]
-  {{ question }}
+  {{question}}
   [End of Question]
 
-  Return the response between [Start of Response] and [End of Response] tags.
+  Return only the response between the tags. Do not add anything outside.
 
 start_tags: ["[Start of Response]"]
 end_tags: ["[End of Response]"]

sdg_hub/configs/skills/router.yaml

@@ -1,12 +1,59 @@
-system: ""
+system: |
+  You are a skill classifier. Given a question or task, classify it into exactly one of the following categories:
+  - coding: Questions about programming, software development, algorithms, or technical implementation.
+  - extraction: Tasks that require identifying and pulling out specific pieces of information from text or data, without analysis or transformation. This includes extracting structured data and transforming it into a specific format.
+  - humanities: Questions about economics, social sciences, literature, history, philosophy, or arts that require analysis, comparison, or evaluation.
+  - math: Questions requiring mathematical calculations, proofs, probability, or numerical reasoning.
+  - reasoning: Tasks requiring logical deduction, problem-solving, or analytical thinking.
+  - roleplay: Scenarios requiring adopting a specific role, perspective, or character.
+  - stem: Questions about science, technology, engineering, or mathematics that require explaining technical or scientific concepts.
+  - writing: Tasks that require creating new content, compositions, or text generation.
 
-introduction: ""
+  Only output the category name, and nothing else. Please do not output anything except one of these exact words.
 
-principles: ""
+introduction: |
+  Classify the given question into one of the predefined categories based on its primary focus and requirements.
 
-examples: ""
+principles: |
+  1. Each question should be classified into exactly one category.
+  2. Choose the category that best represents the primary skill or domain required.
+  3. If a question could fit multiple categories, choose the most specific one.
+  4. Consider the main task or goal of the question, not just the subject matter.
+  5. For extraction tasks, focus on questions that require pulling out specific information or data, often with specific formatting requirements, without significant analysis or creative generation.
+  6. For humanities tasks, focus on questions that require analysis, comparison, or evaluation of social, cultural, or historical topics.
+  7. For writing tasks, focus on questions that require creative or open-ended content generation or composition, rather than tasks with strict formatting or content extraction rules.
 
-generation: "{{ question }}"
+examples: |
+  Question: Extract and categorize the health-related themes, such as 'symptoms', 'treatments', and 'prevention'
+  Category: extraction
+
+  Question: Summarize the story with three bullet points using only nouns and adjectives, without verbs.
+  Category: extraction
+
+  Question: Can you analyze the relationship between economic indicators like GDP, interest rates, and consumer behavior?
+  Category: humanities
+
+  Question: Calculate the area of a triangle with base 6 units and height 8 units
+  Category: math
+
+  Question: Suppose you are in a marathon and you just passed the person in third place. What position are you in?
+  Category: reasoning
+
+  Question: You are a medieval blacksmith. Describe your typical workday and the challenges you face
+  Category: roleplay
+
+  Question: In the realm of quantum mechanics, what is wave-particle duality, and how does it contribute to the understanding of quantum systems?
+  Category: STEM
+
+  Question: Write a travel blog post about exploring the ancient temples and street food scene in Bangkok, Thailand
+  Category: writing
+
+generation: |
+  Question: {{ question }}
+  
+  Based on the above question, classify it into exactly one of these categories: coding, extraction, humanities, math, reasoning, roleplay, STEM, or writing.
+  
+  Category:
 
 start_tags: [""]
-end_tags: [""]
+end_tags: [""]