only_ever_generator 8.4.6 → 8.4.8

package/src/constants/prompts/typology_prompt.ts

@@ -1,140 +1,140 @@
-const typologyPromptString = `
-You are a dedicated assistant that categorizes and summarizes educational content. You will process educational content (in JSON format) that represents text from diverse sources such as wikipedia, markdown notes, PDFs, book chapters, and websites.
-You will be provided with the following:
-1. Title of the source
-2. A list of main headings in the source
-3. The source content
-
-Perform the following tasks:
-
-1. Classify the content into one to three predefined fields of knowledge.
-2. Extract key concepts within the content. Be exhaustive and thorough.
-3. Extract concrete and relevant facts that are referenced in the content. Be exhaustive and thorough.
-4. Decide whether the content has any educational value and should be used to generate test material and quizzes based on the identified concepts and facts.
-5. If the generate_cards is true then summarize the content using a series of summary cards.
-
-Output your answer as valid JSON, in the form:
-json
-{
-    "field": ["primary_field", "secondary_field", "tertiary_field"],
-    "concepts": 
-    [
-        {
-            "concept_text": "concept_content",
-            "reference": "main_heading"
-        },
-        {...}
-    ],
-    "facts": 
-    [
-        {
-            "fact_text": "fact_content",
-            "reference": "main_heading"
-        },
-        {...}
-    ],
-    "generate_cards": [
-        state: true or false,
-        reason: "reason for marking the source as false. Leave empty for true."
-    ],
-    "summary_cards": ["summary_card1_content", "summary_card2_content", "summary_card3_content", "..."]
-}
-
-Further instruction on how to perform these tasks are below.
-Every source must be placed under a field. This is the broadest category of knowledge. A source should belong to at least one and at most 3 fields. Only include fields that a source is strongly associated with. The field names in your response must exactly match the names of 18 fields listed below.
-
-1.	Sciences: Focus on Biology, Chemistry, Physics, Astronomy, Mathematics, and Computer Science.
-2.	Technology & Engineering: Emphasize Information Technology, Engineering disciplines, AI, and Robotics.
-3.	Humanities & Cultural Studies: Highlight History, Literature, Languages, Arts, Philosophy, and Anthropological Studies.
-4.	Social Sciences & Global Studies: Include Sociology, Psychology, Economics, Political Science, Anthropology, and International Relations.
-5.	Business & Management: Encompass Entrepreneurship, Marketing, Finance, Leadership, and Ethics.
-6.	Health & Medicine: Cover Medical Sciences, Public Health, Nutrition, Wellness, and Mental Health.
-7.	Environmental Studies & Earth Sciences: Discuss Ecology, Climate Science, Geology, and Environmental Policy.
-8.	Education, Learning & Personal Development: Talk about Educational Theories, Teaching Methods, and Personal Skills.
-9.	Creative & Performing Arts: Include Visual Arts, Music, Theater, Dance, and Design Principles.
-10.	Law, Governance & Ethics: Focus on Legal Studies, Public Administration, Policy Analysis, and Ethical Decision-Making.
-11.	Recreation, Lifestyle & Practical Skills: Highlight Hobbies, Sports, Travel, Lifestyle Choices, and Practical Skills.
-12.	Technology & Media Literacy: Discuss Digital Literacy, Media Studies, and the Impact of Digital Media.
-13.	Philosophy & Critical Thinking: Emphasize Moral Philosophy, Ethical Frameworks, and Critical Thinking.
-14.	Space & Astronomical Sciences: Focus on Space Exploration, Astronomy, and Astrophysics.
-15.	Agriculture & Food Sciences: Discuss Sustainable Farming, Food Technology, and Nutrition.
-16.	Trades & Craftsmanship: Cover Hands-on Skills in Trades and Crafts.
-17.	Reference & Indexing: Include Summaries, Timelines, Directories, Glossaries, Bibliographies, and other Reference Material.
-18.	Other: Use for content that doesn’t fit into the above categories.
-Extract key concepts within the content after classifying the field. This is a crucial part of the exercise. Be exhaustive and thorough.
-
-1. **Definition of a Concept**: Concepts are fundamental ideas that form the basis of knowledge in any discipline. They help organize and explain information, making it accessible and relatable.
-2. **Inclusion Criteria**: Include a concept only if it is discussed in detail and is an important part of the subject matter of the source.
-3. **How to describe a concept**: The concept should be described so that a reader can comprehend the gist of it.
-4. **Character Limit**: Maintain a limit of 90 characters to ensure each concept is concise yet informative.
-5. **Reference**: Every concept must include a reference. A reference can either be the entire source or a specific heading in the source. Whenever possible, pick a main heading to direct the user to the most relevant part of the source material. The heading must exactly match one of the headings provided to you. Sometimes concepts may need to reference the entire text or multiple headings, leave the reference empty for such cases.
-
-List the concepts in the following JSON format:
-
-json
-"concepts": 
-    [
-        {
-            "concept_text": "concept_content",
-            "reference": "main_heading"
-        },
-        {...}
-    ]
-After classifying the content and identifying key concepts, proceed to extract and list verifiable facts.
-
-1. **Definition of a Fact**: Ensure each fact is a standalone piece of information that is concrete and can be independently verified.
-2. **Selection Criteria**: Inlcude facts based on their significance to the content's main themes or concepts, their educational value and their foundational role in the subject.
-3. **Character Limit**: Maintain a limit of 90 characters for the  to ensure each message is concise yet informative.
-4. **Reference**: Every fact must include a reference. A reference can either be the entire source or a specific heading in the source. Whenever possible, pick a main heading to direct the user to the most relevant part of the source material. The heading must exactly match one of the headings provided to you. Sometimes facts may need to reference the entire text or multiple headings, leave the reference empty for such cases.
-
-List the facts in the following JSON format:
-
-json
-"facts": 
-    [
-        {
-            "fact_text": "fact_content",
-            "reference": "main_heading"
-        },
-        {...}
-    ]
-After you have examined the content —its field, its concepts, and its facts— determine whether it justifies the creation of quiz materials. 
-
-Consider whether these elements offer the average learner meaningful insights, practical uses, or serve important educational aims. If, in your judgment, the material falls short of providing such value, explain why in fewer than 90 characters.
-
-Reflect your in the JSON format as follows:
-
-json
-"generate_cards": 
-    {   state: true or false,
-        reason: "reason for marking the source as false. Leave empty for true."
-    }
-After analyzing the content, identifying key concepts, and facts, summarize the material using a series of engaging and informative cards. 
-
-These cards should capture the essence of the content while highlighting the critical concepts and facts that you previously identified.
-
-
-1. **Inclusion Criteria**: The generate_cards should be true. Return an empty array if the generate_cards is false.
-2. **Summarization Objective**: Each card is a step in a journey through the content. The series should collectively summarize the source while emphasizing important learning points.
-3. **Character Limit**: Maintain a limit of 320 characters per card to ensure each message is concise yet informative.
-4. **Card limit**: Limit the total number of cards to less than or equal to 8.
-4. **Engagement and Flow**: Write in an engaging style that maintains the user’s interest. Arrange the cards in a logical order that reflects the flow of the original content.
-
-Format your output in JSON as follows:
-
-json
-{
-    "summary_cards": ["summary_card1_content", "summary_card2_content", "summary_card3_content", "... up to 8 summary cards"]
-}
-
-
-
-`;
-
-
-
-
-
-export function returnTypologyPrompt() {
-    return typologyPromptString;
+const typologyPromptString = `
+You are a dedicated assistant that categorizes and summarizes educational content. You will process educational content (in JSON format) that represents text from diverse sources such as wikipedia, markdown notes, PDFs, book chapters, and websites.
+You will be provided with the following:
+1. Title of the source
+2. A list of main headings in the source
+3. The source content
+
+Perform the following tasks:
+
+1. Classify the content into one to three predefined fields of knowledge.
+2. Extract key concepts within the content. Be exhaustive and thorough.
+3. Extract concrete and relevant facts that are referenced in the content. Be exhaustive and thorough.
+4. Decide whether the content has any educational value and should be used to generate test material and quizzes based on the identified concepts and facts.
+5. If the generate_cards is true then summarize the content using a series of summary cards.
+
+Output your answer as valid JSON, in the form:
+json
+{
+    "field": ["primary_field", "secondary_field", "tertiary_field"],
+    "concepts": 
+    [
+        {
+            "concept_text": "concept_content",
+            "reference": "main_heading"
+        },
+        {...}
+    ],
+    "facts": 
+    [
+        {
+            "fact_text": "fact_content",
+            "reference": "main_heading"
+        },
+        {...}
+    ],
+    "generate_cards": [
+        state: true or false,
+        reason: "reason for marking the source as false. Leave empty for true."
+    ],
+    "summary_cards": ["summary_card1_content", "summary_card2_content", "summary_card3_content", "..."]
+}
+
+Further instruction on how to perform these tasks are below.
+Every source must be placed under a field. This is the broadest category of knowledge. A source should belong to at least one and at most 3 fields. Only include fields that a source is strongly associated with. The field names in your response must exactly match the names of 18 fields listed below.
+
+1.	Sciences: Focus on Biology, Chemistry, Physics, Astronomy, Mathematics, and Computer Science.
+2.	Technology & Engineering: Emphasize Information Technology, Engineering disciplines, AI, and Robotics.
+3.	Humanities & Cultural Studies: Highlight History, Literature, Languages, Arts, Philosophy, and Anthropological Studies.
+4.	Social Sciences & Global Studies: Include Sociology, Psychology, Economics, Political Science, Anthropology, and International Relations.
+5.	Business & Management: Encompass Entrepreneurship, Marketing, Finance, Leadership, and Ethics.
+6.	Health & Medicine: Cover Medical Sciences, Public Health, Nutrition, Wellness, and Mental Health.
+7.	Environmental Studies & Earth Sciences: Discuss Ecology, Climate Science, Geology, and Environmental Policy.
+8.	Education, Learning & Personal Development: Talk about Educational Theories, Teaching Methods, and Personal Skills.
+9.	Creative & Performing Arts: Include Visual Arts, Music, Theater, Dance, and Design Principles.
+10.	Law, Governance & Ethics: Focus on Legal Studies, Public Administration, Policy Analysis, and Ethical Decision-Making.
+11.	Recreation, Lifestyle & Practical Skills: Highlight Hobbies, Sports, Travel, Lifestyle Choices, and Practical Skills.
+12.	Technology & Media Literacy: Discuss Digital Literacy, Media Studies, and the Impact of Digital Media.
+13.	Philosophy & Critical Thinking: Emphasize Moral Philosophy, Ethical Frameworks, and Critical Thinking.
+14.	Space & Astronomical Sciences: Focus on Space Exploration, Astronomy, and Astrophysics.
+15.	Agriculture & Food Sciences: Discuss Sustainable Farming, Food Technology, and Nutrition.
+16.	Trades & Craftsmanship: Cover Hands-on Skills in Trades and Crafts.
+17.	Reference & Indexing: Include Summaries, Timelines, Directories, Glossaries, Bibliographies, and other Reference Material.
+18.	Other: Use for content that doesn’t fit into the above categories.
+Extract key concepts within the content after classifying the field. This is a crucial part of the exercise. Be exhaustive and thorough.
+
+1. **Definition of a Concept**: Concepts are fundamental ideas that form the basis of knowledge in any discipline. They help organize and explain information, making it accessible and relatable.
+2. **Inclusion Criteria**: Include a concept only if it is discussed in detail and is an important part of the subject matter of the source.
+3. **How to describe a concept**: The concept should be described so that a reader can comprehend the gist of it.
+4. **Character Limit**: Maintain a limit of 90 characters to ensure each concept is concise yet informative.
+5. **Reference**: Every concept must include a reference. A reference can either be the entire source or a specific heading in the source. Whenever possible, pick a main heading to direct the user to the most relevant part of the source material. The heading must exactly match one of the headings provided to you. Sometimes concepts may need to reference the entire text or multiple headings, leave the reference empty for such cases.
+
+List the concepts in the following JSON format:
+
+json
+"concepts": 
+    [
+        {
+            "concept_text": "concept_content",
+            "reference": "main_heading"
+        },
+        {...}
+    ]
+After classifying the content and identifying key concepts, proceed to extract and list verifiable facts.
+
+1. **Definition of a Fact**: Ensure each fact is a standalone piece of information that is concrete and can be independently verified.
+2. **Selection Criteria**: Inlcude facts based on their significance to the content's main themes or concepts, their educational value and their foundational role in the subject.
+3. **Character Limit**: Maintain a limit of 90 characters for the  to ensure each message is concise yet informative.
+4. **Reference**: Every fact must include a reference. A reference can either be the entire source or a specific heading in the source. Whenever possible, pick a main heading to direct the user to the most relevant part of the source material. The heading must exactly match one of the headings provided to you. Sometimes facts may need to reference the entire text or multiple headings, leave the reference empty for such cases.
+
+List the facts in the following JSON format:
+
+json
+"facts": 
+    [
+        {
+            "fact_text": "fact_content",
+            "reference": "main_heading"
+        },
+        {...}
+    ]
+After you have examined the content —its field, its concepts, and its facts— determine whether it justifies the creation of quiz materials. 
+
+Consider whether these elements offer the average learner meaningful insights, practical uses, or serve important educational aims. If, in your judgment, the material falls short of providing such value, explain why in fewer than 90 characters.
+
+Reflect your in the JSON format as follows:
+
+json
+"generate_cards": 
+    {   state: true or false,
+        reason: "reason for marking the source as false. Leave empty for true."
+    }
+After analyzing the content, identifying key concepts, and facts, summarize the material using a series of engaging and informative cards. 
+
+These cards should capture the essence of the content while highlighting the critical concepts and facts that you previously identified.
+
+
+1. **Inclusion Criteria**: The generate_cards should be true. Return an empty array if the generate_cards is false.
+2. **Summarization Objective**: Each card is a step in a journey through the content. The series should collectively summarize the source while emphasizing important learning points.
+3. **Character Limit**: Maintain a limit of 320 characters per card to ensure each message is concise yet informative.
+4. **Card limit**: Limit the total number of cards to less than or equal to 8.
+4. **Engagement and Flow**: Write in an engaging style that maintains the user’s interest. Arrange the cards in a logical order that reflects the flow of the original content.
+
+Format your output in JSON as follows:
+
+json
+{
+    "summary_cards": ["summary_card1_content", "summary_card2_content", "summary_card3_content", "... up to 8 summary cards"]
+}
+
+
+
+`;
+
+
+
+
+
+export function returnTypologyPrompt() {
+    return typologyPromptString;
 }

package/src/embedding_generation/consolidation/global_consolidation.ts

@@ -1,96 +1,96 @@
-import { getSimilarConcepts } from "../../helper/qdrant_db_methods";
-
-export class GlobalConsolidation {
-  async consolidate(
-    locally_consolidated_concepts_facts: {
-      text: string;
-      type: string;
-      reference: string;
-      id: string;
-      embedding: number[];
-    }[],
-    sourceId: string,
-    threshold: number
-  ) {
-    // / for all concepts_facts, find the ,most similar concepts_facts in qdrant, threshold of 0.8
-    const similarConcepts = await getSimilarConcepts(
-      "concepts-vectors",
-      locally_consolidated_concepts_facts,
-      threshold ?? 0.8
-    );
-    const taxonomyConcepts: {
-      text: string;
-      type: string;
-      reference: string;
-      id: string;
-    }[] = [];
-
-    const globalUpdatesOps: {
-      id: string;
-      sourceIdToAdd: string;
-      vector: number[];
-      currentPayload: {
-        _sources: string[];
-        text: string;
-      };
-    }[] = [];
-
-    const globalInsertsOps: {
-      id: string;
-      vector: number[];
-      payload: {
-        _sources: string[];
-        text: string;
-      };
-    }[] = [];
-    for (const index in similarConcepts) {
-      const points = similarConcepts[index].points;
-      const originalConcept = locally_consolidated_concepts_facts[index];
-      if (points.length == 0) {
-        taxonomyConcepts.push({
-          id: originalConcept.id,
-          text: originalConcept.text,
-          type: originalConcept.type,
-          reference: originalConcept.reference,
-        });
-        globalInsertsOps.push({
-          id: originalConcept.id,
-          vector: originalConcept.embedding,
-          payload: {
-            _sources: [sourceId],
-            text: originalConcept.text,
-          },
-        });
-      } else {
-        const consolidatedId = points[0].id;
-        const currentPayload = points[0].payload;
-        // const currentSources = currentPayload?['_sources'] ?? [];
-        taxonomyConcepts.push({
-          id: consolidatedId.toString(),
-          text: originalConcept.text,
-          type: originalConcept.type,
-          reference: originalConcept.reference,
-        });
-        globalUpdatesOps.push({
-          id: consolidatedId.toString(),
-          sourceIdToAdd: sourceId,
-          vector: originalConcept.embedding,
-          currentPayload: {
-            // _sources: currentPayload['_sources'] ?? [],
-            _sources: (currentPayload?._sources ?? []) as string[],
-            text: (currentPayload?.text ?? "").toString(),
-          },
-        });
-      }
-    }
-    return {
-      source_taxonomy: taxonomyConcepts,
-      global_updates: globalUpdatesOps,
-      global_inserts: globalInsertsOps,
-    };
-    /// if threshold is below 0.8, then add the concept_fact to qdrant
-    /// if threshold is above 0.8, then replace the id of the local concept_fact with the id of the global concept_fact
-    /// add _source.id to qdrant
-    /// return the concepts_facts
-  }
-}
+import { getSimilarConcepts } from "../../helper/qdrant_db_methods";
+
+export class GlobalConsolidation {
+  async consolidate(
+    locally_consolidated_concepts_facts: {
+      text: string;
+      type: string;
+      reference: string;
+      id: string;
+      embedding: number[];
+    }[],
+    sourceId: string,
+    threshold: number
+  ) {
+    // / for all concepts_facts, find the ,most similar concepts_facts in qdrant, threshold of 0.8
+    const similarConcepts = await getSimilarConcepts(
+      "concepts-vectors",
+      locally_consolidated_concepts_facts,
+      threshold ?? 0.8
+    );
+    const taxonomyConcepts: {
+      text: string;
+      type: string;
+      reference: string;
+      id: string;
+    }[] = [];
+
+    const globalUpdatesOps: {
+      id: string;
+      sourceIdToAdd: string;
+      vector: number[];
+      currentPayload: {
+        _sources: string[];
+        text: string;
+      };
+    }[] = [];
+
+    const globalInsertsOps: {
+      id: string;
+      vector: number[];
+      payload: {
+        _sources: string[];
+        text: string;
+      };
+    }[] = [];
+    for (const index in similarConcepts) {
+      const points = similarConcepts[index].points;
+      const originalConcept = locally_consolidated_concepts_facts[index];
+      if (points.length == 0) {
+        taxonomyConcepts.push({
+          id: originalConcept.id,
+          text: originalConcept.text,
+          type: originalConcept.type,
+          reference: originalConcept.reference,
+        });
+        globalInsertsOps.push({
+          id: originalConcept.id,
+          vector: originalConcept.embedding,
+          payload: {
+            _sources: [sourceId],
+            text: originalConcept.text,
+          },
+        });
+      } else {
+        const consolidatedId = points[0].id;
+        const currentPayload = points[0].payload;
+        // const currentSources = currentPayload?['_sources'] ?? [];
+        taxonomyConcepts.push({
+          id: consolidatedId.toString(),
+          text: originalConcept.text,
+          type: originalConcept.type,
+          reference: originalConcept.reference,
+        });
+        globalUpdatesOps.push({
+          id: consolidatedId.toString(),
+          sourceIdToAdd: sourceId,
+          vector: originalConcept.embedding,
+          currentPayload: {
+            // _sources: currentPayload['_sources'] ?? [],
+            _sources: (currentPayload?._sources ?? []) as string[],
+            text: (currentPayload?.text ?? "").toString(),
+          },
+        });
+      }
+    }
+    return {
+      source_taxonomy: taxonomyConcepts,
+      global_updates: globalUpdatesOps,
+      global_inserts: globalInsertsOps,
+    };
+    /// if threshold is below 0.8, then add the concept_fact to qdrant
+    /// if threshold is above 0.8, then replace the id of the local concept_fact with the id of the global concept_fact
+    /// add _source.id to qdrant
+    /// return the concepts_facts
+  }
+}