dev-laiser 0.2.0__py3-none-any.whl

dev_laiser-0.2.0.dist-info/LICENSE

@@ -0,0 +1,30 @@
+Copyright (c) 2025, LAiSER.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+       notice, this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above
+       copyright notice, this list of conditions and the following
+       disclaimer in the documentation and/or other materials provided
+       with the distribution.
+
+    * Neither the name of the LAiSER nor the names of any
+       contributors may be used to endorse or promote products derived
+       from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

dev_laiser-0.2.0.dist-info/METADATA

@@ -0,0 +1,147 @@
+Metadata-Version: 2.2
+Name: dev-laiser
+Version: 0.2.0
+Summary: LAiSER (Leveraging Artificial Intelligence for Skill Extraction & Research) is a tool designed to help learners, educators, and employers extract and share trusted information about skills. It uses a fine-tuned language model to extract raw skill keywords from text, then aligns them with a predefined taxonomy. You can find more technical details in the project’s paper.md and an overview in the README.md.
+Home-page: https://github.com/LAiSER-Software/extract-module
+Author: Satya Phanindra Kumar Kalaga, Bharat Khandelwal, Prudhvi Chekuri
+Author-email: phanindra.connect@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: psutil
+Requires-Dist: scikit_learn
+Requires-Dist: skillNer
+Requires-Dist: spacy
+Requires-Dist: transformers
+Requires-Dist: accelerate
+Requires-Dist: bitsandbytes
+Requires-Dist: datasets
+Requires-Dist: huggingface_hub
+Requires-Dist: peft
+Requires-Dist: torch
+Requires-Dist: trl
+Requires-Dist: ipython
+Requires-Dist: python-dotenv
+Requires-Dist: vllm
+Requires-Dist: tqdm
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+<div align="center">
+<img src="https://i.imgur.com/XznvjNi.png" width="70%"/>
+<h2>Leveraging ​Artificial ​Intelligence for ​Skill ​Extraction &​ Research (LAiSER)</h2>
+</div>
+
+### Contents
+LAiSER is a tool that helps learners, educators and employers share trusted and mutually intelligible information about skills​.
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Setup and Installation](#setup-and-installation)
+  - [i. Download the repository](#i-download-the-repository)
+  - [ii. Install the dependencies](#ii-install-the-dependencies)
+- [Usage](#usage)
+  - [Google Colab Setup(preferred)](#google-colab-setuppreferred)
+  - [Command Line Setup](#command-line-setup)
+- [Funding](#funding)
+- [Authors](#authors)
+- [Partners](#partners)
+<!-- - [Examples](#examples) -->
+- [Funding](#funding)
+- [Authors](#authors)
+- [Partners](#partners)
+
+## About
+## Requirements
+- Python version >= Python 3.12. 
+- A GPU with atelast 15GB video memory is essential for running this tool on large datasets.
+
+
+## Setup and Installation
+
+### i. Download the repository
+Before proceeding to  LAiSER, you'd want to follow the steps below to install the required dependencies:
+- Clone the repository using 
+  ```shell
+  git clone https://github.com/Micah-Sanders/LAiSER.git
+  ```
+  or download the [zip(link)](https://github.com/Micah-Sanders/LAiSER/archive/refs/heads/main.zip) file and extract it.
+
+### ii. Install the dependencies
+> [!NOTE]
+> If you intend to use the Jupyter Notebook interface, you can skip this step as the dependencies will be installed seperately in the Google Colab environment.
+
+Install the required dependencies using the command below:
+  ```shell
+    pip install -r requirements.txt
+```
+**NOTE**: Python 3.9 or later, *preferably 3.12*, is expected to be installed on your system. If you don't have Python installed, you can download it from [here](https://www.python.org/downloads/).
+
+
+## Usage
+
+As of now LAiSER can be used a command line tool or from the Jupyter notebook(Google Colab). The steps to setup the tool are as follows:
+
+### Google Colab Setup(preferred)
+LAiSER's Jupyter notebook is, currently, the fastest way to get started with the tool. You can access the notebook [here](https://github.com/LAiSER-Software/extract-module/blob/main/dev_space/Extract%20Function%20Colab%20Execution.ipynb).
+
+- Once the notebook is imported in google colaboratory, connect to a GPU-accelerated runtime(T4 GPU) and run the cells in the notebook.
+
+### Command Line Setup
+To use LAiSER as a command line tool, follow the steps below:
+
+- Navigate to the root directory of the repository and run the command below:
+  ```shell
+  pip install laiser-dev
+  ```
+
+- Once the installation is complete, you can run the tool using the command below:
+  <!-- TODO: add an example of importing and initiating the skillExtractor class -->
+  ```shell
+  
+  ```
+
+<!-- > [!CAUTION]
+> - If you encounter any `*.dll` file missing errors, make sure you downgrade the pytorch version to `2.2.2`.
+```shell
+pip install pytorch=2.2.2
+``` -->
+
+
+<!-- ## Examples -->
+
+
+## Funding
+<div align="center">
+<img src="https://i.imgur.com/XtgngBz.png" width="100px"/>
+<img src="https://i.imgur.com/a2SNYma.jpeg" width="130px"/>
+</div>
+
+## Authors
+<a href="https://github.com/LAiSER-Software/extract-module/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=LAiSER-Software/extract-module" />
+</a>
+
+## Partners
+<div align="center">
+<img src="https://i.imgur.com/hMb5n6T.png" width="120px"/>
+<img src="https://i.imgur.com/dxz2Udo.png" width="70px"/>
+<img src="https://i.imgur.com/5O1EuFU.png" width="100px"/>
+</div>
+
+
+
+</br>
+<!-- <p align='center'> <b> Made with Passion💖, Data Science📊, and a little magic!🪄 </b></p> -->

dev_laiser-0.2.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+laiser/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+laiser/llm_methods.py,sha256=6uMAdRwTDSxc0KIVlOIkN3sDs37EUcT-LJwaXp9qKwQ,15306
+laiser/params.py,sha256=FuCMi_GYYFUQ3e_SL-5S-ieGtXFXrTxh2w9wBIg6s4A,2350
+laiser/skill_extractor.py,sha256=wzonq0FQDw3bjTdxul2o8e1GD19wmYKR02dp4nr0T6g,15655
+laiser/utils.py,sha256=AU2ClhkbG2GXygKEOc-C96iX_K-IM21cA2zyWPWWDe0,3881
+dev_laiser-0.2.0.dist-info/LICENSE,sha256=yzvCtVuCrHO8861qLJbm1a2-XQmzh8VDXdcV-JXx1Pc,1546
+dev_laiser-0.2.0.dist-info/METADATA,sha256=rhYE1utXzJ5uHhtfvmGAKqpzAHAokxAVeGvNDDId5X8,5431
+dev_laiser-0.2.0.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
+dev_laiser-0.2.0.dist-info/top_level.txt,sha256=uiCW59yq2qCbyBOTlTtRRhtCMLjPM91nSoELBVJ4QB0,7
+dev_laiser-0.2.0.dist-info/RECORD,,

dev_laiser-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (75.8.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

dev_laiser-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+laiser

laiser/llm_methods.py

@@ -0,0 +1,382 @@
+"""
+Module Description:
+-------------------
+Class to extract skills from text and align them to existing taxonomy
+
+Ownership:
+----------
+Project: Leveraging Artificial intelligence for Skills Extraction and Research (LAiSER)
+Owner:  George Washington University Institute of Public Policy
+        Program on Skills, Credentials and Workforce Policy
+        Media and Public Affairs Building
+        805 21st Street NW
+        Washington, DC 20052
+        PSCWP@gwu.edu
+        https://gwipp.gwu.edu/program-skills-credentials-workforce-policy-pscwp
+
+License:
+--------
+Copyright 2024 George Washington University Institute of Public Policy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the “Software”), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Input Requirements:
+-------------------
+- All the libraries in the requirements.txt should be installed
+
+Output/Return Format:
+----------------------------
+- List of extracted skills from text
+
+"""
+"""
+Revision History:
+-----------------
+Rev No.     Date            Author              Description
+[1.0.0]     07/10/2024      Satya Phanindra K.  Define all the LLM methods being used in the project
+[1.0.1]     07/19/2024      Satya Phanindra K.  Add descriptions to each method
+[1.0.2]     11/24/2024      Prudhvi Chekuri     Add support for skills extraction from syllabi data
+[1.0.3]     11/25/2024      Satya Phanindra K.  Add support for skills extraction from course outcomes data
+
+TODO:
+-----
+
+"""
+
+import re
+import torch
+import numpy as np
+from trl import SFTTrainer
+from vllm import SamplingParams
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+
+torch.cuda.empty_cache()
+
+def fetch_model_output(response):
+    """
+    Format the model's output to extract the skill keywords from the get_completion() response
+    
+    Parameters
+    ----------
+    input_text : text
+        The model's response after processing the prompt. 
+        Contains special tags to identify the start and end of the model's response.
+        
+    Returns
+    -------
+    list: List of extracted skills from text
+    
+    """
+    # Find the content between the model start tag and the last <eos> tag
+    pattern = r'<start_of_turn>model\s*<eos>(.*?)<eos>\s*$'
+    match = re.search(pattern, response, re.DOTALL)
+
+    if match:
+        content = match.group(1).strip()
+
+        # Split the content by lines and filter out empty lines
+        lines = [line.strip() for line in content.split('\n') if line.strip()]
+
+        # Extract skills (lines starting with '-')
+        skills = [line[1:].strip() for line in lines if line.startswith('-')]
+
+        return skills
+
+def get_completion_batch(queries: list, model, tokenizer, batch_size=2) -> list:
+    """
+    Get completions for a list of queries using the model
+    
+    Parameters
+    ----------
+    queries : list
+        List of queries to get completions for using the model
+    model : model
+        The model to use for generating completions
+    tokenizer : tokenizer
+        The tokenizer to use for encoding the queries
+    batch_size : int, optional
+        Preferred batch size to use for generating completions
+        
+    Returns
+    -------
+    list: List of extracted skills from the text(s)
+    
+    """
+    
+    device = "cuda:0"
+    results = []
+
+    prompt_template = """
+    <start_of_turn>user
+    Name all the skills present in the following description in a single list. Response should be in English and have only the skills, no other information or words. Skills should be keywords, each being no more than 3 words.
+    Below text is the Description:
+
+    {query}
+    <end_of_turn>\n<start_of_turn>model
+    """
+
+    for i in range(0, len(queries), batch_size):
+        batch = queries[i:i+batch_size]
+        prompts = [prompt_template.format(query=query) for query in batch]
+
+        encodeds = tokenizer(prompts, return_tensors="pt", add_special_tokens=True, padding=True, truncation=True)
+        model_inputs = encodeds.to(device)
+
+        with torch.no_grad():
+            generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
+
+        decoded = tokenizer.batch_decode(generated_ids, skip_special_tokens=False)
+
+        for full_output in decoded:
+            # Extract only the model's response
+            response = full_output.split("<start_of_turn>model<eos>")[-1].strip()
+            processed_response = fetch_model_output(response)
+            results.append(processed_response)
+
+        # Clear CUDA cache after each batch
+        torch.cuda.empty_cache()
+
+        print(f"Processed batch {i//batch_size + 1}/{(len(queries)-1)//batch_size + 1}")
+
+    return results
+
+def get_completion(input_text, text_columns, input_type, model, tokenizer) -> str:
+    """
+    Get completion for a single query using the model
+    
+    Parameters
+    ----------
+    input_text : pandas Series with text data related to Job Description / Syllabus Description / Course Outcomes etc.
+        The query to get completions for using the model
+    text_columns : list
+        List of columns in the input_text dataframe that contain the text data. (Default: ['description'])
+    input_type : str
+        Type of input data - 'job_desc' / 'syllabus' etc. (Default: 'job_desc')
+    model : model
+        The model to use for generating completions
+    tokenizer : tokenizer
+        The tokenizer to use for encoding the queries
+        
+    Returns
+    -------
+    list: List of extracted skills from the text
+        
+    """
+    
+    device = "cuda:0"
+
+    if input_type == "job_desc":
+        prompt_template = """
+        <start_of_turn>user
+        Name all the skills present in the following description in a single list. Response should be in English and have only the skills, no other information or words. Skills should be keywords, each being no more than 3 words.
+        Below text is the Description:
+
+        {query}
+        <end_of_turn>\n<start_of_turn>model
+        """
+        prompt = prompt_template.format(query=input_text[text_columns[0]])
+    elif input_type == "syllabus":
+        prompt_template = """
+        <start_of_turn>user
+        Name all the skills present in the following course details in a single list. Response should be in English and have only the skills, no other information or words. Skills should be keywords, each being no more than 3 words.
+
+        Course Description:
+        {description}
+
+
+        Learning Outcomes:
+        {learning_outcomes}
+
+        <end_of_turn>
+        <start_of_turn>model
+        """
+        prompt = prompt_template.format(description=input_text[text_columns[0]], learning_outcomes=input_text[text_columns[1]])
+
+    encodeds = tokenizer(prompt, return_tensors="pt", add_special_tokens=True)
+
+    model_inputs = encodeds.to(device)
+    
+    with torch.no_grad():
+        generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
+
+
+    generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
+    decoded = tokenizer.decode(generated_ids[0], skip_special_tokens=False)
+    response = decoded.split("<start_of_turn>model<eos>")[-1].strip()
+    processed_response = fetch_model_output(response)
+    return (processed_response)
+
+
+def parse_output_vllm(response):
+    # TODO: Verify the docstring and update missing/incorrect information
+    """
+    Parse the output from the VLLM model to extract skills, levels, knowledge required, and task abilities.
+    
+    Parameters
+    ----------
+    response : str
+        The model's response containing the structured information about skills.
+        
+    Returns
+    -------
+    list: List of dictionaries containing the extracted skills, levels, knowledge required, and task abilities.
+    """
+        
+    out = []
+    # Split into items, handling optional '->' prefix and multi-line input
+    items = [item.strip() for item in response.split('->') if item.strip()]
+
+    for item in items:
+        skill_data = {}
+
+        # Extract skill
+        skill_match = re.search(r"Skill:\s*([^,\n]+)", item)
+        if skill_match:
+            skill_data['Skill'] = skill_match.group(1).strip()
+
+        # Extract level
+        level_match = re.search(r"Level:\s*(\d+)", item)
+        if level_match:
+            skill_data['Level'] = int(level_match.group(1).strip())
+
+        # Extract knowledge required (multi-line support with re.DOTALL)
+        knowledge_match = re.search(r"Knowledge Required:\s*(.*?)(?=\s*Task Abilities:|\s*$)", item, re.DOTALL)
+        if knowledge_match:
+            knowledge_raw = knowledge_match.group(1).strip()
+            skill_data['Knowledge Required'] = [k.strip() for k in knowledge_raw.split(',') if k.strip()]
+
+        # Extract task abilities (multi-line support with re.DOTALL)
+        task_match = re.search(r"Task Abilities:\s*(.*?)(?=\s*$)", item, re.DOTALL)
+        if task_match:
+            task_raw = task_match.group(1).strip()
+            skill_data['Task Abilities'] = [t.strip() for t in task_raw.split(',') if t.strip()]
+
+        out.append(skill_data)
+
+    return out
+
+
+def create_ksa_prompt(query, input_type, num_key_skills, num_key_kr, num_key_tas):
+    # TODO: Verify the docstring and update missing/incorrect information
+    """
+    Create a structured prompt for the KSA (Knowledge, Skills, Abilities) extraction task.
+    
+    Parameters
+    ----------
+    query : dict
+        A dictionary containing the input data, including 'description' and optionally 'learning_outcomes'.
+    input_type : str
+        The type of input data - 'job_desc' or 'syllabi'.
+    num_key_skills : int
+        The number of key skills to extract.
+    num_key_kr : str
+        The number of key knowledge areas to extract (e.g., '3-5').
+    num_key_tas : str
+        The number of key task abilities to extract (e.g., '3-5').
+    Returns
+    ------- 
+    str
+        The formatted prompt for the KSA extraction task.       
+    """
+
+    prompt_template = """user
+**Objective:** Given a {input_desc}, complete the following tasks with structured outputs.
+
+### Tasks:
+1. **Skills Extraction:** Identify {num_key_skills} key skills mentioned in the {input_desc}.
+  - Extract skill keywords or phrases of no more than three words.
+
+2. **Skill Level Assignment:** Assign a proficiency level to each extracted skill based on the SCQF Level Descriptors (see below).
+
+3. **Knowledge Required:** For each skill, list {num_key_kr} broad areas of understanding or expertise necessary to develop the skill.
+
+4. **Task Abilities:** For each skill, list {num_key_tas} general tasks or capabilities enabled by the skill.
+
+### Guidelines:
+- **Skill Extraction:** Identify skills explicitly stated or implied through {input_desc}.
+- **Skill Level Assignment:** Use the SCQF Level Descriptors to classify proficiency:
+  - 1: Basic awareness of simple concepts.
+  - 2: Limited operational understanding, guided application.
+  - 3: Moderate knowledge, supervised application of techniques.
+  - 4: Clear understanding, independent work in familiar contexts.
+  - 5: Advanced knowledge, autonomous problem-solving.
+  - 6: Specialized knowledge, critical analysis within defined areas.
+  - 7: Advanced specialization, leadership in problem-solving.
+  - 8: Expert knowledge, innovation in complex contexts.
+  - 9: Highly specialized expertise, contributing original thought.
+  - 10: Sustained mastery, influential in areas of specialization.
+  - 11: Groundbreaking innovation, professional or academic mastery.
+  - 12: Global expertise, leading advancements at the highest level.
+
+- **Knowledge and Task Abilities:**
+  - **Knowledge Required:** Broad areas, e.g., "data visualization techniques."
+  - **Task Abilities:** General tasks or capabilities, e.g., "data analysis."
+  - Each item in these two lists should be no more than three words.
+  - Avoid overly specific or vague terms.
+
+### Answer Format:
+- Use this format strictly in the response:
+  -> Skill: [Skill Name], Level: [1–12], Knowledge Required: [list], Task Abilities: [list].
+
+{input_text}
+
+**Response:** Provide only the requested structured information without additional explanations.
+
+
+model
+"""
+
+
+    input_desc = "job description" if input_type == "syllabi" else "course syllabus description and its learning outcomes"
+    
+    if input_type == "syllabi":
+        input_text = f"""### Input:\n**Course Description:** {query["description"]}\n**Learning Outcomes:** {query["learning_outcomes"]}"""
+    else:
+        input_text = f"""### Input:\n{query["description"]}"""
+        
+    prompt = prompt_template.format(input_desc=input_desc, num_key_skills=num_key_skills, num_key_kr=num_key_kr, num_key_tas=num_key_tas, input_text=input_text)
+    return prompt
+
+
+def vllm_batch_generate(llm, queries, input_type, batch_size=32, num_key_skills=5, num_key_kr='3-5', num_key_tas='3-5'):
+
+    result = []
+
+    sampling_params = SamplingParams(max_tokens=1000)
+
+    for i in range(0, len(queries), batch_size):
+        prompts = [create_ksa_prompt(queries.iloc[i], input_type, num_key_skills, num_key_kr, num_key_tas) for i in range(i, min(i+batch_size, len(queries)))]
+        output = llm.generate(prompts, sampling_params=sampling_params)
+
+        result.extend(output)
+
+    return result
+
+
+def get_completion_vllm(input_text, text_columns, id_column, input_type, llm, batch_size=4) -> list:
+
+    result = vllm_batch_generate(llm, input_text, input_type=input_type, batch_size=batch_size)
+    
+    parsed_output = []
+    for i in range(len(result)):
+        parsed = parse_output_vllm(result[i].outputs[0].text)
+        for item in parsed:
+            item[id_column] = input_text.iloc[i][id_column]
+            item['description'] = input_text.iloc[i]['description']
+            if 'learning_outcomes' in input_text.columns:
+                item['learning_outcomes'] = input_text.iloc[i]['learning_outcomes']
+        parsed_output.extend(parsed)
+
+    return parsed_output

laiser/params.py

@@ -0,0 +1,60 @@
+
+"""
+Module Description:
+-------------------
+A python file with global constants
+
+Ownership:
+----------
+Project: Leveraging Artificial intelligence for Skills Extraction and Research (LAiSER)
+Owner:  George Washington University Institute of Public Policy
+        Program on Skills, Credentials and Workforce Policy
+        Media and Public Affairs Building
+        805 21st Street NW
+        Washington, DC 20052
+        PSCWP@gwu.edu
+        https://gwipp.gwu.edu/program-skills-credentials-workforce-policy-pscwp
+
+License:
+--------
+Copyright 2024 George Washington University Institute of Public Policy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the “Software”), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+"""
+"""
+Revision History
+-----------------
+Rev No.     Date            Author                Description
+[1.0.0]     06/01/2024      Vedant M.             Initial Version
+[1.0.1]     06/10/2024      Vedant M.             added paths for input and output
+[1.0.2]     07/01/2024      Satya Phanindra K.    updated threshold for similarity and AI model ID
+
+
+TODO:
+-----
+- 1:
+"""
+
+import os
+from dotenv import load_dotenv
+
+ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
+INPUT_PATH = os.path.join(ROOT_DIR, 'input')
+OUTPUT_PATH = os.path.join(ROOT_DIR, 'output')
+
+SKILL_DB_PATH = os.path.join(INPUT_PATH, 'combined.csv')
+
+
+SIMILARITY_THRESHOLD = 0.85

laiser/skill_extractor.py

@@ -0,0 +1,367 @@
+"""
+Module Description:
+-------------------
+Class to extract skills from text and align them to existing taxonomy
+
+Ownership:
+----------
+Project: Leveraging Artificial intelligence for Skills Extraction and Research (LAiSER)
+Owner:  George Washington University Institute of Public Policy
+        Program on Skills, Credentials and Workforce Policy
+        Media and Public Affairs Building
+        805 21st Street NW
+        Washington, DC 20052
+        PSCWP@gwu.edu
+        https://gwipp.gwu.edu/program-skills-credentials-workforce-policy-pscwp
+
+License:
+--------
+Copyright 2024 George Washington University Institute of Public Policy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the “Software”), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Input Requirements:
+-------------------
+- Pandas Dataframe with ID and Text Column
+
+Output/Return Format:
+----------------------------
+- Pandas dataframe with below columns:
+    - "Research ID": text_id
+    - "Skill Name": Raw skill extracted,
+    - "Skill Tag": skill tag from taxonomy,
+    - "Correlation Coefficient": similarity_score
+
+
+"""
+"""
+Revision History:
+-----------------
+Rev No.     Date            Author              Description
+[1.0.0]     05/30/2024      Vedant M.           Initial Version
+[1.0.1]     06/01/2024      Vedant M.           Referencing utils.py and params.py
+[1.0.2]     06/08/2024      Satya Phanindra K.  Modify get_aligned_skills function to JSON output
+[1.0.3]     06/10/2024      Vedant M.           Updated functions extract_raw and align_skills for input and output
+[1.0.4]     06/13/2024      Vedant M.           Added function extractor to encapsulate both functions
+[1.0.5]     06/15/2024      Satya Phanindra K.  Replaced OpenAI API with HuggingFace API for skill extraction
+[1.0.6]     06/20/2024      Satya Phanindra K.  Added function to extract skills from text using Fine-Tuned Language Model's API
+[1.0.7]     07/03/2024      Satya Phanindra K.  Added CONDITIONAL GPU support for Fine-Tuned Language Model and error handling
+[1.0.9]     07/08/2024      Satya Phanindra K.  Added support for SkillNer model for skill extraction, if GPU not available
+[1.0.8]     07/11/2024      Satya Phanindra K.  Calculate cosine similarities in bulk for optimal performance.
+[1.0.9]     07/15/2024      Satya Phanindra K.  Error handling for empty list outputs from extract_raw function
+[1.0.10]    11/24/2024      Prudhvi Chekuri     Added support for skills extraction from syllabi data
+[1.0.11]    03/12/2025      Satya Phanindra K.  Update extractor function to handle syllabus data
+
+
+TODO:
+-----
+- 1: Add references to utils and global parameter file
+- 2: sort taxonomy inputs
+- 3: include rsd_name instead of keywords from osn
+- 4: Optimize the `align_skills` function.
+"""
+
+# native packages
+import sys
+import os
+
+# installed packages
+import spacy
+import torch
+import numpy as np
+import pandas as pd
+from vllm import LLM
+from tqdm.auto import tqdm
+from spacy.matcher import PhraseMatcher
+from skillNer.general_params import SKILL_DB
+from sklearn.metrics.pairwise import cosine_similarity
+from skillNer.skill_extractor_class import SkillExtractor
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+from scipy.spatial.distance import cdist
+
+
+# internal packages
+from laiser.utils import get_embedding, cosine_similarity
+from laiser.params import SIMILARITY_THRESHOLD, SKILL_DB_PATH
+from laiser.llm_methods import get_completion, get_completion_vllm
+
+
+class Skill_Extractor:
+    """
+    Class to extract skills from text and align them to existing taxonomy
+    ...
+
+    Attributes
+    ----------
+    client : HuggingFace API client
+    nlp : spacy nlp model
+    
+    Methods
+    -------
+    extract_raw(input_text: text)
+        The function extracts skills from text using NER model
+
+    align_skills(raw_skills: list, document_id='0': string):
+        This function aligns the skills provided to the desired taxonomy
+        
+    align_KSAs(extracted_df: pandas dataframe, id_column='Research ID'):
+        This function aligns the skills provided to the desired taxonomy
+        
+    extractor(data: pandas dataframe, id_column='Research ID', text_column='Text'):
+        Function takes text dataset to extract and aligns skills based on available taxonomies
+    ....
+
+    """
+
+    def __init__(self, AI_MODEL_ID, HF_TOKEN, use_gpu):
+        self.model_id = AI_MODEL_ID
+        self.HF_TOKEN=HF_TOKEN
+        self.use_gpu=use_gpu
+        self.nlp = spacy.load("en_core_web_lg")
+        self.skill_db_df = pd.read_csv(SKILL_DB_PATH)
+        self.skill_db_embeddings = np.array([get_embedding(self.nlp, label) for label in self.skill_db_df['SkillLabel']])
+        if torch.cuda.is_available() and self.use_gpu:
+            print("GPU is available. Using GPU for Large Language model initialization...")
+            torch.cuda.empty_cache()
+            
+            # Use quantization to reduce the model size and memory usage
+            if self.model_id:
+                self.llm = LLM(model=self.model_id, dtype="float16", quantization='gptq')
+            else:
+                self.llm = LLM(model="marcsun13/gemma-2-9b-it-GPTQ", dtype="float16", quantization='gptq')              
+        else:
+            print("GPU is not available. Using CPU for SkillNer model initialization...")
+            self.ner_extractor = SkillExtractor(self.nlp, SKILL_DB, PhraseMatcher)
+        return
+
+
+    # Declaring a private method for extracting raw skills from input text
+    def extract_raw(self, input_text, text_columns, id_column, input_type):
+        """
+        The function extracts skills from text using Fine-Tuned Language Model's API
+
+        Parameters
+        ----------
+        input_text : pandas Series with text data
+            Job advertisement / Job Description / Syllabus Description / Course Outcomes etc.
+        text_columns: list
+            Name of the text columns in the dataset. Defaults to 'description'
+        input_type: string
+            Type of input data. Defaults to 'job_desc'
+
+        Returns
+        -------
+        list: List of extracted skills from text
+
+        Notes
+        -----
+        More details on which (pre-trained) language model is fine-tuned can be found in llm_methods.py
+        The Function is designed only to return list of skills based on prompt passed to OpenAI's Fine-tuned model.
+
+        """    
+        
+        if torch.cuda.is_available() and self.use_gpu:
+            # GPU is available. Using Language model for extraction.
+            extracted_skills_set = get_completion_vllm(input_text, text_columns, id_column, input_type, self.llm)
+            torch.cuda.empty_cache()
+        else: 
+            # GPU is not available. Using SkillNer model for extraction.
+            ner_extractor = self.ner_extractor
+            extracted_skills_set = set()
+            annotations = None
+            try:
+                
+                if input_type == "job_desc":
+                  input_text = input_text[text_columns][0]
+                elif input_type == "syllabus":
+                  input_text = f"Course Description: {input_text[text_columns[0]]}\n\nLearning Outcomes: {input_text[text_columns[1]]}"
+                
+                annotations = ner_extractor.annotate(input_text)
+            except ValueError as e:
+                print(f"Skipping example, ValueError encountered: {e}")
+            except Exception as e:
+                print(f"Skipping example, An unexpected error occurred: {e}")
+
+            for item in annotations['results']['full_matches']:
+                extracted_skills_set.add(item['doc_node_value'])
+
+            # get ngram_scored
+            for item in annotations['results']['ngram_scored']:
+                extracted_skills_set.add(item['doc_node_value'])
+
+        return list(extracted_skills_set)
+            
+
+    def align_skills(self, raw_skills, document_id='0'):
+        """
+        This function aligns the skills provided to the available taxonomy
+
+        Parameters
+        ----------
+        raw_skills : list
+            Provide list of skill extracted from Job Descriptions / Syllabus.
+
+        Returns
+        -------
+        list: List of taxonomy skills from text in JSON format
+            [
+                {
+                    "Research ID": text_id,
+                    "Skill Name": Raw skill extracted,
+                    "Skill Tag": taxonomy skill tag,
+                    "Correlation Coefficient": similarity_score
+                },
+                ...
+            ]
+
+        """
+        raw_skill_embeddings = np.array([get_embedding(self.nlp, skill) for skill in raw_skills])
+
+        # Calculate cosine similarities in bulk
+        similarities = 1 - cdist(raw_skill_embeddings, self.skill_db_embeddings, metric='cosine')
+
+        matches = []
+        for i, raw_skill in enumerate(raw_skills):
+            skill_matches = np.where(similarities[i] > SIMILARITY_THRESHOLD)[0]
+            for match in skill_matches:
+                matches.append({
+                    "Research ID": document_id,
+                    "Raw Skill": raw_skill,
+                    "Skill Tag": self.skill_db_df.iloc[match]['SkillTag'],
+                    "Correlation Coefficient": similarities[i, match]
+                })
+
+        return matches
+
+
+    def align_KSAs(self, extracted_df, id_column):
+        """
+        This function aligns the skills provided to the available taxonomy
+        
+        Parameters
+        ----------
+        extracted_df : pandas dataframe
+            Provide dataframe of skills extracted from Job Descriptions / Syllabus.
+        id_column: string
+            Name of id column in the dataset. Defaults to 'Research ID'
+            
+        Returns
+        -------
+        list: List of taxonomy skills from text in JSON format
+
+            [
+                {
+                    "Research ID": text_id,
+                    "Description": String, (optional)
+                    "Learning Outcomes": List of Strings, (optional)
+                    "Raw Skill": String,
+                    "Level": int, (optional)
+                    "Knowledge Required": String, (optional)
+                    "Task Abilities": String, (optional)
+                    "Skill Tag": String,
+                    "Correlation Coefficient": float
+                },
+                ...
+            ]
+        """
+        
+        matches = []
+
+        raw_skill_embeddings = np.array([get_embedding(self.nlp, skill) for skill in extracted_df['Skill']])
+
+        # Calculate cosine similarities in bulk
+        similarities = 1 - cdist(raw_skill_embeddings, self.skill_db_embeddings, metric='cosine')
+
+        for i, raw_skill in tqdm(enumerate(extracted_df['Skill'])):
+            skill_matches = np.where(similarities[i] > SIMILARITY_THRESHOLD)[0]
+            for match in skill_matches:
+                matches.append({
+                    "Research ID": extracted_df.iloc[i][id_column],
+                    "Description": extracted_df.iloc[i]['description'],
+                    "Learning Outcomes": extracted_df.iloc[i]['learning_outcomes'],
+                    "Raw Skill": raw_skill,
+                    "Level": extracted_df.iloc[i]['Level'],
+                    "Knowledge Required": extracted_df.iloc[i]['Knowledge Required'],
+                    "Task Abilities": extracted_df.iloc[i]['Task Abilities'],
+                    "Skill Tag": f"ESCO.{match}",
+                    "Correlation Coefficient": similarities[i, match]
+                })
+
+        return matches
+
+
+    def extractor(self, data, id_column='Research ID', text_columns=["description"], input_type="job_desc"):
+        """
+        Function takes text dataset to extract and aligns skills based on available taxonomies
+
+        Parameters
+        ----------
+        data : pandas dataframe
+            Dataset containing text id and actual text to extract skills.
+        id_column: string
+            Name of id column in the dataset. Defaults to 'Research ID'
+        text_columns: list
+            Name of the text columns in the dataset. Defaults to 'description'
+        input_type: string
+            Type of input data. Defaults to 'job_desc'
+
+        Returns
+        -------
+        list: List of skill tags and similarity_score for all texts in  from text in JSON format
+            [
+                {
+                    "Research ID": text_id,
+                    "Skill Tag": String, 
+                    "Description": String, (optional)
+                    "Learning Outcomes": List of Strings, (optional)
+                    "Raw Skill": String,
+                    "Level": int, (optional)
+                    "Knowledge Required": String, (optional)
+                    "Task Abilities": String, (optional)
+                    "Correlation Coefficient": float
+                },
+                ...
+            ]
+
+        """
+        
+        if torch.cuda.is_available() and self.use_gpu:
+            KSAs = self.extract_raw(data, text_columns, id_column, input_type)
+        
+            extracted_df = pd.DataFrame(KSAs)
+            if input_type != 'syllabus':
+                extracted_df["learning_outcomes"] = ''
+
+            extracted_df = extracted_df[[id_column, "description", "learning_outcomes", "Skill", "Level", "Knowledge Required", "Task Abilities"]]
+            matches = self.align_KSAs(extracted_df, id_column)
+            
+            extracted = pd.DataFrame(columns=['Research ID', 'Description', 'Learning Outcomes', 'Raw Skill', 'Level', 'Knowledge Required', 'Task Abilities', 'Skill Tag', 'Correlation Coefficient'])
+            extracted = extracted._append(matches, ignore_index=True)
+
+            if input_type != "syllabus":
+                extracted.drop("Learning Outcomes", axis=1, inplace=True)
+
+        else:
+            extracted = pd.DataFrame(columns=['Research ID', 'Raw Skill', 'Skill Tag', 'Correlation Coefficient'])
+            for _, row in data.iterrows():
+                research_id = row[id_column]
+                raw_skills = self.extract_raw(row, text_columns, id_column, input_type)
+                if(len(raw_skills) == 0):
+                    continue
+                else:
+                    aligned_skills = self.align_skills(raw_skills, research_id)
+                    extracted = extracted._append(aligned_skills, ignore_index=True)
+
+        return extracted

laiser/utils.py

@@ -0,0 +1,122 @@
+"""
+Module Description:
+-------------------
+A Class with utility functions
+
+Ownership:
+----------
+Project: Leveraging Artificial intelligence for Skills Extraction and Research (LAiSER)
+Owner:  George Washington University Institute of Public Policy
+        Program on Skills, Credentials and Workforce Policy
+        Media and Public Affairs Building
+        805 21st Street NW
+        Washington, DC 20052
+        PSCWP@gwu.edu
+        https://gwipp.gwu.edu/program-skills-credentials-workforce-policy-pscwp
+
+License:
+--------
+Copyright 2024 George Washington University Institute of Public Policy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the “Software”), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+"""
+"""
+Revision History:
+-----------------
+Rev No.     Date            Author              Description
+[1.0.0]     06/01/2024      Vedant M.           Initial Version
+[1.0.1]     06/10/2024      Vedant M.           added logging function
+
+
+TODO:
+-----
+- 1:
+"""
+
+import numpy as np
+import psutil
+import logging
+
+
+def cosine_similarity(vec1, vec2):
+    """
+    Calculates cosine similarity between 2 vectors
+
+    Parameters
+    ----------
+    vec1, vec2 : numpy array of vectorized text
+
+    Returns
+    -------
+    numeric value
+    """
+    product_of_magnitude = np.linalg.norm(vec1) * np.linalg.norm(vec2)
+    if product_of_magnitude == 0.0:
+        return 0.0
+    return np.dot(vec1, vec2) / product_of_magnitude
+
+
+def get_embedding(nlp, input_text):
+    """
+    Creates vector embeddings for input text based on nlp object
+
+    Parameters
+    ----------
+    nlp : object of spacy nlp model
+    input_text : text
+        Provide text to be vectorized, usually skill, extracted of referenced
+
+    Returns
+    -------
+    numpy array of vectorized text
+
+
+    """
+    doc = nlp(input_text)
+    if len(doc) == 0:
+        return np.zeros(300)  # Return zeros for empty texts
+    return np.mean([word.vector for word in doc], axis=0)
+
+
+def log_performance(function_name, start_time, end_time):
+    """
+    Utility function to log performance in unit of time for a function
+
+    Parameters
+    ----------
+    function_name : text
+        Name of the function
+    start_time : time
+        execution start time of the function
+    end_time : time
+        execution end time of the function
+
+    """
+    execution_time = end_time - start_time
+    process = psutil.Process()
+    cpu_percent = process.cpu_percent()
+    memory_info = process.memory_info()
+    memory_usage = memory_info.rss / (1024 ** 2)  # Convert to MB
+
+    log_message = (
+        f"Function: {function_name}\n"
+        f"Execution time: {execution_time:.2f} seconds\n"
+        f"CPU usage: {cpu_percent:.2f}%\n"
+        f"Memory usage: {memory_usage:.2f} MB\n"
+        "-------------------------------"
+    )
+    logging.info(log_message)
+    print(log_message)