SinaTools 0.1.26__tar.gz → 0.1.28__tar.gz

SinaTools-0.1.28/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.1
+Name: SinaTools
+Version: 0.1.28
+Summary: Open-source Python toolkit for Arabic Natural Understanding, allowing people to integrate it in their system workflow.
+Home-page: https://github.com/SinaLab/sinatools
+License: MIT license
+Description: SinaTools
+        ======================
+        Open Source Toolkit for Arabic NLP and NLU developed by [SinaLab](http://sina.birzeit.edu/) at Birzeit University. SinaTools is available through Python APIs, command lines, colabs, and online demos.
+        
+        See the full list of [Available Packages](https://sina.birzeit.edu/sinatools/), which include: (1) [Morphology Tagging](https://sina.birzeit.edu/sinatools/index.html#morph), (2) [Named Entity Recognition (NER)](https://sina.birzeit.edu/sinatools/index.html#ner), (3) [Word Sense Disambiguation (WSD)](https://sina.birzeit.edu/sinatools/index.html#wsd), (4) [Semantic Relatedness](https://sina.birzeit.edu/sinatools/index.html#sr), (5) [Synonymy Extraction and Evaluation](https://sina.birzeit.edu/sinatools/index.html#se), (6) [Relation Extraction](https://sina.birzeit.edu/sinatools/index.html#re), (7) [Utilities](https://sina.birzeit.edu/sinatools/index.html#u) (diacritic-based word matching, Jaccard similarly, parser, tokenizers, corpora processing, transliteration, etc).
+        
+        See [Demo Pages](https://sina.birzeit.edu/sinatools/).
+        
+        See the [benchmarking](https://www.jarrar.info/publications/HJK24.pdf), which shows that SinaTools outperformed all related toolkits. 
+        
+        Installation 
+        --------
+        To install SinaTools, ensure you are using Python version 3.10.8, then clone the [GitHub](git://github.com/SinaLab/SinaTools) repository.
+        
+        Alternatively, you can execute the following command:
+        
+        ```bash
+        pip install sinatools
+        ```
+        
+        Installing Models and Data Files
+        --------
+        Some modules in SinaTools require some data files and fine-tuned models to be downloaded. To download these models, please consult the [DataDownload](https://sina.birzeit.edu/sinatools/documentation/cli_tools/DataDownload/DataDownload.html).
+        
+        Documentation
+        --------
+        For information, please refer to the [main page](https://sina.birzeit.edu/sinatools) or the [online domuementation](https://sina.birzeit.edu/sinatools/documentation). 
+        
+        Citation
+        -------
+        Tymaa Hammouda, Mustafa Jarrar, Mohammed Khalilia: [SinaTools: Open Source Toolkit for Arabic Natural Language Understanding](http://www.jarrar.info/publications/HJK24.pdf). In Proceedings of the 2024 AI in Computational Linguistics (ACLing 2024), Procedia Computer Science, Dubai. ELSEVIER.
+        
+        License
+        --------
+        SinaTools is available under the MIT License. See the [LICENSE](https://github.com/SinaLab/sinatools/blob/main/LICENSE) file for more information.
+        
+        Reporting Issues
+        --------
+        To report any issues or bugs, please contact us at "sina.institute.bzu@gmail.com" or visit [SinaTools Issues](https://github.com/SinaLab/sinatools/issues).
+        
+        
+Keywords: sinatools
+Platform: UNKNOWN
+Description-Content-Type: text/markdown

SinaTools-0.1.28/README.rst

@@ -0,0 +1,39 @@
+SinaTools
+======================
+Open Source Toolkit for Arabic NLP and NLU developed by [SinaLab](http://sina.birzeit.edu/) at Birzeit University. SinaTools is available through Python APIs, command lines, colabs, and online demos.
+
+See the full list of [Available Packages](https://sina.birzeit.edu/sinatools/), which include: (1) [Morphology Tagging](https://sina.birzeit.edu/sinatools/index.html#morph), (2) [Named Entity Recognition (NER)](https://sina.birzeit.edu/sinatools/index.html#ner), (3) [Word Sense Disambiguation (WSD)](https://sina.birzeit.edu/sinatools/index.html#wsd), (4) [Semantic Relatedness](https://sina.birzeit.edu/sinatools/index.html#sr), (5) [Synonymy Extraction and Evaluation](https://sina.birzeit.edu/sinatools/index.html#se), (6) [Relation Extraction](https://sina.birzeit.edu/sinatools/index.html#re), (7) [Utilities](https://sina.birzeit.edu/sinatools/index.html#u) (diacritic-based word matching, Jaccard similarly, parser, tokenizers, corpora processing, transliteration, etc).
+
+See [Demo Pages](https://sina.birzeit.edu/sinatools/).
+
+See the [benchmarking](https://www.jarrar.info/publications/HJK24.pdf), which shows that SinaTools outperformed all related toolkits. 
+
+Installation 
+--------
+To install SinaTools, ensure you are using Python version 3.10.8, then clone the [GitHub](git://github.com/SinaLab/SinaTools) repository.
+
+Alternatively, you can execute the following command:
+
+```bash
+pip install sinatools
+```
+
+Installing Models and Data Files
+--------
+Some modules in SinaTools require some data files and fine-tuned models to be downloaded. To download these models, please consult the [DataDownload](https://sina.birzeit.edu/sinatools/documentation/cli_tools/DataDownload/DataDownload.html).
+
+Documentation
+--------
+For information, please refer to the [main page](https://sina.birzeit.edu/sinatools) or the [online domuementation](https://sina.birzeit.edu/sinatools/documentation). 
+
+Citation
+-------
+Tymaa Hammouda, Mustafa Jarrar, Mohammed Khalilia: [SinaTools: Open Source Toolkit for Arabic Natural Language Understanding](http://www.jarrar.info/publications/HJK24.pdf). In Proceedings of the 2024 AI in Computational Linguistics (ACLing 2024), Procedia Computer Science, Dubai. ELSEVIER.
+
+License
+--------
+SinaTools is available under the MIT License. See the [LICENSE](https://github.com/SinaLab/sinatools/blob/main/LICENSE) file for more information.
+
+Reporting Issues
+--------
+To report any issues or bugs, please contact us at "sina.institute.bzu@gmail.com" or visit [SinaTools Issues](https://github.com/SinaLab/sinatools/issues).

SinaTools-0.1.28/SinaTools.egg-info/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.1
+Name: SinaTools
+Version: 0.1.28
+Summary: Open-source Python toolkit for Arabic Natural Understanding, allowing people to integrate it in their system workflow.
+Home-page: https://github.com/SinaLab/sinatools
+License: MIT license
+Description: SinaTools
+        ======================
+        Open Source Toolkit for Arabic NLP and NLU developed by [SinaLab](http://sina.birzeit.edu/) at Birzeit University. SinaTools is available through Python APIs, command lines, colabs, and online demos.
+        
+        See the full list of [Available Packages](https://sina.birzeit.edu/sinatools/), which include: (1) [Morphology Tagging](https://sina.birzeit.edu/sinatools/index.html#morph), (2) [Named Entity Recognition (NER)](https://sina.birzeit.edu/sinatools/index.html#ner), (3) [Word Sense Disambiguation (WSD)](https://sina.birzeit.edu/sinatools/index.html#wsd), (4) [Semantic Relatedness](https://sina.birzeit.edu/sinatools/index.html#sr), (5) [Synonymy Extraction and Evaluation](https://sina.birzeit.edu/sinatools/index.html#se), (6) [Relation Extraction](https://sina.birzeit.edu/sinatools/index.html#re), (7) [Utilities](https://sina.birzeit.edu/sinatools/index.html#u) (diacritic-based word matching, Jaccard similarly, parser, tokenizers, corpora processing, transliteration, etc).
+        
+        See [Demo Pages](https://sina.birzeit.edu/sinatools/).
+        
+        See the [benchmarking](https://www.jarrar.info/publications/HJK24.pdf), which shows that SinaTools outperformed all related toolkits. 
+        
+        Installation 
+        --------
+        To install SinaTools, ensure you are using Python version 3.10.8, then clone the [GitHub](git://github.com/SinaLab/SinaTools) repository.
+        
+        Alternatively, you can execute the following command:
+        
+        ```bash
+        pip install sinatools
+        ```
+        
+        Installing Models and Data Files
+        --------
+        Some modules in SinaTools require some data files and fine-tuned models to be downloaded. To download these models, please consult the [DataDownload](https://sina.birzeit.edu/sinatools/documentation/cli_tools/DataDownload/DataDownload.html).
+        
+        Documentation
+        --------
+        For information, please refer to the [main page](https://sina.birzeit.edu/sinatools) or the [online domuementation](https://sina.birzeit.edu/sinatools/documentation). 
+        
+        Citation
+        -------
+        Tymaa Hammouda, Mustafa Jarrar, Mohammed Khalilia: [SinaTools: Open Source Toolkit for Arabic Natural Language Understanding](http://www.jarrar.info/publications/HJK24.pdf). In Proceedings of the 2024 AI in Computational Linguistics (ACLing 2024), Procedia Computer Science, Dubai. ELSEVIER.
+        
+        License
+        --------
+        SinaTools is available under the MIT License. See the [LICENSE](https://github.com/SinaLab/sinatools/blob/main/LICENSE) file for more information.
+        
+        Reporting Issues
+        --------
+        To report any issues or bugs, please contact us at "sina.institute.bzu@gmail.com" or visit [SinaTools Issues](https://github.com/SinaLab/sinatools/issues).
+        
+        
+Keywords: sinatools
+Platform: UNKNOWN
+Description-Content-Type: text/markdown

{SinaTools-0.1.26 → SinaTools-0.1.28}/SinaTools.egg-info/SOURCES.txt

@@ -155,6 +155,7 @@ sinatools/ner/datasets.py
 sinatools/ner/entity_extractor.py
 sinatools/ner/helpers.py
 sinatools/ner/metrics.py
+sinatools/ner/relation_extractor.py
 sinatools/ner/transforms.py
 sinatools/ner/data/__init__.py
 sinatools/ner/data/datasets.py
@@ -173,14 +174,14 @@ sinatools/synonyms/__init__.py
 sinatools/synonyms/synonyms_generator.py
 sinatools/utils/__init__.py
 sinatools/utils/charsets.py
-sinatools/utils/implication.py
-sinatools/utils/jaccard.py
 sinatools/utils/parser.py
 sinatools/utils/readfile.py
+sinatools/utils/similarity.py
 sinatools/utils/text_dublication_detector.py
 sinatools/utils/text_transliteration.py
 sinatools/utils/tokenizer.py
 sinatools/utils/tokenizers_words.py
+sinatools/utils/word_compare.py
 sinatools/wsd/__init__.py
 sinatools/wsd/disambiguator.py
 sinatools/wsd/settings.py

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/DataDownload/download_files.py

@@ -2,7 +2,7 @@
 About:
 ------
 
-The download_files is a command-line interface for downloading various NLP resources from pre-specified URLs. It is a part of the sinatools package and provides options to choose which files to download and to specify a download directory. The tool automatically handles file extraction for zip and tar.gz files.
+The download_files command, allows users to select specific files and models to download and use it within SinaTools modules. Additionally, it automatically manages the extraction of compressed files, including zip and tar.gz formats.
 
 Usage:
 ------
@@ -18,7 +18,7 @@ Below is the usage information that can be generated by running download_files -
 
         Options:
         -f, --files FILES
-            Names of the files to download. Available files are: ner, morph, wsd_model, wsd_tokenizer, glosses_dic, five_grams, four_grams, three_grams, two_grams, synonyms_level2, synonyms_level3. 
+            Names of the files to download. Available files are: ner, morph, wsd, synonyms. 
             If no file is specified, all files will be downloaded.
 
 Examples:
@@ -28,7 +28,6 @@ Examples:
 
     download_files -f morph ner 
     This command will download only the `morph` and `ner` files to the default directory.
-
 """
 
 import argparse
@@ -56,14 +55,14 @@ def main():
                 download_file(urls["ner"])
                 download_file(urls["wsd_model"])
                 download_file(urls["wsd_tokenizer"])
-                download_file(urls["glosses_dic"])
+                download_file(urls["one_gram"])
                 download_file(urls["five_grams"])
                 download_file(urls["four_grams"])
                 download_file(urls["three_grams"])
                 download_file(urls["two_grams"])
             elif file == "synonyms":
-                download_file(urls["synonyms_level2"])
-                download_file(urls["synonyms_level3"])
+                download_file(urls["graph_l2"])
+                download_file(urls["graph_l3"])
             else:
                url = urls[file]
                download_file(url)
@@ -72,5 +71,3 @@ def main():
 
 if __name__ == '__main__':
     main()
-
-#download_files -f morph ner

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/morphology/ALMA_multi_word.py

@@ -1,37 +1,3 @@
-"""
-About:
-------
-The alma_multi_word tool performs multi-word morphological analysis using SinaTools' `ALMA_multi_word` utility. Given a multi-word Arabic text input, it returns a detailed analysis in JSON format.
-
-Usage:
-------
-Below is the usage information that can be generated by running alma_multi_word --help.
-
-.. code-block:: none
-
-    alma_multi_word --multi_word=MULTI_WORD_TEXT
-    alma_multi_word --file
-
-Options:
---------
-
-.. code-block:: none
-
-  --multi_word MULTI_WORD_TEXT
-        The multi-word Arabic text that needs to be analyzed.
-  --file
-        File containing the multi-word text to be analyzed
-
-Examples:
----------
-
-.. code-block:: none
-
-  alma_multi_word --multi_word "Your multi-word text here"
-  alma_multi_word --file "path/to/your/file.txt"
-
-"""
-
 import argparse
 from sinatools.morphology.ALMA_multi_word import ALMA_multi_word
 import json

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/morphology/morph_analyzer.py

@@ -1,7 +1,7 @@
 """
 About:
 ------
-The morphology_analyzer command is designed to provide morphological analysis for Arabic text using the SinaTools morph_analyzer component. Users can specify the language and desired analysis task (lemmatization, part-of-speech tagging, or full morphological analysis), and flag.
+The morphology_analyzer command is designed to provide morphological analysis for Arabic text using the SinaTools morph_analyzer API. Users can specify the language and desired analysis task (lemmatization, part-of-speech tagging, or full morphological analysis), and flag.
 
 Usage:
 ------

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/ner/corpus_entity_extractor.py

@@ -7,13 +7,26 @@ import argparse
 from sinatools.ner.entity_extractor import extract
 
 """
-This tool processes a csv file and returns named entites for each token within the text, based on the specified batch size. As follows:
+The following command takes a CSV file as input. It splits a specific column into tokens and tags them using named entity recognition (NER). It retains all other columns as they are, and it also adds sentences and tokens. Additionally, it assigns an auto-incrementing ID, a sentence ID, and a global sentence ID to each token. As follows:
 
 Usage:
 ------
-Run the script with the following command:
-
-corpus_entity_extractor  input.csv --text-columns "TextColumn1,TextColumn2" --additional-columns "Column3,Column4" --output-csv output.csv
+Below is the usage information that can be generated by running corpus_entity_extractor --help.
+
+corpus_entity_extractor --input_csv path/to/csv/file --text-columns "name of the column to be tokenized" --additional-columns "Column3,Column4" --output-csv path/to/csv/file
+
+Options:
+-------
+    --input_csv CSV_FILE_PATH
+          Path of csv file 
+    --text-columns STR
+          Name of the text column that need to be tagged
+    -- additional-columns      
+          name of columns that returned as they are
+    -- output-csv
+          path to csv file 
+
+corpus_entity_extractor  --input_csv "input.csv" --text-columns "TextColumn1" --additional-columns "Column3,Column4" --output-csv "output.csv"          
 """
 
 def jsons_to_list_of_lists(json_list):

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/ner/entity_extractor.py

@@ -1,7 +1,7 @@
 """
 About:
 ------
-This tool processes an input text and returns named entites for each token within the text, based on the specified batch size. As follows:
+This command processes an input text and returns named entites for each token within the text. As follows:
 
 Usage:
 ------
@@ -10,7 +10,7 @@ Below is the usage information that can be generated by running entity_extractor
 .. code-block:: none
 
     entity_extractor --text=INPUT_TEXT
-    entity_extractor --dir=INPUT_FILE --output_csv=OUTPUT_FILE_NAME
+    entity_extractor --dir=DIRECTORY_PATH --output_csv "path/to/csv/file"
 
 Options:
 --------
@@ -18,11 +18,11 @@ Options:
 .. code-block:: none
 
   --text INPUT_TEXT
-        The text that needs to be analyzed for Named Entity Recognition.
-  --file INPUT_FILE
-        File containing the text to be analyzed for Named Entity Recognition.
-  --output_csv OUTPUT_FILE_NAME
-        A file containing the tokenized text and its Named Entity tags.
+      The text that needs to be analyzed for Named Entity Recognition.
+  --dir DIRECTORY_PATH
+        Directory containing the text files to be analyzed for Named Entity Recognition
+  --output_csv CSV_FILE
+        The path for output csv file 
 
 
 Examples:
@@ -31,7 +31,7 @@ Examples:
 .. code-block:: none
 
     entity_extractor --text "Your text here"
-    entity_extractor --dir "/path/to/your/directory" --output_csv "output.csv"
+    entity_extractor --dir "path/to/your/dir" --output_csv "path/to/your/file"
 
 """
 

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/utils/implication.py

@@ -39,7 +39,7 @@ Examples:
 
 """
 import argparse
-from sinatools.utils.implication import Implication
+from sinatools.utils.word_compare import Implication
 
 def read_file(file_path):
     with open(file_path, 'r', encoding='utf-8') as file:
@@ -72,8 +72,8 @@ def main():
     # Instantiate the Implication class
     implication_obj = Implication(word1, word2)
     
-    # For this example, assuming there is a method `get_result()` in the Implication class.
-    result = implication_obj.get_result()
+    # For this example, assuming there is a method `get_verdict()` in the Implication class.
+    result = implication_obj.get_verdict()
     print(result)
 
 if __name__ == '__main__':

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/CLI/utils/jaccard.py

@@ -46,7 +46,7 @@ Examples:
 """
 
 import argparse
-from sinatools.utils.jaccard import jaccard
+from sinatools.utils.similarity import get_jaccard
 from sinatools.utils.readfile import read_file
 
 
@@ -76,7 +76,7 @@ def main():
         print("Either --file1 and --file2 arguments or both --set1 and --set2 arguments must be provided.")
         return
 
-    similarity = jaccard(args.delimiter, set1, set2, args.selection, args.ignoreAllDiacriticsButNotShadda, args.ignoreShaddaDiacritic)
+    similarity = get_jaccard(args.delimiter, set1, set2, args.selection, args.ignoreAllDiacriticsButNotShadda, args.ignoreShaddaDiacritic)
     
     print("Jaccard Result:", similarity)
 

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/DataDownload/downloader.py

@@ -10,13 +10,13 @@ urls = {
     'ner': 'https://sina.birzeit.edu/Wj27012000.tar.gz',
     'wsd_model': 'https://sina.birzeit.edu/bert-base-arabertv02_22_May_2021_00h_allglosses_unused01.zip',
     'wsd_tokenizer': 'https://sina.birzeit.edu/bert-base-arabertv02.zip',
-    'glosses_dic': 'https://sina.birzeit.edu/glosses_dic.pickle',
+    'one_gram': 'https://sina.birzeit.edu/one_gram.pickle',
     'five_grams': 'https://sina.birzeit.edu/five_grams.pickle',
     'four_grams':'https://sina.birzeit.edu/four_grams.pickle',
     'three_grams':'https://sina.birzeit.edu/three_grams.pickle',
     'two_grams':'https://sina.birzeit.edu/two_grams.pickle',
-    'synonyms_level2':'https://sina.birzeit.edu/synonyms_level2.pkl',
-    'synonyms_level3':'https://sina.birzeit.edu/synonyms_level3.pkl'
+    'graph_l2':'https://sina.birzeit.edu/graph_l2.pkl',
+    'graph_l3':'https://sina.birzeit.edu/graph_l3.pkl'
 }
 
 def get_appdatadir():

SinaTools-0.1.28/sinatools/VERSION

@@ -0,0 +1 @@
+0.1.28

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/morphology/morph_analyzer.py

@@ -24,27 +24,27 @@ def find_solution(token, language, flag):
 
 def analyze(text, language ='MSA', task ='full', flag="1"):
    """
-    This method processes an input text and returns morphological analysis for each token within the text, based on the specified language, task, and flag. As follows:
-    If:
-        The task is lemmatization, the morphological solution includes only the lemma_id, lemma, token, and token frequency.
-        The task is pos, the morphological solution includes only the part-of-speech, token, and token frequency.
-        The task is root, the morphological solution includes only the root, token, and token frequency.
-        The task is full, the morphological solution includes the lemma_id, lemma, part-of-speech, root, token, and token frequency.
+    This method processes an input text and returns morphological analysis for each token within the text, based on the specified language, task, and flag. You can try the demo online. See article for more details
+    
+        * If the task is lemmatization, the morphological solution includes only the lemma_id, lemma, token, and token frequency.
+        * If the task is pos, the morphological solution includes only the part-of-speech, token, and token frequency.
+        * If the task is root, the morphological solution includes only the root, token, and token frequency.
+        * If the task is full, the morphological solution includes the lemma_id, lemma, part-of-speech, root, token, and token frequency.    
      
-    Args:
+    Parameters:
         text (:obj:`str`): The Arabic text to be morphologically analyzed.
-        language (:obj:`str`): The type of the input text. Currently, only Modern Standard Arabic (MSA) is supported.
+        language (:obj:`str`): Currently, only Modern Standard Arabic (MSA) is supported.
         task (:obj:`str`): The task to filter the results by. Options are [lemmatization, pos, root, full]. The default task if not specified is `full`.
-        flag (:obj:`str`): The flag to filter the returned results. If the flag is `1`, the solution with the highest frequency will be returned. If the flag is `*`, all solutions will be returned, ordered descendingly, with the highest frequency solution first. The default flag if not specified is `1`.
+        flag (:obj:`str`):  The flag to filter the returned results. If the flag is `1`, the solution with the highest frequency will be returned. If the flag is `*`, all solutions will be returned, ordered descendingly, with the highest frequency solution first. The default flag if not specified is `1`.
          
     Returns:
         list (:obj:`list`): A list of JSON objects, where each JSON could be contains:
             token: The token from the original text.
-            lemma: The lemma of the token.
-            lemma_id: The id of the lemma.
-            pos: The part-of-speech of the token.
-            root: The root of the token.
-            frequency: The frequency of the token.
+            lemma: The lemma of the token (Lemmas from the Qabas lexicon).
+            lemma_id: The id of the lemma (qabas lemma ids).
+            pos: The part-of-speech of the token (see Qabas POS tags).
+            root: The root of the token (qabas roots).
+            frequency: The frequency of the token (see section 3 in article).        
 
     **Example:**
 
@@ -57,37 +57,36 @@ def analyze(text, language ='MSA', task ='full', flag="1"):
         #Example: task = full 
         analyze('ذهب الولد الى المدرسة')
 
-        [
-            { 
-              "token": "ذهب",
-              "lemma": "ذَهَبَ",
-              "lemma_id": "202001617",
-              "root": "ذ ه ب",
-              "pos": "فعل ماضي",
-              "frequency": "82202"
-            },{ 
-               "token": "الولد",
-               "lemma": "وَلَدٌ",
-               "lemma_id": "202003092",
-               "root": "و ل د",
-               "pos": "اسم",
-               "frequency": "19066"
-            },{ 
-               "token": "إلى",
-               "lemma": "إِلَى",
-               "lemma_id": "202000856",
-               "root": "إ ل ى",
-               "pos": "حرف جر",
-               "frequency": "7367507"
-            },{ 
-               "token": "المدرسة",
-               "lemma": "مَدْرَسَةٌ",
-               "lemma_id": "202002620",
-               "root": "د ر س",
-               "pos": "اسم",
-               "frequency": "145285"
-            }
-        ]
+        [{ 
+            "token": "ذهب",
+            "lemma": "ذَهَبَ",
+            "lemma_id": "202001617",
+            "root": "ذ ه ب",
+            "pos": "فعل ماضي",
+            "frequency": "82202"
+          },{ 
+            "token": "الولد",
+            "lemma": "وَلَدٌ",
+            "lemma_id": "202003092",
+            "root": "و ل د",
+            "pos": "اسم",
+            "frequency": "19066"
+          },{ 
+            "token": "إلى",
+            "lemma": "إِلَى",
+            "lemma_id": "202000856",
+            "root": "إ ل ى",
+            "pos": "حرف جر",
+            "frequency": "7367507"
+          },{ 
+            "token": "المدرسة",
+            "lemma": "مَدْرَسَةٌ",
+            "lemma_id": "202002620",
+            "root": "د ر س",
+            "pos": "اسم",
+            "frequency": "145285"
+        }]
+
    """
 
    output_list = []

{SinaTools-0.1.26 → SinaTools-0.1.28}/sinatools/ner/__init__.py

@@ -7,6 +7,8 @@ import torch
 import pickle
 import json
 from argparse import Namespace
+from transformers import pipeline
+#from transformers import AutoModelForSequenceClassification
 
 tagger = None
 tag_vocab = None
@@ -35,4 +37,7 @@ if torch.cuda.is_available():
 
 train_config.trainer_config["kwargs"]["model"] = model
 tagger = load_object(train_config.trainer_config["fn"], train_config.trainer_config["kwargs"])
-tagger.load(os.path.join(model_path,"checkpoints"))
+tagger.load(os.path.join(model_path,"checkpoints"))
+
+pipe = pipeline("sentiment-analysis", model= os.path.join(path, "best_model"), return_all_scores =True, max_length=128, truncation=True)
+#pipe = AutoModelForSequenceClassification.from_pretrained(os.path.join(path, "best_model"))

SinaTools-0.1.28/sinatools/ner/entity_extractor.py

@@ -0,0 +1,104 @@
+import os
+from collections import namedtuple
+from sinatools.ner.data_format import get_dataloaders, text2segments
+from . import tagger, tag_vocab, train_config
+
+
+def convert_nested_to_flat(nested_tags):
+    flat_tags = []
+    
+    for entry in nested_tags:
+        word = entry['token']
+        tags = entry['tags'].split()
+        
+        # Initialize with the first tag in the sequence
+        flat_tag = tags[0]
+        
+        for tag in tags[1:]:
+            # Check if the tag is an "I-" tag, indicating continuation of an entity
+            if tag.startswith('I-'):
+                flat_tag = tag
+                break
+        
+        flat_tags.append({
+            'token': word,
+            'tags': flat_tag
+        })
+    
+    return flat_tags
+
+def extract(text, ner_method="nested"):
+    """
+    This method processes an input text and returns named entites for each token within the text. It support 21 class of entites. The method also support flat and nested NER. You can try the demo online. See article for details.
+    
+    Args:
+        * text (:obj:`str`) – The Arabic text to be tagged.
+        * ner_method (:obj:`str`) – The NER method can produce either flat or nested output formats. The default method is nested.
+            nested method: If the method is nested, the output will include nested tags.
+            flat method: If the method is flat, the output will consist of only flat tags.
+        The choice between flat and nested methods determines the structure and detail of the named entity recognition output.    
+    
+    Returns:
+        A list of JSON objects, where each object could be contains:
+            token: The token from the original text.
+            NER tag: The label pairs for each segment.
+    
+    **Example:**
+    
+    .. highlight:: python
+    .. code-block:: python
+    
+        from sinatools.ner.entity_extractor import extract
+        #Example of nested ner. Notice that the last word in this sentense contains nested tags.
+        extract('ذهب محمد الى جامعة بيرزيت')
+        #the output
+        [{
+            "token":"ذهب",
+            "tags":"O"
+          },{
+            "token":"محمد",
+            "tags":"B-PERS"
+          },{
+            "token":"إلى",
+            "tags":"O"
+          },{
+            "token":"جامعة",
+            "tags":"B-ORG"
+          },{
+            "token":"بيرزيت",
+            "tags":"B-GPE I-ORG"
+        }]    
+    """
+
+    dataset, token_vocab = text2segments(text)
+
+    vocabs = namedtuple("Vocab", ["tags", "tokens"])
+    vocab = vocabs(tokens=token_vocab, tags=tag_vocab)
+
+    dataloader = get_dataloaders(
+        (dataset,),
+        vocab,
+        train_config.data_config,
+        batch_size=32,
+        shuffle=(False,),
+    )[0]
+
+
+    segments = tagger.infer(dataloader)
+    segments_lists = []
+    
+    for segment in segments:
+        for token in segment:
+            segments_list = {}
+            segments_list["token"] = token.text
+            list_of_tags = [t['tag'] for t in token.pred_tag]
+            list_of_tags = [i for i in list_of_tags if i not in('O',' ','')]
+            if list_of_tags == []:
+               segments_list["tags"] = ' '.join(['O'])
+            else:
+               segments_list["tags"] = ' '.join(list_of_tags)
+            segments_lists.append(segments_list)  
+    
+    if ner_method == "flat":
+      segments_lists = convert_nested_to_flat(segments_lists)          
+    return segments_lists