dstklib 1.0.0__py3-none-any.whl

dstklib-1.0.0.dist-info/METADATA

@@ -0,0 +1,360 @@
+Metadata-Version: 2.1
+Name: dstklib
+Version: 1.0.0
+Requires-Python: ==3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# Distributional Semantics Toolkit
+
+This library is based on the book *Distributional Semantics* by Alessandro Lenci and Magnus Sahlgren. It attempts to incorporate some of the algorithms described in the book, commonly used in distributional semantics.
+
+## Table of Contents
+
+1. [Introduction](#introduction)
+2. [Installation](#installation)
+3. [Usage](#usage)
+4. [Algorithms](#algorithms)
+5. [Contributing](#contributing)
+6. [License](#license)
+7. [Current Status](#current-status)
+
+## Introduction
+
+The toolkit provides a set of classes and methods for conducting research in distributional semantics. It groups its methods by the common tasks followed in distributional semantics. Each task has its own substages, which should be followed in order. For the list of tasks see [Algorithms](#algorithms). Fore more information about the different tasks consult the book *Distributional Semantics* by Alessandro Lenci and Magnus Sahlgren
+
+## Installation
+
+To install it just run the command:
+
+```bash
+pip install dstklib
+```
+
+DSTK requires python 3.11 to work.
+
+# Usage
+
+The library can be used in three modes:
+
+## Standalone mode
+
+In standalone mode you can use the methods individually. Just select the class that contains the method you want to use (without passing any argument) and select the
+method:
+
+```python
+from dstk import TextProcessor
+
+tokens = ["The", "Quick", "Brown", "Fox", "Jumps", "Over", "The", "Lazy", "Dog"]
+
+# Do not pass any argument to the class. Doing it will activate workflow mode.
+# Also, arguments must be keyword arguments. Positional arguments are not supported for methods.
+lower_tokens = TextProcessor().to_lower(tokens=tokens)
+
+print(lower_tokens)
+
+# Output: ["the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]
+```
+
+## Workflow mode
+
+In workflow mode you can chain the desired methods (as long as they followed the order of the stages in which they can be used) and then just call 'result'. To use workflow mode just do:
+
+```python
+from dstk import TextProcessor
+
+text = "The quick brown fox jumps over the lazy dog while the sun sets behind the hills."
+model = "my_spacy_model"
+
+# Calling result is important. Otherwise, it will return an instance of the class
+tokens = TextProcessor(text=text).set_model(model=model).get_tokens().remove_stop_words().get_text().result
+
+print(tokens)
+
+# Output: ["quick", "brown", "fox", "jumps", "lazy", "dog", "sun", "sets", "behind", "hills"]
+```
+
+### Automate workflows:
+
+If there is a specific workflow you use multiple times, you can automate it by using WorkflowBuilder. Just input the name of the methods (in the correct order) you use and its correspondent arguments as a dictionary, along witu the class you are using:
+
+```python
+from dstk import TextProcessor, WorkflowBuilder
+
+text = "The quick brown fox jumps over the lazy dog while the sun sets behind the hills."
+model = "my_spacy_model"
+
+CustomTextWorkflow = WorkflowBuilder(
+        work_class=TextProcessor, 
+        method_representation={
+            "set_model": {"model": model},
+            "get_tokens": {},
+            "remove_stop_words": {},
+            "get_text": {}
+        }
+    )
+
+# Pass as an argument the input required by the class
+tokens = CustomWorkflow(text=text)
+
+print(tokens)
+
+# Output: ["quick", "brown", "fox", "jumps", "lazy", "dog", "sun", "sets", "behind", "hills"]
+```
+
+## Pipeline mode
+
+A pipeline is just a set of workflows running one after another. If there are a lot of workflows that you constantly use, you can automate the process by using PipelineBuilder. Just pass your workflows as a list:
+
+```python
+from dstk import TextProcessor, Collocations, WorkflowBuilder, PipelineBuilder
+
+text = "The quick brown fox jumps over the lazy dog while the sun sets behind the hills."
+
+CustomTextWorkflow = WorkflowBuilder(
+        work_class=TextProcessor, 
+        method_representation={
+            "set_model": {"model": model},
+            "get_tokens": {},
+            "remove_stop_words": {},
+            "get_text": {}
+        }
+    )
+
+CustomCollocationsWorkflow = WorkflowBuilder(
+        work_class=Collocations, 
+        method_representation={
+            "extract_ngrams": {"target_word": "fox", "window_size": [2, 2]},
+            "count_collocates": {}
+        }
+    )
+
+CustomPipeline = PipelineBuilder(
+    workflows=[
+        CustomTextWorkflow,
+        CustomCollocationsWorkflow
+    ]
+)
+
+# Pass as an argument the input required by the class in the first workflow. In this example, the first class is TextProcessor
+result = CustomPipeline(text=text)
+
+# Output: Counter({'quick': 1, 'brown': 1, 'jumps': 1, 'over': 1})
+```
+
+### Hooks
+
+You can add hooks (functions with custom logic) to a pipeline. You must only follow two rules: 
+
+1. It must only accept one input and return one output
+2. The type of its input must be the same as the one returned from the previous workflow. Also, the type it returns must match the input of the next workflow.
+
+Following these rules you can insert your custom hooks this way:
+
+```python
+from dstk import TextProcessor, Collocations, WorkflowBuilder, PipelineBuilder
+
+text = "The quick brown fox jumps over the lazy dog while the sun sets behind the hills."
+
+CustomTextWorkflow = WorkflowBuilder(
+        work_class=TextProcessor, 
+        method_representation={
+            "set_model": {"model": model},
+            "get_tokens": {},
+            "remove_stop_words": {},
+            "get_text": {}
+        }
+    )
+
+CustomCollocationsWorkflow = WorkflowBuilder(
+        work_class=Collocations, 
+        method_representation={
+            "extract_ngrams": {"target_word": "fox_hook", "window_size": [2, 2]},
+            "count_collocates": {}
+        }
+    )
+
+def custom_hook(tokens): 
+    return [token + "_hook" for token in tokens]
+
+CustomPipeline = PipelineBuilder(
+    workflows=[
+        CustomTextWorkflow,
+        custom_hook,
+        CustomCollocationsWorkflow
+    ]
+)
+
+# Pass as an argument the input required by the class in the first workflow. In this example, the first class is TextProcessor
+result = CustomPipeline(text=text)
+
+# Output: Counter({'quick_hook': 1, 'brown_hook': 1, 'jumps_hook': 1, 'over_hook': 1})
+```
+
+
+# Algorithms
+
+This library groups its methods by the common tasks commonly followed while doing distributional semantics:
+
+## Text pre-processing:
+
+### Class: TextProcessor
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *set_model*: Takes a text and analyzes it using a language model.
+
+**Stage: model**
+
+- *get_tokens*: Returns a list of spaCy tokens from a Doc object.
+- *get_sentences*: Returns a list containing sentences as strings or as spaCy Span objects.
+
+**Stage: token_manipulation**
+
+- *remove_stop_words*: Filters tokens, returning only alphanumeric tokens that are not stop words.
+- *raw_tokenizer*: Tokenizes a text including punctuation and stop words.
+- *alphanumeric_raw_tokenizer*: Tokenizes a text including only alphanumeric characters and stop words.
+- *filter_by_pos*: Returns a list of spaCy tokens filtered by a spacific part-of-speech tag.
+- *pos_tagger*: Returns a list of (Token, POS) tuples, pairing each token with its part-of-speech tag.
+- *get_text*: Returns the text content from a list of spaCy tokens, Span objects or list of spaCy tokens.
+
+**Stage: text_processing**
+
+- *to_lower*: Returns a list of lower cased words.
+- *corpus_by_context_window*: Splits the tokens into groups of window_size consecutive words and joins each group into a string.
+- *get_vocabulary*: Returns the vocabulary a text.
+- *join*: Joins a list of strings into a single string text.
+- *save_to_file*: Saves a list of strings or (Token, POS) tuples in the specified path.
+
+## Find collocations:
+
+### Class: Collocations
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *extract_ngrams*: Extracts both the context words of the target collocation, returned as tuples whose lenght corresponds to the specified window_size, and the collocations of the target word, in either directed or undirected manner.
+
+**Stage: collocates**
+
+- *count_collocates*:  Counts the collocates of the target word.
+
+**Stage: count**
+
+- *plot*: Plots the count of the collocates.
+
+## Build a matrix from a text corpus:
+
+### Class: TextMatrixBuilder
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *create_dtm*: Creates Document Term Matrix (DTM).
+
+**Stage: matrix_operations**
+
+- *create_co_ocurrence_matrix*: Creates a Co-occurrence matrix.
+- *to_dataframe*: Creates a dataframe from a matrix representation.
+
+## Weight the co-occurrence matrix:
+
+### Class: WeightMatrix
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *pmi*: Weights a Co-occurrence matrix by PMI or PPMI.
+- *tf_idf*: Weights a Co-occurrence matrix by Tf-idf.
+
+## Generate word embeddings from a co-occurrence matrix:
+
+### Class: CountModels
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *scale_matrix*: Scales the input matrix to have zero mean and unit variance for each feature.
+
+**Stage: embeddings**
+
+- *svd_embeddings*: Generates word embeddings using truncated Single Value Descomposition (SVD).
+- *pca_embeddings*: Generates word embeddings using Principal Component Analysis (PCA).
+- *to_dataframe*: Creates a dataframe from a matrix representation.
+
+## Measure the distance between two words (after generating the word embeddings):
+
+### Class: GeometricDistance
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *euclidean_distance*: Computes the Euclidean distance between the embeddings of two words.
+- *manhattan_distance*: Computes the Manhattan distance between the embeddings of two words.
+- *cos_similarity*: Computes the cosine similarity between the embeddings of two words.
+- *nearest_neighbors*: Returns the top N most semantically similar words to a given target word, based on the specified distance or similarity metric.
+
+## Generate word embeddings using neural networks:
+
+### Class: PredictModels
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *word2vec*: Creates word embeddings using the Word2Vec algorithm.
+- *fastText*: Creates word embeddings using the FastText algorithm.
+- *load_model*: Loads the trained embeddings in .model (Word2Vec) or .bin (FastText) format, depending on the algorithm used.
+
+**Stage: predict_model**
+
+- *save_model*: Saves the trained embeddings in .model (Word2Vec) or .bin (FastText) format, depending on the algorithm used. Can also be used in stage embeddings_operations.
+- *nearest_neighbors*: Returns the top N most semantically similar words to a given target word. Can also be used in stage embeddings_operations.
+- *cos_similarity*: Computes the cosine similarity between the embeddings of two words. Can also be used in stage embeddings_operations.
+- *to_matrix*: Returns a matrix represenation of the word embeddings and their associated labels.
+
+## Plot the word embeddings:
+
+### Class: PlotEmbeddings
+
+The available methods, grouped by stages, are the following:
+
+**Stage: start**
+
+- *elbow_analysis*: Generates an Elbow plot to help determine the optimal number of clusters for the word embeddings.
+- *extract_silhouette_score*: Extracts and plots the Silhouette score to help determine the optimal number of clusters for the word embeddings.
+
+**Stage: clusters**
+
+- *plot_embeddings_2D*: Generates a 2D plot of the word embedddings.
+- *plot_embeddings_3D*: Generates a 3D plot of the word embedddings.
+
+## Predefined pipelines
+
+DSTK has some pipelines included that already cover most of the frequent tasks in distributional semantics:
+
+- *StandardModel*: This pipeline generates word embeddings using the standard model as defined by (Lenci & Sahlgren 97). It preprocesses the text by removing stop words, lowering the words and segmenting the text using a context window. The co-occurrence matrix is weighted with PPMI and reduced with truncated SVD. 
+- *SGNSModel*:  This pipeline generates word embeddings using Skip-Gram with Negative Sampling (SGNS) as defined by (Lenci & Sahlgren 162). It preprocesses the text by extracting the sentences, removing stop words and lowering them. The embeddings are extracted by using word2vec to do SGNS. Returns an instance of PredictModels.
+
+## Other tools:
+
+You can also convert from MatrixRepresentation to a dataframe and viceversa using 'matrix_to_dataframe' and 'dataframe_to_matrix' from matrix_base. 
+
+# Contributing
+
+I welcome contributions to improve this toolkit. If you have ideas or fixes, feel free to fork the repository and submit a pull request. Here are some ways you can help:
+
+* Report bugs or issues.
+
+* Suggest new features or algorithms to add.
+
+# License
+
+This project is licensed under the GPL-3 License - see the [LICENSE](https://gitlab.com/CesarACabrera/distributional-semantics-toolkit/-/blob/master/LICENSE?ref_type=heads) file for details.

dstklib-1.0.0.dist-info/RECORD

@@ -0,0 +1,28 @@
+dstk/__init__.py,sha256=X58NNxR1EZ3CCpPMM569T_O-DUNGNjhpe6UoSn7Rrn0,356
+dstk/collocations.py,sha256=fn47xrVHW1cgROMSLRbH5M9YxdNvO35tOIKCrXbpJvc,4985
+dstk/count_models.py,sha256=D4OINB2Z2jUTyN3scLuNGREV0jHnvcITST3l-kHV9Jo,5489
+dstk/geometric_distance.py,sha256=ddlUFv1nl0OKRgNb9_6fFf7loCDC-CQMDffQ6k0lxvE,5107
+dstk/matrix_base.py,sha256=zLTM05dYUu7M7WUTV8i7ZQdDkJZ65iddR3ksd4Gy4T4,4199
+dstk/pipeline_tools.py,sha256=ve3MfzZMoff6EQrJGyrlLW6ZQHWHEcGJSC_5aMpangk,933
+dstk/pipelines.py,sha256=Za7DuCHSaZBSxzJyMOWHgTH3jm0aYkW4lav-nIMNdZo,4530
+dstk/plot_embeddings.py,sha256=gfjXJGulRbV3uNC45gqMQoJN4YmfP5VVlpKB6eV7PTU,11210
+dstk/predict_models.py,sha256=Cfv3ykzN5-ZIaeuqea0fkjZUy-bFHaDafwWdriBAwe0,8290
+dstk/text_matrix_builder.py,sha256=VBs6pDS-YLAw0cuwVRHjgYHRXCNoo449p2IDl1-dKso,3405
+dstk/text_processor.py,sha256=tHstoWJIHyC0s0zs0B8PIN2_L9woN7GlkiBPVixYsX4,18433
+dstk/weight_matrix.py,sha256=wPBZeNro2ceSQukFzov_6HRFtIklEC_tDVs1Ze-KKQM,2346
+dstk/workflow_tools.py,sha256=62zH2N91V9OYT0P9Jk97ahTmevnjmGF1Yda2BoQtSrU,11574
+dstk/lib_types/__init__.py,sha256=Ka4bfePHC9HWUTiACBdEsaU8Go2J-C1D7ixeMG89lm4,252
+dstk/lib_types/dstk_types.py,sha256=KCQwKav65nAD4VV1kixcWyF-m32HOKjbG0JO0Z6Vjsg,1011
+dstk/lib_types/fasttext_types.py,sha256=5LXE77kgCPJHRx0zXlLTs7wRIQOGZiz30Pq0trIXcBA,51
+dstk/lib_types/gensim_types.py,sha256=tg3OASG_EWuqFQw_pKM4HNjRk1yrMnmlBqdKm-orxag,34
+dstk/lib_types/matplotlib_types.py,sha256=FSP2c6ryTscbuES7w1ccTtcMS1g3k_m-zD7ZlLkfb6I,177
+dstk/lib_types/nltk_types.py,sha256=s_UVeJWIEmh2tzvS3ttuRjWXo84quMjIrm4OQf3vms4,21
+dstk/lib_types/numpy_types.py,sha256=zxgVrHcRJ-_NGO3LE1aba0d4JQDLYN26us5ljlhIq7E,64
+dstk/lib_types/pandas_types.py,sha256=bR27h-xyZ3FccROIHxqYpVvqMNoi1bvIzpq25cf8kkg,43
+dstk/lib_types/sklearn_types.py,sha256=W59yIEkZM_E_tW061x1bY-LpRC2aCzLgtYmXANNSN3Q,47
+dstk/lib_types/spacy_types.py,sha256=hUiaw4AywSW8o42h5lp3t6a4yosG_GasdJX2RCKgW7o,125
+dstklib-1.0.0.dist-info/LICENSE,sha256=LpSgNPBfwn5F4CVhnTbhpiX2f0YgRMzGWQ7Sphuuwuc,35139
+dstklib-1.0.0.dist-info/METADATA,sha256=MQ9P-jmmX_ezDUR7sZeA8XzDr43kqExBQI0Oe2UHOhM,13099
+dstklib-1.0.0.dist-info/WHEEL,sha256=VyG4dJCdJcxE1baiVBm9NET3Nj7Wne1lZZq7UFNxRpg,97
+dstklib-1.0.0.dist-info/top_level.txt,sha256=b_MNmKso0-ra2M7snsy5fZBW-l9MItjrwMYBd-tiOYo,5
+dstklib-1.0.0.dist-info/RECORD,,

dstklib-1.0.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (75.1.1.post0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

dstklib-1.0.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+dstk