wordtangible 0.1.0__tar.gz

wordtangible-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Jason Robison
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wordtangible-0.1.0/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.1
+Name: wordtangible
+Version: 0.1.0
+Summary: 
+Author: Jason Robison
+Author-email: jrrobison1@hoodsen.me
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: nltk (>=3.9.1,<4.0.0)
+Description-Content-Type: text/markdown
+
+# WordTangible
+
+WordTangible is a Python library for analyzing the concreteness and imageability of words and text. It provides tools to measure how abstract or concrete the language in a given text is, which can be useful for various natural language processing tasks, readability analysis, and linguistic research.
+
+## Features
+
+- Get concreteness ratings for individual words
+- Calculate average concreteness for a given text
+- Compute the ratio of concrete to abstract words in a text
+- Customizable thresholds for concrete and abstract word classification
+- Option to include or exclude stopwords in analysis
+
+## Installation
+
+You can install WordTangible using pip:
+
+```bash
+pip install wordtangible
+```
+
+## Usage
+
+Here are some basic examples of how to use WordTangible:
+
+```python
+from wordtangible import word_concreteness, avg_text_concreteness, concrete_abstract_ratio
+
+# Get concreteness rating for a single word
+print(word_concreteness("apple"))  # Output: 5.0 (highly concrete)
+
+# Calculate average concreteness of a text
+text = "The abstract concept of love is as tangible as the apple in your hand."
+print(avg_text_concreteness(text))  # Output: ~3.5 (mix of concrete and abstract)
+
+# Get the ratio of concrete to abstract words
+print(concrete_abstract_ratio(text))  # Output: ~1.0 (balanced concrete and abstract words)
+```
+
+For more detailed usage instructions and API documentation, please refer to our [documentation](link-to-your-docs).
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- The concreteness ratings are derived from multiple sources, including the MRC Psycholinguistic Database, Brysbaert et al. concreteness ratings, and Glasgow concreteness ratings.
+- This project uses NLTK for tokenization and stopword filtering.
+
+## Citation
+
+If you use WordTangible in your research, please cite it as follows:
+
+```
+Robison, J. (2024). WordTangible: A Python library for word concreteness and imageability analysis. [Software]. Available from https://github.com/jrrobison1/wordtangible
+```
+

wordtangible-0.1.0/README.md

@@ -0,0 +1,60 @@
+# WordTangible
+
+WordTangible is a Python library for analyzing the concreteness and imageability of words and text. It provides tools to measure how abstract or concrete the language in a given text is, which can be useful for various natural language processing tasks, readability analysis, and linguistic research.
+
+## Features
+
+- Get concreteness ratings for individual words
+- Calculate average concreteness for a given text
+- Compute the ratio of concrete to abstract words in a text
+- Customizable thresholds for concrete and abstract word classification
+- Option to include or exclude stopwords in analysis
+
+## Installation
+
+You can install WordTangible using pip:
+
+```bash
+pip install wordtangible
+```
+
+## Usage
+
+Here are some basic examples of how to use WordTangible:
+
+```python
+from wordtangible import word_concreteness, avg_text_concreteness, concrete_abstract_ratio
+
+# Get concreteness rating for a single word
+print(word_concreteness("apple"))  # Output: 5.0 (highly concrete)
+
+# Calculate average concreteness of a text
+text = "The abstract concept of love is as tangible as the apple in your hand."
+print(avg_text_concreteness(text))  # Output: ~3.5 (mix of concrete and abstract)
+
+# Get the ratio of concrete to abstract words
+print(concrete_abstract_ratio(text))  # Output: ~1.0 (balanced concrete and abstract words)
+```
+
+For more detailed usage instructions and API documentation, please refer to our [documentation](link-to-your-docs).
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- The concreteness ratings are derived from multiple sources, including the MRC Psycholinguistic Database, Brysbaert et al. concreteness ratings, and Glasgow concreteness ratings.
+- This project uses NLTK for tokenization and stopword filtering.
+
+## Citation
+
+If you use WordTangible in your research, please cite it as follows:
+
+```
+Robison, J. (2024). WordTangible: A Python library for word concreteness and imageability analysis. [Software]. Available from https://github.com/jrrobison1/wordtangible
+```

wordtangible-0.1.0/pyproject.toml

@@ -0,0 +1,19 @@
+[tool.poetry]
+name = "wordtangible"
+version = "0.1.0"
+description = ""
+authors = ["Jason Robison <jrrobison1@hoodsen.me>"]
+readme = "README.md"
+include = ["wordtangible/resources/*.csv"]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+nltk = "^3.9.1"
+
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.3.2"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

wordtangible-0.1.0/wordtangible/__init__.py

@@ -0,0 +1,5 @@
+from .concrete import word_concreteness
+from .concrete import concrete_abstract_ratio
+from .concrete import avg_text_concreteness
+
+__all__ = ["word_concreteness", "concrete_abstract_ratio", "avg_text_concreteness"]

wordtangible-0.1.0/wordtangible/concrete.py

@@ -0,0 +1,156 @@
+import csv
+from pathlib import Path
+import nltk
+from nltk.tokenize import word_tokenize
+from nltk.corpus import stopwords
+from importlib import resources
+
+nltk.download("punkt", quiet=True)
+nltk.download("stopwords", quiet=True)
+
+
+def _load_concreteness_ratings() -> dict[str, float]:
+    concreteness_dict = {}
+
+    # Use importlib.resources to access the CSV file
+    with resources.open_text(
+        "wordtangible.resources", "concreteness_ratings.csv"
+    ) as csvfile:
+        reader = csv.DictReader(csvfile)
+        for row in reader:
+            concreteness_dict[row["Word"]] = float(row["Concreteness"])
+
+    return concreteness_dict
+
+
+CONCRETENESS_RATINGS = _load_concreteness_ratings()
+
+
+def word_concreteness(word: str) -> float | None:
+    """
+    Get the concreteness rating for a given word.
+
+    The concreteness ratings are derived from three sources:
+    1. MRC Psycholinguistic Database
+    2. Brysbaert et al. concreteness ratings
+    3. Glasgow concreteness ratings
+
+    All ratings were normalized to a 1-5 scale, where:
+    - 1 represents highly abstract words
+    - 5 represents highly concrete words
+
+    If a word was rated in only one list, that list's rating was used.
+    If a word was rated in multiple lists, the average of those ratings was used.
+
+    Args:
+        word (str): The word to look up.
+
+    Returns:
+        float | None: The concreteness rating of the word if available, None otherwise.
+    """
+    return CONCRETENESS_RATINGS.get(word, None)
+
+
+def avg_text_concreteness(
+    text: str, include_stopwords: bool = False, only_rated_words: bool = True
+) -> float:
+    """
+    Calculate the average concreteness rating for a given text.
+
+    This function tokenizes the input text, retrieves concreteness ratings for each token,
+    and calculates the average concreteness score.
+
+    Args:
+        text (str): The input text to analyze.
+        include_stopwords (bool, optional): Whether to include stopwords in the analysis.
+            Defaults to False.
+        only_rated_words (bool, optional): Whether to only consider words with known
+            concreteness ratings in the average calculation. Defaults to True.
+
+    Returns:
+        float: The average concreteness rating of the text. Returns 0.0 if no words
+        are found or if no words have concreteness ratings.
+
+    Note:
+        - Concreteness ratings range from 1 (highly abstract) to 5 (highly concrete).
+        - If only_rated_words is True, words without concreteness ratings are excluded
+          from both the numerator and denominator of the average calculation.
+        - If only_rated_words is False, all words are included in the denominator,
+          but only rated words contribute to the numerator.
+    """
+    tokens = _get_tokens(text, include_stopwords)
+
+    if len(tokens) == 0:
+        return 0.0
+
+    concreteness_ratings = [
+        concreteness
+        for token in tokens
+        if (concreteness := word_concreteness(token)) is not None
+    ]
+    num_tokens = len(concreteness_ratings if only_rated_words else tokens)
+    total_concreteness = sum(concreteness_ratings)
+
+    return (total_concreteness / num_tokens) if num_tokens > 0 else 0.0
+
+
+def concrete_abstract_ratio(
+    text: str,
+    include_stopwords: bool = False,
+    very_concrete_threshold: float = 4.0,
+    very_abstract_threshold: float = 2.0,
+) -> float:
+    """
+    Calculate the ratio of very concrete words to very abstract words in a given text.
+
+    This function tokenizes the input text, determines the concreteness of each word,
+    and calculates the ratio of words that are considered very concrete to those
+    considered very abstract based on the provided thresholds.
+
+    Args:
+        text (str): The input text to analyze.
+        include_stopwords (bool, optional): Whether to include stopwords in the analysis.
+            Defaults to False.
+        very_concrete_threshold (float, optional): The concreteness rating threshold
+            for a word to be considered very concrete. Defaults to 4.0.
+        very_abstract_threshold (float, optional): The concreteness rating threshold
+            for a word to be considered very abstract. Defaults to 2.0.
+
+    Returns:
+        float: The ratio of very concrete words to very abstract words.
+            Returns float('inf') if there are concrete words but no abstract words.
+            Returns 0.0 if there are no concrete words or if the text is empty.
+
+    Note:
+        - Concreteness ratings range from 1 (highly abstract) to 5 (highly concrete).
+        - Words with concreteness ratings between the two thresholds are not counted
+          in either category.
+        - Words without known concreteness ratings are ignored.
+    """
+    tokens = _get_tokens(text, include_stopwords)
+
+    concrete_words = 0
+    abstract_words = 0
+
+    for token in tokens:
+        concreteness = word_concreteness(token)
+        if concreteness is not None:
+            if concreteness >= very_concrete_threshold:
+                concrete_words += 1
+            elif concreteness <= very_abstract_threshold:
+                abstract_words += 1
+
+    if abstract_words == 0:
+        return float("inf") if concrete_words > 0 else 0.0
+
+    return concrete_words / abstract_words
+
+
+def _get_tokens(text: str, include_stopwords: bool = False):
+    tokens = [token for token in word_tokenize(text.lower()) if token.isalpha()]
+
+    if not include_stopwords:
+        stop_words = set(stopwords.words("english"))
+        tokens = [token for token in tokens if token not in stop_words]
+
+    return tokens