lemo-vocabulate 0.1.4__tar.gz

lemo_vocabulate-0.1.4/LICENSE

@@ -0,0 +1,29 @@
+MIT License
+
+Copyright (c) 2025 Steven Mesquiti
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+This software is a Python port of Vocabulate, originally developed by:
+Vine, V., Boyd, R. L., & Pennebaker, J. W. (2020).
+Natural emotion vocabularies as windows on distress and well-being.
+Nature Communications, 11(1), 4525.
+https://doi.org/10.1038/s41467-020-18349-0

lemo_vocabulate-0.1.4/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include LICENSE
+include requirements.txt
+recursive-include lemo_vocabulate/data *.csv
+recursive-include lemo_vocabulate/data *.txt

lemo_vocabulate-0.1.4/PKG-INFO

@@ -0,0 +1,415 @@
+Metadata-Version: 2.4
+Name: lemo-vocabulate
+Version: 0.1.4
+Summary: Dictionary-based text analysis tool for emotion and sentiment analysis. Python port of Vocabulate (Vine et al., 2020).
+Author-email: Steven Mesquiti <sm9518@princeton.edu>
+License: MIT License
+        
+        Copyright (c) 2025 Steven Mesquiti
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+        ---
+        
+        This software is a Python port of Vocabulate, originally developed by:
+        Vine, V., Boyd, R. L., & Pennebaker, J. W. (2020).
+        Natural emotion vocabularies as windows on distress and well-being.
+        Nature Communications, 11(1), 4525.
+        https://doi.org/10.1038/s41467-020-18349-0
+Project-URL: Homepage, https://github.com/Bushel-of-Lemons/LEMO_Vocabulate
+Project-URL: Issues, https://github.com/Bushel-of-Lemons/LEMO_Vocabulate/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: tqdm>=4.62.0
+Dynamic: license-file
+
+# Vocabulate Python Edition
+
+**Vocabulate** is a dictionary-based text analysis tool originally converted from C#.  
+This Python package allows you to tokenize, clean, and analyze texts based on custom dictionaries.
+
+**DISCLAIMER:** I do not take credit for this software. I simply reconfigured it to run using a higher-level programming language (i.e., python instead of C#). All credit goes to the original authors: [Vine et al. (2020)](https://www.nature.com/articles/s41467-020-18349-0)
+
+```bib
+@article{vine2020natural,
+  title={Natural emotion vocabularies as windows on distress and well-being},
+  author={Vine, Vera, Boyd, Ryan L. and Pennebaker, James W.},
+  journal={Nature Communications},
+  volume={11},
+  number={1},
+  pages={4525},
+  year={2020},
+  doi={10.1038/s41467-020-18349-0}
+}
+```
+
+---
+
+## Installation
+
+Clone this repository and navigate to the project directory:
+
+```bash
+git clone https://github.com/Bushel-of-Lemons/LEMO_Vocabulate.git
+cd LEMO_Vocabulate
+```
+
+### Option 1: Install with pip
+
+```bash
+pip install -r requirements.txt
+```
+
+### Option 2: Create a Conda environment (recommended)
+
+```bash
+# Create conda environment with Python 3.8
+conda create -n lemons python=3.8 pandas>=2.0 numpy>=1.24 pip -y
+
+# Activate the environment
+conda activate lemons
+
+# Install remaining dependencies
+pip install tqdm>=4.65
+```
+
+### Option 3: Install from TestPyPI
+
+```bash
+pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ lemo-vocabulate
+```
+
+**Note:** This package requires Python >= 3.8 and assumes you have python and pip already installed. 
+
+---
+
+## Quick Start
+
+```python
+import pandas as pd
+from lemo_vocabulate import run_vocabulate_analysis, get_data_path
+
+# Example using a DataFrame with included data files
+df = pd.DataFrame({
+    "user_id": ["user_1", "user_2"],
+    "text": ["This is a sample text.", "Another example text."]
+})
+
+# Use the included dictionary and stopwords
+results = run_vocabulate_analysis(
+    dict_file=get_data_path("AEV_Dict.csv"),
+    input_data=df,
+    text_column="text",
+    stopwords_file=get_data_path("stopwords.txt"),
+    raw_counts=True
+)
+
+print(results.head())
+```
+
+### Using Custom Files
+
+You can still use your own dictionary and stopwords files:
+
+```python
+# Use custom files
+results = run_vocabulate_analysis(
+    dict_file="path/to/your/custom_dict.csv",
+    input_data=df,
+    text_column="text",
+    stopwords_file="path/to/your/custom_stopwords.txt",
+    raw_counts=True
+)
+```
+
+For more examples, see the `lemo_vocabulate_example.ipynb` notebook.
+
+---
+
+## Features
+
+- **Tokenization designed for social media text**
+    - Twitter-aware tokenizer that handles:
+        - Usernames (@user)
+        - Hashtags (#happy)
+        - Emojis and emoticons
+        - URLs
+        - Repeated characters (soooo good)
+        - Punctuation-heavy social media text
+
+- **Stopword removal**
+    - Flexible stopword handling via file or string input
+
+- **Dictionary matching with multi-word wildcards**
+    - Compatible with custom dictionaries in CSV format
+    - Example dictionary provided: `Dictionary/AEV_Dict.csv`
+    
+    **Dictionary breakdown:**
+    ```
+    Neg          94
+    Pos          53
+    AnxFear      20
+    Anger        16
+    Sadness      36
+    NegUndiff    15
+    Total words in dictionary: 162
+    ```
+
+- **Comprehensive text metrics**
+    - Word count, type-token ratio, dictionary coverage
+    - Category-level statistics (CWR, CCR, counts, unique counts)
+
+- **Flexible output**
+    - Returns Pandas DataFrame
+    - Optional CSV export
+
+---
+
+## Usage Examples
+
+### Analyzing a DataFrame
+
+```python
+import pandas as pd
+from lemo_vocabulate import run_vocabulate_analysis
+
+# Create sample data
+df = pd.DataFrame({
+    "text_id": [1, 2, 3],
+    "text": [
+        "I am so agitated and aggravated!",
+        "He was afraid of the dark.",
+        "I am so happy happy happy! And sad."
+    ]
+})
+
+# Run analysis
+results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data=df,
+    text_column="text",
+    stopwords_file="stopwords.txt",
+    raw_counts=True,
+    output_csv="results.csv"
+)
+
+print(results.head())
+```
+
+### Analyzing Text Files
+
+```python
+# Analyze a single text file
+results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data="path/to/file.txt",
+    stopwords_file="stopwords.txt",
+    raw_counts=True
+)
+
+# Analyze all .txt files in a folder
+results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data="path/to/folder",
+    stopwords_file="stopwords.txt",
+    raw_counts=False
+)
+```
+
+### Merging Results with Original Data
+
+```python
+# Run analysis
+df_results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data=df,
+    text_column="text",
+    stopwords_file="stopwords.txt",
+    raw_counts=True
+)
+
+# Create text_id for merging
+df_results['text_id'] = df_results.index
+
+# Merge with original data
+df_complete = df_results.drop(['text', 'Filename'], axis=1).merge(
+    df,
+    on='text_id',
+    how='left'
+)
+
+# Reorder columns
+cols = ['text_id', 'text'] + [col for col in df_complete.columns if col not in ['text_id', 'text']]
+df_complete = df_complete[cols]
+```
+
+---
+
+## Stopwords
+
+Stopword removal allows you to exclude very common function words (e.g., `the`, `and`, `is`, `I`, `you`) that typically do not carry psychological or semantic content. In Vocabulate, stopwords are removed **after tokenization** and **before dictionary matching**, which improves the interpretability of dictionary categories.
+
+### Using a Stopwords File (Recommended)
+
+Create a `.txt` file with one word per line:
+
+```txt
+the
+and
+is
+i
+you
+to
+```
+
+Use it in your analysis:
+
+```python
+results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data=df,
+    text_column="text",
+    stopwords_file="stopwords.txt"
+)
+```
+
+### Using a Stopwords String
+
+```python
+stopwords = "the\nand\nis\nbe\nnot\n"
+results = run_vocabulate_analysis(
+    dict_file="Dictionary/AEV_Dict.csv",
+    input_data=df,
+    text_column="text",
+    stopwords_text=stopwords
+)
+```
+
+### How Stopwords Affect Output Metrics
+
+**Stopword removal does NOT affect:**
+- `WC` (whitespace word count)
+- `TC_Raw` (raw token count)
+- `TTR_Raw` (raw type-token ratio)
+
+**Stopword removal DOES affect:**
+
+| Column | Effect |
+|--------|--------|
+| `TC_Clean` | Tokens after stopword removal |
+| `TTR_Clean` | Based on clean tokens |
+| `TC_NonDict` | Non-dictionary tokens after cleaning |
+| `DictPercent` | Higher if stopwords filtered out |
+| Category metrics | Only meaningful content words remain |
+
+---
+
+## Understanding the Output
+
+The `run_vocabulate_analysis` function returns a Pandas DataFrame where each row corresponds to a single input text. Below is a detailed explanation of all output columns.
+
+### General Text Metrics
+
+| Column Name    | Description                                                                            |
+| -------------- | -------------------------------------------------------------------------------------- |
+| `Filename`     | Name of the file or index of the row from the input DataFrame                          |
+| `text`         | The full original text that was analyzed                                               |
+| `WC`           | Word count: total number of whitespace-separated tokens in the original text           |
+| `TC_Raw`       | Token count after tokenizer processing (including punctuation, emoticons, etc.)        |
+| `TTR_Raw`      | Type-Token Ratio for raw tokens: `#unique tokens / TC_Raw * 100`                       |
+| `TC_Clean`     | Token count after removing stopwords                                                   |
+| `TTR_Clean`    | Type-Token Ratio for cleaned tokens: `#unique tokens / TC_Clean * 100`                 |
+| `TC_NonDict`   | Number of tokens not matched to any dictionary concept                                 |
+| `TTR_NonDict`  | Type-Token Ratio of non-dictionary tokens                                              |
+| `DictPercent`  | Percent of tokens matched to dictionary concepts: `num_matched_tokens / TC_Raw * 100`  |
+| `CapturedText` | Concatenated string of all dictionary-matched words from the text                      |
+
+### Category-Specific Metrics
+
+For each category in the loaded dictionary (e.g., `Neg`, `Pos`, `AnxFear`, `Anger`, `Sadness`, `NegUndiff`), four metrics are provided:
+
+| Column Suffix | Description                                                                                                                                                  |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `_CWR`        | **Category Word Ratio**: percentage of unique words in the category relative to total words in text: `unique_count / WC * 100`                               |
+| `_CCR`        | **Category Concept Ratio**: percentage of unique words in the category relative to all matched tokens in that category: `unique_count / total_count * 100`   |
+| `_Count`      | **Raw Count**: total number of occurrences of words from this category in the text (only if `raw_counts=True`)                                              |
+| `_Unique`     | **Unique Count**: number of unique words in the text that matched this category (only if `raw_counts=True`)                                                 |
+
+#### Example: Category "Neg"
+
+- `Neg_CWR` → % of total words in the text that were unique Neg words
+- `Neg_CCR` → % of category words that were unique
+- `Neg_Count` → Total Neg words matched
+- `Neg_Unique` → Number of unique Neg words matched
+
+---
+
+## Example Output
+
+| Filename | text | WC | TC_Raw | TTR_Raw | TC_Clean | TTR_Clean | TC_NonDict | TTR_NonDict | DictPercent | CapturedText | Neg_CWR | Neg_CCR | Neg_Count | Neg_Unique | Pos_CWR | Pos_CCR | Pos_Count | Pos_Unique | AnxFear_CWR | AnxFear_CCR | AnxFear_Count | AnxFear_Unique |
+|----------|------|-----|--------|---------|----------|-----------|------------|-------------|-------------|--------------|---------|---------|-----------|------------|---------|---------|-----------|------------|-------------|-------------|---------------|----------------|
+| 0 | I am so agitated and aggravated! | 6 | 7 | 100.0 | 2 | 100.0 | 0 | 0.0 | 28.57 | agitated aggravated | 33.33 | 100.0 | 2 | 2 | 0.0 | 0.0 | 0 | 0 | 0.0 | 0.0 | 0 | 0 |
+| 1 | He was afraid of the dark.... | 6 | 8 | 100.0 | 3 | 100.0 | 2 | 100.0 | 12.5 | afraid | 16.67 | 100.0 | 1 | 1 | 0.0 | 0.0 | 0 | 0 | 16.67 | 100.0 | 1 | 1 |
+| 2 | Nothing to be afraid or agitated about. Yet I'm afraid, and it makes me want to agitate!! | 17 | 21 | 85.71 | 6 | 83.33 | 2 | 100.0 | 19.05 | afraid agitated afraid agitate | 11.76 | 50.0 | 4 | 2 | 0.0 | 0.0 | 0 | 0 | 5.88 | 50.0 | 2 | 1 |
+| 3 | I am so happy happy happy! And sad. | 8 | 10 | 80.0 | 4 | 50.0 | 0 | 0.0 | 40.0 | happy happy happy sad | 12.5 | 100.0 | 1 | 1 | 12.5 | 33.33 | 3 | 1 | 0.0 | 0.0 | 0 | 0 |
+| 4 | dislike disliked dislikes disliking/doo | 5 | 6 | 100.0 | 5 | 100.0 | 1 | 100.0 | 66.67 | dislike disliked dislikes disliking | 20.0 | 25.0 | 4 | 1 | 0.0 | 0.0 | 0 | 0 | 0.0 | 0.0 | 0 | 0 |
+
+---
+
+## Function Parameters
+
+```python
+run_vocabulate_analysis(
+    dict_file: str = None,           # Path to dictionary CSV file (required)
+    input_data = None,               # DataFrame, file path, or folder path (required)
+    text_column: str = None,         # Column name for text (required for DataFrame)
+    stopwords_text: str = None,      # Stopwords as newline-separated string
+    stopwords_file: str = None,      # Path to stopwords file
+    raw_counts: bool = True,         # Include raw counts in output
+    encoding: str = "utf-8",         # File encoding
+    csv_delimiter: str = ",",        # CSV delimiter
+    csv_quote: str = '"',            # CSV quote character
+    output_csv: str = None,          # Optional output CSV path
+    whitespace_method: str = 'new'   # Whitespace tokenization method
+)
+```
+---
+
+## Citation
+
+If you use this software in your research, please cite the original paper:
+
+```bib
+@article{vine2020natural,
+  title={Natural emotion vocabularies as windows on distress and well-being},
+  author={Vine, Vera, Boyd, Ryan L. and Pennebaker, James W.},
+  journal={Nature Communications},
+  volume={11},
+  number={1},
+  pages={4525},
+  year={2020},
+  doi={10.1038/s41467-020-18349-0}
+}
+```