PerSent 0.2.4__tar.gz → 0.299.0__tar.gz

persent-0.299.0/PKG-INFO

@@ -0,0 +1,365 @@
+Metadata-Version: 2.4
+Name: PerSent
+Version: 0.299.0
+Summary: Persian Sentiment Analysis Toolkit
+Home-page: https://github.com/RezaGooner/PerSent
+Author: RezaGooner
+Author-email: RezaAsadiProgrammer@Gmail.com
+Keywords: persian sentiment analysis nlp
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: hazm>=0.7.0
+Requires-Dist: gensim>=4.0.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: tqdm>=4.62.0
+Requires-Dist: joblib>=1.1.0
+Requires-Dist: requests>=2.26.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# PerSent (Persian Sentiment Analyzer) 
+
+[![Persian](https://img.shields.io/badge/Persian-فارسی-blue.svg)](README.fa.md)
+
+![PerSent Logo](https://github.com/user-attachments/assets/6bb1633b-6ed3-47fa-aae2-f97886dc4e22)
+
+## Introduction
+PerSent is a practical Python library designed for Persian sentiment analysis. The name stands for "Persian Sentiment Analyzer". Currently in its early testing phase, the library provides basic functionality and is available on [PyPI](https://pypi.org/project/PerSent/). Install it using:
+
+```bash
+pip install PerSent
+```
+
+Current capabilities include:
+
+- Sentiment analysis of opinions/comments
+
+- Emotion analysis of texts (happiness, sadness, anger, surprise, fear, disgust, calmness)
+
+- Analysis of product/service reviews (recommended/not recommended/no idea)
+
+- Both single-text and batch CSV processing
+
+- Output displayed in terminal or saved to CSV with summary statistics
+
+Initial repository that evolved into this library:
+[Click here](https://github.com/RezaGooner/Sentiment-Survey-Analyzer)
+
+We welcome user testing and feedback to improve the library. If you encounter bugs or have suggestions, please:
+
+- [Contribute](#contribution)
+
+For installation issues due to dependency conflicts (especially with mingw-w64), consider using online platforms like DeepNote.com.
+
+## Structure
+### Comment Analysis Functions
+
+```train(train_csv, test_size=0.2, vector_size=100, window=5)```
+
+| Parameter     | Data Type | Default Value | Description                                                                 | Optional/Required |
+|--------------|-----------|---------------|-----------------------------------------------------------------------------|-------------------|
+| `train_csv`  | str       | -             | Path to CSV file containing training data with `body` and `recommendation_status` columns | Required          |
+| `test_size`  | float     | 0.2           | Proportion of test data (between 0.0 and 1.0)                               | Optional          |
+| `vector_size`| int       | 100           | Output vector dimension for Word2Vec model (embedding size)                 | Optional          |
+| `window`     | int       | 5             | Context window size for Word2Vec model                                      | Optional          |
+
+recommendation_status must be one of:
+
+- no_idea
+
+- recommended
+
+- not_recommended
+
+Null/NaN values are converted to no_idea, affecting model accuracy.
+
+- Returns test accuracy score.
+
+---
+
+  ```analyzeText(text)```
+  
+| Parameter | Data Type | Description                          | Optional/Required |
+|-----------|-----------|--------------------------------------|-------------------|
+| `text`    | str       | The Persian text to be analyzed      | Required          |
+
+The core function that analyzes a text and returns one of:
+"not_recommended", "recommended", or "no_idea".
+
+---
+
+```saveModel()```
+
+```loadModel()```
+  
+  Model persistence functions. Models are saved in the model directory.
+
+---
+
+```analyzeCSV(input_csv, output_path, summary_path=None, text_column=0)```
+
+| Parameter      | Data Type       | Default Value | Description                                                                 | Optional/Required |
+|----------------|-----------------|---------------|-----------------------------------------------------------------------------|-------------------|
+| `input_csv`    | str             | -             | Path to input CSV file containing comments to analyze                       | Required          |
+| `output_path`  | str             | -             | Path where analyzed results CSV will be saved                               | Required          |
+| `summary_path` | str or None     | None          | Optional path to save summary statistics CSV                                | Optional          |
+| `text_column`  | int or str      | 0             | Column index (int) or name (str) containing the text to analyze             | Optional          |
+  
+
+Batch processes comments from a CSV file. For single-column files, text_column isn't needed. Otherwise specify column name/index (0-based, negative indices supported). Output contains:
+
+1- Original text
+
+2- Recommendation status
+Optional summary_path generates statistics:
+
+- Total count
+
+- Recommended count
+
+- Not recommended count
+
+- No idea count
+
+- Model accuracy (not implemented in current version)
+
+Returns a DataFrame and saves results.
+
+---
+
+### Emotion Analysis Functions
+
+```loadLex(csv_file, word_col=0, emotion_col=1, weight_col=2)```
+
+| Parameter      | Data Type       | Default Value | Description                                                                 | Optional/Required |
+|----------------|-----------------|---------------|-----------------------------------------------------------------------------|-------------------|
+| `csv_file`     | str             | -             | Path to CSV lexicon file                                                    | Required          |
+| `word_col`     | int or str      | 0             | Column index (int) or name (str) containing words                           | Optional          |
+| `emotion_col`  | int or str      | 1             | Column index (int) or name (str) containing emotion labels                  | Optional          |
+| `weight_col`   | int or str      | 2             | Column index (int) or name (str) containing weight values                   | Optional          |
+
+Loads a CSV with three columns:
+
+1- Keywords
+
+2- Emotion (happiness, sadness, anger, fear, disgust, calmness)
+
+3- Emotion weight (defaults to 1 if unspecified, affecting accuracy)
+
+Column indices are optional.
+
+---
+
+```train(train_csv, text_col='text', emotion_col='sentiment', weight_col='weight')```
+
+| Parameter      | Data Type       | Default Value | Description                                                                 | Optional/Required |
+|----------------|-----------------|---------------|-----------------------------------------------------------------------------|-------------------|
+| `train_csv`    | str             | -             | Path to training CSV file                                                   | Required          |
+| `text_col`     | str or int      | 'text'        | Column name/index containing text data                                      | Optional          |
+| `emotion_col`  | str or int      | 'emotion'     | Column name/index containing emotion labels                                 | Optional          |
+| `weight_col`   | str or int      | 'weight'      | Column name/index containing weight values                                  | Optional          |
+
+Trains the emotion model using a CSV with specified column names (optional).
+
+---
+
+```saveModel(model_name='weighted_sentiment_model')```
+
+| Parameter     | Type  | Default Value               | Description                                                                 | Optional/Required |
+|--------------|-------|-----------------------------|-----------------------------------------------------------------------------|-------------------|
+| `model_name` | str   | 'weighted_sentiment_model'  | Base filename for saving model (without extension)                          | Optional          |
+
+
+```loadModel(model_name='weighted_sentiment_model')```
+
+| Parameter     | Type  | Default Value               | Description                                                                 | Optional/Required |
+|--------------|-------|-----------------------------|-----------------------------------------------------------------------------|-------------------|
+| `model_name` | str   | 'weighted_sentiment_model'  | Base filename of model to load (without extension)                         | Optional          |
+
+
+Model persistence functions (saved in model directory).
+
+---
+
+```analyzeText(text)```
+
+| Parameter | Type | Description                          | Optional/Required |
+|-----------|------|--------------------------------------|-------------------|
+| `text`    | str  | Persian text to analyze              | Required          |
+
+Analyzes a single text, returning percentage scores for each emotion.
+
+---
+
+```analyzeCSV(input_csv, output_csv, text_col='text', output_col='sentiment_analysis')```
+
+| Parameter           | Type          | Default Value          | Description                                                                 | Optional/Required |
+|---------------------|---------------|------------------------|-----------------------------------------------------------------------------|-------------------|
+| `input_csv`         | str           | -                      | Path to input CSV file containing text to analyze                           | Required          |
+| `output_csv`        | str           | -                      | Path to save analyzed results                                               | Required          |
+| `text_col`          | str/int       | 'text'                 | Column name/index containing text to analyze                                | Optional          |
+| `output_col`        | str           | 'sentiment_analysis'   | Column name for output results                                              | Optional          |
+
+Batch processes texts from CSV. Returns True on success. Requires:
+
+- input_csv path
+
+- output_csv path
+Optional column names.
+
+---
+
+## Installation
+Install via pip:
+
+```bash
+pip install PerSent
+```
+
+For specific versions:
+
+```bash
+pip install PerSent==<VERSION_NUMBER>
+```
+
+## Usage
+- Comment Analysis
+
+Basic single-text analysis:
+
+```bash
+from PerSent import CommentAnalyzer
+
+analyzer = CommentAnalyzer()
+
+'''
+Training (if you have data):
+Requires CSV with comments and recommendation status columns
+Status must be: recommended/not_recommended/no_idea
+'''
+analyzer.train("train.csv")
+
+# Load pre-trained model
+analyzer.loadModel()
+
+# Predict
+text = "کیفیت عالی داشت" # "Excellent quality"
+result = analyzer.analyzeText(text)
+print(f"Sentiment: {result}")  # Output: Sentiment: recommended
+```
+
+The included pre-trained model has ~70% accuracy. For better results, you can train with larger datasets. I've prepared a split dataset (due to size):
+
+[Download Here](https://github.com/RezaGooner/Sentiment-Survey-Analyzer/tree/main/Dataset/big_train)
+
+---
+
+Batch CSV processing:
+
+```bash
+from PerSent import CommentAnalyzer
+analyzer = CommentAnalyzer()
+analyzer.loadModel()
+
+# Basic usage (single-column CSV)
+analyzer.analyzeCSV(
+    input_csv="comments.csv",
+    output_path="results.csv"
+)
+
+# Alternative usage patterns:
+# 1. Using column index (0-based)
+analyzer.analyzeCSV("comments.csv", "results.csv", None, 0)
+
+# 2. Negative indices (count from end)
+analyzer.analyzeCSV("comments.csv", "results.csv", None, -1)
+
+# 3. Column name
+analyzer.analyzeCSV("comments.csv", "results.csv", None, "نظرات") # "Comments" column
+
+# 4. With summary (single-column)
+analyzer.analyzeCSV("comments.csv", "results.csv", "summary.csv")
+
+# 5. With summary and column specification
+analyzer.analyzeCSV("comments.csv", "results.csv", "summary.csv", 2)
+```
+
+- Emotion Analysis
+
+Single text analysis with pre-trained model:
+
+```bash
+from PerSent import SentimentAnalyzer
+
+analyzer = SentimentAnalyzer()
+analyzer.loadModel()
+
+sample_text = "امتحانم رو خراب کردم. احساس می‌کنم یک شکست خورده‌ی تمام عیارم."
+# "I failed my exam. I feel like a complete failure."
+
+result = analyzer.analyzeText(sample_text)
+for emotion, score in sorted(result.items(), key=lambda x: x[1], reverse=True):
+    print(f"{emotion}: {score:.2f}%")
+```
+
+output :
+
+```bash
+غم: 36.00%                     #Sadness
+عصبانیت: 36.00%                 #anger
+ترس: 28.00%                    #fear
+شادی: 0.00%                     #happiness
+تنفر: 0.00%                      #disgust
+شگفتی: 0.00%                    #surprise
+آرامش: 0.00%                    #calmness
+```
+
+To train your own model:
+
+``` bash
+analyzer.train('emotion_dataset.csv')
+```
+
+Required CSV columns:
+
+1- Keywords
+
+2- Emotion (happiness, sadness, anger, disgust, fear, calmness)
+
+3- Emotion weight
+
+Model persistence:
+
+```bash
+analyzer.saveModel("custom_model_name")
+analyzer.loadModel("custom_model_name")
+```
+
+Batch CSV processing:
+
+```bash
+analyzer.analyzeCSV("input.csv", "output.csv")
+```
+
+## Contribution
+As mentioned, this library needs community collaboration. Please share suggestions, bugs, or feedback via:
+
+- [Fork Repository & Pull Request](https://github.com/RezaGooner/PerSent/fork)
+
+- [Create Issue](https://github.com/RezaGooner/PerSent/issues/new)
+
+- Email: ```RezaAsadiProgrammer@gmail.com```
+
+- Telegram: ```@RezaGooner```

persent-0.299.0/PerSent/CommentAnalyzer.py

@@ -0,0 +1,263 @@
+#import necessary library
+import pandas as pd
+from hazm import Normalizer, word_tokenize, Stemmer, stopwords_list
+import re
+from tqdm import tqdm
+from gensim.models import Word2Vec
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+import os
+import joblib
+
+class CommentAnalyzer:
+    def __init__(self, model_dir='PerSent/model'):
+        self.normalizer = Normalizer()
+        self.stemmer = Stemmer()
+        self.stopwords = set(stopwords_list())
+        self.model_dir = model_dir
+        self.vectorizer = None
+        self.classifier = None
+        
+        # make /model Directory if not exist
+        os.makedirs(self.model_dir, exist_ok=True)
+        
+    def _preprocess_text(self, text):
+        """PreProcess Persian Text"""
+        try:
+            # Normalizing
+            text = self.normalizer.normalize(str(text))
+            
+            # remove number and sign
+            text = re.sub(r'[!()-\[\]{};:\'",؟<>./?@#$%^&*_~۰-۹\d]+', ' ', text)
+            text = re.sub(r'\s+', ' ', text).strip()
+            
+            # tokenize and stemming
+            tokens = word_tokenize(text)
+            processed_tokens = [
+                self.stemmer.stem(token)
+                for token in tokens
+                if token not in self.stopwords and len(token) > 1
+            ]
+            
+            return processed_tokens
+        except Exception as e:
+            raise Exception(f"Text preprocessing failed: {str(e)}")
+    
+    def _sentence_vector(self, sentence, model):
+        """convert sentences to vector by word2vec model"""
+        try:
+            vectors = []
+            for word in sentence:
+                try:
+                    vectors.append(model.wv[word])
+                except KeyError:
+                    vectors.append(np.zeros(100))
+            return np.mean(vectors, axis=0) if vectors else np.zeros(100)
+        except Exception as e:
+            raise Exception(f"Sentence vectorization failed: {str(e)}")
+    
+    def train(self, train_csv, test_size=0.2, vector_size=100, window=5):
+        """Train model"""
+        try:
+            # read data
+            if not os.path.exists(train_csv):
+                raise FileNotFoundError(f"Training file {train_csv} not found")
+                
+            df = pd.read_csv(train_csv)
+            
+            if 'body' not in df.columns or 'recommendation_status' not in df.columns:
+                raise ValueError("CSV file must contain 'body' and 'recommendation_status' columns")
+            
+            df['tokens'] = df['body'].apply(self._preprocess_text)
+            
+            # train Word2Vec model
+            self.vectorizer = Word2Vec(
+                sentences=df['tokens'],
+                vector_size=vector_size,
+                window=window,
+                min_count=1,
+                workers=4
+            )
+            
+            # convert sentences to vector
+            X = np.array([self._sentence_vector(s, self.vectorizer) for s in df['tokens']])
+            y = df['recommendation_status'].map({
+                "no_idea": 2,
+                "recommended": 1,
+                "not_recommended": 0
+            }).values
+            
+            # make train and test data
+            X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size)
+
+            self.classifier = LogisticRegression(max_iter=1000)
+            self.classifier.fit(X_train, y_train)
+            
+            # save model
+            self.saveModel()
+            
+            # evaluation
+            accuracy = self.classifier.score(X_test, y_test)
+            return accuracy
+            
+        except FileNotFoundError as e:
+            raise FileNotFoundError(f"Training file error: {str(e)}")
+        except ValueError as e:
+            raise ValueError(f"Data validation error: {str(e)}")
+        except Exception as e:
+            raise Exception(f"Training failed: {str(e)}")
+    
+    def analyzeText(self, text):
+        """Predict text sentiment"""
+        try:
+            if not self.classifier or not self.vectorizer:
+                raise Exception("Model not trained! Call train() first or load a pretrained model.")
+                
+            tokens = self._preprocess_text(text)
+            vector = self._sentence_vector(tokens, self.vectorizer)
+            prediction = self.classifier.predict([vector])[0]
+            
+            return {
+                0: "not_recommended",
+                1: "recommended",
+                2: "no_idea"
+            }[prediction]
+            
+        except Exception as e:
+            raise Exception(f"Text analysis failed: {str(e)}")
+    
+    def saveModel(self):
+        """save trained model"""
+        try:
+            if not self.classifier or not self.vectorizer:
+                raise Exception("No trained model to save")
+                
+            joblib.dump(self.classifier, os.path.join(self.model_dir, 'classifier.joblib'))
+            self.vectorizer.save(os.path.join(self.model_dir, 'word2vec.model'))
+        except IOError as e:
+            raise IOError(f"Failed to save model: {str(e)}")
+        except Exception as e:
+            raise Exception(f"Error while saving model: {str(e)}")
+    
+    def loadModel(self):
+        """reload from file"""
+        try:
+            classifier_path = os.path.join(self.model_dir, 'classifier.joblib')
+            vectorizer_path = os.path.join(self.model_dir, 'word2vec.model')
+            
+            if not os.path.exists(classifier_path) or not os.path.exists(vectorizer_path):
+                raise FileNotFoundError("Model files not found in the specified directory")
+                
+            self.classifier = joblib.load(classifier_path)
+            self.vectorizer = Word2Vec.load(vectorizer_path)
+        except FileNotFoundError as e:
+            raise FileNotFoundError(f"Model files not found: {str(e)}")
+        except Exception as e:
+            raise Exception(f"Failed to load model: {str(e)}")
+        
+    def analyzeCSV(self, input_csv, output_path, summary_path=None, text_column=0):
+        """
+        Analyze sentiment for comments in a CSV file and save results
+        
+        Parameters:
+            input_csv (str): Path to input CSV file
+            output_path (str): Path to save output CSV file
+            text_column (str/int, optional): Name or index (0-based) of column containing comments. 
+                                          Defaults to 0 (first column).
+            summary_path (str, optional): Path to save prediction summary report.
+                                       If None, no summary will be saved.
+                                       
+        Returns:
+            pd.DataFrame: DataFrame containing analysis results or None if failed
+            
+        Raises:
+            FileNotFoundError: If input file not found
+            ValueError: If invalid column specified
+            Exception: For other processing errors
+        """
+        try:
+            # Read input CSV
+            if not os.path.exists(input_csv):
+                raise FileNotFoundError(f"Input file {input_csv} not found")
+                
+            df = pd.read_csv(input_csv)
+            
+            # Determine the correct column
+            if isinstance(text_column, int):
+                # Handle negative indices
+                if text_column < 0:
+                    text_column = len(df.columns) + text_column
+                    
+                if text_column >= len(df.columns) or text_column < 0:
+                    raise ValueError(f"Column index {text_column} is out of range")
+                    
+                column_name = df.columns[text_column]
+            else:
+                if text_column not in df.columns:
+                    raise ValueError(f"Column '{text_column}' not found in CSV file")
+                column_name = text_column
+            
+            # Analyze each comment
+            tqdm.pandas(desc="Analyzing comments")
+            df['sentiment'] = df[column_name].progress_apply(self.analyzeText)
+            
+            # Save results
+            df.to_csv(output_path, index=False, encoding='utf-8-sig')
+            print(f"Results saved to {output_path}")
+            
+            # Generate and save summary if requested
+            if summary_path:
+                summary = self._generate_summary(df)
+                summary.to_csv(summary_path, index=False, encoding='utf-8-sig')
+                print(f"Summary report saved to {summary_path}")
+            
+            return df
+            
+        except FileNotFoundError as e:
+            print(f"Error: File not found - {str(e)}")
+            return None
+        except ValueError as e:
+            print(f"Error: Invalid column specification - {str(e)}")
+            return None
+        except Exception as e:
+            print(f"Error during CSV analysis: {str(e)}")
+            return None
+
+    def _generate_summary(self, df):
+        """Generate prediction summary statistics"""
+        try:
+            # Count each sentiment
+            counts = df['sentiment'].value_counts().to_dict()
+            
+            # Create summary dataframe
+            summary = pd.DataFrame({
+                'Category': [
+                    'Recommended',
+                    'Not Recommended', 
+                    'No Idea',
+                    'Total',
+                    'Model Accuracy'
+                ],
+                'Count': [
+                    counts.get('recommended', 0),
+                    counts.get('not_recommended', 0),
+                    counts.get('no_idea', 0),
+                    len(df),
+                    'N/A'  # Accuracy needs to be calculated during training
+                ],
+                'Percentage': [
+                    f"{100 * counts.get('recommended', 0) / len(df):.2f}%",
+                    f"{100 * counts.get('not_recommended', 0) / len(df):.2f}%",
+                    f"{100 * counts.get('no_idea', 0) / len(df):.2f}%",
+                    '100%',
+                    'N/A'
+                ]
+            })
+            
+            return summary
+        except Exception as e:
+            raise Exception(f"Failed to generate summary: {str(e)}")
+
+
+# Github : RezaGooner