centering-lgram 1.0.0__tar.gz

centering_lgram-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 iatagun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

centering_lgram-1.0.0/MANIFEST.in

@@ -0,0 +1,30 @@
+# Include important files in the distribution
+include README.md
+include LICENSE
+include requirements.txt
+include pyproject.toml
+
+# Include all Python files
+recursive-include lgram *.py
+
+# Include model and data files
+recursive-include models *.py
+recursive-include ngrams *.pkl *.json *.txt
+recursive-include logs *.txt
+
+# Exclude unnecessary files
+exclude *.pyc
+exclude .DS_Store
+exclude .gitignore
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+recursive-exclude * .DS_Store
+
+# Include documentation
+recursive-include docs *.md *.rst *.txt
+
+# Include tests if present
+recursive-include tests *.py
+
+# Include examples if present
+recursive-include examples *.py *.txt *.md

centering_lgram-1.0.0/PKG-INFO

@@ -0,0 +1,260 @@
+Metadata-Version: 2.4
+Name: centering-lgram
+Version: 1.0.0
+Summary: Advanced Language Model with Centering Theory for Coherent Text Generation
+Home-page: https://github.com/iatagun/Lgram
+Author: İlker Atagün
+Author-email: İlker Atagün <ilker.atagun@gmail.com>
+Project-URL: Homepage, https://github.com/iatagun/Lgram
+Project-URL: Bug Reports, https://github.com/iatagun/Lgram/issues
+Project-URL: Source, https://github.com/iatagun/Lgram
+Project-URL: Documentation, https://github.com/iatagun/Lgram/blob/main/README.md
+Keywords: nlp,natural language processing,text generation,centering theory,coherence,language model,n-gram,discourse analysis,computational linguistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=1.9.0
+Requires-Dist: transformers>=4.20.0
+Requires-Dist: spacy>=3.4.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: scipy>=1.7.0
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: tqdm>=4.62.0
+Provides-Extra: django
+Requires-Dist: django>=3.2.0; extra == "django"
+Provides-Extra: full
+Requires-Dist: django>=3.2.0; extra == "full"
+Requires-Dist: jupyter>=1.0.0; extra == "full"
+Requires-Dist: matplotlib>=3.5.0; extra == "full"
+Requires-Dist: plotly>=5.0.0; extra == "full"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: flake8>=4.0.0; extra == "dev"
+Requires-Dist: mypy>=0.950; extra == "dev"
+Requires-Dist: build>=0.8.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+
+# 🧠 Centering-Lgram: Advanced Language Model with Centering Theory
+
+[![PyPI version](https://badge.fury.io/py/centering-lgram.svg)](https://badge.fury.io/py/centering-lgram)
+[![Python versions](https://img.shields.io/pypi/pyversions/centering-lgram.svg)](https://pypi.org/project/centering-lgram/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Downloads](https://pepy.tech/badge/centering-lgram)](https://pepy.tech/project/centering-lgram)
+
+A sophisticated natural language processing library that combines **N-gram language models** with **Centering Theory** to generate coherent and contextually appropriate text. Lgram provides state-of-the-art discourse coherence analysis and text generation capabilities.
+
+## ✨ Key Features
+
+- **🎯 Coherent Text Generation**: Advanced n-gram models (2-gram to 6-gram) with centering theory
+- **🧠 Discourse Analysis**: Implementation of centering theory for coherence evaluation
+- **🔧 Grammar Correction**: T5 transformer-based grammar and style correction
+- **🌐 Semantic Analysis**: SpaCy-powered semantic relationship detection
+- **📊 Collocation Analysis**: Statistical collocation and thematic consistency
+- **⚡ Django Ready**: Production-ready integration with Django framework
+- **🎨 CLI Interface**: Easy-to-use command line tools
+- **📈 Progress Tracking**: Visual progress bars and detailed logging
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Install from PyPI
+pip install centering-lgram
+
+# Install with all optional dependencies
+pip install centering-lgram[full]
+
+# Install for Django projects
+pip install centering-lgram[django]
+```
+
+### Basic Usage
+
+```python
+from lgram import create_language_model
+
+# Create or load a language model
+model = create_language_model()
+
+# Generate coherent text
+text = model.generate_text(
+    num_sentences=3,
+    input_words=["The", "weather"],
+    length=12,
+    use_progress_bar=True
+)
+print(text)
+```
+
+### Advanced Usage with Centering Theory
+
+```python
+from lgram import EnhancedLanguageModel
+
+# Initialize with custom settings
+model = EnhancedLanguageModel(n=3)  # Use 3-gram model
+
+# Generate with centering theory for better coherence
+coherent_text = model.generate_text_with_centering(
+    num_sentences=5,
+    input_words=["She", "founded"],
+    length=15
+)
+
+# Apply grammar correction
+corrected_text = model.correct_grammar_t5(coherent_text)
+print(corrected_text)
+```
+
+### Command Line Interface
+
+```bash
+# Generate text from command line
+centering-lgram generate --input "The weather today" --sentences 3 --correct
+
+# Use centering theory
+centering-lgram generate --input "She founded" --sentences 5 --centering --progress
+
+# Show system information
+centering-lgram info
+
+# Train a new model
+centering-lgram train --text-file data.txt --model-file my_model.pkl
+
+# Backward compatibility - old command still works
+lgram generate --input "Hello world" --sentences 3
+```
+- Output: Transition scores, types, and detailed pairwise information
+
+### 2. `TransitionAnalyzer`
+Analyzes sentence pairs to extract:
+- **Noun phrases** (`noun_chunks`)
+- **Anaphoric relations**
+- **Transition types**
+
+This analysis supports both statistical and linguistic evaluation.
+
+### 3. `EnhancedLanguageModel`
+Generates context-aware, fluent sentences using a **Kneser-Ney smoothed n-gram model** enhanced with POS tagging.
+
+#### Key Features:
+- Generation using 2- to 6-gram models
+- Syntactic analysis and centering using `SpaCy`
+- Linguistic center tracking via `get_center_from_sentence`
+- Contextual word selection via `choose_word_with_context`
+- Completeness check via `is_complete_thought`
+- Theme consistency via `post_process_sentences`
+
+### 4. `dynamicngramparaphraser.py`
+Performs **contextual paraphrasing** based on n-grams. Selects the **best alternative match** for each word depending on its position and syntactic role.
+
+- Supports **dependency-based reordering** (`reorder_sentence`)
+- Combines vector similarity and frequency with `select_best_match`
+
+### 5. `analyze_transitions.py`
+Invokes the `CenteringModel` to analyze all sentence transitions in a text and returns the results as a `DataFrame`, including:
+- `current_sentence`
+- `next_sentence`
+- `transition_type`
+- `score`
+- `total_score`
+
+## 🗂 File Structure
+
+.
+├── analyze_transitions.py
+├── centering_model.py
+├── chunk.py
+├── dynamicngramparaphraser.py
+├── simple_language_model.py
+├── get_gender.py
+├── transition_analyzer.py
+├── corrections.json
+├── ngrams/
+│   ├── bigram_model.pkl
+│   ├── trigram_model.pkl
+│   ├── fourgram_model.pkl
+│   ├── fivegram_model.pkl
+│   ├── sixgram_model.pkl
+│   └── text_data.txt
+
+## 🚀 Usage Example
+
+### Transition Analysis
+```python
+from analyze_transitions import analyze_transitions
+
+text = "Least of all do they thus dispose of the murdered. Guardsman take small farmer well who loathe every precaution the officer."
+df = analyze_transitions(text)
+print(df)
+```
+
+### Sentence Generation
+```python
+from simple_language_model import EnhancedLanguageModel
+
+model = EnhancedLanguageModel("Some training text.")
+sentence = model.generate_sentence(start_words=["The", "man"], length=12)
+print("Generated:", sentence)
+```
+
+### Coherence Report
+```python
+sentences = ["The man left.", "She stayed at home."]
+cleaned_sentences, report = model.post_process_sentences(sentences)
+print(report)
+```
+
+## 🛠 Requirements
+
+- Python 3.8+
+- `spacy`
+- `numpy`
+- `scikit-learn`
+- `tqdm`
+- `pandas`
+
+### SpaCy Model
+```bash
+python -m spacy download en_core_web_lg
+```
+
+## 🎯 Purpose
+
+This project provides a powerful infrastructure for researchers, developers, and linguistics enthusiasts working in **textual coherence**, **discourse transition**, and **automated sentence generation**. Whether you're generating text, analyzing transitions, or evaluating textual consistency — this is your **linguistic lab**.
+
+## Outputs
+
+The victim must get to go to the woods and learn the true meaning of the word, "she continued. " According to Gallup, in 2002, they would not say willingly. Your opinion is needed. At some moment he might not think of himself as a genius, but as the mere fact of being alive. He chose art.
+
+The murder of Agamemnon would send shivers down your backbone. It is no use trying to suppress that side of myself. In her dead mind there is nothing which appears to her as being outside, and what is outside is what He has left behind. There is evidence to support the view that he has seen such a mention as an occasional burst of electricity and, I am sure, no trace of it. His plan would have been to attack the house and burn down the garden with some kind of fire. While her husband was away, she had made an excuse for being late. The size of these ships is unknown.
+
+The murderer will say that this way if indeed the former is the case. In spring of 1992, I would have noticed. The gun swung and he asked the woman what she wanted, but she did not say anything. With the information that we have, we are able to attach more importance to what we do not need. To be near the rest of the world in this case is a matter of great importance.
+
+The crime was committed by an old man. Indeed, so bad is the weather that we sometimes talk about the reason why he was in Berlin and the city itself. The reason is that he is a rich man. Grass has not grown in this day and age, so it is not suitable for making friends or building communities. I find myself in a bar and ask for a drink when I see a new world around me.
+
+The time has come for you to do something in return and to observe what is happening in the present as closely as possible to one possibility. Mr. Phillips's protest was that softening the blow by a gentle breeze and creating a myriad of other sounds was wrong for the foot to be moved. He just pointed out that the world around him had gone wrong and that the course of events in his country would be too difficult for him to change. Of course, Miss Diana told me that she found it very pleasant to hear that he would return to private life. So say much as well as a bird would be a woman's best friend.
+
+The victims of the fire could not have been more remarkable and sad to see. The victim wants to say that he has a young wife who loves him. It seems to him that his life has been entirely smothered by his work. He is on the verge of death and reaches for the moon.
+
+The murder of Lady Godfrey does not make the heart race as much as anybody else. The murderer is the light sleeper and nothing less than Mr. Bingley will do. It is out of the question whether his own vanity has done much good for him. Is it rather subtler to say that some things are better than others? I like it pretty well and am willing to give it a try.

centering_lgram-1.0.0/README.md

@@ -0,0 +1,206 @@
+
+# 🧠 Centering-Lgram: Advanced Language Model with Centering Theory
+
+[![PyPI version](https://badge.fury.io/py/centering-lgram.svg)](https://badge.fury.io/py/centering-lgram)
+[![Python versions](https://img.shields.io/pypi/pyversions/centering-lgram.svg)](https://pypi.org/project/centering-lgram/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Downloads](https://pepy.tech/badge/centering-lgram)](https://pepy.tech/project/centering-lgram)
+
+A sophisticated natural language processing library that combines **N-gram language models** with **Centering Theory** to generate coherent and contextually appropriate text. Lgram provides state-of-the-art discourse coherence analysis and text generation capabilities.
+
+## ✨ Key Features
+
+- **🎯 Coherent Text Generation**: Advanced n-gram models (2-gram to 6-gram) with centering theory
+- **🧠 Discourse Analysis**: Implementation of centering theory for coherence evaluation
+- **🔧 Grammar Correction**: T5 transformer-based grammar and style correction
+- **🌐 Semantic Analysis**: SpaCy-powered semantic relationship detection
+- **📊 Collocation Analysis**: Statistical collocation and thematic consistency
+- **⚡ Django Ready**: Production-ready integration with Django framework
+- **🎨 CLI Interface**: Easy-to-use command line tools
+- **📈 Progress Tracking**: Visual progress bars and detailed logging
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Install from PyPI
+pip install centering-lgram
+
+# Install with all optional dependencies
+pip install centering-lgram[full]
+
+# Install for Django projects
+pip install centering-lgram[django]
+```
+
+### Basic Usage
+
+```python
+from lgram import create_language_model
+
+# Create or load a language model
+model = create_language_model()
+
+# Generate coherent text
+text = model.generate_text(
+    num_sentences=3,
+    input_words=["The", "weather"],
+    length=12,
+    use_progress_bar=True
+)
+print(text)
+```
+
+### Advanced Usage with Centering Theory
+
+```python
+from lgram import EnhancedLanguageModel
+
+# Initialize with custom settings
+model = EnhancedLanguageModel(n=3)  # Use 3-gram model
+
+# Generate with centering theory for better coherence
+coherent_text = model.generate_text_with_centering(
+    num_sentences=5,
+    input_words=["She", "founded"],
+    length=15
+)
+
+# Apply grammar correction
+corrected_text = model.correct_grammar_t5(coherent_text)
+print(corrected_text)
+```
+
+### Command Line Interface
+
+```bash
+# Generate text from command line
+centering-lgram generate --input "The weather today" --sentences 3 --correct
+
+# Use centering theory
+centering-lgram generate --input "She founded" --sentences 5 --centering --progress
+
+# Show system information
+centering-lgram info
+
+# Train a new model
+centering-lgram train --text-file data.txt --model-file my_model.pkl
+
+# Backward compatibility - old command still works
+lgram generate --input "Hello world" --sentences 3
+```
+- Output: Transition scores, types, and detailed pairwise information
+
+### 2. `TransitionAnalyzer`
+Analyzes sentence pairs to extract:
+- **Noun phrases** (`noun_chunks`)
+- **Anaphoric relations**
+- **Transition types**
+
+This analysis supports both statistical and linguistic evaluation.
+
+### 3. `EnhancedLanguageModel`
+Generates context-aware, fluent sentences using a **Kneser-Ney smoothed n-gram model** enhanced with POS tagging.
+
+#### Key Features:
+- Generation using 2- to 6-gram models
+- Syntactic analysis and centering using `SpaCy`
+- Linguistic center tracking via `get_center_from_sentence`
+- Contextual word selection via `choose_word_with_context`
+- Completeness check via `is_complete_thought`
+- Theme consistency via `post_process_sentences`
+
+### 4. `dynamicngramparaphraser.py`
+Performs **contextual paraphrasing** based on n-grams. Selects the **best alternative match** for each word depending on its position and syntactic role.
+
+- Supports **dependency-based reordering** (`reorder_sentence`)
+- Combines vector similarity and frequency with `select_best_match`
+
+### 5. `analyze_transitions.py`
+Invokes the `CenteringModel` to analyze all sentence transitions in a text and returns the results as a `DataFrame`, including:
+- `current_sentence`
+- `next_sentence`
+- `transition_type`
+- `score`
+- `total_score`
+
+## 🗂 File Structure
+
+.
+├── analyze_transitions.py
+├── centering_model.py
+├── chunk.py
+├── dynamicngramparaphraser.py
+├── simple_language_model.py
+├── get_gender.py
+├── transition_analyzer.py
+├── corrections.json
+├── ngrams/
+│   ├── bigram_model.pkl
+│   ├── trigram_model.pkl
+│   ├── fourgram_model.pkl
+│   ├── fivegram_model.pkl
+│   ├── sixgram_model.pkl
+│   └── text_data.txt
+
+## 🚀 Usage Example
+
+### Transition Analysis
+```python
+from analyze_transitions import analyze_transitions
+
+text = "Least of all do they thus dispose of the murdered. Guardsman take small farmer well who loathe every precaution the officer."
+df = analyze_transitions(text)
+print(df)
+```
+
+### Sentence Generation
+```python
+from simple_language_model import EnhancedLanguageModel
+
+model = EnhancedLanguageModel("Some training text.")
+sentence = model.generate_sentence(start_words=["The", "man"], length=12)
+print("Generated:", sentence)
+```
+
+### Coherence Report
+```python
+sentences = ["The man left.", "She stayed at home."]
+cleaned_sentences, report = model.post_process_sentences(sentences)
+print(report)
+```
+
+## 🛠 Requirements
+
+- Python 3.8+
+- `spacy`
+- `numpy`
+- `scikit-learn`
+- `tqdm`
+- `pandas`
+
+### SpaCy Model
+```bash
+python -m spacy download en_core_web_lg
+```
+
+## 🎯 Purpose
+
+This project provides a powerful infrastructure for researchers, developers, and linguistics enthusiasts working in **textual coherence**, **discourse transition**, and **automated sentence generation**. Whether you're generating text, analyzing transitions, or evaluating textual consistency — this is your **linguistic lab**.
+
+## Outputs
+
+The victim must get to go to the woods and learn the true meaning of the word, "she continued. " According to Gallup, in 2002, they would not say willingly. Your opinion is needed. At some moment he might not think of himself as a genius, but as the mere fact of being alive. He chose art.
+
+The murder of Agamemnon would send shivers down your backbone. It is no use trying to suppress that side of myself. In her dead mind there is nothing which appears to her as being outside, and what is outside is what He has left behind. There is evidence to support the view that he has seen such a mention as an occasional burst of electricity and, I am sure, no trace of it. His plan would have been to attack the house and burn down the garden with some kind of fire. While her husband was away, she had made an excuse for being late. The size of these ships is unknown.
+
+The murderer will say that this way if indeed the former is the case. In spring of 1992, I would have noticed. The gun swung and he asked the woman what she wanted, but she did not say anything. With the information that we have, we are able to attach more importance to what we do not need. To be near the rest of the world in this case is a matter of great importance.
+
+The crime was committed by an old man. Indeed, so bad is the weather that we sometimes talk about the reason why he was in Berlin and the city itself. The reason is that he is a rich man. Grass has not grown in this day and age, so it is not suitable for making friends or building communities. I find myself in a bar and ask for a drink when I see a new world around me.
+
+The time has come for you to do something in return and to observe what is happening in the present as closely as possible to one possibility. Mr. Phillips's protest was that softening the blow by a gentle breeze and creating a myriad of other sounds was wrong for the foot to be moved. He just pointed out that the world around him had gone wrong and that the course of events in his country would be too difficult for him to change. Of course, Miss Diana told me that she found it very pleasant to hear that he would return to private life. So say much as well as a bird would be a woman's best friend.
+
+The victims of the fire could not have been more remarkable and sad to see. The victim wants to say that he has a young wife who loves him. It seems to him that his life has been entirely smothered by his work. He is on the verge of death and reaches for the moon.
+
+The murder of Lady Godfrey does not make the heart race as much as anybody else. The murderer is the light sleeper and nothing less than Mr. Bingley will do. It is out of the question whether his own vanity has done much good for him. Is it rather subtler to say that some things are better than others? I like it pretty well and am willing to give it a try.