toxic-comment-classifier 0.2.4__tar.gz → 0.4.0__tar.gz

toxic_comment_classifier-0.4.0/CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.4.0] - 2026-10-31
+
+### Changed
+
+- **Tokenizer no longer depends on Keras internals or pickle.** 0.3.0 unpickled a Keras
+  `Tokenizer` whose class lives at the private path `keras.src.legacy.preprocessing.text`,
+  which breaks whenever Keras moves it (and needs `tensorflow.keras.preprocessing`, which
+  Keras 3 deprecates). The vocabulary now ships as JSON with a small built-in tokenizer and
+  padding function. Outputs are identical to 0.3.0 (verified: same token ids, max score
+  difference 1.2e-7), and the package is about 9 MB smaller because only the 19,999 words
+  the model can use are kept.
+- Model is built with the public Keras 3 API (`import keras`).
+- `predict_batch()` runs one batched model call instead of one call per text.
+- `predict()`/`predict_batch()` return plain Python `float` instead of `numpy.float64`.
+- Requires Python 3.10+ (tested on 3.10-3.13) and `tensorflow>=2.16` / `keras>=3`.
+  Removed the unused `pandas` and `scikit-learn` dependencies.
+- Packaging modernized (`pyproject.toml` only, PEP 639 `license = "MIT"`, `LICENSE.txt` is
+  included in distributions again).
+
+### Added
+
+- `classify_batch(texts)` and `flagged_labels(text, threshold=0.5)`.
+- `ToxicCommentClassifier(clean=True)` to apply `clean_text()` automatically.
+- `toxic-classifier` command-line tool (JSON Lines output, reads stdin).
+- `toxic_classifier.__version__`, `LIST_CLASSES` and `clean_text` exported at package level,
+  type hints and `py.typed`.
+- Input validation with clear `TypeError`s.
+- pytest suite (tokenizer parity, reference scores, batch/single consistency, CLI) and GitHub
+  Actions CI on Python 3.10-3.13 with Ruff; community files.
+
+### Fixed
+
+- `clean_text` URL pattern `www.\S+` matched any character after `www`; now `www\.\S+`.
+
+## [0.3.0] - 2025
+
+- Previous release on PyPI.

toxic_comment_classifier-0.4.0/CITATION.cff

@@ -0,0 +1,12 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+title: "toxic_comment_classifier"
+abstract: "Pre-trained BiLSTM classifier that scores comments for six kinds of toxicity"
+authors:
+  - family-names: Ali
+    given-names: Irfan
+version: "0.4.0"
+date-released: "2026-10-31"
+license: MIT
+repository-code: "https://github.com/irfanalidv/toxic_comment_classifier"
+url: "https://pypi.org/project/toxic_comment_classifier/"

toxic_comment_classifier-0.4.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include README.md
+include LICENSE.txt
+include CHANGELOG.md
+include CITATION.cff
+include toxic_classifier/py.typed
+recursive-include toxic_classifier/model *.h5 *.json
+recursive-include tests *.py
+recursive-include examples *.py
+recursive-include scripts *.py
+global-exclude *.py[cod] __pycache__ .DS_Store

toxic_comment_classifier-0.4.0/PKG-INFO

@@ -0,0 +1,226 @@
+Metadata-Version: 2.4
+Name: toxic_comment_classifier
+Version: 0.4.0
+Summary: Pre-trained BiLSTM classifier that scores comments for six kinds of toxicity (toxic, severe_toxic, obscene, threat, insult, identity_hate). Works offline, one line to use.
+Author-email: Md Irfan Ali <irfanali29@hotmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/irfanalidv/toxic_comment_classifier
+Project-URL: Documentation, https://github.com/irfanalidv/toxic_comment_classifier#readme
+Project-URL: Repository, https://github.com/irfanalidv/toxic_comment_classifier
+Project-URL: Issues, https://github.com/irfanalidv/toxic_comment_classifier/issues
+Project-URL: Changelog, https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CHANGELOG.md
+Keywords: toxicity,toxic-comments,content-moderation,hate-speech,text-classification,multi-label-classification,nlp,deep-learning,lstm,keras,tensorflow,jigsaw
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: numpy>=1.23
+Requires-Dist: tensorflow>=2.16
+Requires-Dist: keras>=3.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: ruff>=0.6.0; extra == "dev"
+Requires-Dist: build>=1.0.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Dynamic: license-file
+
+<div align="center">
+
+# Toxic Comment Classifier
+
+**A pre-trained deep-learning model that scores comments for six kinds of toxicity, offline, in one line of Python.**
+
+[![PyPI version](https://img.shields.io/pypi/v/toxic-comment-classifier.svg)](https://pypi.org/project/toxic-comment-classifier/)
+[![Python versions](https://img.shields.io/pypi/pyversions/toxic-comment-classifier.svg)](https://pypi.org/project/toxic-comment-classifier/)
+[![Tests](https://github.com/irfanalidv/toxic_comment_classifier/actions/workflows/tests.yml/badge.svg)](https://github.com/irfanalidv/toxic_comment_classifier/actions/workflows/tests.yml)
+[![PyPI Downloads](https://static.pepy.tech/badge/toxic-comment-classifier)](https://pepy.tech/projects/toxic-comment-classifier)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/LICENSE.txt)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+
+</div>
+
+A Python library for classifying toxic comments using deep learning.
+It supports detecting multiple types of toxicity including obscene language, threats, and identity hate.
+
+## ✨ Features
+
+- **Six labels, multi-label**: `toxic`, `severe_toxic`, `obscene`, `threat`, `insult`, `identity_hate`, each scored independently from 0 to 1.
+- **Pre-trained and bundled**: the model weights and vocabulary ship inside the package, so there is nothing to download and it works offline.
+- **Simple API**: `classify()` for per-label scores, `predict()` for one overall score, batch versions of both, and `flagged_labels()` for threshold-based moderation.
+- **Fast batch scoring**: batches go through the network in a single call.
+- **Command line**: `toxic-classifier "some comment"` prints JSON.
+- **Small model**: a BiLSTM (about 4.6 MB of weights) that runs comfortably on CPU.
+
+**The model:** a bidirectional LSTM (embedding 20,000 x 50, BiLSTM 2 x 50, global max pooling, dense layers) with sigmoid outputs for the six labels of the [Jigsaw Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) dataset. Its vocabulary was built from that dataset's 159,571 training comments.
+
+## 📦 Installation
+
+```bash
+pip install toxic-comment-classifier
+```
+
+Requires Python 3.10-3.13. TensorFlow (>= 2.16, with Keras 3) is installed automatically.
+
+## 🚀 Usage
+
+### 🔹 Import and Initialize the Model
+
+```python
+from toxic_classifier import ToxicCommentClassifier
+
+# Load the classifier
+model = ToxicCommentClassifier()
+```
+
+(`from toxic_classifier.model import ToxicCommentClassifier` still works too.)
+
+---
+
+### 🔹 Classify a Single Comment
+
+```python
+text = "You are so dumb and stupid!"
+scores = model.classify(text)
+
+scores
+```
+
+**Example Output:**
+
+```python
+{'toxic': 0.9889402985572815,
+ 'severe_toxic': 0.07256769388914108,
+ 'obscene': 0.6204292178153992,
+ 'threat': 0.019348442554473877,
+ 'insult': 0.8664074540138245,
+ 'identity_hate': 0.040729500353336334}
+```
+
+---
+
+### 🔹 Get Overall Toxicity Score
+
+The overall score is the mean of the six label scores.
+
+```python
+toxicity = model.predict(text)
+print(f"Overall Toxicity Score: {toxicity:.4f}")
+```
+
+**Example Output:**
+
+```python
+Overall Toxicity Score: 0.4347
+```
+
+---
+
+### 🔹 Classify Multiple Comments
+
+```python
+texts = [
+    "I hate you so much!",
+    "This is wonderful news.",
+    "You're disgusting!",
+    "Absolutely love your energy!",
+    "You're the worst person ever!",
+    "Have a nice day :)"
+]
+
+scores = model.predict_batch(texts)
+
+for txt, score in zip(texts, scores):
+    print(f"Text: {txt} --> Toxicity Score: {score:.4f}")
+```
+
+**Example Output:**
+
+```python
+Text: I hate you so much! --> Toxicity Score: 0.1395
+Text: This is wonderful news. --> Toxicity Score: 0.0013
+Text: You're disgusting! --> Toxicity Score: 0.3110
+Text: Absolutely love your energy! --> Toxicity Score: 0.0088
+Text: You're the worst person ever! --> Toxicity Score: 0.0937
+Text: Have a nice day :) --> Toxicity Score: 0.0115
+```
+
+Use `model.classify_batch(texts)` to get the per-label dictionaries for a list of texts.
+
+---
+
+### 🔹 Flag Comments Above a Threshold
+
+```python
+model.flagged_labels("You are so dumb and stupid!", threshold=0.5)
+# ['toxic', 'obscene', 'insult']
+```
+
+---
+
+### 🔹 Command Line
+
+```bash
+toxic-classifier "go kill yourself"
+# {"text": "go kill yourself", "toxicity": 0.374754, "flagged": ["toxic", "insult"], "scores": {"toxic": 0.888507, ...}}
+
+# One comment per line from stdin, JSON Lines out
+cat comments.txt | toxic-classifier --threshold 0.7
+```
+
+## 💡 Use Cases
+
+- **Comment moderation**: hold back or review comments where `flagged_labels()` is non-empty.
+- **Community health dashboards**: track the share of toxic messages over time with `predict_batch()`.
+- **Dataset filtering**: drop toxic samples before training or fine-tuning another model.
+- **Triage**: route `threat` or `identity_hate` hits to a human reviewer first.
+
+Scores are model estimates, not ground truth. Like any model trained on Jigsaw data, it can be wrong on sarcasm, reclaimed slurs, non-English text, and identity terms in neutral contexts, so keep a human in the loop for consequential decisions.
+
+## 🧩 API Overview
+
+| Method | Returns |
+| --- | --- |
+| `ToxicCommentClassifier(clean=False, batch_size=256)` | Loads the bundled model. `clean=True` applies `clean_text()` before scoring. |
+| `classify(text)` | `dict[str, float]`: score per label |
+| `classify_batch(texts)` | `list[dict[str, float]]` |
+| `predict(text)` | `float`: mean of the six label scores |
+| `predict_batch(texts)` | `list[float]` |
+| `flagged_labels(text, threshold=0.5)` | `list[str]`: labels with score >= threshold |
+| `toxic_classifier.clean_text(text)` | Lowercases and removes URLs and special characters |
+| `toxic_classifier.LIST_CLASSES` | The six label names, in output order |
+
+## ⚙️ Configuration
+
+- `TF_CPP_MIN_LOG_LEVEL`: the library sets it to `2` (if unset) to hide TensorFlow's C++ info logs.
+- `KERAS_BACKEND`: the model is built with Keras 3. TensorFlow is the default and tested backend.
+
+## 🤝 Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CONTRIBUTING.md).
+
+```bash
+git clone https://github.com/irfanalidv/toxic_comment_classifier.git
+cd toxic_comment_classifier
+pip install -e ".[dev]"
+pytest
+ruff check . && ruff format --check .
+```
+
+See the [CHANGELOG](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CHANGELOG.md) for release notes.
+
+## 📄 License
+
+MIT. See [LICENSE.txt](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/LICENSE.txt).

toxic_comment_classifier-0.4.0/README.md

@@ -0,0 +1,187 @@
+<div align="center">
+
+# Toxic Comment Classifier
+
+**A pre-trained deep-learning model that scores comments for six kinds of toxicity, offline, in one line of Python.**
+
+[![PyPI version](https://img.shields.io/pypi/v/toxic-comment-classifier.svg)](https://pypi.org/project/toxic-comment-classifier/)
+[![Python versions](https://img.shields.io/pypi/pyversions/toxic-comment-classifier.svg)](https://pypi.org/project/toxic-comment-classifier/)
+[![Tests](https://github.com/irfanalidv/toxic_comment_classifier/actions/workflows/tests.yml/badge.svg)](https://github.com/irfanalidv/toxic_comment_classifier/actions/workflows/tests.yml)
+[![PyPI Downloads](https://static.pepy.tech/badge/toxic-comment-classifier)](https://pepy.tech/projects/toxic-comment-classifier)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/LICENSE.txt)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+
+</div>
+
+A Python library for classifying toxic comments using deep learning.
+It supports detecting multiple types of toxicity including obscene language, threats, and identity hate.
+
+## ✨ Features
+
+- **Six labels, multi-label**: `toxic`, `severe_toxic`, `obscene`, `threat`, `insult`, `identity_hate`, each scored independently from 0 to 1.
+- **Pre-trained and bundled**: the model weights and vocabulary ship inside the package, so there is nothing to download and it works offline.
+- **Simple API**: `classify()` for per-label scores, `predict()` for one overall score, batch versions of both, and `flagged_labels()` for threshold-based moderation.
+- **Fast batch scoring**: batches go through the network in a single call.
+- **Command line**: `toxic-classifier "some comment"` prints JSON.
+- **Small model**: a BiLSTM (about 4.6 MB of weights) that runs comfortably on CPU.
+
+**The model:** a bidirectional LSTM (embedding 20,000 x 50, BiLSTM 2 x 50, global max pooling, dense layers) with sigmoid outputs for the six labels of the [Jigsaw Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) dataset. Its vocabulary was built from that dataset's 159,571 training comments.
+
+## 📦 Installation
+
+```bash
+pip install toxic-comment-classifier
+```
+
+Requires Python 3.10-3.13. TensorFlow (>= 2.16, with Keras 3) is installed automatically.
+
+## 🚀 Usage
+
+### 🔹 Import and Initialize the Model
+
+```python
+from toxic_classifier import ToxicCommentClassifier
+
+# Load the classifier
+model = ToxicCommentClassifier()
+```
+
+(`from toxic_classifier.model import ToxicCommentClassifier` still works too.)
+
+---
+
+### 🔹 Classify a Single Comment
+
+```python
+text = "You are so dumb and stupid!"
+scores = model.classify(text)
+
+scores
+```
+
+**Example Output:**
+
+```python
+{'toxic': 0.9889402985572815,
+ 'severe_toxic': 0.07256769388914108,
+ 'obscene': 0.6204292178153992,
+ 'threat': 0.019348442554473877,
+ 'insult': 0.8664074540138245,
+ 'identity_hate': 0.040729500353336334}
+```
+
+---
+
+### 🔹 Get Overall Toxicity Score
+
+The overall score is the mean of the six label scores.
+
+```python
+toxicity = model.predict(text)
+print(f"Overall Toxicity Score: {toxicity:.4f}")
+```
+
+**Example Output:**
+
+```python
+Overall Toxicity Score: 0.4347
+```
+
+---
+
+### 🔹 Classify Multiple Comments
+
+```python
+texts = [
+    "I hate you so much!",
+    "This is wonderful news.",
+    "You're disgusting!",
+    "Absolutely love your energy!",
+    "You're the worst person ever!",
+    "Have a nice day :)"
+]
+
+scores = model.predict_batch(texts)
+
+for txt, score in zip(texts, scores):
+    print(f"Text: {txt} --> Toxicity Score: {score:.4f}")
+```
+
+**Example Output:**
+
+```python
+Text: I hate you so much! --> Toxicity Score: 0.1395
+Text: This is wonderful news. --> Toxicity Score: 0.0013
+Text: You're disgusting! --> Toxicity Score: 0.3110
+Text: Absolutely love your energy! --> Toxicity Score: 0.0088
+Text: You're the worst person ever! --> Toxicity Score: 0.0937
+Text: Have a nice day :) --> Toxicity Score: 0.0115
+```
+
+Use `model.classify_batch(texts)` to get the per-label dictionaries for a list of texts.
+
+---
+
+### 🔹 Flag Comments Above a Threshold
+
+```python
+model.flagged_labels("You are so dumb and stupid!", threshold=0.5)
+# ['toxic', 'obscene', 'insult']
+```
+
+---
+
+### 🔹 Command Line
+
+```bash
+toxic-classifier "go kill yourself"
+# {"text": "go kill yourself", "toxicity": 0.374754, "flagged": ["toxic", "insult"], "scores": {"toxic": 0.888507, ...}}
+
+# One comment per line from stdin, JSON Lines out
+cat comments.txt | toxic-classifier --threshold 0.7
+```
+
+## 💡 Use Cases
+
+- **Comment moderation**: hold back or review comments where `flagged_labels()` is non-empty.
+- **Community health dashboards**: track the share of toxic messages over time with `predict_batch()`.
+- **Dataset filtering**: drop toxic samples before training or fine-tuning another model.
+- **Triage**: route `threat` or `identity_hate` hits to a human reviewer first.
+
+Scores are model estimates, not ground truth. Like any model trained on Jigsaw data, it can be wrong on sarcasm, reclaimed slurs, non-English text, and identity terms in neutral contexts, so keep a human in the loop for consequential decisions.
+
+## 🧩 API Overview
+
+| Method | Returns |
+| --- | --- |
+| `ToxicCommentClassifier(clean=False, batch_size=256)` | Loads the bundled model. `clean=True` applies `clean_text()` before scoring. |
+| `classify(text)` | `dict[str, float]`: score per label |
+| `classify_batch(texts)` | `list[dict[str, float]]` |
+| `predict(text)` | `float`: mean of the six label scores |
+| `predict_batch(texts)` | `list[float]` |
+| `flagged_labels(text, threshold=0.5)` | `list[str]`: labels with score >= threshold |
+| `toxic_classifier.clean_text(text)` | Lowercases and removes URLs and special characters |
+| `toxic_classifier.LIST_CLASSES` | The six label names, in output order |
+
+## ⚙️ Configuration
+
+- `TF_CPP_MIN_LOG_LEVEL`: the library sets it to `2` (if unset) to hide TensorFlow's C++ info logs.
+- `KERAS_BACKEND`: the model is built with Keras 3. TensorFlow is the default and tested backend.
+
+## 🤝 Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CONTRIBUTING.md).
+
+```bash
+git clone https://github.com/irfanalidv/toxic_comment_classifier.git
+cd toxic_comment_classifier
+pip install -e ".[dev]"
+pytest
+ruff check . && ruff format --check .
+```
+
+See the [CHANGELOG](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CHANGELOG.md) for release notes.
+
+## 📄 License
+
+MIT. See [LICENSE.txt](https://github.com/irfanalidv/toxic_comment_classifier/blob/main/LICENSE.txt).

toxic_comment_classifier-0.4.0/examples/quickstart.py

@@ -0,0 +1,35 @@
+"""Quick start for toxic_comment_classifier (run: python examples/quickstart.py)."""
+
+from toxic_classifier import ToxicCommentClassifier, clean_text
+
+classifier = ToxicCommentClassifier()
+
+# === Single comment ===
+raw_text = "YOU are an amazing person!"
+cleaned = clean_text(raw_text)
+print(f"\nCleaned text: {cleaned}")
+
+scores = classifier.classify(cleaned)
+print("Toxicity scores (single):")
+for label, score in scores.items():
+    print(f"  {label}: {score:.4f}")
+
+avg = classifier.predict(cleaned)
+print(f"Average toxicity score: {avg:.4f}")
+
+# === Batch ===
+print("\nBatch predictions:")
+texts = [
+    "I hate you so much!",
+    "This is wonderful news.",
+    "You're disgusting!",
+    "Absolutely love your energy!",
+    "You're the worst person ever!",
+    "Have a nice day :)",
+]
+
+for txt, score in zip(texts, classifier.predict_batch(texts), strict=True):
+    print(f"Text: {txt}\n-> Toxicity Score: {score:.4f}\n")
+
+# Labels at or above a threshold
+print(classifier.flagged_labels("You are so dumb and stupid!", threshold=0.5))

toxic_comment_classifier-0.4.0/pyproject.toml

@@ -0,0 +1,87 @@
+[build-system]
+requires = ["setuptools>=77.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "toxic_comment_classifier"
+version = "0.4.0"
+description = "Pre-trained BiLSTM classifier that scores comments for six kinds of toxicity (toxic, severe_toxic, obscene, threat, insult, identity_hate). Works offline, one line to use."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE.txt"]
+authors = [{ name = "Md Irfan Ali", email = "irfanali29@hotmail.com" }]
+keywords = [
+    "toxicity",
+    "toxic-comments",
+    "content-moderation",
+    "hate-speech",
+    "text-classification",
+    "multi-label-classification",
+    "nlp",
+    "deep-learning",
+    "lstm",
+    "keras",
+    "tensorflow",
+    "jigsaw",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Text Processing :: Linguistic",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Typing :: Typed",
+]
+dependencies = [
+    "numpy>=1.23",
+    "tensorflow>=2.16",
+    "keras>=3.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "ruff>=0.6.0",
+    "build>=1.0.0",
+    "twine>=5.0.0",
+]
+
+[project.scripts]
+toxic-classifier = "toxic_classifier.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/irfanalidv/toxic_comment_classifier"
+Documentation = "https://github.com/irfanalidv/toxic_comment_classifier#readme"
+Repository = "https://github.com/irfanalidv/toxic_comment_classifier"
+Issues = "https://github.com/irfanalidv/toxic_comment_classifier/issues"
+Changelog = "https://github.com/irfanalidv/toxic_comment_classifier/blob/main/CHANGELOG.md"
+
+[tool.setuptools]
+packages = ["toxic_classifier"]
+
+[tool.setuptools.package-data]
+toxic_classifier = ["py.typed", "model/*.h5", "model/*.json"]
+
+[tool.ruff]
+target-version = "py310"
+line-length = 100
+extend-exclude = ["*.md"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP"]
+
+[tool.ruff.lint.isort]
+known-first-party = ["toxic_classifier"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra -q"

toxic_comment_classifier-0.4.0/scripts/export_tokenizer.py

@@ -0,0 +1,45 @@
+"""Convert the original pickled Keras ``Tokenizer`` into the JSON vocabulary shipped with
+the package (``toxic_classifier/model/tokenizer.json``).
+
+Only words the model can actually see are kept (index < ``num_words``). Requires Keras 3,
+because the pickle references ``keras.src.legacy.preprocessing.text.Tokenizer``.
+
+Usage:
+    python scripts/export_tokenizer.py tokenizer.pickle toxic_classifier/model/tokenizer.json
+
+(The original ``tokenizer.pickle`` is in the git history, release 0.3.0.)
+"""
+
+from __future__ import annotations
+
+import json
+import pickle
+import sys
+
+
+def main(src: str, dst: str) -> None:
+    with open(src, "rb") as handle:
+        tok = pickle.load(handle)  # noqa: S301 - trusted, project-owned artifact
+
+    num_words = tok.num_words
+    word_index = {
+        word: idx for word, idx in tok.word_index.items() if num_words is None or idx < num_words
+    }
+    payload = {
+        "config": {
+            "num_words": num_words,
+            "filters": tok.filters,
+            "lower": tok.lower,
+            "split": tok.split,
+            "char_level": tok.char_level,
+            "oov_token": tok.oov_token,
+        },
+        "word_index": dict(sorted(word_index.items(), key=lambda kv: kv[1])),
+    }
+    with open(dst, "w", encoding="utf-8") as handle:
+        json.dump(payload, handle, ensure_ascii=False, separators=(",", ":"))
+    print(f"wrote {len(word_index)} words to {dst}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2])