sentimentizer 0.6.5__tar.gz

sentimentizer-0.6.5/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2018-Present Edward Yang
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

sentimentizer-0.6.5/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include sentimentizer/data/yelp.dictionary
+include sentimentizer/data/weights.pth 

sentimentizer-0.6.5/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.1
+Name: sentimentizer
+Version: 0.6.5
+Summary: straight forward rnn model
+Author-email: Edward Yang <edwardpyang@gmail.com>
+License: Copyright (c) 2018-Present Edward Yang
+        
+        Permission is hereby granted, free of charge, to any person obtaining
+        a copy of this software and associated documentation files (the
+        "Software"), to deal in the Software without restriction, including
+        without limitation the rights to use, copy, modify, merge, publish,
+        distribute, sublicense, and/or sell copies of the Software, and to
+        permit persons to whom the Software is furnished to do so, subject to
+        the following conditions:
+        
+        The above copyright notice and this permission notice shall be
+        included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+        EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+        MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+        NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+        LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+        OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+        WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+        
+Keywords: rnn,pytorch,nlp,sentiment
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+License-File: LICENSE
+
+# Introduction
+
+[![PyPI Latest Release](https://img.shields.io/pypi/v/sentimentizer.svg)](https://pypi.org/project/sentimentizer/)
+![GitHub CI](https://github.com/eddiepyang/sentimentizer/actions/workflows/ci.yaml/badge.svg)
+  
+Beta release, api subject to change. Install with:  
+
+```
+pip install sentimentizer
+```  
+  
+This repo contains Neural Nets written with the pytorch framework for sentiment analysis.  
+A LSTM based torch model can be found in the rnn folder. In spite of large language models (GPT3.5 as of 2023) 
+dominating the conversation, small models can be pretty effective and are nice to learn from. This model focuses on sentiment analysis and was trained on 
+a single gpu in minutes and requires less than 1GB of memory.
+
+  
+## Usage
+```
+# where 0 is very negative and 1 is very positive
+from sentimentizer.tokenizer import get_trained_tokenizer
+from sentimentizer.rnn.model import get_trained_model
+
+model = get_trained_model(64, 'cpu')
+tokenizer = get_trained_tokenizer()
+review_text = "greatest pie ever, best in town!"
+positive_ids = tokenizer.tokenize_text(review_text)
+model.predict(positive_ids)
+  
+>> tensor(0.9701)
+```
+
+## Install for development with miniconda:  
+```
+conda create -n {env}  
+conda install pip  
+pip install -e .  
+```
+
+## Retrain model
+To rerun the model:
+* get the yelp [dataset](https://www.yelp.com/dataset), 
+* get the glove 6B 100D [dataset](https://nlp.stanford.edu/projects/glove/)
+* place both files in the package data directory 
+* run the training script in workflows
+

sentimentizer-0.6.5/README.md

@@ -0,0 +1,46 @@
+# Introduction
+
+[![PyPI Latest Release](https://img.shields.io/pypi/v/sentimentizer.svg)](https://pypi.org/project/sentimentizer/)
+![GitHub CI](https://github.com/eddiepyang/sentimentizer/actions/workflows/ci.yaml/badge.svg)
+  
+Beta release, api subject to change. Install with:  
+
+```
+pip install sentimentizer
+```  
+  
+This repo contains Neural Nets written with the pytorch framework for sentiment analysis.  
+A LSTM based torch model can be found in the rnn folder. In spite of large language models (GPT3.5 as of 2023) 
+dominating the conversation, small models can be pretty effective and are nice to learn from. This model focuses on sentiment analysis and was trained on 
+a single gpu in minutes and requires less than 1GB of memory.
+
+  
+## Usage
+```
+# where 0 is very negative and 1 is very positive
+from sentimentizer.tokenizer import get_trained_tokenizer
+from sentimentizer.rnn.model import get_trained_model
+
+model = get_trained_model(64, 'cpu')
+tokenizer = get_trained_tokenizer()
+review_text = "greatest pie ever, best in town!"
+positive_ids = tokenizer.tokenize_text(review_text)
+model.predict(positive_ids)
+  
+>> tensor(0.9701)
+```
+
+## Install for development with miniconda:  
+```
+conda create -n {env}  
+conda install pip  
+pip install -e .  
+```
+
+## Retrain model
+To rerun the model:
+* get the yelp [dataset](https://www.yelp.com/dataset), 
+* get the glove 6B 100D [dataset](https://nlp.stanford.edu/projects/glove/)
+* place both files in the package data directory 
+* run the training script in workflows
+

sentimentizer-0.6.5/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires      = ["setuptools>=61.0.1", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sentimentizer"
+version = "0.6.5"
+description = "straight forward rnn model"
+readme = "README.md"
+authors = [{ name = "Edward Yang", email = "edwardpyang@gmail.com" }]
+license = { file = "LICENSE" }
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python"
+]
+
+keywords = ["rnn", "pytorch", "nlp", "sentiment"]
+dependencies = [
+        "numpy >= 1.24.0",
+        "polars >= 0.15.16",
+        "pandas >= 1.5.2",
+        "torch >= 1.13.1",
+        "orjson >= 3.8.5",
+        "pyarrow >= 10.0.1",
+        "attrs >= 22.2.0",
+        "scikit-learn",
+        "gensim",
+        "structlog",
+        "psutil",
+]
+requires-python = ">=3.10"
+
+[project.optional-dependencies]
+dev = ["black", "bumpver", "isort", "pip-tools", "pytest"]

sentimentizer-0.6.5/sentimentizer/__init__.py

@@ -0,0 +1,57 @@
+from functools import wraps
+import logging
+from pathlib import Path
+import sys
+import time
+from typing import TextIO
+
+import psutil
+import structlog
+
+
+file_path = Path(__file__)
+root = file_path.parent.parent.absolute()
+
+
+def new_logger(level: int = 20, output: TextIO = sys.stderr) -> structlog.PrintLogger:
+    """
+    creates instance of struct logger
+    """
+    structlog.configure(
+        cache_logger_on_first_use=True,
+        wrapper_class=structlog.make_filtering_bound_logger(level),
+        processors=[
+            structlog.contextvars.merge_contextvars,
+            structlog.processors.add_log_level,
+            structlog.processors.format_exc_info,
+            structlog.processors.TimeStamper(fmt="iso", utc=True),
+            structlog.processors.JSONRenderer(),
+        ],
+        logger_factory=structlog.PrintLoggerFactory(file=output),
+    )
+    return structlog.getLogger(__name__)
+
+
+logger = new_logger(logging.INFO)
+
+
+def time_decorator(func):
+    """logs time stats of function"""
+
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        ts = time.perf_counter()
+        result = func(*args, **kwargs)
+        te = time.perf_counter()
+        event = "function completed successfully"
+        logger.info(
+            event,
+            function=func.__name__,
+            run_time=f"{te-ts: 2.4f} seconds",
+            available_memory=f"{psutil.virtual_memory().available/1024**3: .2f} GBs",
+            free_memory=f"{psutil.virtual_memory().free/1024**3: .2f} GBs",
+            used_memory=f"{psutil.virtual_memory().used/1024**3: .2f} GBs",
+        )
+        return result
+
+    return wrapper

sentimentizer-0.6.5/sentimentizer/config.py

@@ -0,0 +1,91 @@
+import os
+import enum
+
+from dataclasses import dataclass
+from logging import NOTSET, DEBUG, INFO, WARN, ERROR, CRITICAL
+
+from typing import Tuple, Callable
+from sentimentizer import root
+
+data_path = os.path.join(root, "sentimentizer")
+
+DEFAULT_LOG_LEVEL = INFO
+
+BATCH_SIZE = 100000
+WRITE_BYTES = "wb"
+READ_BYTES = "rb"
+TEXT_COLUMN = "text"
+
+Devices = set(("cpu", "cuda", "mps"))
+
+class Device(enum.Enum):
+    CPU = 0
+    CUDA = 1
+    MPS = 2
+
+class FitModes(enum.Enum):
+    fitting = 0
+    training = 1
+    evaluation = 2
+
+
+@dataclass
+class OptimizationParams:
+    lr: float = 0.005
+    betas: Tuple[float, float] = (0.7, 0.99)
+    weight_decay: float = 1e-4
+
+
+@dataclass
+class SchedulerParams:
+    T_max: int = 100
+    eta_min: int = 0
+    last_epoch: int = -1
+
+
+@dataclass(frozen=True)
+class TokenizerConfig:
+    text_col: str = "text"
+    label_col: str = "stars"
+    inputs: str = "data"
+    labels: str = "target"
+    stop: int = 10000
+    max_len: int = 200
+    dict_min: int = 3
+    dict_keep: int = 20000
+    no_above: float = 0.99999
+    save_dictionary: bool = True
+
+
+@dataclass(frozen=True)
+class FileConfig:
+    archive_file_path: str = f"{data_path}/data/archive.zip"
+    raw_file_path: str = "yelp_academic_dataset_review.json"
+    dictionary_file_path: str = f"{data_path}/data/yelp.dictionary"
+    raw_reviews_file_path: str = f"{data_path}/data/review_data.arrow"
+    processed_reviews_file_path: str = f"{data_path}/data/review_data.parquet"
+    weights_file_path: str = f"{data_path}/data/weights.pth"
+
+
+@dataclass
+class TrainerConfig:
+    batch_size: int = 64
+    epochs: int = 4
+    workers: int = 10
+    device: str = "cuda"
+    memory: bool = True
+
+
+@dataclass
+class EmbeddingsConfig:
+    file_path: str = f"{data_path}/data/glove.6B.zip"
+    sub_file_path: str = "glove.6B.100d.txt"
+    emb_length: int = 100
+
+
+@dataclass
+class DriverConfig:
+    files: Callable = FileConfig
+    embeddings: Callable = EmbeddingsConfig
+    tokenizer: Callable = TokenizerConfig
+    trainer: Callable = TrainerConfig

sentimentizer-0.6.5/sentimentizer/extractor.py

@@ -0,0 +1,122 @@
+import zipfile
+from itertools import islice
+from typing import IO, Generator
+
+import numpy as np
+import orjson as json
+import pyarrow as pa
+from gensim import corpora
+
+from sentimentizer import new_logger, time_decorator
+from sentimentizer.config import DEFAULT_LOG_LEVEL, EmbeddingsConfig
+from sentimentizer.tokenizer import tokenize
+
+logger = new_logger(DEFAULT_LOG_LEVEL)
+
+BATCH_SIZE = 100000
+WRITE_BYTES = "wb"
+
+
+def generate_batch(
+    generator_input: Generator[dict, str, None], iter_size: int
+) -> Generator[pa.RecordBatch, list, None]:
+
+    for start in range(0, iter_size, BATCH_SIZE):
+        end = min(start + BATCH_SIZE, iter_size)
+        review_dicts = []
+        review_dicts.extend(islice(generator_input, BATCH_SIZE))
+        yield review_dicts, start, end
+
+
+def process_json(json_file: IO[bytes], stop: int = 0) -> Generator:
+    for i, line in enumerate(json_file):
+        if i % 100000 == 0:
+            logger.debug(f"processing line {i}")
+        dc = json.loads(line)
+        dc["text"] = tokenize(dc.get("text"))
+        if i >= stop and stop != 0:
+            break
+        yield dc
+
+
+@time_decorator
+def extract_data(file_path: str, compressed_file_name: str, stop: int = 0) -> Generator:
+    "reads from zipped yelp data file"
+
+    with zipfile.ZipFile(file_path) as zfile:
+        inf = zfile.open(compressed_file_name)
+    return process_json(inf, stop)
+
+
+def write_arrow(
+    generator_input: Generator,
+    iter_size: int,
+    write_path: str,
+    schema: pa.Schema = None,
+) -> None:
+    gen = generate_batch(generator_input, iter_size)
+
+    in_schema = schema
+    if schema is None:
+        records, _, _ = next(gen)
+        batch = pa.RecordBatch.from_pylist(records)
+        in_schema = batch.schema
+
+    with pa.OSFile(write_path, WRITE_BYTES) as sink, pa.ipc.RecordBatchFileWriter(
+        sink, in_schema
+    ) as writer:
+        if schema is None:
+            writer.write(batch)
+
+        for records, _, end in gen:
+            try:
+                batch = pa.RecordBatch.from_pylist(records)
+                writer.write(batch)
+            except pa.ArrowInvalid:
+                logger.info(f"file completed, last item count was {end}")
+
+
+@time_decorator
+def extract_embeddings(
+    dictionary: corpora.Dictionary, cfg: EmbeddingsConfig
+) -> dict[str, np.ndarray]:
+    """load glove vectors"""
+
+    embeddings_dict: dict = {}
+
+    with zipfile.ZipFile(cfg.file_path, "r") as f, f.open(cfg.sub_file_path, "r") as z:
+        for line in z:
+            values = line.split()
+            key = values[0].decode()
+
+            if key in dictionary.token2id:
+                embeddings_dict.setdefault(
+                    dictionary.token2id[key] + 1,
+                    np.asarray(values[1:], dtype=np.float32),  # noqa: E501
+                )
+
+    return embeddings_dict
+
+
+@time_decorator
+def new_embedding_weights(
+    dictionary: corpora.Dictionary, cfg: EmbeddingsConfig
+) -> np.ndarray:
+
+    """converts local dictionary to embeddings from glove"""
+
+    embeddings_dict: dict = extract_embeddings(dictionary, cfg)
+
+    for word in dictionary.values():
+        if word not in embeddings_dict:
+            embeddings_dict.setdefault(
+                dictionary.token2id[word] + 1, np.random.normal(0, 0.32, cfg.emb_length)
+            )
+
+    return np.vstack(
+        (
+            np.zeros(cfg.emb_length),
+            list(embeddings_dict.values()),
+            np.random.randn(cfg.emb_length),
+        )
+    )

sentimentizer-0.6.5/sentimentizer/loader.py

@@ -0,0 +1,42 @@
+from attr import define
+import pandas as pd
+
+import torch
+from typing import Tuple
+from torch.utils.data import Dataset
+from sklearn.model_selection import train_test_split
+from sentimentizer import new_logger
+from sentimentizer.config import DEFAULT_LOG_LEVEL
+
+logger = new_logger(DEFAULT_LOG_LEVEL)
+
+
+@define
+class CorpusDataset(Dataset):
+    """Dataset class required for pytorch to output items by index"""
+
+    data: pd.DataFrame
+    x_labels: str = "data"
+    y_labels: str = "target"
+
+    def __attr_pre__init__(self):
+
+        super().__init__()
+
+    def __len__(self):
+        return self.data.__len__()
+
+    def __getitem__(self, i):
+        return torch.tensor(self.data[self.x_labels].iat[i]), torch.tensor(
+            self.data[self.y_labels].iat[i]
+        )
+
+
+def load_train_val_corpus_datasets(
+    data_path: str, test_size=0.2
+) -> Tuple[CorpusDataset, CorpusDataset]:
+
+    df = pd.read_parquet(data_path)
+    train_df, val_df = train_test_split(df, test_size=test_size)
+    del df
+    return CorpusDataset(data=train_df), CorpusDataset(val_df)

sentimentizer-0.6.5/sentimentizer/models/decoder.py

@@ -0,0 +1,116 @@
+from importlib.resources import files
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from gensim import corpora
+from torch import nn
+
+from sentimentizer import new_logger
+from sentimentizer.config import DEFAULT_LOG_LEVEL, EmbeddingsConfig, Devices
+from sentimentizer.extractor import new_embedding_weights
+
+logger = new_logger(DEFAULT_LOG_LEVEL)
+
+
+class Decoder(nn.Module):
+    """model class"""
+
+    def __init__(
+        self,
+        batch_size: int,
+        input_len: int,
+        d_model: int,
+        n_heads: int,
+        emb_weights: torch.Tensor,  # weights are vocabsize x embedding length
+        verbose: bool = True,
+        dropout: float = 0.2,
+    ):
+        super().__init__()
+        # vocab size in, hidden size out
+        self.batch_size = batch_size
+        self.emb_weights = emb_weights
+
+        self.embed_layer = nn.Embedding(emb_weights.shape[0], emb_weights.shape[1])
+        self.fc0 = nn.Linear(emb_weights.shape[1], emb_weights.shape[1])
+
+        self.dropout = dropout
+        self.dropout_layer = nn.Dropout1d(p=self.dropout, inplace=True)
+        # input of shape (seq_len, batch, input_size)
+        # https://pytorch.org/docs/stable/nn.html
+
+        decoder_layer = nn.TransformerDecoderLayer(d_model, n_heads)
+        layer_norm = nn.LayerNorm(d_model)
+        self.encoder = nn.TransformerDecoder(
+            encoder_layer=decoder_layer, num_layers=1, norm=layer_norm
+        )
+
+        self.fc1 = nn.Linear(input_len, 1)
+        self.fc2 = nn.Linear(emb_weights.shape[1], 1)
+        self.verbose = verbose
+
+    def load_weights(self):
+        self.embed_layer.load_state_dict({"weight": self.emb_weights})  # type: ignore
+        return self
+
+    def forward(self, inputs: torch.Tensor):
+        embeds = self.embed_layer(inputs)
+        self.dropout_layer(embeds)
+
+        logger.debug("embedding shape %s" % (embeds.shape,))
+        embeds = F.relu(self.fc0(embeds))
+        encoded_out = self.encoder(embeds.permute(0, 2, 1))
+
+        logger.debug("lstm out shape %s" % (encoded_out.shape,))
+        out = self.fc1(encoded_out)
+        logger.debug("fc1 out shape %s" % (out.shape,))
+        fout = self.fc2(out.permute(0, 2, 1))
+        logger.debug("final %s" % (fout.shape,))
+
+        return torch.squeeze(fout)
+
+    def predict(self, converted_text: np.ndarray) -> torch.Tensor:
+        with torch.no_grad():
+            self.eval()
+            output = torch.from_numpy(converted_text)
+            return torch.sigmoid(self.forward(output))
+
+
+def new_model(
+    dict_path: str, embeddings_config: EmbeddingsConfig, batch_size: int, input_len: int
+):
+    dict_yelp = corpora.Dictionary.load(dict_path)
+    embedding_matrix = new_embedding_weights(dict_yelp, embeddings_config)
+    emb_t = torch.from_numpy(embedding_matrix)
+    model = Encoder(
+        batch_size=batch_size,
+        d_model=200,
+        n_heads=4,
+        input_len=input_len,
+        emb_weights=emb_t,
+    )
+    model.load_weights()
+    return model
+
+
+def get_trained_model(batch_size: int, device: str) -> Encoder:
+    """loads pre-trained model"""
+    if device not in Devices:
+        raise ValueError("device must be cpu, cuda, or mps")
+
+    weights = torch.load(
+        str(files("sentimentizer.data").joinpath("embed_weights.pth")),
+        map_location=torch.device(device=device),
+    )
+    empty_embeddings = torch.zeros(weights["embed_layer.weight"].shape)
+    model = Encoder(
+        batch_size=batch_size,
+        d_model=200,
+        n_heads=4,
+        input_len=200,
+        emb_weights=empty_embeddings,
+    )
+
+    model.load_state_dict(weights)
+
+    return model