semantic-reducer 0.1.0__tar.gz

semantic_reducer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ab_Kafi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

semantic_reducer-0.1.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: semantic-reducer
+Version: 0.1.0
+Summary: O(1) semantic vocabulary reduction using BERT and FAISS.
+Author-email: Your Name <your.email@example.com>
+License: MIT License
+Project-URL: Homepage, https://github.com/abkafi1234/Semantic-Reducer
+Project-URL: Repository, https://github.com/abkafi1234/Semantic-Reducer
+Project-URL: Bug Tracker, https://github.com/abkafi1234/Semantic-Reducer
+Keywords: nlp,bert,faiss,embeddings,text-reduction,machine-learning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=1.10.0
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: faiss-cpu>=1.7.0
+Requires-Dist: tqdm>=4.60.0
+Requires-Dist: transformers>=4.20.0
+Dynamic: license-file
+
+# Semantic Reducer
+
+A high-performance Python library that bridges the gap between **Deep Learning** and **traditional Machine Learning**.
+
+`semantic-reducer` uses **BERT contextual embeddings** and **FAISS vector search** to semantically normalize and reduce the vocabulary of a text corpus.
+
+By mapping semantically equivalent words (for example: *fast*, *quick*, *rapid*) to a single, highly frequent canonical word, it creates cleaner and denser input for traditional ML models such as **TF-IDF + SVM**, **Random Forest**, or **Logistic Regression**.
+
+After the initial processing stage, inference on new text requires only a dictionary lookup, enabling **O(1) runtime complexity**.
+
+---
+
+# Features
+
+- **Context-Aware Processing**  
+  Uses Transformer models (default: `bert-base-multilingual-cased`) to capture contextual meaning instead of relying on static embeddings.
+
+- **Smart Vocabulary Reduction**  
+  Automatically replaces rare or obscure words with their most frequent semantic equivalents.
+
+- **O(1) Inference Speed**  
+  After training, the BERT and FAISS pipeline is no longer required. Text reduction happens via a lightweight Python dictionary.
+
+- **Production Ready**  
+  Supports saving and loading compiled reduction maps for deployment.
+
+- **Hardware Agnostic**  
+  Automatically detects and uses GPU (CUDA) if available, while remaining fully functional on CPU.
+
+---
+
+# Installation
+
+Install via pip:
+
+```bash
+pip install semantic-reducer
+```
+
+---
+
+# Quick Start
+
+## 1. Training and Building the Reduction Map
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize the reducer
+# (downloads the BERT model on first run)
+reducer = SemanticReducer()
+
+# Training corpus
+corpus = [
+    "The quick brown fox jumps over the lazy dog.",
+    "A fast dark-colored canine leaps above a tired hound.",
+    "Speedy foxes jump over sleepy dogs."
+]
+
+# Extract contextual representations
+reducer.process_corpus_contextually(corpus, batch_size=2)
+
+# Finalize embeddings and create FAISS index
+reducer.finalize_embeddings()
+reducer.build_index()
+
+# Build the semantic reduction dictionary
+reducer.build_reduction_map(
+    threshold=0.85,   # similarity threshold
+    top_k=5           # number of neighbors to consider
+)
+
+# Save the trained system
+reducer.save_system(prefix="my_corpus")
+```
+
+---
+
+## 2. Lightning-Fast Inference
+
+Once the reduction map is built, **BERT is no longer required**. New text is processed through a dictionary lookup.
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize reducer
+reducer = SemanticReducer()
+
+# Load saved system
+reducer.load_system(prefix="my_corpus")
+
+# Reduce new text instantly
+new_text = "The rapid fox leaped."
+reduced_text = reducer.reduce_text(new_text)
+
+print(f"Original: {new_text}")
+print(f"Reduced:  {reduced_text}")
+```
+
+---
+
+# Workflow Overview
+
+1. **Contextual Encoding**  
+   Extract contextual word embeddings using BERT.
+
+2. **Vector Indexing**  
+   Store embeddings in a FAISS similarity search index.
+
+3. **Semantic Clustering**  
+   Identify semantically similar words.
+
+4. **Canonical Replacement**  
+   Replace each cluster with the most frequent word in the corpus.
+
+5. **Dictionary Compilation**  
+   Store the final mappings for fast inference.
+
+---
+
+# Requirements
+
+- Python ≥ 3.8
+- torch
+- numpy
+- faiss-cpu
+- transformers
+- tqdm
+
+Install dependencies manually if needed:
+
+```bash
+pip install torch numpy faiss-cpu transformers tqdm
+```
+
+---
+
+# License
+
+This project is licensed under the **MIT License**.

semantic_reducer-0.1.0/README.md

@@ -0,0 +1,140 @@
+# Semantic Reducer
+
+A high-performance Python library that bridges the gap between **Deep Learning** and **traditional Machine Learning**.
+
+`semantic-reducer` uses **BERT contextual embeddings** and **FAISS vector search** to semantically normalize and reduce the vocabulary of a text corpus.
+
+By mapping semantically equivalent words (for example: *fast*, *quick*, *rapid*) to a single, highly frequent canonical word, it creates cleaner and denser input for traditional ML models such as **TF-IDF + SVM**, **Random Forest**, or **Logistic Regression**.
+
+After the initial processing stage, inference on new text requires only a dictionary lookup, enabling **O(1) runtime complexity**.
+
+---
+
+# Features
+
+- **Context-Aware Processing**  
+  Uses Transformer models (default: `bert-base-multilingual-cased`) to capture contextual meaning instead of relying on static embeddings.
+
+- **Smart Vocabulary Reduction**  
+  Automatically replaces rare or obscure words with their most frequent semantic equivalents.
+
+- **O(1) Inference Speed**  
+  After training, the BERT and FAISS pipeline is no longer required. Text reduction happens via a lightweight Python dictionary.
+
+- **Production Ready**  
+  Supports saving and loading compiled reduction maps for deployment.
+
+- **Hardware Agnostic**  
+  Automatically detects and uses GPU (CUDA) if available, while remaining fully functional on CPU.
+
+---
+
+# Installation
+
+Install via pip:
+
+```bash
+pip install semantic-reducer
+```
+
+---
+
+# Quick Start
+
+## 1. Training and Building the Reduction Map
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize the reducer
+# (downloads the BERT model on first run)
+reducer = SemanticReducer()
+
+# Training corpus
+corpus = [
+    "The quick brown fox jumps over the lazy dog.",
+    "A fast dark-colored canine leaps above a tired hound.",
+    "Speedy foxes jump over sleepy dogs."
+]
+
+# Extract contextual representations
+reducer.process_corpus_contextually(corpus, batch_size=2)
+
+# Finalize embeddings and create FAISS index
+reducer.finalize_embeddings()
+reducer.build_index()
+
+# Build the semantic reduction dictionary
+reducer.build_reduction_map(
+    threshold=0.85,   # similarity threshold
+    top_k=5           # number of neighbors to consider
+)
+
+# Save the trained system
+reducer.save_system(prefix="my_corpus")
+```
+
+---
+
+## 2. Lightning-Fast Inference
+
+Once the reduction map is built, **BERT is no longer required**. New text is processed through a dictionary lookup.
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize reducer
+reducer = SemanticReducer()
+
+# Load saved system
+reducer.load_system(prefix="my_corpus")
+
+# Reduce new text instantly
+new_text = "The rapid fox leaped."
+reduced_text = reducer.reduce_text(new_text)
+
+print(f"Original: {new_text}")
+print(f"Reduced:  {reduced_text}")
+```
+
+---
+
+# Workflow Overview
+
+1. **Contextual Encoding**  
+   Extract contextual word embeddings using BERT.
+
+2. **Vector Indexing**  
+   Store embeddings in a FAISS similarity search index.
+
+3. **Semantic Clustering**  
+   Identify semantically similar words.
+
+4. **Canonical Replacement**  
+   Replace each cluster with the most frequent word in the corpus.
+
+5. **Dictionary Compilation**  
+   Store the final mappings for fast inference.
+
+---
+
+# Requirements
+
+- Python ≥ 3.8
+- torch
+- numpy
+- faiss-cpu
+- transformers
+- tqdm
+
+Install dependencies manually if needed:
+
+```bash
+pip install torch numpy faiss-cpu transformers tqdm
+```
+
+---
+
+# License
+
+This project is licensed under the **MIT License**.

semantic_reducer-0.1.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "semantic-reducer"
+version = "0.1.0"
+authors = [
+  { name="Your Name", email="your.email@example.com" },
+]
+description = "O(1) semantic vocabulary reduction using BERT and FAISS."
+readme = "README.md"
+requires-python = ">=3.8"
+license = { text = "MIT License" }
+keywords = ["nlp", "bert", "faiss", "embeddings", "text-reduction", "machine-learning"]
+
+# Classifiers help users filter and find your project on PyPI
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Text Processing :: Linguistic",
+]
+
+# The libraries installed automatically when someone pip installs your package
+dependencies = [
+    "torch>=1.10.0",
+    "numpy>=1.21.0",
+    "faiss-cpu>=1.7.0",
+    "tqdm>=4.60.0",
+    "transformers>=4.20.0"
+]
+
+# These links will appear on the left sidebar of your PyPI page
+[project.urls]
+Homepage = "https://github.com/abkafi1234/Semantic-Reducer"
+Repository = "https://github.com/abkafi1234/Semantic-Reducer"
+"Bug Tracker" = "https://github.com/abkafi1234/Semantic-Reducer"
+
+# Explicitly tells the build tool to look inside the src/ folder for your code
+[tool.setuptools.packages.find]
+where = ["src"]

semantic_reducer-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

semantic_reducer-0.1.0/src/semantic_reducer/__init__.py

@@ -0,0 +1,3 @@
+__version__ = "0.1.0"
+
+from .reducer import SemanticReducer

semantic_reducer-0.1.0/src/semantic_reducer/reducer.py

@@ -0,0 +1,150 @@
+import torch
+import numpy as np
+import faiss
+import pickle
+import os
+from tqdm import tqdm
+from collections import defaultdict, Counter
+from transformers import AutoTokenizer, AutoModel
+
+class SemanticReducer:
+    def __init__(self, model_name='bert-base-multilingual-cased', cache_dir='./cache'):
+        self.model_name = model_name
+        self.cache_dir = cache_dir
+        os.makedirs(cache_dir, exist_ok=True)
+        
+        # Use Fast tokenizer for word_ids mapping
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+        self.model = AutoModel.from_pretrained(model_name)
+        self.model.eval() # Ensure evaluation mode
+        
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        self.model.to(self.device)
+
+        self.word_counts = Counter()
+        self.word_embeddings_sum = defaultdict(lambda: np.zeros(self.model.config.hidden_size))
+        self.vocab_embeddings = None
+        self.vocab_list = []
+        self.index = None
+        self.reduction_map = {} # O(1) inference dictionary
+
+    def process_corpus_contextually(self, sentences, batch_size=16):
+        """
+        Processes entire sentences to extract sense-averaged contextual embeddings.
+        Addresses Reviewer 3's concern about isolated token embeddings.
+        """
+        print("Extracting contextual embeddings from corpus...")
+        
+        # Pre-tokenize by whitespace to maintain word boundaries
+        split_sentences = [str(sent).strip().split() for sent in sentences]
+        
+        for i in tqdm(range(0, len(split_sentences), batch_size), desc="Processing Batches"):
+            batch = split_sentences[i:i+batch_size]
+            
+            # Tokenize batch with word alignment (returns a BatchEncoding object)
+            inputs = self.tokenizer(batch, is_split_into_words=True, return_tensors='pt', 
+                                    padding=True, truncation=True, max_length=512)
+            
+            # Move tensors to device FOR THE MODEL, but do NOT overwrite 'inputs'
+            model_inputs = {k: v.to(self.device) for k, v in inputs.items()}
+            
+            with torch.no_grad():
+                outputs = self.model(**model_inputs)
+            
+            # Use last_hidden_state (explicitly addressing Rev 4, Q2)
+            hidden_states = outputs.last_hidden_state.cpu().numpy()
+            
+            # Aggregate subwords into whole words
+            for batch_idx in range(len(batch)):
+                # Now this works perfectly because inputs is still a BatchEncoding
+                word_ids = inputs.word_ids(batch_index=batch_idx) 
+                original_words = batch[batch_idx]
+                
+                # Group subword embeddings by word_id
+                word_embeddings_temp = defaultdict(list)
+                for seq_idx, word_idx in enumerate(word_ids):
+                    if word_idx is not None: # Ignore [CLS], [SEP], [PAD]
+                        word_embeddings_temp[word_idx].append(hidden_states[batch_idx, seq_idx, :])
+                
+                # Average subwords to get the word embedding, add to global corpus sum
+                for word_idx, embeddings in word_embeddings_temp.items():
+                    # Safeguard just in case word_idx is out of bounds
+                    if word_idx < len(original_words):
+                        word = original_words[word_idx]
+                        avg_word_emb = np.mean(embeddings, axis=0)
+                        
+                        self.word_embeddings_sum[word] += avg_word_emb
+                        self.word_counts[word] += 1
+
+    def finalize_embeddings(self):
+        """Averages the accumulated contextual embeddings to create static representations."""
+        print("Finalizing sense-averaged embeddings...")
+        self.vocab_list = list(self.word_counts.keys())
+        self.vocab_embeddings = np.zeros((len(self.vocab_list), self.model.config.hidden_size), dtype=np.float32)
+        
+        for idx, word in enumerate(self.vocab_list):
+            # Divide sum by count to get the mean contextual embedding
+            self.vocab_embeddings[idx] = self.word_embeddings_sum[word] / self.word_counts[word]
+            
+        # L2 Normalize for FAISS Inner Product (Cosine Similarity)
+        faiss.normalize_L2(self.vocab_embeddings)
+
+    def build_index(self):
+        """Builds the FAISS index. Explicitly uses Inner Product (Cosine)."""
+        dim = self.vocab_embeddings.shape[1]
+        self.index = faiss.IndexFlatIP(dim) # Cosine similarity since vectors are L2 normalized
+        self.index.add(self.vocab_embeddings)
+        print(f"Built FAISS index with {self.index.ntotal} vectors of dimension {dim}.")
+
+    def build_reduction_map(self, threshold=0.9, top_k=10):
+        """
+        Creates the global mapping dictionary.
+        Uses FREQUENCY instead of length to pick the canonical word (Fixes Rev 3, Concern 3).
+        """
+        print(f"Building semantic reduction map (threshold={threshold})...")
+        self.reduction_map = {}
+        
+        # Search all words against the index at once for efficiency
+        distances, indices = self.index.search(self.vocab_embeddings, top_k)
+        
+        for i, word in enumerate(tqdm(self.vocab_list, desc="Mapping Vocabulary")):
+            candidates = []
+            for j in range(top_k):
+                dist = distances[i][j]
+                neighbor_idx = indices[i][j]
+                
+                if dist >= threshold:
+                    neighbor_word = self.vocab_list[neighbor_idx]
+                    candidates.append(neighbor_word)
+            
+            if candidates:
+                # The core fix: Sort candidates by corpus frequency (descending)
+                # If tied, fall back to shorter length as secondary heuristic
+                candidates.sort(key=lambda w: (self.word_counts[w], -len(w)), reverse=True)
+                
+                # The most frequent semantic neighbor becomes the representative token
+                self.reduction_map[word] = candidates[0]
+            else:
+                self.reduction_map[word] = word
+
+    def reduce_text(self, text):
+        """O(1) inference for new text."""
+        words = text.split()
+        return ' '.join([self.reduction_map.get(w, w) for w in words])
+
+    def save_system(self, prefix="semred"):
+        """Saves the minimal artifacts needed for inference."""
+        file_path = os.path.join(self.cache_dir, f"{prefix}_map.pkl")
+        with open(file_path, 'wb') as f:
+            pickle.dump(self.reduction_map, f)
+        print(f"System saved successfully to {file_path}")
+
+    def load_system(self, prefix="semred"):
+        """Loads a previously saved mapping dictionary for fast inference."""
+        file_path = os.path.join(self.cache_dir, f"{prefix}_map.pkl")
+        if not os.path.exists(file_path):
+            raise FileNotFoundError(f"Could not find a saved system at {file_path}")
+            
+        with open(file_path, 'rb') as f:
+            self.reduction_map = pickle.load(f)
+        print(f"System loaded successfully from {file_path}")

semantic_reducer-0.1.0/src/semantic_reducer.egg-info/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: semantic-reducer
+Version: 0.1.0
+Summary: O(1) semantic vocabulary reduction using BERT and FAISS.
+Author-email: Your Name <your.email@example.com>
+License: MIT License
+Project-URL: Homepage, https://github.com/abkafi1234/Semantic-Reducer
+Project-URL: Repository, https://github.com/abkafi1234/Semantic-Reducer
+Project-URL: Bug Tracker, https://github.com/abkafi1234/Semantic-Reducer
+Keywords: nlp,bert,faiss,embeddings,text-reduction,machine-learning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=1.10.0
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: faiss-cpu>=1.7.0
+Requires-Dist: tqdm>=4.60.0
+Requires-Dist: transformers>=4.20.0
+Dynamic: license-file
+
+# Semantic Reducer
+
+A high-performance Python library that bridges the gap between **Deep Learning** and **traditional Machine Learning**.
+
+`semantic-reducer` uses **BERT contextual embeddings** and **FAISS vector search** to semantically normalize and reduce the vocabulary of a text corpus.
+
+By mapping semantically equivalent words (for example: *fast*, *quick*, *rapid*) to a single, highly frequent canonical word, it creates cleaner and denser input for traditional ML models such as **TF-IDF + SVM**, **Random Forest**, or **Logistic Regression**.
+
+After the initial processing stage, inference on new text requires only a dictionary lookup, enabling **O(1) runtime complexity**.
+
+---
+
+# Features
+
+- **Context-Aware Processing**  
+  Uses Transformer models (default: `bert-base-multilingual-cased`) to capture contextual meaning instead of relying on static embeddings.
+
+- **Smart Vocabulary Reduction**  
+  Automatically replaces rare or obscure words with their most frequent semantic equivalents.
+
+- **O(1) Inference Speed**  
+  After training, the BERT and FAISS pipeline is no longer required. Text reduction happens via a lightweight Python dictionary.
+
+- **Production Ready**  
+  Supports saving and loading compiled reduction maps for deployment.
+
+- **Hardware Agnostic**  
+  Automatically detects and uses GPU (CUDA) if available, while remaining fully functional on CPU.
+
+---
+
+# Installation
+
+Install via pip:
+
+```bash
+pip install semantic-reducer
+```
+
+---
+
+# Quick Start
+
+## 1. Training and Building the Reduction Map
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize the reducer
+# (downloads the BERT model on first run)
+reducer = SemanticReducer()
+
+# Training corpus
+corpus = [
+    "The quick brown fox jumps over the lazy dog.",
+    "A fast dark-colored canine leaps above a tired hound.",
+    "Speedy foxes jump over sleepy dogs."
+]
+
+# Extract contextual representations
+reducer.process_corpus_contextually(corpus, batch_size=2)
+
+# Finalize embeddings and create FAISS index
+reducer.finalize_embeddings()
+reducer.build_index()
+
+# Build the semantic reduction dictionary
+reducer.build_reduction_map(
+    threshold=0.85,   # similarity threshold
+    top_k=5           # number of neighbors to consider
+)
+
+# Save the trained system
+reducer.save_system(prefix="my_corpus")
+```
+
+---
+
+## 2. Lightning-Fast Inference
+
+Once the reduction map is built, **BERT is no longer required**. New text is processed through a dictionary lookup.
+
+```python
+from semantic_reducer import SemanticReducer
+
+# Initialize reducer
+reducer = SemanticReducer()
+
+# Load saved system
+reducer.load_system(prefix="my_corpus")
+
+# Reduce new text instantly
+new_text = "The rapid fox leaped."
+reduced_text = reducer.reduce_text(new_text)
+
+print(f"Original: {new_text}")
+print(f"Reduced:  {reduced_text}")
+```
+
+---
+
+# Workflow Overview
+
+1. **Contextual Encoding**  
+   Extract contextual word embeddings using BERT.
+
+2. **Vector Indexing**  
+   Store embeddings in a FAISS similarity search index.
+
+3. **Semantic Clustering**  
+   Identify semantically similar words.
+
+4. **Canonical Replacement**  
+   Replace each cluster with the most frequent word in the corpus.
+
+5. **Dictionary Compilation**  
+   Store the final mappings for fast inference.
+
+---
+
+# Requirements
+
+- Python ≥ 3.8
+- torch
+- numpy
+- faiss-cpu
+- transformers
+- tqdm
+
+Install dependencies manually if needed:
+
+```bash
+pip install torch numpy faiss-cpu transformers tqdm
+```
+
+---
+
+# License
+
+This project is licensed under the **MIT License**.

semantic_reducer-0.1.0/src/semantic_reducer.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+pyproject.toml
+src/semantic_reducer/__init__.py
+src/semantic_reducer/reducer.py
+src/semantic_reducer.egg-info/PKG-INFO
+src/semantic_reducer.egg-info/SOURCES.txt
+src/semantic_reducer.egg-info/dependency_links.txt
+src/semantic_reducer.egg-info/requires.txt
+src/semantic_reducer.egg-info/top_level.txt

semantic_reducer-0.1.0/src/semantic_reducer.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

semantic_reducer-0.1.0/src/semantic_reducer.egg-info/requires.txt

@@ -0,0 +1,5 @@
+torch>=1.10.0
+numpy>=1.21.0
+faiss-cpu>=1.7.0
+tqdm>=4.60.0
+transformers>=4.20.0

semantic_reducer-0.1.0/src/semantic_reducer.egg-info/top_level.txt

@@ -0,0 +1 @@
+semantic_reducer