semantic-compressor 1.2__tar.gz → 1.4__tar.gz

{semantic_compressor-1.2/semantic_compressor.egg-info → semantic_compressor-1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: semantic_compressor
-Version: 1.2
+Version: 1.4
 Author: Carlo Moro
 Author-email: Carlo Moro <cnmoro@gmail.com>
 Classifier: Programming Language :: Python :: 3
@@ -15,9 +15,10 @@ Requires-Dist: scikit-learn
 Requires-Dist: fasttext
 Requires-Dist: onnxruntime
 Requires-Dist: onnxruntime-extensions
+Requires-Dist: pyspellchecker
 
 ```python
-from compressor.semantic import compress_text
+from compressor.semantic import compress_text, find_needle_in_haystack
 
 text = """
 Akin to France's heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They're said to be home to more cattle than people, and despite being home to less than 3% of Portugal's population, the islands produce 30% of Portugal's dairy products and 13% of its beef.   Beef is part of everyday life in the Azores, and come spring on one particular island, the ingredient even crosses paths with religion.   In the days following Easter, Azorean people kick off a series of religious celebrations called Festas do Espírito Santo (Festivals of the Holy Spirit). During the 13th Century, a Catholic sect called the Cult of the Holy Spirit predicted a utopian era on Earth. This fringe faith was discouraged in mainland Europe but lived on in these remote islands in the middle of the Atlantic Ocean. The sect was also promoted by Portuguese queen Elizabeth of Aragon (also known as Elizabeth of Portugal), who was known for her charity.  Over the subsequent centuries, a series of festivals emerged on the Azores that blended these utopian aspirations with the queen's alleged generosity. Between Easter and the week following Whitsunday, a total of eight weeks, the islands host a series of parades and other cultural and religious festivals that revolve around brightly coloured community houses called impérios. During this time, the community houses also collect donations from locals, which is then redistributed to people in the form of bread, beef and wine.  These three elements generally come together in the form of a soup, called sopa do Espírito Santo, that's served at the impérios during the festivals. But on the island of Terceira, locals combine these ingredients in a different and delicious way, one that's become synonymous with the island's culinary identity.  Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush)Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush) "People eat alcatra year round, but especially during the celebrations in spring and summer," explains Duarte Fournier. He is the Grand Master of the Brotherhood of Alcatra, a culinary fraternity on Terceira, and is telling me about the island's signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of France's famed boeuf bourguignon.  We're sitting at a cafe in Angra do Heroísmo, Terceira's largest city, and as we chat, children race to and from a nearby império delivering massive trays of raw beef to neighbours. Fournier tells me that alcatra likely has its origins in northern Portugal, where there's a tradition of baking goat in wine.  "We don't know why it's called alcatra," he says. "We suppose it's from Arabic. Al catar means 'small pieces of meat'."  According to Fournier, alcatra differs from mainland Portugal's baked meat dishes in that it includes dried spices, generally allspice and black peppercorns, but also sometimes clove or cinnamon.
@@ -30,4 +31,15 @@ print(compressed_text_90_percent)
 compressed_text_to_100_tokens = compress_text(text, target_token_count=100)
 print(compressed_text_to_100_tokens)
 # 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.'
+
+text_reference = "Archipelago Islands"
+compressed_text_with_steering = compress_text(text, compression_rate=0.7, reference_text_steering=text_reference)
+# 'Akin to france\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote azores archipelago. The azores, an archipelago of nine islands belonging to portugal and located roughly between europe and the us, are cow country. Beef is part of everyday life in the azores, and come spring on one particular island, the ingredient even crosses paths with religion. But on the island of terceira, locals combine these ingredients in a different and delicious way, one that\'s become synonymous with the island\'s culinary identity. He is the grand master of the brotherhood of alcatra, a culinary fraternity on terceira, and is telling me about the island\'s signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of france\'s famed boeuf bourguignon.'
+
+needle_in_haystack = find_needle_in_haystack(
+    haystack = text,
+    needle = "Archipelago Islands",
+    block_size = 200
+)
+# 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They\'re said to be home to more cattle than people, and despite being home to less than 3% of Portugal\'s population, the islands'
 ```

{semantic_compressor-1.2 → semantic_compressor-1.4}/README.md

@@ -1,5 +1,5 @@
 ```python
-from compressor.semantic import compress_text
+from compressor.semantic import compress_text, find_needle_in_haystack
 
 text = """
 Akin to France's heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They're said to be home to more cattle than people, and despite being home to less than 3% of Portugal's population, the islands produce 30% of Portugal's dairy products and 13% of its beef.   Beef is part of everyday life in the Azores, and come spring on one particular island, the ingredient even crosses paths with religion.   In the days following Easter, Azorean people kick off a series of religious celebrations called Festas do Espírito Santo (Festivals of the Holy Spirit). During the 13th Century, a Catholic sect called the Cult of the Holy Spirit predicted a utopian era on Earth. This fringe faith was discouraged in mainland Europe but lived on in these remote islands in the middle of the Atlantic Ocean. The sect was also promoted by Portuguese queen Elizabeth of Aragon (also known as Elizabeth of Portugal), who was known for her charity.  Over the subsequent centuries, a series of festivals emerged on the Azores that blended these utopian aspirations with the queen's alleged generosity. Between Easter and the week following Whitsunday, a total of eight weeks, the islands host a series of parades and other cultural and religious festivals that revolve around brightly coloured community houses called impérios. During this time, the community houses also collect donations from locals, which is then redistributed to people in the form of bread, beef and wine.  These three elements generally come together in the form of a soup, called sopa do Espírito Santo, that's served at the impérios during the festivals. But on the island of Terceira, locals combine these ingredients in a different and delicious way, one that's become synonymous with the island's culinary identity.  Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush)Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush) "People eat alcatra year round, but especially during the celebrations in spring and summer," explains Duarte Fournier. He is the Grand Master of the Brotherhood of Alcatra, a culinary fraternity on Terceira, and is telling me about the island's signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of France's famed boeuf bourguignon.  We're sitting at a cafe in Angra do Heroísmo, Terceira's largest city, and as we chat, children race to and from a nearby império delivering massive trays of raw beef to neighbours. Fournier tells me that alcatra likely has its origins in northern Portugal, where there's a tradition of baking goat in wine.  "We don't know why it's called alcatra," he says. "We suppose it's from Arabic. Al catar means 'small pieces of meat'."  According to Fournier, alcatra differs from mainland Portugal's baked meat dishes in that it includes dried spices, generally allspice and black peppercorns, but also sometimes clove or cinnamon.
@@ -12,4 +12,15 @@ print(compressed_text_90_percent)
 compressed_text_to_100_tokens = compress_text(text, target_token_count=100)
 print(compressed_text_to_100_tokens)
 # 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.'
+
+text_reference = "Archipelago Islands"
+compressed_text_with_steering = compress_text(text, compression_rate=0.7, reference_text_steering=text_reference)
+# 'Akin to france\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote azores archipelago. The azores, an archipelago of nine islands belonging to portugal and located roughly between europe and the us, are cow country. Beef is part of everyday life in the azores, and come spring on one particular island, the ingredient even crosses paths with religion. But on the island of terceira, locals combine these ingredients in a different and delicious way, one that\'s become synonymous with the island\'s culinary identity. He is the grand master of the brotherhood of alcatra, a culinary fraternity on terceira, and is telling me about the island\'s signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of france\'s famed boeuf bourguignon.'
+
+needle_in_haystack = find_needle_in_haystack(
+    haystack = text,
+    needle = "Archipelago Islands",
+    block_size = 200
+)
+# 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They\'re said to be home to more cattle than people, and despite being home to less than 3% of Portugal\'s population, the islands'
 ```

{semantic_compressor-1.2 → semantic_compressor-1.4}/compressor/semantic.py

@@ -6,14 +6,24 @@ from onnxruntime_extensions import get_library_path
 from compressor.minbpe.regex import RegexTokenizer
 from nltk.tokenize import sent_tokenize
 from multiprocessing import cpu_count
+from spellchecker import SpellChecker
+from nltk.stem import PorterStemmer
+from nltk.stem import RSLPStemmer
 from collections import Counter
 import onnxruntime as ort
+import nltk
+
+# Inicializando os stemmers
+stemmer_english = PorterStemmer()
+stemmer_portuguese = RSLPStemmer()
 
 tokenizer = RegexTokenizer()
 nltk_data_path = str(importlib.resources.files('compressor').joinpath('resources/nltk_data'))
 
 os.environ['NLTK_DATA'] = nltk_data_path
 
+nltk.download('rslp')
+
 english_stopwords_path = str(importlib.resources.files('compressor').joinpath('resources/en_stopwords.pkl'))
 portuguese_stopwords_path = str(importlib.resources.files('compressor').joinpath('resources/pt_stopwords.pkl'))
 fasttext_model_path = str(importlib.resources.files('compressor').joinpath('resources/lid.176.ftz'))
@@ -213,8 +223,6 @@ def compress_text(text, *, target_token_count=None, compression_rate=0.7, refere
     try:
         if target_token_count is None:
             compression_rate = 1 - compression_rate
-            original_token_count = count_tokens(text)
-            target_token_count = int(original_token_count * compression_rate)
         else:
             original_token_count = count_tokens(text)
             if original_token_count <= target_token_count:
@@ -232,9 +240,28 @@ def compress_text(text, *, target_token_count=None, compression_rate=0.7, refere
 
     return text
 
+def stem_text(text, lang='en'):
+    if lang == 'en':
+        stems = [stemmer_english.stem(word) for word in text.split()]
+        stemmed_text = " ".join(stems)
+    else:
+        stems = [stemmer_portuguese.stem(word) for word in text.split()]
+        stemmed_text = " ".join(stems)
+
+    return stemmed_text
+
+def correct_spelling(frase, detected_lang="pt"):
+    spell = SpellChecker(language=detected_lang)
+    words = frase.split()
+    fixed = [spell.correction(word) for word in words]
+    return " ".join(fixed)
+
 def find_needle_in_haystack(
-        *, haystack: str, needle: str, block_size = 350,
-        semantic_embeddings_weight: float = 0.3, textual_embeddings_weight: float = 0.7
+        *, haystack: str, needle: str, block_size = 300,
+        semantic_embeddings_weight: float = 0.3,
+        textual_embeddings_weight: float = 0.7,
+        use_stemming: bool = False,
+        correct_spelling_needle: bool = False
     ):
     """
     Finds the string block in the haystack that contains the needle.
@@ -245,7 +272,9 @@ def find_needle_in_haystack(
         block_size (int, optional): The size of each string block. The needle will be searched in each block. Defaults to 350.
         semantic_embeddings_weight (float, optional): The weight of the semantic embeddings in the similarity calculation. Defaults to 0.3.
         textual_embeddings_weight (float, optional): The weight of the textual embeddings in the similarity calculation. Defaults to 0.7.
-
+        use_stemming (bool, optional): Whether to use stemming for the text. Defaults to False.
+        correct_spelling_needle (bool, optional): Whether to correct the spelling of the needle. Defaults to False.
+        
     Returns:
         str: The string block in the haystack that contains the needle. The size of the needle will be less than or equal to the block size.
     """
@@ -253,14 +282,19 @@ def find_needle_in_haystack(
     try:
         # Split the haystack into blocks
         blocks = structurize_text(haystack, tokens_per_chunk=block_size)
+
+        lang = detect_language(f"{needle}\n\n{haystack}")
+        
+        if correct_spelling_needle:
+            needle = correct_spelling(needle, lang)
         
         # Compute the embeddings of the needle
         needle_semantic_embedding = extract_semantic_embeddings(needle)
-        needle_textual_embedding = extract_textual_embeddings(needle.lower())
+        needle_textual_embedding = extract_textual_embeddings(needle.lower() if not use_stemming else stem_text(needle, lang))
 
         # Compute the embeddings of the haystack (each block)
         haystack_semantic_embeddings = [extract_semantic_embeddings(block) for block in blocks]
-        haystack_textual_embeddings = [extract_textual_embeddings(block.lower()) for block in blocks]
+        haystack_textual_embeddings = [extract_textual_embeddings(block.lower() if not use_stemming else stem_text(block.lower(), lang)) for block in blocks]
 
         # Compute the similarity between the needle and each block
         semantic_similarities = [calculate_similarity(needle_semantic_embedding, block_embedding) for block_embedding in haystack_semantic_embeddings]
@@ -275,7 +309,9 @@ def find_needle_in_haystack(
         # Find the index of the needle in all the blocks
         most_similar_block_index = blocks.index(most_similar_block)
 
-        needle_region = blocks[most_similar_block_index-1:most_similar_block_index+2]
+        start_index = most_similar_block_index-1 if most_similar_block_index > 0 else 0
+
+        needle_region = blocks[start_index:most_similar_block_index+2]
 
         return ''.join(needle_region).strip()
     except Exception:

{semantic_compressor-1.2 → semantic_compressor-1.4}/pyproject.toml

@@ -1,10 +1,10 @@
 [build-system]
-requires = ["setuptools>=61.0", "numpy<2", "nltk", "scikit-learn", "fasttext", "onnxruntime", "onnxruntime-extensions"]
+requires = ["setuptools>=61.0", "numpy<2", "nltk", "scikit-learn", "fasttext", "onnxruntime", "onnxruntime-extensions", "pyspellchecker"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantic_compressor"
-version = "1.2"
+version = "1.4"
 authors = [
   { name="Carlo Moro", email="cnmoro@gmail.com" },
 ]
@@ -22,5 +22,6 @@ dependencies = [
     "scikit-learn",
     "fasttext",
     "onnxruntime",
-    "onnxruntime-extensions"
+    "onnxruntime-extensions",
+    "pyspellchecker"
 ]

{semantic_compressor-1.2 → semantic_compressor-1.4/semantic_compressor.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: semantic_compressor
-Version: 1.2
+Version: 1.4
 Author: Carlo Moro
 Author-email: Carlo Moro <cnmoro@gmail.com>
 Classifier: Programming Language :: Python :: 3
@@ -15,9 +15,10 @@ Requires-Dist: scikit-learn
 Requires-Dist: fasttext
 Requires-Dist: onnxruntime
 Requires-Dist: onnxruntime-extensions
+Requires-Dist: pyspellchecker
 
 ```python
-from compressor.semantic import compress_text
+from compressor.semantic import compress_text, find_needle_in_haystack
 
 text = """
 Akin to France's heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They're said to be home to more cattle than people, and despite being home to less than 3% of Portugal's population, the islands produce 30% of Portugal's dairy products and 13% of its beef.   Beef is part of everyday life in the Azores, and come spring on one particular island, the ingredient even crosses paths with religion.   In the days following Easter, Azorean people kick off a series of religious celebrations called Festas do Espírito Santo (Festivals of the Holy Spirit). During the 13th Century, a Catholic sect called the Cult of the Holy Spirit predicted a utopian era on Earth. This fringe faith was discouraged in mainland Europe but lived on in these remote islands in the middle of the Atlantic Ocean. The sect was also promoted by Portuguese queen Elizabeth of Aragon (also known as Elizabeth of Portugal), who was known for her charity.  Over the subsequent centuries, a series of festivals emerged on the Azores that blended these utopian aspirations with the queen's alleged generosity. Between Easter and the week following Whitsunday, a total of eight weeks, the islands host a series of parades and other cultural and religious festivals that revolve around brightly coloured community houses called impérios. During this time, the community houses also collect donations from locals, which is then redistributed to people in the form of bread, beef and wine.  These three elements generally come together in the form of a soup, called sopa do Espírito Santo, that's served at the impérios during the festivals. But on the island of Terceira, locals combine these ingredients in a different and delicious way, one that's become synonymous with the island's culinary identity.  Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush)Austin Bush The Festas do Espírito Santo revolve around community houses called impérios (Credit: Austin Bush) "People eat alcatra year round, but especially during the celebrations in spring and summer," explains Duarte Fournier. He is the Grand Master of the Brotherhood of Alcatra, a culinary fraternity on Terceira, and is telling me about the island's signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of France's famed boeuf bourguignon.  We're sitting at a cafe in Angra do Heroísmo, Terceira's largest city, and as we chat, children race to and from a nearby império delivering massive trays of raw beef to neighbours. Fournier tells me that alcatra likely has its origins in northern Portugal, where there's a tradition of baking goat in wine.  "We don't know why it's called alcatra," he says. "We suppose it's from Arabic. Al catar means 'small pieces of meat'."  According to Fournier, alcatra differs from mainland Portugal's baked meat dishes in that it includes dried spices, generally allspice and black peppercorns, but also sometimes clove or cinnamon.
@@ -30,4 +31,15 @@ print(compressed_text_90_percent)
 compressed_text_to_100_tokens = compress_text(text, target_token_count=100)
 print(compressed_text_to_100_tokens)
 # 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.'
+
+text_reference = "Archipelago Islands"
+compressed_text_with_steering = compress_text(text, compression_rate=0.7, reference_text_steering=text_reference)
+# 'Akin to france\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote azores archipelago. The azores, an archipelago of nine islands belonging to portugal and located roughly between europe and the us, are cow country. Beef is part of everyday life in the azores, and come spring on one particular island, the ingredient even crosses paths with religion. But on the island of terceira, locals combine these ingredients in a different and delicious way, one that\'s become synonymous with the island\'s culinary identity. He is the grand master of the brotherhood of alcatra, a culinary fraternity on terceira, and is telling me about the island\'s signature dish: cuts of beef braised in local wine, smoked pork fat and dried spices, resulting in something of a heartier, spicier, richer version of france\'s famed boeuf bourguignon.'
+
+needle_in_haystack = find_needle_in_haystack(
+    haystack = text,
+    needle = "Archipelago Islands",
+    block_size = 200
+)
+# 'Akin to France\'s heartier, spicier, richer boeuf bourguignon, "alcatra" is synonymous with a single island in the remote Azores archipelago.  The Azores, an archipelago of nine islands belonging to Portugal and located roughly between Europe and the US, are cow country. They\'re said to be home to more cattle than people, and despite being home to less than 3% of Portugal\'s population, the islands'
 ```

{semantic_compressor-1.2 → semantic_compressor-1.4}/semantic_compressor.egg-info/requires.txt

@@ -4,3 +4,4 @@ scikit-learn
 fasttext
 onnxruntime
 onnxruntime-extensions
+pyspellchecker

{semantic_compressor-1.2 → semantic_compressor-1.4}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
 
 setup(
     name='semantic_compressor',
-    version='1.2',
+    version='1.4',
     author='Carlo Moro',
     author_email='cnmoro@gmail.com',
     description="Semantic text compression",
@@ -17,7 +17,8 @@ setup(
         "scikit-learn",
         "fasttext",
         "onnxruntime",
-        "onnxruntime-extensions"
+        "onnxruntime-extensions",
+        "pyspellchecker"
     ],
     classifiers=[
         'Programming Language :: Python :: 3',