LanguageStatisticsLibPy 1.0.3__tar.gz → 1.0.4__tar.gz

{languagestatisticslibpy-1.0.3 → languagestatisticslibpy-1.0.4}/PKG-INFO

@@ -1,8 +1,9 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: LanguageStatisticsLibPy
-Version: 1.0.3
+Version: 1.0.4
 Summary: Quick and robust Python functions to do cryptanalysis using language statistics data for 15 languages
 License: Apache-2.0
+License-File: LICENSE
 Author: Nils Kopal and Bernhard Esslinger
 Author-email: kopal@cryptool.org
 Requires-Python: >=3.10,<4.0
@@ -13,6 +14,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Project-URL: Homepage, https://github.com/CrypToolProject/LanguageStatisticsLibPy
 Description-Content-Type: text/markdown
 

{languagestatisticslibpy-1.0.3 → languagestatisticslibpy-1.0.4}/README.md

@@ -1,106 +1,106 @@
-# LanguageStatisticsLibPy
-
-LanguageStatisticsLibPy is a Python library designed to facilitate the fast analysis and manipulation of language statistics data. It originates from a C# library that was first used in the widespread cryptography e-learning software CrypTool 2. From now on "CrypTool 2" is abbreviated "CT2". CT2 is an open-source e-learning program for Windows to do cryptography and cryptanalysis (https://www.cryptool.org/en/ct2/).
-
-This Python library supports 15 different languages and offers functionality for generating and handling n-gram data, specifically for calculating n-gram frequencies using the language statistics files from CT2.
-Additionally, it facilitates the use of CT2's dictionaries through a "Word Tree", an efficient data structure for rapid word searches within a language.
-
-The language statistics files (for example, `en-5gram-nocs.gz` indicates an English 5-gram file that is not case-sensitive and excludes spaces) can be found in the "LanguageStatistics" subdirectory of CT2, if you have installed CT2 on Windows.
-If you don't have a Windows machine or you don't want to install CT2, you may download the language statistics files and dictionaries from the CT2 Github repo: [Language Statistics](https://github.com/CrypToolProject/CrypTool-2/tree/main/LanguageStatistics).
-
-Remark: This package contains the implemented algorithms without the language statistics files. These files have to be downloaded separately as they occupy many megabytes.
-
-## Features
-
-- **Support for multiple languages**: The library includes predefined support for 15 languages, including English, German, Spanish, French -- each with its own set of unigram frequencies and alphabets.
-  
-- **N-gram loading**: Users can load unigrams, bigrams, trigrams, tetragrams, pentagrams, and hexagrams as n-gram objects in supported languages, with the option to include or exclude spaces. The n-grams delivered in the language statistics files range from 1 to 5 (we don't deliver 6-grams within CT2, since these files are too big). All language statistics files delivered are case-insensitive, denoted as "nocs" in the filename. Each language statistics is available in two forms: with space/blank ("sp" in the filename) and without space/blank (indicated by the absence of "sp" in the filename) within the alphabet.
-
-- **Index of Coincidence calculation**: It offers a method to calculate the index of coincidence (IoC) for a given plaintext, which is useful for cryptanalysis and language pattern recognition.
-  
-- **Alphabet and number mapping**: The library provides functionality to map characters to their respective positions in a language's alphabet and vice versa, supporting operations on encoded messages or language data.
-
-- **Dynamic n-gram support**: Depending on the available data, the library dynamically supports various n-gram types. 
-- **Word tree data structure**: It supports a word tree data structure for fast word lookups (true = part of language, false = not part of language) of a specific language.
-
-## Usage
-
-Prerequisites: LanguageStatisticsLibPy is installed on your computer via 
-**$ pip3 install LanguageStatisticsLibPy**
-
-1. **Initialization**: Start by importing the `LanguageStatistics` class and specify the language code for your analysis.
-2. **Loading n-grams**: To load n-grams of your chosen type (e.g., unigrams, bigrams) for a specific language, use the `create_grams` method with the appropriate .gz file from the LanguageStatistics directory in CT2. For instance, to load English 4-grams that are case-insensitive and include the space/blank symbol, use the file named `en-4gram-nocs-sp.gz`.
-3. **Calculating IoC**: Calculate the index of coincidence for a given plaintext using the `calculate_ioc` method.
-4. **Word tree loading**: For advanced language analysis, load a pre-built word tree for a specific language using the `load_word_tree` method.
-
-Sample usage (from file `test1.py`):
-
-```python
-from languagestatisticslibpy.LanguageStatistics import LanguageStatistics as LS
-
-plaintext = LS.map_text_into_number_space("HELLOWORD", LS.alphabets['en'])
-ioc = LS.calculate_ioc(plaintext)
-
-print(ioc)
-```
-
-You can find further example usages in the file `test2.py` within the package.
-
-## Supported languages
-
-The library includes predefined configurations for the following languages:
-- English (en)
-- German (de)
-- Spanish (es)
-- French (fr)
-- Italian (it)
-- Hungarian (hu)
-- Russian (ru)
-- Czech (cs)
-- Greek (el)
-- Latin (la)
-- Dutch (nl)
-- Swedish (sv)
-- Portuguese (pt)
-- Polish (pl)
-- Turkish (tr)
-
-
-## Some more technical details
-
-### Where are the package files stored after installing the package and how to find this out
-
-```bash
-% pip3 list | grep  LanguageStatisticsLibPy
-% pip3 show  LanguageStatisticsLibPy
-
-# show package content (including the test files) for example on Mac
-% tree /Users/be/Library/Python/3.13/lib/python/site-packages/LanguageStatisticsLibPy
-# on Linux this could be in:
-# /home/user/.local/lib/python3.10/site-packages/LanguageStatisticsLibPy/
-...
-# show content of a directory where the statistics files had been copied to
-tree /Users/be/Documents/Python/LanguageStatisticsLibPy_PIP-Test/LSLP
-...
-```
-
-### How to call the test files
-```bash
-% pwd
-/Users/be/Documents/Python/LanguageStatisticsLibPy_PIP-Test/testen2
-
-% ls -l
--rwx------  1 be  staff   956 27 Dez 09:44 test1.py
--rwx------  1 be  staff  2944 27 Dez 09:42 test2.py
-
-% python3 test1.py                                                     
-0.08333333333333333
-
-% python3 test2.py
-Grams size: 1
-	Grams loaded in 0:00:00.000097
-	Grams normalized in 0:00:00.000007
-	Text: HELLOWORLDTHISISATEST
-	Cost value: 771793.56
-...
+# LanguageStatisticsLibPy
+
+LanguageStatisticsLibPy is a Python library designed to facilitate the fast analysis and manipulation of language statistics data. It originates from a C# library that was first used in the widespread cryptography e-learning software CrypTool 2. From now on "CrypTool 2" is abbreviated "CT2". CT2 is an open-source e-learning program for Windows to do cryptography and cryptanalysis (https://www.cryptool.org/en/ct2/).
+
+This Python library supports 15 different languages and offers functionality for generating and handling n-gram data, specifically for calculating n-gram frequencies using the language statistics files from CT2.
+Additionally, it facilitates the use of CT2's dictionaries through a "Word Tree", an efficient data structure for rapid word searches within a language.
+
+The language statistics files (for example, `en-5gram-nocs.gz` indicates an English 5-gram file that is not case-sensitive and excludes spaces) can be found in the "LanguageStatistics" subdirectory of CT2, if you have installed CT2 on Windows.
+If you don't have a Windows machine or you don't want to install CT2, you may download the language statistics files and dictionaries from the CT2 Github repo: [Language Statistics](https://github.com/CrypToolProject/CrypTool-2/tree/main/LanguageStatistics).
+
+Remark: This package contains the implemented algorithms without the language statistics files. These files have to be downloaded separately as they occupy many megabytes.
+
+## Features
+
+- **Support for multiple languages**: The library includes predefined support for 15 languages, including English, German, Spanish, French -- each with its own set of unigram frequencies and alphabets.
+  
+- **N-gram loading**: Users can load unigrams, bigrams, trigrams, tetragrams, pentagrams, and hexagrams as n-gram objects in supported languages, with the option to include or exclude spaces. The n-grams delivered in the language statistics files range from 1 to 5 (we don't deliver 6-grams within CT2, since these files are too big). All language statistics files delivered are case-insensitive, denoted as "nocs" in the filename. Each language statistics is available in two forms: with space/blank ("sp" in the filename) and without space/blank (indicated by the absence of "sp" in the filename) within the alphabet.
+
+- **Index of Coincidence calculation**: It offers a method to calculate the index of coincidence (IoC) for a given plaintext, which is useful for cryptanalysis and language pattern recognition.
+  
+- **Alphabet and number mapping**: The library provides functionality to map characters to their respective positions in a language's alphabet and vice versa, supporting operations on encoded messages or language data.
+
+- **Dynamic n-gram support**: Depending on the available data, the library dynamically supports various n-gram types. 
+- **Word tree data structure**: It supports a word tree data structure for fast word lookups (true = part of language, false = not part of language) of a specific language.
+
+## Usage
+
+Prerequisites: LanguageStatisticsLibPy is installed on your computer via 
+**$ pip3 install LanguageStatisticsLibPy**
+
+1. **Initialization**: Start by importing the `LanguageStatistics` class and specify the language code for your analysis.
+2. **Loading n-grams**: To load n-grams of your chosen type (e.g., unigrams, bigrams) for a specific language, use the `create_grams` method with the appropriate .gz file from the LanguageStatistics directory in CT2. For instance, to load English 4-grams that are case-insensitive and include the space/blank symbol, use the file named `en-4gram-nocs-sp.gz`.
+3. **Calculating IoC**: Calculate the index of coincidence for a given plaintext using the `calculate_ioc` method.
+4. **Word tree loading**: For advanced language analysis, load a pre-built word tree for a specific language using the `load_word_tree` method.
+
+Sample usage (from file `test1.py`):
+
+```python
+from languagestatisticslibpy.LanguageStatistics import LanguageStatistics as LS
+
+plaintext = LS.map_text_into_number_space("HELLOWORD", LS.alphabets['en'])
+ioc = LS.calculate_ioc(plaintext)
+
+print(ioc)
+```
+
+You can find further example usages in the file `test2.py` within the package.
+
+## Supported languages
+
+The library includes predefined configurations for the following languages:
+- English (en)
+- German (de)
+- Spanish (es)
+- French (fr)
+- Italian (it)
+- Hungarian (hu)
+- Russian (ru)
+- Czech (cs)
+- Greek (el)
+- Latin (la)
+- Dutch (nl)
+- Swedish (sv)
+- Portuguese (pt)
+- Polish (pl)
+- Turkish (tr)
+
+
+## Some more technical details
+
+### Where are the package files stored after installing the package and how to find this out
+
+```bash
+% pip3 list | grep  LanguageStatisticsLibPy
+% pip3 show  LanguageStatisticsLibPy
+
+# show package content (including the test files) for example on Mac
+% tree /Users/be/Library/Python/3.13/lib/python/site-packages/LanguageStatisticsLibPy
+# on Linux this could be in:
+# /home/user/.local/lib/python3.10/site-packages/LanguageStatisticsLibPy/
+...
+# show content of a directory where the statistics files had been copied to
+tree /Users/be/Documents/Python/LanguageStatisticsLibPy_PIP-Test/LSLP
+...
+```
+
+### How to call the test files
+```bash
+% pwd
+/Users/be/Documents/Python/LanguageStatisticsLibPy_PIP-Test/testen2
+
+% ls -l
+-rwx------  1 be  staff   956 27 Dez 09:44 test1.py
+-rwx------  1 be  staff  2944 27 Dez 09:42 test2.py
+
+% python3 test1.py                                                     
+0.08333333333333333
+
+% python3 test2.py
+Grams size: 1
+	Grams loaded in 0:00:00.000097
+	Grams normalized in 0:00:00.000007
+	Text: HELLOWORLDTHISISATEST
+	Cost value: 771793.56
+...
 ```

{languagestatisticslibpy-1.0.3 → languagestatisticslibpy-1.0.4}/pyproject.toml

@@ -1,9 +1,9 @@
 [tool.poetry]
 name = "LanguageStatisticsLibPy"
-version = "1.0.3"
+version = "1.0.4"
 description = "Quick and robust Python functions to do cryptanalysis using language statistics data for 15 languages"
 readme = "README.md"
-authors = ["Nils Kopal and Bernhard Esslinger <kopal@cryptool.org>"] 
+authors = ["Nils Kopal and Bernhard Esslinger <kopal@cryptool.org>"]
 license = "Apache-2.0"
 classifiers = [
     "Programming Language :: Python :: 3",
@@ -24,4 +24,4 @@ dependencies = [
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]
-build-backend = "poetry.core.masonry.api"
+build-backend = "poetry.core.masonry.api"

{languagestatisticslibpy-1.0.3 → languagestatisticslibpy-1.0.4}/src/languagestatisticslibpy/LanguageStatisticsFile.py

@@ -1,4 +1,4 @@
-'''
+"""
    Copyright 2024 Nils Kopal, Bernhard Esslinger, CrypTool Team
 
    Licensed under the Apache License, Version 2.0 (the "License");
@@ -12,15 +12,12 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-'''
-import struct
-import numpy as np
-import gzip
-
+"""
 import gzip
 import struct
 import numpy as np
 
+
 class LanguageStatisticsFile:
     """
     Class for handling the loading of language statistics from a compressed file.
@@ -28,29 +25,28 @@ class LanguageStatisticsFile:
     Attributes:
     - FILE_FORMAT_MAGIC_NUMBER (str): The expected magic number to identify valid language statistics files.
     - file_path (str): The path to the language statistics file.
-    - alphabet (str): The alphabet used in the language statistics.
+    - alphabet (str): The decoded alphabet used in the language statistics.
+    - alphabet_length (int): Number of symbols (characters) in `alphabet` (NOT bytes).
+    - alphabet_length_bytes (int): Length prefix from file (= number of bytes used to store the alphabet string).
     - language_code (str): The language code extracted from the statistics file.
     """
 
     FILE_FORMAT_MAGIC_NUMBER = "CTLS"
 
-    def __init__(self, file_path):
+    def __init__(self, file_path: str):
         """
         Initializes the LanguageStatisticsFile class.
 
         Parameters:
         - file_path (str): The path to the language statistics file.
-
-        Initializes:
-        - self.file_path (str): Stores the path to the file.
-        - self.alphabet (str): Initially empty, set after file loading.
-        - self.language_code (str): Initially empty, set after file loading.
         """
         self.file_path = file_path
-        self.alphabet = ''
-        self.language_code = ''
+        self.alphabet = ""
+        self.alphabet_length = 0
+        self.alphabet_length_bytes = 0
+        self.language_code = ""
 
-    def load_frequencies(self, array_dimensions):
+    def load_frequencies(self, array_dimensions: int) -> np.ndarray:
         """
         Loads the frequency data from the language statistics file.
 
@@ -61,48 +57,60 @@ class LanguageStatisticsFile:
         - np.ndarray: A numpy array containing the frequency data.
 
         Raises:
-        - Exception: If the file format is invalid or the dimensions of the frequency array do not match the expected value.
-
-        Process:
-        1. Validates the file by checking the magic number.
-        2. Reads the language code, gram length, and alphabet.
-        3. Verifies that the gram length matches the required dimensions.
-        4. Reads the frequency data and reshapes it into the appropriate numpy array format.
-        5. Copies the data into a new numpy array to allow modification.
+        - Exception: If the file format is invalid, gram length mismatches, or the file content size is inconsistent.
+
+        Notes:
+        - The alphabet length stored in the file is a *byte length* (UTF-8). For non-ASCII alphabets (e.g., German ÄÖÜß),
+          the number of bytes differs from the number of symbols. We therefore decode first and then compute the symbol count.
         """
-        with gzip.open(self.file_path, 'rb') as file:
+        if array_dimensions < 1:
+            raise Exception("array_dimensions must be >= 1")
+
+        with gzip.open(self.file_path, "rb") as file:
             # Validate the file format by checking the magic number.
-            magic_number = file.read(4).decode('utf-8')
+            magic_number = file.read(4).decode("utf-8")
             if magic_number != self.FILE_FORMAT_MAGIC_NUMBER:
                 raise Exception("File does not start with the expected magic number for language statistics.")
 
             # Read the language code (length-prefixed string).
             language_code_length_bytes = file.read(1)[0]
-            self.language_code = file.read(language_code_length_bytes).decode('utf-8')
+            self.language_code = file.read(language_code_length_bytes).decode("utf-8")
 
             # Read the gram length (32-bit signed integer).
-            gram_length = struct.unpack('<i', file.read(4))[0]
+            gram_length = struct.unpack("<i", file.read(4))[0]
 
             # Ensure the gram length matches the required dimensions.
             if gram_length != array_dimensions:
                 raise Exception("Gram length of statistics file differs from required dimensions of frequency array.")
 
-            # Read the alphabet (length-prefixed string).
-            self.alphabet_length = file.read(1)[0]
-            self.alphabet = file.read(self.alphabet_length).decode('utf-8')
+            # Read the alphabet (length-prefixed BYTES, UTF-8).
+            self.alphabet_length_bytes = file.read(1)[0]
+            alphabet_bytes = file.read(self.alphabet_length_bytes)
+            self.alphabet = alphabet_bytes.decode("utf-8")
+
+            # IMPORTANT: use number of symbols (characters), not bytes, for frequency tensor dimensions.
+            self.alphabet_length = len(self.alphabet)
 
             # Calculate the total number of frequency entries.
             frequency_entries = self.alphabet_length ** gram_length
+            expected_frequency_bytes = frequency_entries * 4  # float32
 
             # Read the frequency data (32-bit float array).
-            frequency_data = file.read(frequency_entries * 4)
-
-            # Reshape the data into the correct dimensionality and copy it for mutability.
+            frequency_data = file.read(expected_frequency_bytes)
+
+            # Hard validation: mismatched files should fail with a clear message.
+            if len(frequency_data) != expected_frequency_bytes:
+                raise Exception(
+                    f"Frequency data size mismatch. "
+                    f"Expected {expected_frequency_bytes} bytes ({frequency_entries} float32 values) "
+                    f"for alphabet_length={self.alphabet_length} and gram_length={gram_length}, "
+                    f"but got {len(frequency_data)} bytes. "
+                    f"(alphabet_length_bytes={self.alphabet_length_bytes}, alphabet='{self.alphabet}')"
+                )
+
+            # Convert and reshape.
+            data = np.frombuffer(frequency_data, dtype=np.float32)
             if array_dimensions == 1:
-                frequencies = np.frombuffer(frequency_data, dtype=np.float32).copy()
-            else:
-                frequencies = np.frombuffer(frequency_data, dtype=np.float32).reshape(
-                    tuple([self.alphabet_length] * array_dimensions)
-                ).copy()
+                return data.copy()
 
-            return frequencies
+            return data.reshape(tuple([self.alphabet_length] * array_dimensions)).copy()

{languagestatisticslibpy-1.0.3 → languagestatisticslibpy-1.0.4}/src/languagestatisticslibpy/Tetragrams.py

@@ -1,125 +1,125 @@
-'''
-   Copyright 2024 Nils Kopal, Bernhard Esslinger, CrypTool Team
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-'''
-import numpy as np
-import os
-from languagestatisticslibpy.Grams import Grams
-from languagestatisticslibpy.GramsType import GramsType
-from languagestatisticslibpy.LanguageStatisticsFile import LanguageStatisticsFile
-
-class Tetragrams(Grams):
-    def __init__(self, language, language_statistics_directory, use_spaces=False):
-        """
-        Initializes the Tetragrams class by calling the parent class (Grams) initializer.
-
-        Parameters:
-        - language (str): The language of the tetragram statistics.
-        - language_statistics_directory (str): Path to the directory containing language statistics files.
-        - use_spaces (bool): Whether to include spaces in the analysis (default: False).
-        """
-        super().__init__(language, language_statistics_directory, use_spaces)
-
-    def load_gz(self, filename, language_statistics_directory):
-        """
-        Loads a gzip-compressed file containing tetragram frequencies.
-
-        Parameters:
-        - filename (str): The name of the file to load.
-        - language_statistics_directory (str): The directory where the statistics file is located.
-
-        Sets:
-        - self.frequencies (np.ndarray): A 4D array of tetragram frequencies.
-        - self.alphabet (list): The alphabet used in the statistics file.
-        - self.max_value (float): The maximum value in the frequencies array, or -∞ if the array is empty.
-        """
-        file_path = os.path.join(language_statistics_directory, filename)
-        language_statistics_file = LanguageStatisticsFile(file_path)
-        self.frequencies = language_statistics_file.load_frequencies(4)
-        self.alphabet = language_statistics_file.alphabet
-        self.max_value = np.max(self.frequencies) if self.frequencies.size > 0 else float('-inf')
-
-    def calculate_cost(self, text):
-        """
-        Calculates the cost of a given text based on tetragram frequencies.
-
-        Parameters:
-        - text (str): The text to analyze.
-
-        Returns:
-        - float: The average cost of tetragrams in the text. Returns 0.0 if the text length is less than 4.
-
-        Notes:
-        - Skips tetragrams containing characters outside the defined alphabet.
-        - If `add_letter_indices` is defined, modifies the index of the characters before computing the cost.
-        """
-        if len(text) < 4:
-            return 0.0
-
-        value = 0.0
-        alphabet_length = len(self.alphabet)
-        end = len(text) - 3
-
-        for i in range(end):
-            a, b, c, d = text[i:i+4]
-
-            if self.add_letter_indices:
-                a += self.add_letter_indices.get(a, 0)
-                b += self.add_letter_indices.get(b, 0)
-                c += self.add_letter_indices.get(c, 0)
-                d += self.add_letter_indices.get(d, 0)
-
-            if 0 <= a < alphabet_length and 0 <= b < alphabet_length and \
-               0 <= c < alphabet_length and 0 <= d < alphabet_length:
-                value += self.frequencies[a, b, c, d]
-
-        return value / end
-
-    def gram_size(self):
-        """
-        Returns the size of the grams being analyzed (tetragrams in this case).
-
-        Returns:
-        - int: The size of the grams (always 4 for tetragrams).
-        """
-        return 4
-
-    def grams_type(self):
-        """
-        Returns the type of grams being analyzed.
-
-        Returns:
-        - GramsType: An enum value representing the type of grams (GramsType.Tetragrams).
-        """
-        return GramsType.Tetragrams
-
-    def normalize(self, max_value):
-        """
-        Normalizes the tetragram frequencies based on the provided maximum value.
-
-        Parameters:
-        - max_value (float): The maximum value used for normalization.
-
-        Notes:
-        - Adjusts all frequencies proportionally to the new maximum value.
-        - Updates `self.max_value` to the new maximum after normalization.
-        """
-        super().normalize(max_value)
-        adjust_value = self.max_value * max_value
-        for a in range(len(self.alphabet)):
-            for b in range(len(self.alphabet)):
-                for c in range(len(self.alphabet)):
-                    for d in range(len(self.alphabet)):
-                        self.frequencies[a, b, c, d] = adjust_value / self.frequencies[a, b, c, d]
-        self.max_value = np.max(self.frequencies) if self.frequencies.size > 0 else float('-inf')
+'''
+   Copyright 2024 Nils Kopal, Bernhard Esslinger, CrypTool Team
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+'''
+import numpy as np
+import os
+from languagestatisticslibpy.Grams import Grams
+from languagestatisticslibpy.GramsType import GramsType
+from languagestatisticslibpy.LanguageStatisticsFile import LanguageStatisticsFile
+
+class Tetragrams(Grams):
+    def __init__(self, language, language_statistics_directory, use_spaces=False):
+        """
+        Initializes the Tetragrams class by calling the parent class (Grams) initializer.
+
+        Parameters:
+        - language (str): The language of the tetragram statistics.
+        - language_statistics_directory (str): Path to the directory containing language statistics files.
+        - use_spaces (bool): Whether to include spaces in the analysis (default: False).
+        """
+        super().__init__(language, language_statistics_directory, use_spaces)
+
+    def load_gz(self, filename, language_statistics_directory):
+        """
+        Loads a gzip-compressed file containing tetragram frequencies.
+
+        Parameters:
+        - filename (str): The name of the file to load.
+        - language_statistics_directory (str): The directory where the statistics file is located.
+
+        Sets:
+        - self.frequencies (np.ndarray): A 4D array of tetragram frequencies.
+        - self.alphabet (list): The alphabet used in the statistics file.
+        - self.max_value (float): The maximum value in the frequencies array, or -∞ if the array is empty.
+        """
+        file_path = os.path.join(language_statistics_directory, filename)
+        language_statistics_file = LanguageStatisticsFile(file_path)
+        self.frequencies = language_statistics_file.load_frequencies(4)
+        self.alphabet = language_statistics_file.alphabet
+        self.max_value = np.max(self.frequencies) if self.frequencies.size > 0 else float('-inf')
+
+    def calculate_cost(self, text):
+        """
+        Calculates the cost of a given text based on tetragram frequencies.
+
+        Parameters:
+        - text (str): The text to analyze.
+
+        Returns:
+        - float: The average cost of tetragrams in the text. Returns 0.0 if the text length is less than 4.
+
+        Notes:
+        - Skips tetragrams containing characters outside the defined alphabet.
+        - If `add_letter_indices` is defined, modifies the index of the characters before computing the cost.
+        """
+        if len(text) < 4:
+            return 0.0
+
+        value = 0.0
+        alphabet_length = len(self.alphabet)
+        end = len(text) - 3
+
+        for i in range(end):
+            a, b, c, d = text[i:i+4]
+
+            if self.add_letter_indices:
+                a += self.add_letter_indices.get(a, 0)
+                b += self.add_letter_indices.get(b, 0)
+                c += self.add_letter_indices.get(c, 0)
+                d += self.add_letter_indices.get(d, 0)
+
+            if 0 <= a < alphabet_length and 0 <= b < alphabet_length and \
+               0 <= c < alphabet_length and 0 <= d < alphabet_length:
+                value += self.frequencies[a, b, c, d]
+
+        return value / end
+
+    def gram_size(self):
+        """
+        Returns the size of the grams being analyzed (tetragrams in this case).
+
+        Returns:
+        - int: The size of the grams (always 4 for tetragrams).
+        """
+        return 4
+
+    def grams_type(self):
+        """
+        Returns the type of grams being analyzed.
+
+        Returns:
+        - GramsType: An enum value representing the type of grams (GramsType.Tetragrams).
+        """
+        return GramsType.Tetragrams
+
+    def normalize(self, max_value):
+        """
+        Normalizes the tetragram frequencies based on the provided maximum value.
+
+        Parameters:
+        - max_value (float): The maximum value used for normalization.
+
+        Notes:
+        - Adjusts all frequencies proportionally to the new maximum value.
+        - Updates `self.max_value` to the new maximum after normalization.
+        """
+        super().normalize(max_value)
+        adjust_value = self.max_value * max_value
+        for a in range(len(self.alphabet)):
+            for b in range(len(self.alphabet)):
+                for c in range(len(self.alphabet)):
+                    for d in range(len(self.alphabet)):
+                        self.frequencies[a, b, c, d] = adjust_value / self.frequencies[a, b, c, d]
+        self.max_value = np.max(self.frequencies) if self.frequencies.size > 0 else float('-inf')