fast-langdetect 0.2.2__tar.gz → 0.2.4__tar.gz

{fast_langdetect-0.2.2 → fast_langdetect-0.2.4}/PKG-INFO

@@ -1,14 +1,13 @@
 Metadata-Version: 2.1
 Name: fast-langdetect
-Version: 0.2.2
+Version: 0.2.4
 Summary: Quickly detect text language and segment language
 Author-Email: sudoskys <coldlando@hotmail.com>
 License: MIT
 Requires-Python: <3.13,>=3.9
-Requires-Dist: fasttext-wheel>=0.9.2
 Requires-Dist: robust-downloader>=0.0.2
-Requires-Dist: numpy<2.0.0,>=1.26.4
 Requires-Dist: requests>=2.32.3
+Requires-Dist: fasttext-predict>=0.9.2.4
 Description-Content-Type: text/markdown
 
 # fast-langdetect 🚀
@@ -19,21 +18,20 @@ Description-Content-Type: text/markdown
 
 ## Overview
 
-**fast-langdetect** provides ultra-fast and highly accurate language detection based on FastText, a library developed by
-Facebook. This package is 80x faster than traditional methods and offers 95% accuracy.
+**`fast-langdetect`** is an ultra-fast and highly accurate language detection library based on FastText, a library developed by Facebook. Its incredible speed and accuracy make it 80x faster than conventional methods and deliver up to 95% accuracy.
 
-It supports Python versions 3.9 to 3.12.
+- Supported Python `3.9` to `3.12`.
+- Works offline  in low memory mode
+- No `numpy` required (thanks to @dalf).
 
-Support offline usage.
+> ### Background
+> 
+> This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark) with enhancements in packaging.
+> For more information about the underlying model, see the official FastText documentation: [Language Identification](https://fasttext.cc/docs/en/language-identification.html).
 
-This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark)
-with enhancements in packaging.
-
-For more information on the underlying FastText model, refer to the official
-documentation: [FastText Language Identification](https://fasttext.cc/docs/en/language-identification.html).
-
-> [!NOTE]
-> This library requires over 200MB of memory to use in low memory mode.
+> ### Possible memory usage
+> 
+> *This library requires at least **200MB memory** in low-memory mode.*
 
 ## Installation 💻
 
@@ -53,20 +51,17 @@ pdm add fast-langdetect
 
 ## Usage 🖥️
 
-For optimal performance and accuracy in language detection, use `detect(text, low_memory=False)` to load the larger
-model.
+In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
 
-> The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
+### Prerequisites
 
-### Native API (Recommended)
+- The “/n” character in the argument string must be removed before calling the function.
+- If the sample is too long or too short, the accuracy will be reduced (e.g. if it is too short, Chinese will be predicted as Japanese).
+- The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
 
-> [!NOTE]
-> This function assumes to be given a single line of text. *You should remove `\n` characters before passing the text.*
-> If the sample is too long or too short, the accuracy will decrease (for example, in the case of too short, Chinese
-> will be predicted as Japanese).
+### Native API (Recommended)
 
 ```python
-
 from fast_langdetect import detect, detect_multilingual
 
 # Single language detection
@@ -82,15 +77,17 @@ multiline_text = """
 Hello, world!
 This is a multiline text.
 But we need remove `\n` characters or it will raise an ValueError.
+REMOVE \n
 """
-multiline_text = multiline_text.replace("\n", "")  # NOTE:ITS IMPORTANT TO REMOVE \n CHARACTERS
+multiline_text = multiline_text.replace("\n", "")  
 print(detect(multiline_text))
 # Output: {'lang': 'en', 'score': 0.8509423136711121}
 
 print(detect("Привет, мир!")["lang"])
 # Output: ru
 
-# Multi-language detection
+# Multi-language detection with low memory mode enabled
+# The accuracy is not as good as it should be
 print(detect_multilingual("Hello, world!你好世界!Привет, мир!"))
 # Output: [{'lang': 'ja', 'score': 0.32009604573249817}, {'lang': 'uk', 'score': 0.27781224250793457}, {'lang': 'zh', 'score': 0.17542070150375366}, {'lang': 'sr', 'score': 0.08751443773508072}, {'lang': 'bg', 'score': 0.05222449079155922}]
 
@@ -99,6 +96,10 @@ print(detect_multilingual("Hello, world!你好世界!Привет, мир!", low
 # Output: [{'lang': 'ru', 'score': 0.39008623361587524}, {'lang': 'zh', 'score': 0.18235979974269867}, {'lang': 'ja', 'score': 0.08473210036754608}, {'lang': 'sr', 'score': 0.057975586503744125}, {'lang': 'en', 'score': 0.05422825738787651}]
 ```
 
+#### Fallbacks
+
+We provide a fallback mechanism: when `use_strict_mode=False`, if the program fails to load the **large model** (`low_memory=False`), it will fall back to the offline **small model** to complete the prediction task.
+
 ### Convenient `detect_language` Function
 
 ```python
@@ -148,4 +149,4 @@ models
   journal={arXiv preprint arXiv:1612.03651},
   year={2016}
 }
-```
+```

{fast_langdetect-0.2.2 → fast_langdetect-0.2.4}/README.md

@@ -6,21 +6,20 @@
 
 ## Overview
 
-**fast-langdetect** provides ultra-fast and highly accurate language detection based on FastText, a library developed by
-Facebook. This package is 80x faster than traditional methods and offers 95% accuracy.
+**`fast-langdetect`** is an ultra-fast and highly accurate language detection library based on FastText, a library developed by Facebook. Its incredible speed and accuracy make it 80x faster than conventional methods and deliver up to 95% accuracy.
 
-It supports Python versions 3.9 to 3.12.
+- Supported Python `3.9` to `3.12`.
+- Works offline  in low memory mode
+- No `numpy` required (thanks to @dalf).
 
-Support offline usage.
+> ### Background
+> 
+> This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark) with enhancements in packaging.
+> For more information about the underlying model, see the official FastText documentation: [Language Identification](https://fasttext.cc/docs/en/language-identification.html).
 
-This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark)
-with enhancements in packaging.
-
-For more information on the underlying FastText model, refer to the official
-documentation: [FastText Language Identification](https://fasttext.cc/docs/en/language-identification.html).
-
-> [!NOTE]
-> This library requires over 200MB of memory to use in low memory mode.
+> ### Possible memory usage
+> 
+> *This library requires at least **200MB memory** in low-memory mode.*
 
 ## Installation 💻
 
@@ -40,20 +39,17 @@ pdm add fast-langdetect
 
 ## Usage 🖥️
 
-For optimal performance and accuracy in language detection, use `detect(text, low_memory=False)` to load the larger
-model.
+In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
 
-> The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
+### Prerequisites
 
-### Native API (Recommended)
+- The “/n” character in the argument string must be removed before calling the function.
+- If the sample is too long or too short, the accuracy will be reduced (e.g. if it is too short, Chinese will be predicted as Japanese).
+- The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
 
-> [!NOTE]
-> This function assumes to be given a single line of text. *You should remove `\n` characters before passing the text.*
-> If the sample is too long or too short, the accuracy will decrease (for example, in the case of too short, Chinese
-> will be predicted as Japanese).
+### Native API (Recommended)
 
 ```python
-
 from fast_langdetect import detect, detect_multilingual
 
 # Single language detection
@@ -69,15 +65,17 @@ multiline_text = """
 Hello, world!
 This is a multiline text.
 But we need remove `\n` characters or it will raise an ValueError.
+REMOVE \n
 """
-multiline_text = multiline_text.replace("\n", "")  # NOTE:ITS IMPORTANT TO REMOVE \n CHARACTERS
+multiline_text = multiline_text.replace("\n", "")  
 print(detect(multiline_text))
 # Output: {'lang': 'en', 'score': 0.8509423136711121}
 
 print(detect("Привет, мир!")["lang"])
 # Output: ru
 
-# Multi-language detection
+# Multi-language detection with low memory mode enabled
+# The accuracy is not as good as it should be
 print(detect_multilingual("Hello, world!你好世界!Привет, мир!"))
 # Output: [{'lang': 'ja', 'score': 0.32009604573249817}, {'lang': 'uk', 'score': 0.27781224250793457}, {'lang': 'zh', 'score': 0.17542070150375366}, {'lang': 'sr', 'score': 0.08751443773508072}, {'lang': 'bg', 'score': 0.05222449079155922}]
 
@@ -86,6 +84,10 @@ print(detect_multilingual("Hello, world!你好世界!Привет, мир!", low
 # Output: [{'lang': 'ru', 'score': 0.39008623361587524}, {'lang': 'zh', 'score': 0.18235979974269867}, {'lang': 'ja', 'score': 0.08473210036754608}, {'lang': 'sr', 'score': 0.057975586503744125}, {'lang': 'en', 'score': 0.05422825738787651}]
 ```
 
+#### Fallbacks
+
+We provide a fallback mechanism: when `use_strict_mode=False`, if the program fails to load the **large model** (`low_memory=False`), it will fall back to the offline **small model** to complete the prediction task.
+
 ### Convenient `detect_language` Function
 
 ```python
@@ -135,4 +137,4 @@ models
   journal={arXiv preprint arXiv:1612.03651},
   year={2016}
 }
-```
+```

{fast_langdetect-0.2.2 → fast_langdetect-0.2.4}/pyproject.toml

@@ -1,15 +1,14 @@
 [project]
 name = "fast-langdetect"
-version = "0.2.2"
+version = "0.2.4"
 description = "Quickly detect text language and segment language"
 authors = [
     { name = "sudoskys", email = "coldlando@hotmail.com" },
 ]
 dependencies = [
-    "fasttext-wheel>=0.9.2",
     "robust-downloader>=0.0.2",
-    "numpy>=1.26.4,<2.0.0",
     "requests>=2.32.3",
+    "fasttext-predict>=0.9.2.4",
 ]
 requires-python = ">=3.9,<3.13"
 readme = "README.md"

fast_langdetect-0.2.4/src/fast_langdetect/__init__.py

@@ -0,0 +1,4 @@
+# -*- coding: utf-8 -*-
+
+
+from .ft_detect import detect, detect_language, detect_multilingual  # noqa: F401

{fast_langdetect-0.2.2 → fast_langdetect-0.2.4}/src/fast_langdetect/ft_detect/__init__.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # @Time    : 2024/1/17 下午4:00
-import logging
 
 from .infer import detect
 from .infer import detect_multilingual  # noqa: F401
@@ -24,14 +23,3 @@ def detect_language(sentence, *, low_memory: bool = True):
     if lang_code == "JA" and not is_japanese(sentence):
         lang_code = "ZH"
     return lang_code
-
-
-def detect_langs(sentence, *, low_memory: bool = True):
-    """
-    Detect language
-    :param sentence: str sentence
-    :param low_memory: bool (default: True) whether to use low memory mode
-    :return: ZH, EN, JA, KO, FR, DE, ES, .... (two uppercase letters)
-    """
-    logging.warning("detect_langs is deprecated, use detect_language instead")
-    return detect_language(sentence, low_memory=low_memory)

fast_langdetect-0.2.4/src/fast_langdetect/ft_detect/infer.py

@@ -0,0 +1,261 @@
+# -*- coding: utf-8 -*-
+# @Time    : 2024/01/17 下午08:30
+# @Author  : sudoskys
+# @File    : infer.py
+# @Software: PyCharm
+import logging
+import os
+from pathlib import Path
+from typing import Dict, List, Optional, Union, Any
+
+import fasttext
+from robust_downloader import download
+
+logger = logging.getLogger(__name__)
+CACHE_DIRECTORY = os.getenv("FTLANG_CACHE", "/tmp/fasttext-langdetect")
+LOCAL_SMALL_MODEL_PATH = Path(__file__).parent / "resources" / "lid.176.ftz"
+
+FASTTEXT_LARGE_MODEL_URL = "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin"
+FASTTEXT_LARGE_MODEL_NAME = "lid.176.bin"
+VERIFY_FASTTEXT_LARGE_MODEL = "01810bc59c6a3d2b79c79e6336612f65"
+
+
+class DetectError(Exception):
+    pass
+
+
+class ModelManager:
+    def __init__(self):
+        self._models = {}
+
+    def get_model(self, key: str):
+        """Retrieve cached model."""
+        return self._models.get(key)
+
+    def cache_model(self, key: str, model) -> None:
+        """Cache loaded FastText model."""
+        self._models[key] = model
+
+
+_model_cache = ModelManager()
+
+import hashlib
+
+
+def calculate_md5(file_path, chunk_size=8192):
+    """
+    Calculate the MD5 hash of a file.
+
+    :param file_path: Path to the file
+    :param chunk_size: Size of each chunk to read from the file
+    :return: MD5 hash of the file
+    """
+    md5 = hashlib.md5()
+    with open(file_path, 'rb') as f:
+        for chunk in iter(lambda: f.read(chunk_size), b''):
+            md5.update(chunk)
+    return md5.hexdigest()
+
+
+def verify_md5(file_path, expected_md5, chunk_size=8192):
+    """
+    Verify the MD5 hash of a file against an expected hash.
+
+    :param file_path: Path to the file
+    :param expected_md5: Expected MD5 hash
+    :param chunk_size: Size of each chunk to read from the file
+    :return: True if the file's MD5 hash matches the expected hash, False otherwise
+    """
+    md5 = calculate_md5(file_path, chunk_size)
+    return md5 == expected_md5
+
+
+def download_model(
+        download_url: str,
+        save_path: Path,
+        proxy: Optional[str] = None
+) -> None:
+    """
+    Download FastText model file if it doesn't exist.
+    :param download_url: URL to download the model from
+    :param save_path: Path to save the downloaded model
+    :param proxy: Proxy URL for downloading the model
+    :raises DetectError: If download fails
+    """
+    if save_path.exists():
+        logger.info(f"fast-langdetect:Model already exists at {save_path}. Skipping download.")
+        return
+
+    logger.info(f"fast-langdetect:Downloading FastText model from {download_url} to {save_path}")
+    try:
+        download(
+            url=download_url,
+            folder=str(save_path.parent),
+            filename=save_path.name,
+            proxy=proxy,
+            retry_max=2,
+            sleep_max=5,
+            timeout=7,
+        )
+    except Exception as e:
+        logger.error(f"fast-langdetect:Failed to download FastText model from {download_url}: {e}")
+        raise DetectError(f"Unable to download model from {download_url}")
+
+
+def load_fasttext_model(
+        model_path: Path,
+        download_url: Optional[str] = None,
+        proxy: Optional[str] = None,
+):
+    """
+    Load a FastText model, downloading it if necessary.
+    :param model_path: Path to the FastText model file
+    :param download_url: URL to download the model from
+    :param proxy: Proxy URL for downloading the model
+    :return: FastText model
+    :raises DetectError: If model loading fails
+    """
+    if all([
+        VERIFY_FASTTEXT_LARGE_MODEL,
+        model_path.exists(),
+        model_path.name == FASTTEXT_LARGE_MODEL_NAME,
+    ]):
+        if not verify_md5(model_path, VERIFY_FASTTEXT_LARGE_MODEL):
+            logger.warning(
+                f"fast-langdetect: MD5 hash verification failed for {model_path}, "
+                f"please check the integrity of the downloaded file from {FASTTEXT_LARGE_MODEL_URL}. "
+                "\n    This may seriously reduce the prediction accuracy. "
+                "If you want to ignore this, please set `fast_langdetect.ft_detect.infer.VERIFY_FASTTEXT_LARGE_MODEL = None` "
+            )
+    if not model_path.exists():
+        if download_url:
+            download_model(download_url, model_path, proxy)
+        if not model_path.exists():
+            raise DetectError(f"FastText model file not found at {model_path}")
+
+    try:
+        # Load FastText model
+        return fasttext.load_model(str(model_path))
+    except Exception as e:
+        logger.warning(f"fast-langdetect:Failed to load FastText model from {model_path}: {e}")
+        raise DetectError(f"Failed to load FastText model: {e}")
+
+
+def load_model(
+        low_memory: bool = False,
+        download_proxy: Optional[str] = None,
+        use_strict_mode: bool = False,
+):
+    """
+    Load a FastText model based on memory preference.
+    :param low_memory: Whether to use a memory-efficient model
+    :param download_proxy: Proxy URL for downloading the model
+    :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
+    """
+    # Model path selection
+    if low_memory:
+        cache_key = "low_memory"
+        model_path = LOCAL_SMALL_MODEL_PATH
+    else:
+        cache_key = "high_memory"
+        model_path = Path(CACHE_DIRECTORY) / FASTTEXT_LARGE_MODEL_NAME
+
+    # Check cache
+    cached_model = _model_cache.get_model(cache_key)
+    if cached_model:
+        return cached_model
+
+    # Load appropriate model
+    try:
+        if low_memory:
+            model = load_fasttext_model(model_path)
+        else:
+            model = load_fasttext_model(model_path, download_url=FASTTEXT_LARGE_MODEL_URL, proxy=download_proxy)
+        _model_cache.cache_model(cache_key, model)
+        return model
+    except Exception as e:
+        logger.warning(f"fast-langdetect:Failed to load model ({'low' if low_memory else 'high'} memory): {e}")
+        if use_strict_mode:
+            raise DetectError("Failed to load FastText model.") from e
+        elif not low_memory:
+            logger.info("Falling back to low-memory model...")
+            return load_model(low_memory=True, use_strict_mode=True)
+        raise e
+
+
+def detect(
+        text: str,
+        *,
+        low_memory: bool = True,
+        model_download_proxy: Optional[str] = None,
+        use_strict_mode: bool = False,
+) -> Dict[str, Union[str, float]]:
+    """
+    Detect the language of a text using FastText.
+
+    - You MUST manually remove line breaks(`n`) from the text to be processed in advance, otherwise a ValueError is raised.
+
+    - In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
+
+    :param text: The text for language detection
+    :param low_memory: Whether to use the compressed version of the model (https://fasttext.cc/docs/en/language-identification.html)
+    :param model_download_proxy: Download proxy for the model if needed
+    :param use_strict_mode: When this parameter is enabled, the fallback after loading failure will be disabled.
+    :return: A dictionary with detected language and confidence score
+    :raises LanguageDetectionError: If detection fails
+    """
+    model = load_model(
+        low_memory=low_memory,
+        download_proxy=model_download_proxy,
+        use_strict_mode=use_strict_mode,
+    )
+    try:
+        labels, scores = model.predict(text)
+        language_label = labels[0].replace("__label__", "")
+        confidence_score = min(float(scores[0]), 1.0)
+        return {"lang": language_label, "score": confidence_score}
+    except Exception as e:
+        logger.error(f"fast-langdetect:Error during language detection: {e}")
+        raise DetectError("Language detection failed.")
+
+
+def detect_multilingual(
+        text: str,
+        *,
+        low_memory: bool = False,
+        model_download_proxy: Optional[str] = None,
+        k: int = 5,
+        threshold: float = 0.0,
+        use_strict_mode: bool = False,
+) -> List[Dict[str, Any]]:
+    """
+    Detect the top-k probable languages for a given text.
+
+    - You MUST manually remove line breaks(`n`) from the text to be processed in advance, otherwise a ValueError is raised.
+
+    - In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
+
+    :param text: The text for language detection
+    :param low_memory: Whether to use the compressed version of the model (https://fasttext.cc/docs/en/language-identification.html)
+    :param model_download_proxy: Download proxy for the model if needed
+    :param k: Number of top languages to return
+    :param threshold: Minimum confidence score to consider
+    :param use_strict_mode: When this parameter is enabled, the fallback after loading failure will be disabled.
+    :return: A list of dictionaries with detected languages and confidence scores
+    """
+    model = load_model(
+        low_memory=low_memory,
+        download_proxy=model_download_proxy,
+        use_strict_mode=use_strict_mode,
+    )
+    # Multilingual detection using FastText
+    try:
+        labels, scores = model.predict(text, k=k, threshold=threshold)
+        results = [
+            {"lang": label.replace("__label__", ""), "score": min(float(score), 1.0)}
+            for label, score in zip(labels, scores)
+        ]
+        return sorted(results, key=lambda x: x["score"], reverse=True)
+    except Exception as e:
+        logger.error(f"fast-langdetect:Error during multilingual detection: {e}")
+        raise DetectError("Multilingual detection failed.")

fast_langdetect-0.2.2/src/fast_langdetect/__init__.py

@@ -1,4 +0,0 @@
-# -*- coding: utf-8 -*-
-
-
-from .ft_detect import detect, detect_language, detect_langs, detect_multilingual  # noqa: F401

fast_langdetect-0.2.2/src/fast_langdetect/ft_detect/infer.py

@@ -1,198 +0,0 @@
-# -*- coding: utf-8 -*-
-# @Time    : 2024/1/17 下午8:30
-# @Author  : sudoskys
-# @File    : infer.py
-# @Software: PyCharm
-import logging
-import os
-from enum import Enum
-from pathlib import Path
-from typing import Dict, Union, List, Optional, Any
-
-import fasttext
-from robust_downloader import download
-
-logger = logging.getLogger(__name__)
-CACHE_DIRECTORY = os.getenv("FTLANG_CACHE", "/tmp/fasttext-langdetect")
-LOCAL_SMALL_MODEL_PATH = Path(__file__).parent / "resources" / "lid.176.ftz"
-
-# Suppress FastText output if possible
-try:
-    fasttext.FastText.eprint = lambda *args, **kwargs: None
-except Exception:
-    pass
-
-
-class ModelType(Enum):
-    LOW_MEMORY = "low_mem"
-    HIGH_MEMORY = "high_mem"
-
-
-class ModelCache:
-    def __init__(self):
-        self._models = {}
-
-    def get_model(self, model_type: ModelType) -> Optional["fasttext.FastText._FastText"]:
-        return self._models.get(model_type)
-
-    def set_model(self, model_type: ModelType, model: "fasttext.FastText._FastText"):
-        self._models[model_type] = model
-
-
-_model_cache = ModelCache()
-
-
-class DetectError(Exception):
-    """Custom exception for language detection errors."""
-    pass
-
-
-def load_model(low_memory: bool = False,
-               download_proxy: Optional[str] = None,
-               use_strict_mode: bool = False) -> "fasttext.FastText._FastText":
-    """
-    Load the FastText model based on memory preference.
-
-    :param low_memory: Indicates whether to load a smaller, memory-efficient model
-    :param download_proxy: Proxy to use for downloading the large model if necessary
-    :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
-    :return: Loaded FastText model
-    :raises DetectError: If the model cannot be loaded
-    """
-    model_type = ModelType.LOW_MEMORY if low_memory else ModelType.HIGH_MEMORY
-
-    # If the model is already loaded, return it
-    cached_model = _model_cache.get_model(model_type)
-    if cached_model:
-        return cached_model
-
-    def load_local_small_model():
-        """Try to load the local small model."""
-        try:
-            _loaded_model = fasttext.load_model(str(LOCAL_SMALL_MODEL_PATH))
-            _model_cache.set_model(ModelType.LOW_MEMORY, _loaded_model)
-            return _loaded_model
-        except Exception as e:
-            logger.error(f"Failed to load the local small model '{LOCAL_SMALL_MODEL_PATH}': {e}")
-            raise DetectError("Unable to load low-memory model from local resources.")
-
-    def load_large_model():
-        """Try to load the large model."""
-        try:
-            loaded_model = fasttext.load_model(str(model_path))
-            _model_cache.set_model(ModelType.HIGH_MEMORY, loaded_model)
-            return loaded_model
-        except Exception as e:
-            logger.error(f"Failed to load the large model '{model_path}': {e}")
-        return None
-
-    if low_memory:
-        # Attempt to load the local small model
-        return load_local_small_model()
-
-    # Path for the large model
-    large_model_name = "lid.176.bin"
-    model_path = Path(CACHE_DIRECTORY) / large_model_name
-
-    # If the large model is already present, load it
-    if model_path.exists():
-        # Model cant be dir
-        if model_path.is_dir():
-            try:
-                model_path.rmdir()
-            except Exception as e:
-                logger.error(f"Failed to remove the directory '{model_path}': {e}")
-                raise DetectError(f"Unexpected directory found in large model file path '{model_path}': {e}")
-        # Attempt to load large model
-        loaded_model = load_large_model()
-        if loaded_model:
-            return loaded_model
-
-    # If the large model is not present, attempt to download (only if necessary)
-    model_url = "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin"
-    try:
-        logger.info(f"Downloading large model from {model_url} to {model_path}")
-        download(
-            url=model_url,
-            folder=CACHE_DIRECTORY,
-            filename=large_model_name,
-            proxy=download_proxy,
-            retry_max=3,
-            timeout=20
-        )
-        # Try loading the model again after download
-        loaded_model = load_large_model()
-        if loaded_model:
-            return loaded_model
-    except Exception as e:
-        logger.error(f"Failed to download the large model: {e}")
-
-    # Handle fallback logic for strict and non-strict modes
-    if use_strict_mode:
-        raise DetectError("Strict mode enabled: Unable to download or load the large model.")
-    else:
-        logger.info("Attempting to fall back to local small model.")
-        return load_local_small_model()
-
-
-def detect(text: str, *,
-           low_memory: bool = True,
-           model_download_proxy: Optional[str] = None,
-           use_strict_mode: bool = False
-           ) -> Dict[str, Union[str, float]]:
-    """
-    Detect the language of a text using FastText.
-    This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed and the null character.
-    If the model is not supervised, this function will throw a ValueError.
-    :param text: The text for language detection
-    :param low_memory: Whether to use a memory-efficient model
-    :param model_download_proxy: Download proxy for the model if needed
-    :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
-    :return: A dictionary with detected language and confidence score
-    :raises LanguageDetectionError: If detection fails
-    """
-    model = load_model(low_memory=low_memory, download_proxy=model_download_proxy, use_strict_mode=use_strict_mode)
-    labels, scores = model.predict(text)
-    language_label = labels[0].replace("__label__", '')
-    confidence_score = min(float(scores[0]), 1.0)
-    return {
-        "lang": language_label,
-        "score": confidence_score,
-    }
-
-
-def detect_multilingual(text: str, *,
-                        low_memory: bool = True,
-                        model_download_proxy: Optional[str] = None,
-                        k: int = 5,
-                        threshold: float = 0.0,
-                        on_unicode_error: str = "strict",
-                        use_strict_mode: bool = False
-                        ) -> List[Dict[str, Any]]:
-    """
-    Detect multiple potential languages and their probabilities in a given text.
-    k controls the number of returned labels. A choice of 5, will return the 5 most probable labels. By default, this returns only the most likely label and probability. threshold filters the returned labels by a threshold on probability. A choice of 0.5 will return labels with at least 0.5 probability. k and threshold will be applied together to determine the returned labels.
-    This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed, and the null character.
-    If the model is not supervised, this function will throw a ValueError.
-
-    :param text: The text for language detection
-    :param low_memory: Whether to use a memory-efficient model
-    :param model_download_proxy: Proxy for downloading the model
-    :param k: Number of top language predictions to return
-    :param threshold: Minimum score threshold for predictions
-    :param on_unicode_error: Error handling for Unicode errors
-    :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
-    :return: A list of dictionaries, each containing a language and its confidence score
-    :raises LanguageDetectionError: If detection fails
-    """
-    model = load_model(low_memory=low_memory, download_proxy=model_download_proxy, use_strict_mode=use_strict_mode)
-    labels, scores = model.predict(text, k=k, threshold=threshold, on_unicode_error=on_unicode_error)
-    results = []
-    for label, score in zip(labels, scores):
-        language_label = label.replace("__label__", '')
-        confidence_score = min(float(score), 1.0)
-        results.append({
-            "lang": language_label,
-            "score": confidence_score,
-        })
-    return sorted(results, key=lambda x: x['score'], reverse=True)