fast-langdetect 0.2.2__tar.gz → 0.2.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,14 +1,13 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: fast-langdetect
3
- Version: 0.2.2
3
+ Version: 0.2.3
4
4
  Summary: Quickly detect text language and segment language
5
5
  Author-Email: sudoskys <coldlando@hotmail.com>
6
6
  License: MIT
7
7
  Requires-Python: <3.13,>=3.9
8
- Requires-Dist: fasttext-wheel>=0.9.2
9
8
  Requires-Dist: robust-downloader>=0.0.2
10
- Requires-Dist: numpy<2.0.0,>=1.26.4
11
9
  Requires-Dist: requests>=2.32.3
10
+ Requires-Dist: fasttext-predict>=0.9.2.4
12
11
  Description-Content-Type: text/markdown
13
12
 
14
13
  # fast-langdetect 🚀
@@ -19,21 +18,20 @@ Description-Content-Type: text/markdown
19
18
 
20
19
  ## Overview
21
20
 
22
- **fast-langdetect** provides ultra-fast and highly accurate language detection based on FastText, a library developed by
23
- Facebook. This package is 80x faster than traditional methods and offers 95% accuracy.
21
+ **`fast-langdetect`** is an ultra-fast and highly accurate language detection library based on FastText, a library developed by Facebook. Its incredible speed and accuracy make it 80x faster than conventional methods and deliver up to 95% accuracy.
24
22
 
25
- It supports Python versions 3.9 to 3.12.
23
+ - Supported Python `3.9` to `3.12`.
24
+ - Works offline in low memory mode
25
+ - No `numpy` required (thanks to @dalf).
26
26
 
27
- Support offline usage.
27
+ > ### Background
28
+ >
29
+ > This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark) with enhancements in packaging.
30
+ > For more information about the underlying model, see the official FastText documentation: [Language Identification](https://fasttext.cc/docs/en/language-identification.html).
28
31
 
29
- This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark)
30
- with enhancements in packaging.
31
-
32
- For more information on the underlying FastText model, refer to the official
33
- documentation: [FastText Language Identification](https://fasttext.cc/docs/en/language-identification.html).
34
-
35
- > [!NOTE]
36
- > This library requires over 200MB of memory to use in low memory mode.
32
+ > ### Possible memory usage
33
+ >
34
+ > *This library requires at least **200MB memory** in low-memory mode.*
37
35
 
38
36
  ## Installation 💻
39
37
 
@@ -53,20 +51,17 @@ pdm add fast-langdetect
53
51
 
54
52
  ## Usage 🖥️
55
53
 
56
- For optimal performance and accuracy in language detection, use `detect(text, low_memory=False)` to load the larger
57
- model.
54
+ In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
58
55
 
59
- > The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
56
+ ### Prerequisites
60
57
 
61
- ### Native API (Recommended)
58
+ - The “/n” character in the argument string must be removed before calling the function.
59
+ - If the sample is too long or too short, the accuracy will be reduced (e.g. if it is too short, Chinese will be predicted as Japanese).
60
+ - The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
62
61
 
63
- > [!NOTE]
64
- > This function assumes to be given a single line of text. *You should remove `\n` characters before passing the text.*
65
- > If the sample is too long or too short, the accuracy will decrease (for example, in the case of too short, Chinese
66
- > will be predicted as Japanese).
62
+ ### Native API (Recommended)
67
63
 
68
64
  ```python
69
-
70
65
  from fast_langdetect import detect, detect_multilingual
71
66
 
72
67
  # Single language detection
@@ -82,15 +77,17 @@ multiline_text = """
82
77
  Hello, world!
83
78
  This is a multiline text.
84
79
  But we need remove `\n` characters or it will raise an ValueError.
80
+ REMOVE \n
85
81
  """
86
- multiline_text = multiline_text.replace("\n", "") # NOTE:ITS IMPORTANT TO REMOVE \n CHARACTERS
82
+ multiline_text = multiline_text.replace("\n", "")
87
83
  print(detect(multiline_text))
88
84
  # Output: {'lang': 'en', 'score': 0.8509423136711121}
89
85
 
90
86
  print(detect("Привет, мир!")["lang"])
91
87
  # Output: ru
92
88
 
93
- # Multi-language detection
89
+ # Multi-language detection with low memory mode enabled
90
+ # The accuracy is not as good as it should be
94
91
  print(detect_multilingual("Hello, world!你好世界!Привет, мир!"))
95
92
  # Output: [{'lang': 'ja', 'score': 0.32009604573249817}, {'lang': 'uk', 'score': 0.27781224250793457}, {'lang': 'zh', 'score': 0.17542070150375366}, {'lang': 'sr', 'score': 0.08751443773508072}, {'lang': 'bg', 'score': 0.05222449079155922}]
96
93
 
@@ -99,6 +96,10 @@ print(detect_multilingual("Hello, world!你好世界!Привет, мир!", low
99
96
  # Output: [{'lang': 'ru', 'score': 0.39008623361587524}, {'lang': 'zh', 'score': 0.18235979974269867}, {'lang': 'ja', 'score': 0.08473210036754608}, {'lang': 'sr', 'score': 0.057975586503744125}, {'lang': 'en', 'score': 0.05422825738787651}]
100
97
  ```
101
98
 
99
+ #### Fallbacks
100
+
101
+ We provide a fallback mechanism: when `use_strict_mode=False`, if the program fails to load the **large model** (`low_memory=False`), it will fall back to the offline **small model** to complete the prediction task.
102
+
102
103
  ### Convenient `detect_language` Function
103
104
 
104
105
  ```python
@@ -148,4 +149,4 @@ models
148
149
  journal={arXiv preprint arXiv:1612.03651},
149
150
  year={2016}
150
151
  }
151
- ```
152
+ ```
@@ -6,21 +6,20 @@
6
6
 
7
7
  ## Overview
8
8
 
9
- **fast-langdetect** provides ultra-fast and highly accurate language detection based on FastText, a library developed by
10
- Facebook. This package is 80x faster than traditional methods and offers 95% accuracy.
9
+ **`fast-langdetect`** is an ultra-fast and highly accurate language detection library based on FastText, a library developed by Facebook. Its incredible speed and accuracy make it 80x faster than conventional methods and deliver up to 95% accuracy.
11
10
 
12
- It supports Python versions 3.9 to 3.12.
11
+ - Supported Python `3.9` to `3.12`.
12
+ - Works offline in low memory mode
13
+ - No `numpy` required (thanks to @dalf).
13
14
 
14
- Support offline usage.
15
+ > ### Background
16
+ >
17
+ > This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark) with enhancements in packaging.
18
+ > For more information about the underlying model, see the official FastText documentation: [Language Identification](https://fasttext.cc/docs/en/language-identification.html).
15
19
 
16
- This project builds upon [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect#benchmark)
17
- with enhancements in packaging.
18
-
19
- For more information on the underlying FastText model, refer to the official
20
- documentation: [FastText Language Identification](https://fasttext.cc/docs/en/language-identification.html).
21
-
22
- > [!NOTE]
23
- > This library requires over 200MB of memory to use in low memory mode.
20
+ > ### Possible memory usage
21
+ >
22
+ > *This library requires at least **200MB memory** in low-memory mode.*
24
23
 
25
24
  ## Installation 💻
26
25
 
@@ -40,20 +39,17 @@ pdm add fast-langdetect
40
39
 
41
40
  ## Usage 🖥️
42
41
 
43
- For optimal performance and accuracy in language detection, use `detect(text, low_memory=False)` to load the larger
44
- model.
42
+ In scenarios **where accuracy is important**, you should not rely on the detection results of small models, use `low_memory=False` to download larger models!
45
43
 
46
- > The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
44
+ ### Prerequisites
47
45
 
48
- ### Native API (Recommended)
46
+ - The “/n” character in the argument string must be removed before calling the function.
47
+ - If the sample is too long or too short, the accuracy will be reduced (e.g. if it is too short, Chinese will be predicted as Japanese).
48
+ - The model will be downloaded to the `/tmp/fasttext-langdetect` directory upon first use.
49
49
 
50
- > [!NOTE]
51
- > This function assumes to be given a single line of text. *You should remove `\n` characters before passing the text.*
52
- > If the sample is too long or too short, the accuracy will decrease (for example, in the case of too short, Chinese
53
- > will be predicted as Japanese).
50
+ ### Native API (Recommended)
54
51
 
55
52
  ```python
56
-
57
53
  from fast_langdetect import detect, detect_multilingual
58
54
 
59
55
  # Single language detection
@@ -69,15 +65,17 @@ multiline_text = """
69
65
  Hello, world!
70
66
  This is a multiline text.
71
67
  But we need remove `\n` characters or it will raise an ValueError.
68
+ REMOVE \n
72
69
  """
73
- multiline_text = multiline_text.replace("\n", "") # NOTE:ITS IMPORTANT TO REMOVE \n CHARACTERS
70
+ multiline_text = multiline_text.replace("\n", "")
74
71
  print(detect(multiline_text))
75
72
  # Output: {'lang': 'en', 'score': 0.8509423136711121}
76
73
 
77
74
  print(detect("Привет, мир!")["lang"])
78
75
  # Output: ru
79
76
 
80
- # Multi-language detection
77
+ # Multi-language detection with low memory mode enabled
78
+ # The accuracy is not as good as it should be
81
79
  print(detect_multilingual("Hello, world!你好世界!Привет, мир!"))
82
80
  # Output: [{'lang': 'ja', 'score': 0.32009604573249817}, {'lang': 'uk', 'score': 0.27781224250793457}, {'lang': 'zh', 'score': 0.17542070150375366}, {'lang': 'sr', 'score': 0.08751443773508072}, {'lang': 'bg', 'score': 0.05222449079155922}]
83
81
 
@@ -86,6 +84,10 @@ print(detect_multilingual("Hello, world!你好世界!Привет, мир!", low
86
84
  # Output: [{'lang': 'ru', 'score': 0.39008623361587524}, {'lang': 'zh', 'score': 0.18235979974269867}, {'lang': 'ja', 'score': 0.08473210036754608}, {'lang': 'sr', 'score': 0.057975586503744125}, {'lang': 'en', 'score': 0.05422825738787651}]
87
85
  ```
88
86
 
87
+ #### Fallbacks
88
+
89
+ We provide a fallback mechanism: when `use_strict_mode=False`, if the program fails to load the **large model** (`low_memory=False`), it will fall back to the offline **small model** to complete the prediction task.
90
+
89
91
  ### Convenient `detect_language` Function
90
92
 
91
93
  ```python
@@ -135,4 +137,4 @@ models
135
137
  journal={arXiv preprint arXiv:1612.03651},
136
138
  year={2016}
137
139
  }
138
- ```
140
+ ```
@@ -1,15 +1,14 @@
1
1
  [project]
2
2
  name = "fast-langdetect"
3
- version = "0.2.2"
3
+ version = "0.2.3"
4
4
  description = "Quickly detect text language and segment language"
5
5
  authors = [
6
6
  { name = "sudoskys", email = "coldlando@hotmail.com" },
7
7
  ]
8
8
  dependencies = [
9
- "fasttext-wheel>=0.9.2",
10
9
  "robust-downloader>=0.0.2",
11
- "numpy>=1.26.4,<2.0.0",
12
10
  "requests>=2.32.3",
11
+ "fasttext-predict>=0.9.2.4",
13
12
  ]
14
13
  requires-python = ">=3.9,<3.13"
15
14
  readme = "README.md"
@@ -0,0 +1,203 @@
1
+ # -*- coding: utf-8 -*-
2
+ # @Time : 2024/01/17 下午08:30
3
+ # @Author : sudoskys
4
+ # @File : infer.py
5
+ # @Software: PyCharm
6
+ import logging
7
+ import os
8
+ from pathlib import Path
9
+ from typing import Dict, List, Optional, Union, Any
10
+
11
+ import fasttext
12
+ from robust_downloader import download
13
+
14
+ logger = logging.getLogger(__name__)
15
+ CACHE_DIRECTORY = os.getenv("FTLANG_CACHE", "/tmp/fasttext-langdetect")
16
+ LOCAL_SMALL_MODEL_PATH = Path(__file__).parent / "resources" / "lid.176.ftz"
17
+
18
+ FASTTEXT_LARGE_MODEL_URL = "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin"
19
+ FASTTEXT_LARGE_MODEL_NAME = "lid.176.bin"
20
+
21
+
22
+ class DetectError(Exception):
23
+ pass
24
+
25
+
26
+ class ModelManager:
27
+ def __init__(self):
28
+ self._models = {}
29
+
30
+ def get_model(self, key: str):
31
+ """Retrieve cached model."""
32
+ return self._models.get(key)
33
+
34
+ def cache_model(self, key: str, model) -> None:
35
+ """Cache loaded FastText model."""
36
+ self._models[key] = model
37
+
38
+
39
+ _model_cache = ModelManager()
40
+
41
+
42
+ def download_model(
43
+ download_url: str,
44
+ save_path: Path,
45
+ proxy: Optional[str] = None
46
+ ) -> None:
47
+ """
48
+ Download FastText model file if it doesn't exist.
49
+ :param download_url: URL to download the model from
50
+ :param save_path: Path to save the downloaded model
51
+ :param proxy: Proxy URL for downloading the model
52
+ :raises DetectError: If download fails
53
+ """
54
+ if save_path.exists():
55
+ logger.info(f"fast-langdetect:Model already exists at {save_path}. Skipping download.")
56
+ return
57
+
58
+ logger.info(f"fast-langdetect:Downloading FastText model from {download_url} to {save_path}")
59
+ try:
60
+ download(
61
+ url=download_url,
62
+ folder=str(save_path.parent),
63
+ filename=save_path.name,
64
+ proxy=proxy,
65
+ retry_max=3,
66
+ timeout=30,
67
+ )
68
+ except Exception as e:
69
+ logger.error(f"fast-langdetect:Failed to download FastText model from {download_url}: {e}")
70
+ raise DetectError(f"Unable to download model from {download_url}")
71
+
72
+
73
+ def load_fasttext_model(
74
+ model_path: Path,
75
+ download_url: Optional[str] = None,
76
+ proxy: Optional[str] = None,
77
+ ):
78
+ """
79
+ Load a FastText model, downloading it if necessary.
80
+ :param model_path: Path to the FastText model file
81
+ :param download_url: URL to download the model from
82
+ :param proxy: Proxy URL for downloading the model
83
+ :return: FastText model
84
+ :raises DetectError: If model loading fails
85
+ """
86
+ if not model_path.exists() and download_url:
87
+ # Attempt to download the model
88
+ download_model(download_url, model_path, proxy)
89
+
90
+ if not model_path.exists():
91
+ raise DetectError(f"FastText model file not found at {model_path}")
92
+
93
+ try:
94
+ # Load FastText model
95
+ return fasttext.load_model(str(model_path))
96
+ except Exception as e:
97
+ logger.error(f"fast-langdetect:Failed to load FastText model from {model_path}: {e}")
98
+ raise DetectError(f"Failed to load FastText model: {e}")
99
+
100
+
101
+ def load_model(
102
+ low_memory: bool = False,
103
+ download_proxy: Optional[str] = None,
104
+ use_strict_mode: bool = False,
105
+ ):
106
+ """
107
+ Load a FastText model based on memory preference.
108
+ :param low_memory: Whether to use a memory-efficient model
109
+ :param download_proxy: Proxy URL for downloading the model
110
+ :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
111
+ """
112
+ # Model path selection
113
+ if low_memory:
114
+ cache_key = "low_memory"
115
+ model_path = LOCAL_SMALL_MODEL_PATH
116
+ else:
117
+ cache_key = "high_memory"
118
+ model_path = Path(CACHE_DIRECTORY) / FASTTEXT_LARGE_MODEL_NAME
119
+
120
+ # Check cache
121
+ cached_model = _model_cache.get_model(cache_key)
122
+ if cached_model:
123
+ return cached_model
124
+
125
+ # Load appropriate model
126
+ try:
127
+ if low_memory:
128
+ model = load_fasttext_model(model_path)
129
+ else:
130
+ model = load_fasttext_model(model_path, download_url=FASTTEXT_LARGE_MODEL_URL, proxy=download_proxy)
131
+ _model_cache.cache_model(cache_key, model)
132
+ return model
133
+ except Exception as e:
134
+ logger.error(f"fast-langdetect:Failed to load model ({'low' if low_memory else 'high'} memory): {e}")
135
+ if use_strict_mode:
136
+ raise DetectError("Failed to load FastText model.") from e
137
+ elif not low_memory:
138
+ logger.info("Falling back to low-memory model...")
139
+ return load_model(low_memory=True, use_strict_mode=True)
140
+ raise e
141
+
142
+
143
+ def detect(
144
+ text: str,
145
+ *,
146
+ low_memory: bool = True,
147
+ model_download_proxy: Optional[str] = None,
148
+ use_strict_mode: bool = False,
149
+ ) -> Dict[str, Union[str, float]]:
150
+ """
151
+ Detect the language of a text using FastText.
152
+ This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed and the null character.
153
+ If the model is not supervised, this function will throw a ValueError.
154
+ :param text: The text for language detection
155
+ :param low_memory: Whether to use a memory-efficient model
156
+ :param model_download_proxy: Download proxy for the model if needed
157
+ :param use_strict_mode: If it was enabled, strictly loads large model or raises error if it fails
158
+ :return: A dictionary with detected language and confidence score
159
+ :raises LanguageDetectionError: If detection fails
160
+ """
161
+ model = load_model(
162
+ low_memory=low_memory,
163
+ download_proxy=model_download_proxy,
164
+ use_strict_mode=use_strict_mode,
165
+ )
166
+ try:
167
+ labels, scores = model.predict(text)
168
+ language_label = labels[0].replace("__label__", "")
169
+ confidence_score = min(float(scores[0]), 1.0)
170
+ return {"lang": language_label, "score": confidence_score}
171
+ except Exception as e:
172
+ logger.error(f"fast-langdetect:Error during language detection: {e}")
173
+ raise DetectError("Language detection failed.")
174
+
175
+
176
+ def detect_multilingual(
177
+ text: str,
178
+ *,
179
+ low_memory: bool = True,
180
+ model_download_proxy: Optional[str] = None,
181
+ k: int = 5,
182
+ threshold: float = 0.0,
183
+ use_strict_mode: bool = False,
184
+ ) -> List[Dict[str, Any]]:
185
+ """
186
+ Detect the top-k probable languages for a given text.
187
+ """
188
+ model = load_model(
189
+ low_memory=low_memory,
190
+ download_proxy=model_download_proxy,
191
+ use_strict_mode=use_strict_mode,
192
+ )
193
+ # Multilingual detection using FastText
194
+ try:
195
+ labels, scores = model.predict(text, k=k, threshold=threshold)
196
+ results = [
197
+ {"lang": label.replace("__label__", ""), "score": min(float(score), 1.0)}
198
+ for label, score in zip(labels, scores)
199
+ ]
200
+ return sorted(results, key=lambda x: x["score"], reverse=True)
201
+ except Exception as e:
202
+ logger.error(f"fast-langdetect:Error during multilingual detection: {e}")
203
+ raise DetectError("Multilingual detection failed.")
@@ -1,198 +0,0 @@
1
- # -*- coding: utf-8 -*-
2
- # @Time : 2024/1/17 下午8:30
3
- # @Author : sudoskys
4
- # @File : infer.py
5
- # @Software: PyCharm
6
- import logging
7
- import os
8
- from enum import Enum
9
- from pathlib import Path
10
- from typing import Dict, Union, List, Optional, Any
11
-
12
- import fasttext
13
- from robust_downloader import download
14
-
15
- logger = logging.getLogger(__name__)
16
- CACHE_DIRECTORY = os.getenv("FTLANG_CACHE", "/tmp/fasttext-langdetect")
17
- LOCAL_SMALL_MODEL_PATH = Path(__file__).parent / "resources" / "lid.176.ftz"
18
-
19
- # Suppress FastText output if possible
20
- try:
21
- fasttext.FastText.eprint = lambda *args, **kwargs: None
22
- except Exception:
23
- pass
24
-
25
-
26
- class ModelType(Enum):
27
- LOW_MEMORY = "low_mem"
28
- HIGH_MEMORY = "high_mem"
29
-
30
-
31
- class ModelCache:
32
- def __init__(self):
33
- self._models = {}
34
-
35
- def get_model(self, model_type: ModelType) -> Optional["fasttext.FastText._FastText"]:
36
- return self._models.get(model_type)
37
-
38
- def set_model(self, model_type: ModelType, model: "fasttext.FastText._FastText"):
39
- self._models[model_type] = model
40
-
41
-
42
- _model_cache = ModelCache()
43
-
44
-
45
- class DetectError(Exception):
46
- """Custom exception for language detection errors."""
47
- pass
48
-
49
-
50
- def load_model(low_memory: bool = False,
51
- download_proxy: Optional[str] = None,
52
- use_strict_mode: bool = False) -> "fasttext.FastText._FastText":
53
- """
54
- Load the FastText model based on memory preference.
55
-
56
- :param low_memory: Indicates whether to load a smaller, memory-efficient model
57
- :param download_proxy: Proxy to use for downloading the large model if necessary
58
- :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
59
- :return: Loaded FastText model
60
- :raises DetectError: If the model cannot be loaded
61
- """
62
- model_type = ModelType.LOW_MEMORY if low_memory else ModelType.HIGH_MEMORY
63
-
64
- # If the model is already loaded, return it
65
- cached_model = _model_cache.get_model(model_type)
66
- if cached_model:
67
- return cached_model
68
-
69
- def load_local_small_model():
70
- """Try to load the local small model."""
71
- try:
72
- _loaded_model = fasttext.load_model(str(LOCAL_SMALL_MODEL_PATH))
73
- _model_cache.set_model(ModelType.LOW_MEMORY, _loaded_model)
74
- return _loaded_model
75
- except Exception as e:
76
- logger.error(f"Failed to load the local small model '{LOCAL_SMALL_MODEL_PATH}': {e}")
77
- raise DetectError("Unable to load low-memory model from local resources.")
78
-
79
- def load_large_model():
80
- """Try to load the large model."""
81
- try:
82
- loaded_model = fasttext.load_model(str(model_path))
83
- _model_cache.set_model(ModelType.HIGH_MEMORY, loaded_model)
84
- return loaded_model
85
- except Exception as e:
86
- logger.error(f"Failed to load the large model '{model_path}': {e}")
87
- return None
88
-
89
- if low_memory:
90
- # Attempt to load the local small model
91
- return load_local_small_model()
92
-
93
- # Path for the large model
94
- large_model_name = "lid.176.bin"
95
- model_path = Path(CACHE_DIRECTORY) / large_model_name
96
-
97
- # If the large model is already present, load it
98
- if model_path.exists():
99
- # Model cant be dir
100
- if model_path.is_dir():
101
- try:
102
- model_path.rmdir()
103
- except Exception as e:
104
- logger.error(f"Failed to remove the directory '{model_path}': {e}")
105
- raise DetectError(f"Unexpected directory found in large model file path '{model_path}': {e}")
106
- # Attempt to load large model
107
- loaded_model = load_large_model()
108
- if loaded_model:
109
- return loaded_model
110
-
111
- # If the large model is not present, attempt to download (only if necessary)
112
- model_url = "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin"
113
- try:
114
- logger.info(f"Downloading large model from {model_url} to {model_path}")
115
- download(
116
- url=model_url,
117
- folder=CACHE_DIRECTORY,
118
- filename=large_model_name,
119
- proxy=download_proxy,
120
- retry_max=3,
121
- timeout=20
122
- )
123
- # Try loading the model again after download
124
- loaded_model = load_large_model()
125
- if loaded_model:
126
- return loaded_model
127
- except Exception as e:
128
- logger.error(f"Failed to download the large model: {e}")
129
-
130
- # Handle fallback logic for strict and non-strict modes
131
- if use_strict_mode:
132
- raise DetectError("Strict mode enabled: Unable to download or load the large model.")
133
- else:
134
- logger.info("Attempting to fall back to local small model.")
135
- return load_local_small_model()
136
-
137
-
138
- def detect(text: str, *,
139
- low_memory: bool = True,
140
- model_download_proxy: Optional[str] = None,
141
- use_strict_mode: bool = False
142
- ) -> Dict[str, Union[str, float]]:
143
- """
144
- Detect the language of a text using FastText.
145
- This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed and the null character.
146
- If the model is not supervised, this function will throw a ValueError.
147
- :param text: The text for language detection
148
- :param low_memory: Whether to use a memory-efficient model
149
- :param model_download_proxy: Download proxy for the model if needed
150
- :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
151
- :return: A dictionary with detected language and confidence score
152
- :raises LanguageDetectionError: If detection fails
153
- """
154
- model = load_model(low_memory=low_memory, download_proxy=model_download_proxy, use_strict_mode=use_strict_mode)
155
- labels, scores = model.predict(text)
156
- language_label = labels[0].replace("__label__", '')
157
- confidence_score = min(float(scores[0]), 1.0)
158
- return {
159
- "lang": language_label,
160
- "score": confidence_score,
161
- }
162
-
163
-
164
- def detect_multilingual(text: str, *,
165
- low_memory: bool = True,
166
- model_download_proxy: Optional[str] = None,
167
- k: int = 5,
168
- threshold: float = 0.0,
169
- on_unicode_error: str = "strict",
170
- use_strict_mode: bool = False
171
- ) -> List[Dict[str, Any]]:
172
- """
173
- Detect multiple potential languages and their probabilities in a given text.
174
- k controls the number of returned labels. A choice of 5, will return the 5 most probable labels. By default, this returns only the most likely label and probability. threshold filters the returned labels by a threshold on probability. A choice of 0.5 will return labels with at least 0.5 probability. k and threshold will be applied together to determine the returned labels.
175
- This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed, and the null character.
176
- If the model is not supervised, this function will throw a ValueError.
177
-
178
- :param text: The text for language detection
179
- :param low_memory: Whether to use a memory-efficient model
180
- :param model_download_proxy: Proxy for downloading the model
181
- :param k: Number of top language predictions to return
182
- :param threshold: Minimum score threshold for predictions
183
- :param on_unicode_error: Error handling for Unicode errors
184
- :param use_strict_mode: If enabled, strictly loads large model or raises error if it fails
185
- :return: A list of dictionaries, each containing a language and its confidence score
186
- :raises LanguageDetectionError: If detection fails
187
- """
188
- model = load_model(low_memory=low_memory, download_proxy=model_download_proxy, use_strict_mode=use_strict_mode)
189
- labels, scores = model.predict(text, k=k, threshold=threshold, on_unicode_error=on_unicode_error)
190
- results = []
191
- for label, score in zip(labels, scores):
192
- language_label = label.replace("__label__", '')
193
- confidence_score = min(float(score), 1.0)
194
- results.append({
195
- "lang": language_label,
196
- "score": confidence_score,
197
- })
198
- return sorted(results, key=lambda x: x['score'], reverse=True)
File without changes