python-iso639 2024.10.22__tar.gz → 2025.1.28__tar.gz

{python_iso639-2024.10.22 → python_iso639-2025.1.28}/CHANGELOG.md

@@ -17,6 +17,21 @@ major/minor/micro version numbers like `05` (it'd have to be just `5`).
 ### Fixed
 ### Security
 
+## [2025.1.28]
+
+### Changed
+* Sped up the retrieval of `Language` objects during runtime.
+
+## [2025.1.27]
+
+### Changed
+* Updated the ISO 639 data from SIL to the latest 2025-01-15 release.
+* The data tables are now included as the original tab-separated value files
+  instead of a compiled SQLite database.
+
+### Fixed
+* Fixed `Language.from_part3` so that it now also works for a retired ISO 639-3 code.
+
 ## [2024.10.22]
 
 ### Added

{python_iso639-2024.10.22/src/python_iso639.egg-info → python_iso639-2025.1.28}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: python-iso639
-Version: 2024.10.22
+Version: 2025.1.28
 Summary: ISO 639 language codes, names, and other associated information
 Author-email: "Jackson L. Lee" <jacksonlunlee@gmail.com>
 License: Apache 2.0
@@ -24,16 +24,17 @@ Classifier: Topic :: Text Processing
 Classifier: Topic :: Text Processing :: General
 Classifier: Topic :: Text Processing :: Indexing
 Classifier: Topic :: Text Processing :: Linguistic
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Provides-Extra: dev
 Requires-Dist: black==24.10.0; extra == "dev"
-Requires-Dist: build==1.2.1; extra == "dev"
+Requires-Dist: build==1.2.2; extra == "dev"
 Requires-Dist: flake8==7.1.1; extra == "dev"
-Requires-Dist: pytest==8.3.3; extra == "dev"
+Requires-Dist: mypy==1.14.1; extra == "dev"
+Requires-Dist: pytest==8.3.4; extra == "dev"
 Requires-Dist: requests==2.32.3; extra == "dev"
-Requires-Dist: twine==5.1.1; extra == "dev"
+Requires-Dist: twine==6.1.0; extra == "dev"
 
 # python-iso639
 
@@ -68,7 +69,7 @@ the actual import name during runtime is `iso639`
 
 ### Creating `Language` Instances
 
-Create a `Language` instance by one of the methods.
+Create a `Language` instance by one of the class methods.
 
 #### `from_part3`, with an ISO 639-3 code
 
@@ -276,7 +277,7 @@ e.g., [`dataclasses.asdict`](https://docs.python.org/3/library/dataclasses.html#
     ```python
     >>> import iso639
     >>> iso639.DATA_LAST_UPDATED
-    datetime.date(2024, 4, 15)
+    datetime.date(2025, 1, 15)
     ```
 
 * `ALL_LANGUAGES`: The list of all `Language` objects based on the included language code data
@@ -284,11 +285,9 @@ e.g., [`dataclasses.asdict`](https://docs.python.org/3/library/dataclasses.html#
     ```python
     >>> import iso639
     >>> type(iso639.ALL_LANGUAGES)
-    <class 'list'>
+    <class 'set'>
     >>> len(iso639.ALL_LANGUAGES)
-    7920
-    >>> iso639.ALL_LANGUAGES[0]
-    Language(part3='aaa', scope='I', type='L', status='A', name='Ghotuo', ...)
+    8307
     ```
 
 ## Links
@@ -304,9 +303,12 @@ for details.
 
 The data source that backs this package is the
 [language code tables published by SIL](https://iso639-3.sil.org/code_tables/download_tables).
+The tables are included in this package under [`src/iso639/_data/`](src/iso639/_data/).
+They are the UTF8-encoded `*.tab` tab-separated files bundled as a ZIP archive file,
+typically found at a URL that looks like
+`https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3_Code_Tables_YYYYMMDD.zip`
+(replace `YYYYMMDD` with the data release date).
 Note that SIL resources have their [terms of use](https://www.sil.org/terms-use).
-For exactly how the data is included in the package,
-see the script at [`scripts/create_languages_db.py`](scripts/create_languages_db.py).
 
 ## Why Another ISO 639 Package?
 

{python_iso639-2024.10.22 → python_iso639-2025.1.28}/README.md

@@ -31,7 +31,7 @@ the actual import name during runtime is `iso639`
 
 ### Creating `Language` Instances
 
-Create a `Language` instance by one of the methods.
+Create a `Language` instance by one of the class methods.
 
 #### `from_part3`, with an ISO 639-3 code
 
@@ -239,7 +239,7 @@ e.g., [`dataclasses.asdict`](https://docs.python.org/3/library/dataclasses.html#
     ```python
     >>> import iso639
     >>> iso639.DATA_LAST_UPDATED
-    datetime.date(2024, 4, 15)
+    datetime.date(2025, 1, 15)
     ```
 
 * `ALL_LANGUAGES`: The list of all `Language` objects based on the included language code data
@@ -247,11 +247,9 @@ e.g., [`dataclasses.asdict`](https://docs.python.org/3/library/dataclasses.html#
     ```python
     >>> import iso639
     >>> type(iso639.ALL_LANGUAGES)
-    <class 'list'>
+    <class 'set'>
     >>> len(iso639.ALL_LANGUAGES)
-    7920
-    >>> iso639.ALL_LANGUAGES[0]
-    Language(part3='aaa', scope='I', type='L', status='A', name='Ghotuo', ...)
+    8307
     ```
 
 ## Links
@@ -267,9 +265,12 @@ for details.
 
 The data source that backs this package is the
 [language code tables published by SIL](https://iso639-3.sil.org/code_tables/download_tables).
+The tables are included in this package under [`src/iso639/_data/`](src/iso639/_data/).
+They are the UTF8-encoded `*.tab` tab-separated files bundled as a ZIP archive file,
+typically found at a URL that looks like
+`https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3_Code_Tables_YYYYMMDD.zip`
+(replace `YYYYMMDD` with the data release date).
 Note that SIL resources have their [terms of use](https://www.sil.org/terms-use).
-For exactly how the data is included in the package,
-see the script at [`scripts/create_languages_db.py`](scripts/create_languages_db.py).
 
 ## Why Another ISO 639 Package?
 

{python_iso639-2024.10.22 → python_iso639-2025.1.28}/pyproject.toml

@@ -4,10 +4,10 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "python-iso639"
-version = "2024.10.22"
+version = "2025.1.28"
 description = "ISO 639 language codes, names, and other associated information"
 readme = "README.md"
-requires-python = ">= 3.8"
+requires-python = ">= 3.9"
 license = { text = "Apache 2.0" }
 authors = [ { name = "Jackson L. Lee", email = "jacksonlunlee@gmail.com" } ]
 keywords = ["ISO 639", "language codes", "languages", "linguistics"]
@@ -35,11 +35,12 @@ classifiers = [
 [project.optional-dependencies]
 dev = [
     "black == 24.10.0",
-    "build == 1.2.1",
+    "build == 1.2.2",
     "flake8 == 7.1.1",
-    "pytest == 8.3.3",
+    "mypy == 1.14.1",
+    "pytest == 8.3.4",
     "requests == 2.32.3",
-    "twine == 5.1.1",
+    "twine == 6.1.0",
 ]
 
 [project.urls]
@@ -51,7 +52,7 @@ where = [ "src" ]
 [tool.setuptools]
 zip-safe = false
 package-dir = { "" = "src" }
-package-data = { "iso639" = ["languages.db"] }
+package-data = { "iso639" = ["_data/*.tab"] }
 
 [tool.pytest.ini_options]
 addopts = "-vv --durations=0 --strict-markers"

{python_iso639-2024.10.22 → python_iso639-2025.1.28}/src/iso639/__init__.py

@@ -1,7 +1,7 @@
 import datetime
 from importlib.metadata import version
 
-from .language import Language, LanguageNotFoundError, _get_all_languages
+from .language import ALL_LANGUAGES, Language, LanguageNotFoundError
 
 
 # __version__ is based on calendar versioning (https://calver.org/).
@@ -9,14 +9,7 @@ from .language import Language, LanguageNotFoundError, _get_all_languages
 # While DATA_LAST_UPDATED is strictly the date for the ISO 639-3 data release,
 # __version__ can be bumped for changes other than data updates.
 __version__ = version("python-iso639")
-DATA_LAST_UPDATED = datetime.date(2024, 4, 15)
-
-
-def __getattr__(name):
-    if name == "ALL_LANGUAGES":
-        return _get_all_languages()
-    else:
-        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+DATA_LAST_UPDATED = datetime.date(2025, 1, 15)
 
 
 __all__ = [

python_iso639-2025.1.28/src/iso639/_data/__init__.py

@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+import csv
+import os
+from collections import defaultdict
+from enum import Enum
+
+from typing import Dict, List, Union
+
+
+class _CodesColumn(Enum):
+    ID = "Id"
+    PART2B = "Part2b"
+    PART2T = "Part2t"
+    PART1 = "Part1"
+    SCOPE = "Scope"
+    TYPE = "Language_Type"
+    REF_NAME = "Ref_Name"
+    COMMENT = "Comment"
+
+
+class _NameIndexColumn(Enum):
+    ID = "Id"
+    PRINT_NAME = "Print_Name"
+    INVERTED_NAME = "Inverted_Name"
+
+
+class _RetirementsColumn(Enum):
+    ID = "Id"
+    REF_NAME = "Ref_Name"
+    RET_REASON = "Ret_Reason"
+    CHANGE_TO = "Change_To"
+    REMEDY = "Ret_Remedy"
+    EFFECTIVE = "Effective"
+
+
+class _MacrolanguagesColumn(Enum):
+    ID = "I_Id"
+    MID = "M_Id"
+    STATUS = "I_Status"
+
+
+_COLUMN_TYPE = Union[
+    _CodesColumn, _NameIndexColumn, _RetirementsColumn, _MacrolanguagesColumn
+]
+_ROW_TYPE = Dict[_COLUMN_TYPE, str]
+
+
+class _Table(Enum):
+    CODES = _CodesColumn
+    NAME_INDEX = _NameIndexColumn
+    RETIREMENTS = _RetirementsColumn
+    MACROLANGUAGES = _MacrolanguagesColumn
+
+
+_THIS_DIR = os.path.dirname(__file__)
+_DATA_TSV_PATHS = {
+    _Table.CODES: os.path.join(_THIS_DIR, "iso-639-3.tab"),
+    _Table.NAME_INDEX: os.path.join(_THIS_DIR, "iso-639-3_Name_Index.tab"),
+    _Table.MACROLANGUAGES: os.path.join(_THIS_DIR, "iso-639-3-macrolanguages.tab"),
+    _Table.RETIREMENTS: os.path.join(_THIS_DIR, "iso-639-3_Retirements.tab"),
+}
+
+
+def _load_tsv(table: _Table) -> List[_ROW_TYPE]:
+    column = table.value
+    with open(_DATA_TSV_PATHS[table], encoding="utf-8", newline="") as tsv_file:
+        tsv_reader = csv.DictReader(tsv_file, delimiter="\t")
+        return [{column(k): v for k, v in row.items()} for row in tsv_reader]
+
+
+_PART3_TO_CODES: Dict[str, _ROW_TYPE] = {
+    row[_CodesColumn.ID]: row for row in _load_tsv(_Table.CODES)
+}
+
+_part3_to_name_index: defaultdict[str, List[_ROW_TYPE]] = defaultdict(list)
+for row in _load_tsv(_Table.NAME_INDEX):
+    _part3_to_name_index[row[_NameIndexColumn.ID]].append(row)
+_PART3_TO_NAME_INDEX: Dict[str, List[_ROW_TYPE]] = dict(_part3_to_name_index)
+
+_PART3_TO_RETIREMENTS: Dict[str, _ROW_TYPE] = {
+    row[_RetirementsColumn.ID]: row for row in _load_tsv(_Table.RETIREMENTS)
+}
+_PART3_TO_MACROLANGUAGES: Dict[str, _ROW_TYPE] = {
+    row[_MacrolanguagesColumn.ID]: row for row in _load_tsv(_Table.MACROLANGUAGES)
+}
+
+_PART2B_TO_PART3: Dict[str, str] = {
+    row[_CodesColumn.PART2B]: part3
+    for part3, row in _PART3_TO_CODES.items()
+    if row[_CodesColumn.PART2B]
+}
+_PART2T_TO_PART3: Dict[str, str] = {
+    row[_CodesColumn.PART2T]: part3
+    for part3, row in _PART3_TO_CODES.items()
+    if row[_CodesColumn.PART2T]
+}
+_PART1_TO_PART3: Dict[str, str] = {
+    row[_CodesColumn.PART1]: part3
+    for part3, row in _PART3_TO_CODES.items()
+    if row[_CodesColumn.PART1]
+}
+_REF_NAME_TO_PART3: Dict[str, str] = {
+    row[_CodesColumn.REF_NAME]: part3
+    for part3, row in _PART3_TO_CODES.items()
+    if row[_CodesColumn.REF_NAME]
+}
+_PRINT_NAME_TO_PART3: Dict[str, str] = {
+    row[_NameIndexColumn.PRINT_NAME]: part3
+    for part3, rows in _PART3_TO_NAME_INDEX.items()
+    for row in rows
+    if row[_NameIndexColumn.PRINT_NAME]
+}
+_INVERTED_NAME_TO_PART3: Dict[str, str] = {
+    row[_NameIndexColumn.INVERTED_NAME]: part3
+    for part3, rows in _PART3_TO_NAME_INDEX.items()
+    for row in rows
+    if row[_NameIndexColumn.INVERTED_NAME]
+}