lokit-python 0.1.0__cp313-cp313-win_amd64.whl

lokit_python-0.1.0.dist-info/METADATA

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: lokit-python
+Version: 0.1.0
+Summary: Add your description here
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: lxml>=6.1.1
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: polib>=1.2.0
+
+# lokit
+
+> [!WARNING]
+> **Beta Release:** lokit is currently in Beta. The API is volatile and subject to rapid, breaking changes prior to the official V1 release.
+
+lokit is a high-performance, strictly type-safe, and highly memory-efficient localization toolkit for Python. 
+Supports Python 3.12+.
+
+Unlike legacy tools that wrap around XML DOM element trees in-memory, lokit represents a shift away from XML-based localization interchange formats towards native language parsing. It ingests localization formats (TMX, XLIFF, PO, XLSX, CSV, JSON, HTML, IDML) and compiles them into a strict, unified structural data model. This enables not just parsing, but robust data manipulation, semantic extraction, and advanced translation memory features out-of-the-box. Lokit focuses on streaming and asynchronous processing rather than synchronous events using in-memory files. 
+
+This format type can be easily converted to JSON for interchange with other systems. I've made parsing and data transfers as native as possible by capturing all elements of traditional interchange formats in a common format structure. This allows for much better compatibility, especially in terms of segment matching and leveraging as it uses flattened strings as standard. Tags are preserved but as a common format, meaning the structure parsed from XLIFF will be the same as the structure parsed from HTML.
+
+These legacy file formats have supported vendor-lock in for many year, making it difficult for any client to move to another system. Seeing that this is a major issue across the domain, something new is needed where vendors do not use hidden, legacy technology to lock in their clients. Localization deserves innovation.
+
+Note: SDKs in other languages are coming soon.
+
+## Core Features
+
+lokit provides a comprehensive suite of tools for managing localization data:
+
+* **Native Structural Modeling:** Converts disjointed interchange formats into a strict, unified Python Data class, ensuring complete type safety across your entire localization pipeline.
+* **Advanced Matching Engine:** Provides Exact Matching, Fuzzy Matching (via SequenceMatcher), and In-Context Exact (ICE) Matching leveraging previous and next segment context, as well as inline tag signatures.
+* **Deep Sub-segment Extraction:** Automatically parses and isolates inline tags, properties, and formatting markers, allowing for safe manipulation of text without corrupting code.
+* **Semantic Querying:** Easily traverse and filter translation units using complex predicates, exact ID lookups, or deep nested JSON path querying (`where()`).
+* **Plural Support:** Native extraction and structuring of pluralized translation units.
+* **Universal Format Conversion:** Instantly import and export between any supported format (e.g., TMX to JSON, HTML to XLIFF) with zero data loss.
+* **Synchronous and Asynchronous Streaming:** Process massive enterprise files natively using Python async generators to keep memory overhead to an absolute minimum.
+
+## Parsing Performance vs Translate-Toolkit
+
+When dealing with enterprise-scale localization environments, parsing performance and memory efficiency are paramount. lokit is designed to be significantly leaner and faster than the industry standard.
+
+In a stress-test benchmark on a **612 MB TMX** file containing **557,058 segments**, parsing to XLIFF and back into TMX over 3 consecutive iterations, lokit yielded the following comparative averages:
+
+| Library | Avg Duration (s) | Peak Memory (MB) | Memory Efficiency |
+|---------|------------------|------------------|-------------------|
+| **lokit (async)** | **57.5s** | **213.8 MB** | **~10.6x Less RAM** |
+| **translate-toolkit** | 60.0s | 2,275.7 MB | ~2.3 GB |
+
+Because translate-toolkit loads whole files into string buffers and C-level DOM trees synchronously, its memory spikes to over 2.2 Gigabytes. lokit leverages generator-based async streaming, allowing it to complete the exact same workload using **10.6x less RAM**, while operating slightly faster overall.
+
+This memory safety allows for parallel processing of events, making it suitable for large-scale localization workflows and backend systems.
+
+## SDK Usage Reference
+
+Lokit operates around a central `BaseStructure` dataclass model, which standardizes localization units and segments. This instructs better standardization and branching in a more language native way compared to XML based file formats. Parsing SDKs are added for both extraction and export tasks for localization interchange formats along with common file types.
+
+### Installation
+
+Install lokit via pip:
+
+```bash
+pip install lokit-python
+```
+
+### Basic Parsing and Conversion
+
+Converting files synchronously is straightforward using the modular importers and exporters APIs.
+
+```python
+from lokit.importers import import_tmx
+from lokit.exporters import export_xliff
+
+# Parse a localization file into a BaseStructure
+document = import_tmx("path/to/source.tmx")
+
+print(f"Loaded {len(document.data)} units")
+print(f"Source Locale: {document.source_locale}")
+print(f"Target Locale: {document.target_locale}")
+
+# Export the BaseStructure out to a new format
+export_xliff(document, "path/to/target.xliff")
+```
+
+### Asynchronous Streaming for Massive Files
+
+For files spanning hundreds of megabytes, parsing the entire DOM structure into memory is inefficient. Lokit supports stream-parsing natively.
+
+```python
+import asyncio
+from lokit.importers import import_tmx_async
+from lokit.exporters.xliff import export_xliff_async
+from lokit.data.structure import BaseStructure
+
+async def process_large_file():
+    units = {}
+    
+    # Stream the file in an asynchronous generator
+    async for unit_id, unit_data in import_tmx_async("massive_file.tmx"):
+        units[unit_id] = unit_data
+
+    # Reconstruct the document safely
+    doc = BaseStructure(
+        source_locale="en_US", 
+        target_locale="de_DE", 
+        data=units
+    )
+    
+    # Export asynchronously
+    await export_xliff_async(doc, "massive_output.xliff")
+
+asyncio.run(process_large_file())
+```
+
+### Advanced Querying and Matching
+
+The `Lokit` logic wrapper provides access to the powerful matching engine and data manipulation features.
+
+```python
+from lokit.logic import Lokit
+
+# Wrap a parsed document or path in the Lokit logic engine
+engine = Lokit.parse("path/to/source.xliff")
+
+# Query specific nested data structures
+button_units = engine.where("extensions.component", "checkout_button")
+
+# Perform fuzzy matching against translation memory
+results = engine.fuzzy_find("Complete your purchase", limit=5, threshold=0.75)
+for match in results:
+    print(f"Match found: {match.unit_id} (Score: {match.score})")
+
+# Perform strict In-Context Exact (ICE) matching
+ice_match = engine.match(
+    source="Submit",
+    target_unit_id="submit_btn_1",
+    previous_source="Enter your email",
+    require_context=True
+)
+```
+
+## Supported Formats
+* TMX (Translation Memory eXchange)
+* XLIFF (XML Localization Interchange File Format)
+* PO/POT (Gettext Portable Object)
+* XLSX / CSV (Spreadsheets)
+* JSON (Key-Value nested localization trees)
+* HTML (Hypertext Markup)
+* IDML (InDesign Markup Language)

lokit_python-0.1.0.dist-info/RECORD

@@ -0,0 +1,97 @@
+821d8b73c2a02cb7980f__mypyc.cp313-win_amd64.pyd,sha256=OZeVtknGA7z7HvYkyUX4Y23CgB1SR_teitbfinPBcLM,1430016
+lokit/__init__.cp313-win_amd64.pyd,sha256=A7_FcVluqfyy72C-kpDvnXRfaGmzec_LvP22VLMRYCA,11264
+lokit/__init__.py,sha256=YhZSX5QrEkFxaogw0N7eLf0TZMdJp97kBjTXGIPrK68,3043
+lokit/format_detection.cp313-win_amd64.pyd,sha256=NqiapDgbj6hKbdlN6GqoK2FK5MslhrEZe6lamxqB4bo,11264
+lokit/format_detection.py,sha256=CqoAQf8_V5iZtMzoY5W_nQmWLYo8Ko3wIlVTM8RlKJg,3775
+lokit/importers.py,sha256=w3CrT2F0USyfRC0natLV27ksyrhGpkK8a3qM717rpvE,10485
+lokit/logic.cp313-win_amd64.pyd,sha256=WnIbIdn_bnl8HmGbgPPMeYjOhLn8UOmeOO_8uZFSR_I,11264
+lokit/logic.py,sha256=Mj5cIE2Dkc9geAFmd6EFwMJy-OWj_XQB1TIGjd04lXk,11705
+lokit/py.typed,sha256=frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN_XKdLCPjaYaY,2
+lokit/core/__init__.cp313-win_amd64.pyd,sha256=NjQ8titXkOvxZ3yMKbp3iSrjVGURNFN2zu9YGmA2LdE,11264
+lokit/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+lokit/core/logger.cp313-win_amd64.pyd,sha256=Gr0dSETcueq1wpZIXej6eLW3yhF4FUqEcBBNBFRXOeA,11264
+lokit/core/logger.py,sha256=yjbitZCKBW25pewWZ83YChNnKj4duWD9JAGjRrvE_xk,580
+lokit/data/__init__.cp313-win_amd64.pyd,sha256=lrMub1sNJc_JP1nJlrLgNKGt8fAnqaiC_ucg_A6qr9c,11264
+lokit/data/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+lokit/data/lang_codes.cp313-win_amd64.pyd,sha256=t3A2TB5etk2Jpk5RJIi_eskampBN3bXB4k__iOInaBE,11264
+lokit/data/lang_codes.py,sha256=BY0MhUKeAd-P9IeoEJNi-dkUEklJVaWpQxVq-iAFfO0,10259
+lokit/data/structure.cp313-win_amd64.pyd,sha256=OkESoa-TFg1DAHB_Cm8T70SvW7CpBnu0B_UL290FFmg,11264
+lokit/data/structure.py,sha256=bEnSWe8whhfUcqt9qTf9LVF9z-KQrS2hHNLERyWXhH8,3105
+lokit/data/tag_types.cp313-win_amd64.pyd,sha256=i5fITnpLqTtVRZioV_az2RGP-jNCFDM239VCzpNs1zQ,11264
+lokit/data/tag_types.py,sha256=sjEQdsTzpJuXX3snq2m4o9wQXiNCS_dU0dtwgMHGfEQ,2116
+lokit/exporters/__init__.cp313-win_amd64.pyd,sha256=u-vV_nsek9zaTuuJGNdsx6bNORcXvJVIiV7dhQvhE10,11264
+lokit/exporters/__init__.py,sha256=dpPkL_ODO0XhqxhuExjvvuuGolSgGG66WNS0gpVPpGI,1072
+lokit/exporters/csv.cp313-win_amd64.pyd,sha256=Dey-4yfq6IWoQ5TxyJstNy0mQPv8Ek3p-q1IlkjQaxU,11264
+lokit/exporters/csv.py,sha256=2_E3956eX8nRTDEZue07JdQSoIVgMkHq5hfXJyCWJPE,1122
+lokit/exporters/html.cp313-win_amd64.pyd,sha256=KVkvqnrXAxcyd87u8fcpZBemWVGO71dYOCMDJVewYd0,11264
+lokit/exporters/html.py,sha256=Og23iToijlGNEy3IvM7DCKlKJT_enOlzTlNveb_6diw,6981
+lokit/exporters/idml.cp313-win_amd64.pyd,sha256=n7FkS--R6C6jA47lAcYuGTiEtbyJQmmIrMDSsZXrXCw,11264
+lokit/exporters/idml.py,sha256=QhscjqyhKvBEUUl1iqBIPF7dKNbfN17mmuQN9BV7qRg,5619
+lokit/exporters/json_i18n.cp313-win_amd64.pyd,sha256=jqen_5mYoJpHn8kqIynuKhxy0wzGl1n5vmzS4mp2SLk,11264
+lokit/exporters/json_i18n.py,sha256=UC3Osu6_uJMR497RtQr5DbGPvOmwy7BlFyPlU9wOO0k,1303
+lokit/exporters/po.cp313-win_amd64.pyd,sha256=H7GzGkrKkPHBhi1cB_ZrRbhE9SM486MCNmNRdnOsQcw,11264
+lokit/exporters/po.py,sha256=Leg6mcJITba6HQGXLuCQUVKfVX702r8ROxtpGsXWocs,5003
+lokit/exporters/tmx.cp313-win_amd64.pyd,sha256=QZ2Llxfob5sva2iqh1WL3JH4WrjCIpE1THg3CBI7Ydg,11264
+lokit/exporters/tmx.py,sha256=42qmdn3YKf4y3a7o4gZVrAt52hqmx8IWOSR6qv4N4_s,8531
+lokit/exporters/xliff.cp313-win_amd64.pyd,sha256=_0y9-dixpd2tSpFQFHIsufTTG4xuFD2uRDRxFFyak7s,11264
+lokit/exporters/xliff.py,sha256=kBvRZfQC7vn9QGdAwe2YdpY1yA-icQljGOZTb9zKicc,5404
+lokit/exporters/xlsx.cp313-win_amd64.pyd,sha256=fEI9z3kR5vdz3y_0xtjerRk5JjJscDH9Gia5cxwwuCc,11264
+lokit/exporters/xlsx.py,sha256=gGrrdf6zKUK9zsqPSKVRzUeX2U8m2Ej2ZrlCaQjnHrM,1059
+lokit/io/__init__.cp313-win_amd64.pyd,sha256=I3XtbOpZoKBDl2-z8waq4_U7E7YlOHTDrvrVfTleLmo,11264
+lokit/io/__init__.py,sha256=-hkm8PkCM216sFF7X8f8F5wxSheGQ9iIaVn5K553ZYU,124
+lokit/io/json.cp313-win_amd64.pyd,sha256=_36bTbQw7USrMSe-xD109fakajmsVUtxqQ_VS1nA6rg,11264
+lokit/io/json.py,sha256=kQKIB04lZ-b4CfL6xgYQ9hGfR6F5DIMiQ5DOSVkBMYA,6484
+lokit/parsers/__init__.cp313-win_amd64.pyd,sha256=Rc2nXEBsGndEIkf62ch5RNRgYu5qFKduBakdh-aDM1M,11264
+lokit/parsers/__init__.py,sha256=frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN_XKdLCPjaYaY,2
+lokit/parsers/csv/__init__.cp313-win_amd64.pyd,sha256=o3r6C22KQK6DVcCwFkDgFEeIwlpaxOIAo91WZPEpQMQ,11264
+lokit/parsers/csv/__init__.py,sha256=frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN_XKdLCPjaYaY,2
+lokit/parsers/csv/extraction.cp313-win_amd64.pyd,sha256=mkaSGIGqcPO1_cyniHKGM3WzWnecwcm4nmPM4Q8fjV8,11776
+lokit/parsers/csv/extraction.py,sha256=k5M9JZde9I8gkviDx9A2iHScbzw_05onPJz1YkkS8bM,5611
+lokit/parsers/html/__init__.cp313-win_amd64.pyd,sha256=I8hiKGr-JitsTxst3q3GBSS6IrN3_RhLAdXlA84emao,11264
+lokit/parsers/html/__init__.py,sha256=dqUcjn2YJzMlwuQJaB5qXLu1PJpv_uDNEOcIK2jKxI0,88
+lokit/parsers/html/extraction.cp313-win_amd64.pyd,sha256=ZMFcLpWRswmTidUcdLptlBk5AcbETy53HrZcBjAuPac,11776
+lokit/parsers/html/extraction.py,sha256=G9UQLqJP8i9iKzjc890TFgj3mccU4oi-zVUjFkbntqI,12671
+lokit/parsers/idml/__init__.cp313-win_amd64.pyd,sha256=QGEQXf1DGEVvxkTIv1vvk7kxdgfRIH87iOcy7w0ddl8,11264
+lokit/parsers/idml/__init__.py,sha256=RSQMEfdXJdmSH3MhtiwyrcojOzxxftvK1AgnhvFgB0Y,88
+lokit/parsers/idml/extraction.cp313-win_amd64.pyd,sha256=HsFFaZtPW_yFfBFJKOqYvxnKLVEzGtYWdcMww-Z2-qY,11776
+lokit/parsers/idml/extraction.py,sha256=SR9dKkg9p0rJe35ySl8s7IBFTMua5IiJwm8UO2xvpAc,8764
+lokit/parsers/json_i18n/__init__.cp313-win_amd64.pyd,sha256=xieZQL1WTl-KuPWWcehmTurbB5YM8w7LU2aOkCG8wmI,11264
+lokit/parsers/json_i18n/__init__.py,sha256=FVMKVux9FZGZ6SXquULU42GM71K5cFlpHoP4Dte2LV0,101
+lokit/parsers/json_i18n/extraction.cp313-win_amd64.pyd,sha256=FE3p_lunmr2c63SmeNrWHkEUl09NJ0jkG1faQT_wGVY,11776
+lokit/parsers/json_i18n/extraction.py,sha256=bgAZJmCwb6tmiWwW5TxhuvWFQUUYjhWxE1NkEQTXtg8,5663
+lokit/parsers/po/__init__.cp313-win_amd64.pyd,sha256=skqlToms0JbGjHAqsWs0OAOfuuNpz1y9T1uBA17PfLY,11264
+lokit/parsers/po/__init__.py,sha256=M5hdXd5VgGLDSJt7s3BT2IYZmGVm0zbB_BnHLGcnPtQ,82
+lokit/parsers/po/extraction.cp313-win_amd64.pyd,sha256=YT6lZhPPMJiItDwaHMHHDA7ueDOidtZT5tIGsRiLaIQ,11776
+lokit/parsers/po/extraction.py,sha256=gt43zg1kdjE_x8PDEbHs1Ksimfxhnmi7I6pho5dpHVQ,7787
+lokit/parsers/tmx/__init__.cp313-win_amd64.pyd,sha256=pZ_qf1G341ZVuAsarrlTLTEha8-G0k5AuUReCtdY_oc,11264
+lokit/parsers/tmx/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+lokit/parsers/tmx/base.cp313-win_amd64.pyd,sha256=1UJ5aKTLMZu5oKAGe5fdqmJTGm9Zd2Wa3FltzXWGVKw,11264
+lokit/parsers/tmx/base.py,sha256=XYLX9hnj_DSrOXZRsGFzR1UD6NqsYtdxx6dTjW6uNNc,5443
+lokit/parsers/tmx/extraction.cp313-win_amd64.pyd,sha256=weLCZ0apOb3E-QTYKCRjLYR6fH4D1chCWljXyjLLqtY,11776
+lokit/parsers/tmx/extraction.py,sha256=l-9Begd82OeN9xRB-KewVpY4fCWjlyISZl8kIEGdRws,5916
+lokit/parsers/tmx/header.cp313-win_amd64.pyd,sha256=ZlpuPEWxPPGCmKp8EfvLBqPTVZAFDGqjRlXB2p2ttgA,11264
+lokit/parsers/tmx/header.py,sha256=8vwBGDHDZSkaAz4qIWrW8KErO8nmmyt7NEtIKXqmA-s,2261
+lokit/parsers/tmx/helpers.cp313-win_amd64.pyd,sha256=fKbN00JbGomth6EMj29KCsp_K3PtRa3t-EzrDovMBRs,11776
+lokit/parsers/tmx/helpers.py,sha256=UZRd0a99-mYOwYEMVkRVuFnrp1yC6vmTHuJE95d1HnI,241
+lokit/parsers/tmx/models.cp313-win_amd64.pyd,sha256=WrVuv1dFwXtoZE1Emni1EU2jPyMGN0cAOVvnsQeZJyQ,11264
+lokit/parsers/tmx/models.py,sha256=h28n0JQtg-QwG9UswZVn3qwyFK1BKmnYUlk8V9DQjx8,175
+lokit/parsers/tmx/props.cp313-win_amd64.pyd,sha256=mTi4yRp7x1dyOmElqz6ZcUBtlODnTF9sj3X-xGNV9_E,11264
+lokit/parsers/tmx/props.py,sha256=OOVzZ0-A9uC8kH7lHuedDMNdiXcLJIdACu1Wa-lZqY4,7666
+lokit/parsers/tmx/tags.cp313-win_amd64.pyd,sha256=ntsGLbin_QOqgFMRNncBl0cjB1OuyXAeXrH74qzSvjk,11264
+lokit/parsers/tmx/tags.py,sha256=cRen1mRokpL-qlb8A2zyNPm4f-nbNfHq6IuUfCB3IwA,1990
+lokit/parsers/tmx/xml_utils.cp313-win_amd64.pyd,sha256=Mu_iMB2wWf3ikLzc0QBHZnV1QZr7K_ehy1PL2aUWIqc,11776
+lokit/parsers/tmx/xml_utils.py,sha256=iCpC8-giF_tzn6XzrbEgVue0cDxGWu2o5A-0V-_cw0g,1142
+lokit/parsers/xliff/__init__.cp313-win_amd64.pyd,sha256=_NuzigRmtBMYAkbW66MMdw6RSHfHMZU5pAVYxOagPKI,11264
+lokit/parsers/xliff/__init__.py,sha256=IhYgtPIzVImD9PtPUR36TqD99riPoU0qFmy7ES08-lM,91
+lokit/parsers/xliff/extraction.cp313-win_amd64.pyd,sha256=DhPT1EOlksFGmzOr6fbU9ZkZqOD78cVbTzmGDNqGEG4,11776
+lokit/parsers/xliff/extraction.py,sha256=bo0XHuVlq8GZSxj6On87ludih6uEhex4lzyZkSplpmk,8493
+lokit/parsers/xliff/tags.cp313-win_amd64.pyd,sha256=opWzcy2dlnF1q9IJw0Rmdcc6_GOZ_Is_dTCkophc6FY,11264
+lokit/parsers/xliff/tags.py,sha256=gKkcqcjOcLi3OiGOVOGyQvYKDORAkcW-u5cvhyxNkxo,4153
+lokit/parsers/xlsx/__init__.cp313-win_amd64.pyd,sha256=DxLLd8Wh_x5CKA888hXxXMZtg8AjK9tJ4CFUcurmfuo,11264
+lokit/parsers/xlsx/__init__.py,sha256=frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN_XKdLCPjaYaY,2
+lokit/parsers/xlsx/extraction.cp313-win_amd64.pyd,sha256=3Yr0e3LDXwCx4VJPQAX_AfFfHjkVnPcBUVOirNaHYNA,11776
+lokit/parsers/xlsx/extraction.py,sha256=rZZJmyu-vmpqfL5DDQJeIiRibh-Ga-L9SGpbk8Uj1tk,6582
+lokit_python-0.1.0.dist-info/METADATA,sha256=HkbOnQeGAd8SQaM0snxdTAFw4AIQ4fOCIKSm-upA1KM,7541
+lokit_python-0.1.0.dist-info/WHEEL,sha256=x5Wpw_tLx5PQKiWdxpqvs0e7Sg-SO0mTWdEADYDGPGA,101
+lokit_python-0.1.0.dist-info/top_level.txt,sha256=4FQaCRBDbt-jLGbmkq51WX9Gesp7Y_RXcWontgzRQrE,34
+lokit_python-0.1.0.dist-info/RECORD,,

lokit_python-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: false
+Tag: cp313-cp313-win_amd64
+

lokit_python-0.1.0.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+821d8b73c2a02cb7980f__mypyc
+lokit