polystring 0.1.0__tar.gz

polystring-0.1.0/.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

polystring-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,78 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+    tags: ["v*"]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    name: Test (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Run tests
+        run: pytest --no-cov -q
+
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+
+      - name: Install ruff
+        run: pip install ruff>=0.1
+
+      - name: Run ruff
+        run: ruff check src/ tests/
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: [test, lint]
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+    environment: pypi
+    permissions:
+      id-token: write
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+
+      - name: Install hatch
+        run: pip install hatch
+
+      - name: Build
+        run: hatch build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

polystring-0.1.0/.gitignore

@@ -0,0 +1,46 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.so
+*.egg
+*.egg-info/
+build/
+.eggs/
+
+# Virtual environments
+.venv/
+venv/
+env/
+.env
+
+# Testing & coverage
+.pytest_cache/
+.coverage
+.coverage.*
+coverage.xml
+htmlcov/
+
+# Type checking
+.mypy_cache/
+.dmypy.json
+
+# Ruff
+.ruff_cache/
+
+# Build / packaging
+dist/
+*.whl
+*.tar.gz
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+desktop.ini
+Thumbs.db

polystring-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,18 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.4.4
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml
+      - id: check-added-large-files
+        args: [--maxkb=500]
+      - id: debug-statements

polystring-0.1.0/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+## [0.1.0] - 2026-06-13
+
+Initial release.
+
+### Detection capabilities
+- Span-level language detection: each span of a mixed-language input is labelled independently, with character offsets into the original string
+- 75 languages via [lingua](https://github.com/pemistahl/lingua-py), plus dedicated lexicons for Roman Urdu (`ur-Latn`), Tagalog (`tl`), Swahili (`sw`), French (`fr`), Spanish (`es`), Portuguese (`pt`), Italian (`it`), German (`de`), and Turkish (`tr`)
+- Character n-gram models for Roman Urdu, Tagalog, and Swahili, enabling detection of content words not in the lexicon. Models are pre-bundled and loaded at import time
+- Non-Latin scripts identified directly from Unicode block ranges (Arabic, Devanagari, CJK, Cyrillic, Thai, Hebrew, Georgian, Korean, Japanese, Bengali, and more) without calling the language model
+
+### Robustness features
+- URLs, @mentions, #hashtags, emoji, and numbers are extracted before detection and reinserted in the output, so they never corrupt the language signal
+- Mid-sentence capitalised tokens inconsistent with surrounding context are tagged `ne` (named entity / proper noun) rather than assigned a spurious language
+- Near-identical language pairs (Spanish/Portuguese, Norwegian/Danish/Swedish, Indonesian/Malay, Croatian/Serbian) are returned as `und` with `ambiguous_candidates` populated, rather than a confident wrong answer
+- Context correction pass resolves undetermined tokens by majority vote over a +/- 3 token window, and absorbs single-token language islands into surrounding context
+
+### API
+- `analyze(text, *, languages, granularity, min_confidence, low_accuracy_mode, normalize, custom_lexicon)`
+- `PolyStringResult`: `.spans`, `.languages`, `.dominant_language`, `.is_mixed`, `.confidence`, `.to_dict()`, `.to_dataframe()`, `.highlight()`, `.linguistic_spans()`
+- `Span`: `text`, `language`, `token_type`, `confidence`, `start`, `end`, `is_foreign`, `ambiguous_candidates`
+- `supported_languages()` returning the full list of detectable ISO codes
+- `PolyStringError`, `UnsupportedLanguageError`, `InputTooShortError`
+- Optional pandas integration: `pip install polystring[pandas]`

polystring-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Saad Khan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

polystring-0.1.0/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.4
+Name: polystring
+Version: 0.1.0
+Summary: Span-level language detection for mixed-language text
+Project-URL: Homepage, https://github.com/saadlohani/polystring
+Project-URL: Documentation, https://github.com/saadlohani/polystring#readme
+Project-URL: Bug Tracker, https://github.com/saadlohani/polystring/issues
+Project-URL: Changelog, https://github.com/saadlohani/polystring/blob/main/CHANGELOG.md
+Author-email: Saad Khan <saadlohani@yahoo.com>
+License: MIT License
+        
+        Copyright (c) 2026 Saad Khan
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: code-switching,language-detection,mixed-language,multilingual,nlp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: lingua-language-detector>=2.0
+Requires-Dist: regex>=2023.0
+Requires-Dist: typing-extensions>=4.0; python_version < '3.11'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.0; extra == 'dev'
+Requires-Dist: pre-commit; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.1; extra == 'dev'
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == 'pandas'
+Description-Content-Type: text/markdown
+
+# polystring
+
+**Span-level language detection for mixed-language text.**
+
+Most language detection libraries return a single label for the whole string. polystring returns a labelled span for _every part_ of the sentence, with character offsets, confidence scores, and special-token extraction baked in.
+
+```python
+from polystring import analyze
+
+result = analyze("je suis tellement tired, this week has been rough")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [fr] 'je suis tellement tired'
+# [en] 'this week has been rough'
+```
+
+## Why span-level detection matters
+
+| Tool           | What it returns for `"hola I love this city, en serio"`       |
+| -------------- | ------------------------------------------------------------- |
+| `langdetect`   | `"es"` (labels the whole string)                              |
+| `lingua`       | `"es"` (labels the whole string)                              |
+| `langid`       | `"es"` (labels the whole string)                              |
+| **polystring** | `[es] "hola"` · `[en] "I love this city"` · `[es] "en serio"` |
+
+Code-switching (mixing languages within a single sentence) is normal on social media, in diaspora communities, in customer support chats, and in any multilingual context. A single label for the whole input misses the structure entirely. polystring is built specifically for this problem.
+
+## Installation
+
+```bash
+pip install polystring
+```
+
+Optional extras:
+
+```bash
+pip install polystring[pandas]     # enables result.to_dataframe()
+```
+
+---
+
+## Examples
+
+### Spanish / English (Spanglish)
+
+```python
+result = analyze("no puedo creer how good this restaurant is, en serio")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [es] 'no puedo creer'
+# [en] 'how good this restaurant is'
+# [es] 'en serio'
+```
+
+### French / English
+
+```python
+result = analyze("je suis tellement tired lately, I need des vacances")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [fr] 'je suis tellement tired lately'
+# [en] 'I need'
+# [fr] 'des vacances'
+```
+
+### Mixed with non-Latin scripts
+
+```python
+result = analyze("this is great هذا رائع جداً I am very impressed")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [en] 'this is great'
+# [ar] 'هذا رائع جداً'
+# [en] 'I am very impressed'
+```
+
+## Working with results
+
+```python
+result = analyze("je suis tellement tired, this week has been rough")
+
+result.dominant_language   # 'fr'
+result.is_mixed            # True
+result.languages           # {'fr', 'en'}
+result.confidence          # 0.87  (mean confidence across linguistic spans)
+```
+
+### Serialise to dict
+
+```python
+result.to_dict()
+# {
+#   'text': 'je suis tellement tired, this week has been rough',
+#   'spans': [{'text': 'je suis tellement tired', 'language': 'fr', ...}, ...],
+#   'dominant_language': 'fr',
+#   'is_mixed': True,
+#   'confidence': 0.87,
+#   ...
+# }
+```
+
+### Serialise to DataFrame
+
+```python
+# pip install polystring[pandas]
+df = result.to_dataframe()
+#                      text language token_type  confidence  start  end  is_foreign
+# 0  je suis tellement tired       fr       text        0.91      0   23       False
+# 1  this week has been rough      en       text        0.84     25   49        True
+```
+
+### ANSI-coloured terminal output
+
+```python
+print(result.highlight())
+# [fr]je suis tellement tired [en]this week has been rough
+# (each language rendered in a distinct colour)
+```
+
+### Filter to linguistic spans only
+
+```python
+result.linguistic_spans()
+# Returns spans with token_type == "text" only (no URLs, emoji, mentions, etc.)
+```
+
+## Span fields
+
+| Field                  | Type        | Description                                                                                   |
+| ---------------------- | ----------- | --------------------------------------------------------------------------------------------- |
+| `text`                 | `str`       | Text as it appears in the input                                                               |
+| `language`             | `str`       | ISO 639-1 code. `"ur-Latn"` for Roman Urdu, `"und"` for undetermined, `"ne"` for proper nouns |
+| `token_type`           | `str`       | `"text"`, `"url"`, `"mention"`, `"hashtag"`, `"emoji"`, `"num"`, or `"ne"`                    |
+| `confidence`           | `float`     | 0.0 to 1.0. Non-text tokens are always 0.0                                                    |
+| `start` / `end`        | `int`       | Character offsets into the original string                                                    |
+| `is_foreign`           | `bool`      | `True` if this span is not the dominant language                                              |
+| `ambiguous_candidates` | `list[str]` | Populated when `language == "und"` due to a near-identical pair (e.g. `["es", "pt"]`)         |
+
+## Language coverage
+
+polystring detects **75 languages** via [lingua](https://github.com/pemistahl/lingua-py). Non-Latin scripts (Arabic, Devanagari, CJK, Cyrillic, Thai, Hebrew, Korean, and more) are identified directly from Unicode ranges, no model call needed.
+
+Nine languages have dedicated lexicons on top of the model, which significantly improves accuracy on short spans and code-switched text: Roman Urdu / Hinglish, Spanish, Portuguese, Italian, German, Turkish, Tagalog, Swahili
+
+```python
+import polystring
+print(polystring.supported_languages())  # full list of 75 ISO 639-1 codes
+```
+
+## Options
+
+```python
+analyze(
+    text,
+    languages=["es", "en"],      # restrict to known language set (faster, fewer false positives)
+    granularity="token",          # "span" (default) or "token" to get per-word data
+    min_confidence=0.70,          # tokens below this threshold become "und"
+    low_accuracy_mode=False,      # lexicon + script detection only, no model (very fast)
+    normalize=True,               # NFC normalisation
+    custom_lexicon={"sw": ["mambo", "vipi"]},   # inject domain-specific words
+)
+```
+
+### `granularity="token"` gives per-word data
+
+```python
+result = analyze("bonjour how are you doing", granularity="token")
+
+for tok in result.tokens:
+    print(f"[{tok.language}] {tok.text!r}  ({tok.confidence:.2f})")
+
+# [fr] 'bonjour'  (0.92)
+# [en] 'how'      (0.83)
+# [en] 'are'      (0.81)
+# [en] 'you'      (0.85)
+# [en] 'doing'    (0.88)
+```
+
+## Contributing
+
+```bash
+git clone https://github.com/saadlohani/polystring
+cd polystring
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT

polystring-0.1.0/README.md

@@ -0,0 +1,198 @@
+# polystring
+
+**Span-level language detection for mixed-language text.**
+
+Most language detection libraries return a single label for the whole string. polystring returns a labelled span for _every part_ of the sentence, with character offsets, confidence scores, and special-token extraction baked in.
+
+```python
+from polystring import analyze
+
+result = analyze("je suis tellement tired, this week has been rough")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [fr] 'je suis tellement tired'
+# [en] 'this week has been rough'
+```
+
+## Why span-level detection matters
+
+| Tool           | What it returns for `"hola I love this city, en serio"`       |
+| -------------- | ------------------------------------------------------------- |
+| `langdetect`   | `"es"` (labels the whole string)                              |
+| `lingua`       | `"es"` (labels the whole string)                              |
+| `langid`       | `"es"` (labels the whole string)                              |
+| **polystring** | `[es] "hola"` · `[en] "I love this city"` · `[es] "en serio"` |
+
+Code-switching (mixing languages within a single sentence) is normal on social media, in diaspora communities, in customer support chats, and in any multilingual context. A single label for the whole input misses the structure entirely. polystring is built specifically for this problem.
+
+## Installation
+
+```bash
+pip install polystring
+```
+
+Optional extras:
+
+```bash
+pip install polystring[pandas]     # enables result.to_dataframe()
+```
+
+---
+
+## Examples
+
+### Spanish / English (Spanglish)
+
+```python
+result = analyze("no puedo creer how good this restaurant is, en serio")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [es] 'no puedo creer'
+# [en] 'how good this restaurant is'
+# [es] 'en serio'
+```
+
+### French / English
+
+```python
+result = analyze("je suis tellement tired lately, I need des vacances")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [fr] 'je suis tellement tired lately'
+# [en] 'I need'
+# [fr] 'des vacances'
+```
+
+### Mixed with non-Latin scripts
+
+```python
+result = analyze("this is great هذا رائع جداً I am very impressed")
+
+for span in result.spans:
+    print(f"[{span.language}] {span.text!r}")
+
+# [en] 'this is great'
+# [ar] 'هذا رائع جداً'
+# [en] 'I am very impressed'
+```
+
+## Working with results
+
+```python
+result = analyze("je suis tellement tired, this week has been rough")
+
+result.dominant_language   # 'fr'
+result.is_mixed            # True
+result.languages           # {'fr', 'en'}
+result.confidence          # 0.87  (mean confidence across linguistic spans)
+```
+
+### Serialise to dict
+
+```python
+result.to_dict()
+# {
+#   'text': 'je suis tellement tired, this week has been rough',
+#   'spans': [{'text': 'je suis tellement tired', 'language': 'fr', ...}, ...],
+#   'dominant_language': 'fr',
+#   'is_mixed': True,
+#   'confidence': 0.87,
+#   ...
+# }
+```
+
+### Serialise to DataFrame
+
+```python
+# pip install polystring[pandas]
+df = result.to_dataframe()
+#                      text language token_type  confidence  start  end  is_foreign
+# 0  je suis tellement tired       fr       text        0.91      0   23       False
+# 1  this week has been rough      en       text        0.84     25   49        True
+```
+
+### ANSI-coloured terminal output
+
+```python
+print(result.highlight())
+# [fr]je suis tellement tired [en]this week has been rough
+# (each language rendered in a distinct colour)
+```
+
+### Filter to linguistic spans only
+
+```python
+result.linguistic_spans()
+# Returns spans with token_type == "text" only (no URLs, emoji, mentions, etc.)
+```
+
+## Span fields
+
+| Field                  | Type        | Description                                                                                   |
+| ---------------------- | ----------- | --------------------------------------------------------------------------------------------- |
+| `text`                 | `str`       | Text as it appears in the input                                                               |
+| `language`             | `str`       | ISO 639-1 code. `"ur-Latn"` for Roman Urdu, `"und"` for undetermined, `"ne"` for proper nouns |
+| `token_type`           | `str`       | `"text"`, `"url"`, `"mention"`, `"hashtag"`, `"emoji"`, `"num"`, or `"ne"`                    |
+| `confidence`           | `float`     | 0.0 to 1.0. Non-text tokens are always 0.0                                                    |
+| `start` / `end`        | `int`       | Character offsets into the original string                                                    |
+| `is_foreign`           | `bool`      | `True` if this span is not the dominant language                                              |
+| `ambiguous_candidates` | `list[str]` | Populated when `language == "und"` due to a near-identical pair (e.g. `["es", "pt"]`)         |
+
+## Language coverage
+
+polystring detects **75 languages** via [lingua](https://github.com/pemistahl/lingua-py). Non-Latin scripts (Arabic, Devanagari, CJK, Cyrillic, Thai, Hebrew, Korean, and more) are identified directly from Unicode ranges, no model call needed.
+
+Nine languages have dedicated lexicons on top of the model, which significantly improves accuracy on short spans and code-switched text: Roman Urdu / Hinglish, Spanish, Portuguese, Italian, German, Turkish, Tagalog, Swahili
+
+```python
+import polystring
+print(polystring.supported_languages())  # full list of 75 ISO 639-1 codes
+```
+
+## Options
+
+```python
+analyze(
+    text,
+    languages=["es", "en"],      # restrict to known language set (faster, fewer false positives)
+    granularity="token",          # "span" (default) or "token" to get per-word data
+    min_confidence=0.70,          # tokens below this threshold become "und"
+    low_accuracy_mode=False,      # lexicon + script detection only, no model (very fast)
+    normalize=True,               # NFC normalisation
+    custom_lexicon={"sw": ["mambo", "vipi"]},   # inject domain-specific words
+)
+```
+
+### `granularity="token"` gives per-word data
+
+```python
+result = analyze("bonjour how are you doing", granularity="token")
+
+for tok in result.tokens:
+    print(f"[{tok.language}] {tok.text!r}  ({tok.confidence:.2f})")
+
+# [fr] 'bonjour'  (0.92)
+# [en] 'how'      (0.83)
+# [en] 'are'      (0.81)
+# [en] 'you'      (0.85)
+# [en] 'doing'    (0.88)
+```
+
+## Contributing
+
+```bash
+git clone https://github.com/saadlohani/polystring
+cd polystring
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT