unidash 0.2.0__tar.gz

unidash-0.2.0/.gitattributes

@@ -0,0 +1 @@
+* text=auto eol=lf

unidash-0.2.0/.github/FUNDING.yml

@@ -0,0 +1 @@
+buy_me_a_coffee: jlianesglrs

unidash-0.2.0/.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Third-party data, fetched by scripts/fetch_vectors.py (see its MANIFEST.txt)
+tests/vectors/unicode/*
+!tests/vectors/unicode/MANIFEST.txt

unidash-0.2.0/CHANGELOG.md

@@ -0,0 +1,42 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0]
+
+### Added
+
+- `fold_dashes(text, form="ascii")` with three forms: `ascii` (every dash to one `-`, wavy
+  dashes to `~`), `tex` (en dash `--`, em dash `---`) and `typographic` (real dashes kept,
+  small, vertical and fullwidth forms replaced by their standard counterparts).
+- Every code point with the Unicode 18.0 Dash property is classified: general dashes fold by
+  default, script hyphens only with `script_hyphens=True`, and the superscript and subscript
+  minus are never folded.
+- Soft hyphens are removed unless `keep_soft_hyphens=True`.
+- `find_dashes` and `DashHit` report line, column and name of each character a fold would
+  change; `is_dash`, `DASHES`, `dash_table` and the streaming `iter_fold_dashes`.
+- Command line tool `unidash` (also `python -m unidash`) with `--check` for pre-commit use.
+- Tests against the Unicode data files and the UTS #39 confusables, fetched by
+  `scripts/fetch_vectors.py` and pinned by size and SHA-256, plus an installed-wheel test.
+
+### Changed
+
+- `to_ascii_punct` now folds every general dash (fullwidth, small, vertical, two-em and
+  three-em dashes, wave dashes to `~`) and removes soft hyphens.
+- Type errors name the type that was passed.
+- `to_ascii_punct` folds fullwidth punctuation (U+FF01 to U+FF5E, not letters or digits) to
+  ASCII, so the Japanese wave dash from CP932 text (U+FF5E) becomes `~`.
+
+## [0.1.0]
+
+- First release.
+- `to_ascii_punct(text)` folds Unicode punctuation to ASCII: curly quotes to `'` and `"`,
+  en and em dashes and the minus sign to `-`, the ellipsis to `...`, non-breaking and
+  other spaces to a normal space, and bullets to `*`.
+- Letters, including accented ones, pass through untouched, unlike `unidecode` and
+  `anyascii`, which transliterate every character.
+- Zero runtime dependencies, with the mapping vendored in.

unidash-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,19 @@
+# Contributing
+
+Thanks for taking the time. This project keeps a small, predictable shape.
+
+## Setup
+
+```
+pip install -r requirements-dev.txt
+```
+
+## Before you open a pull request
+
+- Keep public API in `src/unidash/__init__.py`; internals live in `_`-prefixed modules.
+- Add a test for the behaviour you change. Tests inject time, disk and network; they
+  never touch the real thing.
+- Run `scripts/check.ps1` (format, lint, types, tests) and make sure it passes.
+- One logical change per commit, imperative subject line.
+
+Open an issue first if the change is large or shifts the public API.

unidash-0.2.0/IMPROVEMENTS-round.md

@@ -0,0 +1,125 @@
+# unidash improvement round
+
+Progress:
+- [x] Brainstorm ranked improvements (this file)
+- [x] Dash classification from the Unicode data: general dashes, script hyphens, kept marks
+- [x] `fold_dashes` with the `ascii`, `tex` and `typographic` forms and soft hyphen control
+- [x] `to_ascii_punct` covers every general dash and drops soft hyphens
+- [x] `find_dashes`, `is_dash`, `DASHES`, `dash_table`, streaming `iter_fold_dashes`
+- [x] Command line tool (`unidash`, `python -m unidash`) with `--check`
+- [x] Contracts: UCD and UTS #39 data, interpreter scan, typing, README, wheel e2e
+- [x] Mutation run (guards removed one at a time, each must fail the suite)
+- [x] Docs: README, CHANGELOG, docstrings, packaging metadata
+- [x] Gates: ruff format, ruff check, mypy --strict, pytest
+- [x] Independent review findings fixed with regression tests
+- [x] Pull request opened (#2), guard clean
+
+## Starting point
+
+`0.1.0` (not on PyPI) ships one function, `to_ascii_punct`, which folds a hand-picked set of
+quotes, seven dashes, bullets, spaces and the ellipsis. Read as a user who cleans text pasted
+from word processors, PDFs, CJK input methods and web pages:
+
+- Only U+2010..U+2015 and U+2212 fold. The fullwidth hyphen-minus from a Japanese keyboard,
+  the small and vertical presentation forms, the two-em and three-em dashes and the wave dash
+  pass through, so "ASCII punctuation" output still holds non-ASCII dashes.
+- The soft hyphen (U+00AD) is kept. It is invisible, breaks search and diffing, and is the most
+  common stray character in text copied from PDFs and hyphenating renderers.
+- There is no choice of target. Everything becomes one `-`, which loses the en and em dash
+  distinction that Markdown and TeX users keep as `--` and `---`, and there is no way to only
+  tidy lookalike forms while keeping real typographic dashes.
+- Nothing says which characters were chosen or why; nothing is checked against the Unicode
+  data, so the next Unicode version silently adds dashes the table never heard of.
+- No way to find dashes (a lint or a pre-commit check), no streaming, no command line tool.
+- Packaging: README example is not executed, no Python classifiers, no wheel test.
+
+## Sources (fetched 2026-10-13)
+
+- UnicodeData.txt 18.0.0, https://www.unicode.org/Public/18.0.0/ucd/UnicodeData.txt: 27 code
+  points have General_Category=Pd (U+002D, U+058A, U+05BE, U+1400, U+1806, U+2010..U+2015,
+  U+2E17, U+2E1A, U+2E3A, U+2E3B, U+2E40, U+2E5D, U+301C, U+3030, U+30A0, U+FE31, U+FE32,
+  U+FE58, U+FE63, U+FF0D, U+10D6E, U+10EAD).
+- PropList.txt 18.0.0, https://www.unicode.org/Public/18.0.0/ucd/PropList.txt: the Dash
+  property is every Pd code point plus U+2053 SWUNG DASH (Po) and three math signs, U+207B,
+  U+208B and U+2212 (Sm).
+- confusables.txt 18.0.0 (UTS #39), https://www.unicode.org/Public/18.0.0/security/confusables.txt:
+  maps to U+002D the hyphen, non-breaking hyphen, figure dash, en dash, minus sign, small em
+  dash and small hyphen-minus, and also non-dashes (letters in Coptic, Buhid and Myanmar, a
+  modifier letter, the Arabic full stop, the hyphen bullet, an emoji, a Roman weight sign).
+  The em dash, horizontal bar and fullwidth hyphen-minus map to U+30FC instead; the wave dash
+  and swung dash to `~`; the double hyphens to `=`.
+- The three files are third-party and not committed: `scripts/fetch_vectors.py` fetches them
+  into a gitignored folder checked against a tracked MANIFEST; the tests that read them skip
+  when the folder is absent. The running interpreter's `unicodedata` gives a second, always-on
+  check for General_Category and NFKC.
+
+## Ranked improvements
+
+1. **Every dash is classified, on purpose.** Three tables built from code points, each with a
+   reason: general dashes (fold by default), script hyphens (Armenian, Hebrew maqaf, Canadian
+   syllabics, Mongolian, katakana double hyphen, Garay, Yezidi: they belong to a script, like
+   letters, so they fold only on request) and kept marks (superscript and subscript minus,
+   which sit next to superscript digits that stay). A contract checks that every Pd and Dash
+   code point in the fetched data lands in exactly one table, and every UTS #39 confusable of
+   `-` is either folded or excluded for a stated reason (letter, emoji, bullet, sentence stop).
+2. **`fold_dashes(text, form=...)`.** `ascii` (default): every general dash becomes `-`, the
+   wave and swung dashes become `~`. `tex`: en dash `--`, em dash and horizontal bar `---`,
+   two-em and three-em dashes six and nine hyphens, the rest as `ascii`. `typographic`: real
+   dashes stay; only compatibility forms (small, vertical, fullwidth) become their NFKC
+   counterparts. `script_hyphens=True` folds the script hyphens too.
+3. **Soft and non-breaking hyphens.** The soft hyphen is dropped by default (it is a line
+   break hint, not a character), `keep_soft_hyphens=True` keeps it. The non-breaking hyphen
+   folds to `-` in the ASCII forms and stays in `typographic`, where it carries meaning.
+4. **Numbers and ranges preserved.** A minus sign is always a single `-` in the ASCII forms, so
+   `-5` stays a negative number; the TeX form never touches characters that are already ASCII,
+   so `1-2` and `--` written by hand come out unchanged.
+5. **`to_ascii_punct` follows suit.** It folds every general dash and drops soft hyphens, so
+   its output is ASCII for every dash the Unicode data knows.
+6. **Find, test and build.** `find_dashes(text)` yields position, character and Unicode name
+   for each non-ASCII dash (for linting); `is_dash(char)`; `DASHES` is the Dash property as a
+   frozenset; `dash_table(...)` returns the `str.translate` table for callers with their own
+   pipeline.
+7. **Streaming.** `iter_fold_dashes(chunks, ...)` folds an iterable of strings lazily. The fold
+   is per code point, so a chunked result equals the whole-text result (property test).
+8. **Command line tool.** `unidash [FILE...]` writes folded text to stdout (`-` is stdin),
+   `--form`, `--script-hyphens`, `--keep-soft-hyphens`, `--punct` for the full punctuation fold,
+   and `--check` lists `file:line:col: U+2014 EM DASH` and exits 1 (pre-commit style). Exit 2
+   on unreadable or non UTF-8 input.
+9. **Contracts that bite.** UCD and UTS #39 data checks, an interpreter scan, chunking
+   properties, a mypy snippet over the typed API, executed README examples, an installed-wheel
+   e2e and a mutation run.
+10. **Docs and packaging.** README with runnable examples and the full classification, CHANGELOG
+    0.2.0, classifiers and keywords.
+
+Left out on purpose: turning ASCII hyphens into typographic dashes (the opposite direction,
+`smartypants` owns it), context rules such as "en dash between digits" (guesswork that breaks
+code and identifiers), in-place file rewriting (shell redirection and editors already do this
+safely), and folding letters that look like hyphens (the library never touches letters).
+
+## Verification
+
+- Gates: `ruff format --check .`, `ruff check .`, `mypy --strict src` (also `src tests`) and
+  `pytest` exit 0 on Windows with Python 3.13 (126 passed, 2 skipped: the wheel e2e and the
+  POSIX file name test). On the second machine with Python 3.14 and `UNIDASH_E2E=1` the suite
+  passes (127 passed, 1 skipped: the POSIX test): it builds the wheel and the sdist, checks
+  that no third-party data or tests ship in them, installs the wheel in a fresh venv,
+  drives the API from an empty folder, and runs the installed `unidash` script and
+  `python -m unidash`.
+- Unicode data: the classes equal the Dash property of PropList.txt 18.0.0 exactly, cover all
+  27 Pd code points of UnicodeData.txt 18.0.0, and the stored names match it. Every UTS #39
+  confusable of `-` either folds to `-` or is a letter or one of five listed non-dashes. The
+  running interpreter's Pd set and NFKC mappings are checked on every run, data or not.
+- Mutations: 53 mutants (classes, tables, option checks, TeX widths, compatibility targets,
+  soft and script hyphen switches, table copying, position counting, name lookup, streaming
+  type checks, the punctuation and fullwidth tables, LF, CR and CRLF line counting, the byte
+  order mark column, the broken pipe exit, and every command line flag, label, encoding and
+  exit code rule), applied one at a time with the bytecode cache disabled and purged before each
+  run: all killed. One first draft of an output-encoding mutant was equivalent and was
+  replaced. The baseline was green before and after.
+- Independent review: 9 findings, all addressed with regression tests: lone CR now ends a
+  line for positions; a leading byte order mark takes no column; a closed output pipe stops
+  quietly; undecodable POSIX file names are printed as given; the typographic minus claim and
+  the scope of minus-like characters are stated exactly in the README, with the reason double
+  hyphens become `-`; `to_ascii_punct` folds fullwidth punctuation, including the CP932 wave
+  dash; every README table cell is checked against the fold; the sdist is checked for data
+  files; `is_dash` raises `ValueError` for a string of the wrong length.

unidash-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Javier Lianes
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

unidash-0.2.0/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.5
+Name: unidash
+Version: 0.2.0
+Summary: Fold Unicode dashes and punctuation to plain ASCII while leaving letters untouched
+Project-URL: Homepage, https://lianes8studio.com
+Project-URL: Repository, https://github.com/JaviLianes8/unidash
+Project-URL: Issues, https://github.com/JaviLianes8/unidash/issues
+Project-URL: Changelog, https://github.com/JaviLianes8/unidash/blob/main/CHANGELOG.md
+Project-URL: LinkedIn, https://www.linkedin.com/in/jlianes/
+Project-URL: Funding, https://buymeacoffee.com/jlianesglrs
+Author-email: Javier Lianes <javier@lianes8studio.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ascii,dash,em-dash,en-dash,hyphen,minus,normalize,punctuation,quotes,soft-hyphen,text,unicode
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Text Processing :: Filters
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# unidash
+
+Fold Unicode dashes and punctuation to plain ASCII while leaving letters untouched.
+
+Text pasted from word processors, PDFs, CJK input methods and web pages is full of
+characters that look like `-` but are not: en and em dashes, the minus sign, the
+non-breaking hyphen, fullwidth and small forms, and the invisible soft hyphen. They break
+search, diffs, command lines and parsers. `unidash` knows every character with the Unicode
+Dash property, plus the soft hyphen, and folds them the way you choose.
+
+## Install
+
+```
+pip install unidash
+```
+
+Python 3.10 or newer, no dependencies.
+
+## Dashes
+
+`fold_dashes(text, form="ascii")` has three forms. `ascii` turns every dash into one `-`
+(wavy dashes into `~`). `tex` keeps the en and em dash apart the way TeX and Markdown do, as
+`--` and `---`. `typographic` keeps the real dashes and only replaces the small, vertical and
+fullwidth compatibility forms with their standard counterparts.
+
+```python
+from unidash import fold_dashes
+
+text = "Pages 10\u201320 \u2014 see \u22125\uff0d6"
+print(fold_dashes(text))
+print(fold_dashes(text, "tex"))
+print(ascii(fold_dashes("a\ufe58b\uff0dc", "typographic")))
+# Pages 10-20 - see -5-6
+# Pages 10--20 --- see -5-6
+# 'a\u2014b-c'
+```
+
+A minus sign is always a single `-` in the `ascii` and `tex` forms (`typographic` keeps
+U+2212), so negative numbers stay negative, and characters that are already ASCII are never
+touched: `1-2` and a hand-written `--` come out as they went in.
+
+## Soft hyphens, non-breaking hyphens and script hyphens
+
+The soft hyphen (U+00AD) is a hint for where a word may break, not a character, so it is
+removed unless you pass `keep_soft_hyphens=True`. The non-breaking hyphen (U+2011) becomes
+`-` in the ASCII forms and stays in `typographic`, where its promise matters.
+
+Hyphens that belong to one script (Armenian, the Hebrew maqaf, Canadian syllabics,
+Mongolian, the katakana double hyphen, Garay and Yezidi) are left alone, like that script's
+letters, unless you pass `script_hyphens=True`.
+
+```python
+from unidash import fold_dashes
+
+print(fold_dashes("co\u00adop\u00aderate"))
+print(ascii(fold_dashes("co\u00adop", keep_soft_hyphens=True)))
+print(ascii(fold_dashes("\u05e2\u05be\u05d1")))
+print(ascii(fold_dashes("\u05e2\u05be\u05d1", script_hyphens=True)))
+# cooperate
+# 'co\xadop'
+# '\u05e2\u05be\u05d1'
+# '\u05e2-\u05d1'
+```
+
+## Finding dashes
+
+`find_dashes` yields a `DashHit` (offset, line, column, character and Unicode name) for
+every character `fold_dashes` would change with the same options, which makes a lint or a
+pre-commit check a one-liner. Lines end at LF, CR or CRLF, columns count code points from 1,
+and a byte order mark at the start of the text takes no column. `is_dash(char)` tells whether a character has the Unicode Dash
+property and `DASHES` is that set. `dash_table(...)` returns the `str.translate` table, and
+`iter_fold_dashes(chunks, ...)` folds an iterable of strings lazily, for large inputs.
+
+```python
+from unidash import find_dashes, is_dash, iter_fold_dashes
+
+for hit in find_dashes("first line\nsecond\u2014line\u00ad"):
+    print(hit.line, hit.column, hit.codepoint, hit.name)
+print(is_dash("\u2212"), is_dash("\u00ad"))
+print("".join(iter_fold_dashes(["a\u2013", "b"])))
+# 2 7 U+2014 EM DASH
+# 2 12 U+00AD SOFT HYPHEN
+# True False
+# a-b
+```
+
+## All punctuation
+
+`to_ascii_punct(text)` folds the rest of the typographic punctuation too: curly quotes become
+`'` and `"`, the ellipsis becomes `...`, non-breaking and other Unicode spaces become a normal
+space, bullets become `*`, fullwidth punctuation from CJK input (U+FF01 to U+FF5E, letters and
+digits excepted) becomes its ASCII character, and dashes follow the `ascii` form. Accented
+letters pass through.
+
+```python
+from unidash import to_ascii_punct
+
+print(to_ascii_punct("\u201cCaf\u00e9\u201d\u2014it\u2019s 100\u00a0%\u2026"))
+# "Café"-it's 100 %...
+```
+
+## Command line
+
+`unidash FILE...` writes the folded text to stdout (`-` or no file reads stdin). It takes
+`--form`, `--script-hyphens`, `--keep-soft-hyphens`, and `--punct` for the full punctuation
+fold. `--check` lists what would change and exits 1, for pre-commit hooks. Input must be
+UTF-8; unreadable files exit 2.
+
+```
+$ unidash --check notes.md
+notes.md:2:6: U+2014 EM DASH
+notes.md:3:3: U+00AD SOFT HYPHEN
+$ unidash --form tex notes.md
+Draft
+Done --- ship it
+coop
+```
+
+## What counts as a dash
+
+The classes follow the Unicode 18.0 Dash property and General_Category=Pd, and the tests
+check them against the Unicode data files and the UTS #39 confusables list.
+
+| Code point | Name | `ascii` | `tex` | `typographic` |
+| --- | --- | --- | --- | --- |
+| U+2010 | HYPHEN | `-` | `-` | kept |
+| U+2011 | NON-BREAKING HYPHEN | `-` | `-` | kept |
+| U+2012 | FIGURE DASH | `-` | `-` | kept |
+| U+2013 | EN DASH | `-` | `--` | kept |
+| U+2014 | EM DASH | `-` | `---` | kept |
+| U+2015 | HORIZONTAL BAR | `-` | `---` | kept |
+| U+2212 | MINUS SIGN | `-` | `-` | kept |
+| U+2E17 | DOUBLE OBLIQUE HYPHEN | `-` | `-` | kept |
+| U+2E1A | HYPHEN WITH DIAERESIS | `-` | `-` | kept |
+| U+2E3A | TWO-EM DASH | `-` | `------` | kept |
+| U+2E3B | THREE-EM DASH | `-` | `---------` | kept |
+| U+2E40 | DOUBLE HYPHEN | `-` | `-` | kept |
+| U+2E5D | OBLIQUE HYPHEN | `-` | `-` | kept |
+| U+FE31 | PRESENTATION FORM FOR VERTICAL EM DASH | `-` | `---` | U+2014 |
+| U+FE32 | PRESENTATION FORM FOR VERTICAL EN DASH | `-` | `--` | U+2013 |
+| U+FE58 | SMALL EM DASH | `-` | `---` | U+2014 |
+| U+FE63 | SMALL HYPHEN-MINUS | `-` | `-` | `-` |
+| U+FF0D | FULLWIDTH HYPHEN-MINUS | `-` | `-` | `-` |
+| U+2053 | SWUNG DASH | `~` | `~` | kept |
+| U+301C | WAVE DASH | `~` | `~` | kept |
+| U+3030 | WAVY DASH | `~` | `~` | kept |
+| U+00AD | SOFT HYPHEN | removed | removed | removed |
+
+Script hyphens, folded only with `script_hyphens=True` (to `-`, or to U+2010 in
+`typographic`): U+058A ARMENIAN HYPHEN, U+05BE HEBREW PUNCTUATION MAQAF, U+1400 CANADIAN
+SYLLABICS HYPHEN, U+1806 MONGOLIAN TODO SOFT HYPHEN, U+30A0 KATAKANA-HIRAGANA DOUBLE HYPHEN,
+U+10D6E GARAY HYPHEN and U+10EAD YEZIDI HYPHENATION MARK.
+
+The double hyphens (U+2E40, and U+1400 and U+30A0 among the script hyphens) become `-`
+although UTS #39 lists them as confusable with `=`: they are hyphens in use, and `=` would
+change the meaning of the text.
+
+Never folded: U+002D HYPHEN-MINUS, and U+207B SUPERSCRIPT MINUS and U+208B SUBSCRIPT MINUS,
+which sit next to superscript and subscript digits that stay as they are. Characters that
+only look like a hyphen are not dashes and are left alone: letters in other scripts, the Arabic
+full stop, the hyphen bullet, U+02D7 MODIFIER LETTER MINUS SIGN (a tone mark), and minus-like
+symbols without the Dash property such as U+2052 COMMERCIAL MINUS SIGN, U+2796 HEAVY MINUS
+SIGN (an emoji), U+2E43 DASH WITH LEFT UPTURN and U+10D8F GARAY MINUS SIGN.
+
+## Why
+
+Tools like `unidecode` and `anyascii` transliterate every character, so `café` turns into
+`cafe`. That is the wrong tool when you only want to tidy punctuation and keep the words.
+`smartypants` goes the opposite way, turning ASCII into typographic dashes. `unidash` folds
+punctuation to ASCII, knows the whole Unicode dash family rather than the three most common
+members, and leaves letters alone. It is zero dependency and typed.
+
+## Development
+
+The Unicode data files are not part of the repository. `python scripts/fetch_vectors.py`
+downloads them into `tests/vectors/unicode/` and checks each against the size and SHA-256 in
+`MANIFEST.txt`; without them the data tests skip. The installed-wheel test runs with
+`UNIDASH_E2E=1`.
+
+## License
+
+MIT. See [LICENSE](LICENSE).
+
+## Links
+
+- Website: https://lianes8studio.com
+- GitHub: https://github.com/JaviLianes8
+- LinkedIn: https://www.linkedin.com/in/jlianes/

unidash-0.2.0/README.md

@@ -0,0 +1,188 @@
+# unidash
+
+Fold Unicode dashes and punctuation to plain ASCII while leaving letters untouched.
+
+Text pasted from word processors, PDFs, CJK input methods and web pages is full of
+characters that look like `-` but are not: en and em dashes, the minus sign, the
+non-breaking hyphen, fullwidth and small forms, and the invisible soft hyphen. They break
+search, diffs, command lines and parsers. `unidash` knows every character with the Unicode
+Dash property, plus the soft hyphen, and folds them the way you choose.
+
+## Install
+
+```
+pip install unidash
+```
+
+Python 3.10 or newer, no dependencies.
+
+## Dashes
+
+`fold_dashes(text, form="ascii")` has three forms. `ascii` turns every dash into one `-`
+(wavy dashes into `~`). `tex` keeps the en and em dash apart the way TeX and Markdown do, as
+`--` and `---`. `typographic` keeps the real dashes and only replaces the small, vertical and
+fullwidth compatibility forms with their standard counterparts.
+
+```python
+from unidash import fold_dashes
+
+text = "Pages 10\u201320 \u2014 see \u22125\uff0d6"
+print(fold_dashes(text))
+print(fold_dashes(text, "tex"))
+print(ascii(fold_dashes("a\ufe58b\uff0dc", "typographic")))
+# Pages 10-20 - see -5-6
+# Pages 10--20 --- see -5-6
+# 'a\u2014b-c'
+```
+
+A minus sign is always a single `-` in the `ascii` and `tex` forms (`typographic` keeps
+U+2212), so negative numbers stay negative, and characters that are already ASCII are never
+touched: `1-2` and a hand-written `--` come out as they went in.
+
+## Soft hyphens, non-breaking hyphens and script hyphens
+
+The soft hyphen (U+00AD) is a hint for where a word may break, not a character, so it is
+removed unless you pass `keep_soft_hyphens=True`. The non-breaking hyphen (U+2011) becomes
+`-` in the ASCII forms and stays in `typographic`, where its promise matters.
+
+Hyphens that belong to one script (Armenian, the Hebrew maqaf, Canadian syllabics,
+Mongolian, the katakana double hyphen, Garay and Yezidi) are left alone, like that script's
+letters, unless you pass `script_hyphens=True`.
+
+```python
+from unidash import fold_dashes
+
+print(fold_dashes("co\u00adop\u00aderate"))
+print(ascii(fold_dashes("co\u00adop", keep_soft_hyphens=True)))
+print(ascii(fold_dashes("\u05e2\u05be\u05d1")))
+print(ascii(fold_dashes("\u05e2\u05be\u05d1", script_hyphens=True)))
+# cooperate
+# 'co\xadop'
+# '\u05e2\u05be\u05d1'
+# '\u05e2-\u05d1'
+```
+
+## Finding dashes
+
+`find_dashes` yields a `DashHit` (offset, line, column, character and Unicode name) for
+every character `fold_dashes` would change with the same options, which makes a lint or a
+pre-commit check a one-liner. Lines end at LF, CR or CRLF, columns count code points from 1,
+and a byte order mark at the start of the text takes no column. `is_dash(char)` tells whether a character has the Unicode Dash
+property and `DASHES` is that set. `dash_table(...)` returns the `str.translate` table, and
+`iter_fold_dashes(chunks, ...)` folds an iterable of strings lazily, for large inputs.
+
+```python
+from unidash import find_dashes, is_dash, iter_fold_dashes
+
+for hit in find_dashes("first line\nsecond\u2014line\u00ad"):
+    print(hit.line, hit.column, hit.codepoint, hit.name)
+print(is_dash("\u2212"), is_dash("\u00ad"))
+print("".join(iter_fold_dashes(["a\u2013", "b"])))
+# 2 7 U+2014 EM DASH
+# 2 12 U+00AD SOFT HYPHEN
+# True False
+# a-b
+```
+
+## All punctuation
+
+`to_ascii_punct(text)` folds the rest of the typographic punctuation too: curly quotes become
+`'` and `"`, the ellipsis becomes `...`, non-breaking and other Unicode spaces become a normal
+space, bullets become `*`, fullwidth punctuation from CJK input (U+FF01 to U+FF5E, letters and
+digits excepted) becomes its ASCII character, and dashes follow the `ascii` form. Accented
+letters pass through.
+
+```python
+from unidash import to_ascii_punct
+
+print(to_ascii_punct("\u201cCaf\u00e9\u201d\u2014it\u2019s 100\u00a0%\u2026"))
+# "Café"-it's 100 %...
+```
+
+## Command line
+
+`unidash FILE...` writes the folded text to stdout (`-` or no file reads stdin). It takes
+`--form`, `--script-hyphens`, `--keep-soft-hyphens`, and `--punct` for the full punctuation
+fold. `--check` lists what would change and exits 1, for pre-commit hooks. Input must be
+UTF-8; unreadable files exit 2.
+
+```
+$ unidash --check notes.md
+notes.md:2:6: U+2014 EM DASH
+notes.md:3:3: U+00AD SOFT HYPHEN
+$ unidash --form tex notes.md
+Draft
+Done --- ship it
+coop
+```
+
+## What counts as a dash
+
+The classes follow the Unicode 18.0 Dash property and General_Category=Pd, and the tests
+check them against the Unicode data files and the UTS #39 confusables list.
+
+| Code point | Name | `ascii` | `tex` | `typographic` |
+| --- | --- | --- | --- | --- |
+| U+2010 | HYPHEN | `-` | `-` | kept |
+| U+2011 | NON-BREAKING HYPHEN | `-` | `-` | kept |
+| U+2012 | FIGURE DASH | `-` | `-` | kept |
+| U+2013 | EN DASH | `-` | `--` | kept |
+| U+2014 | EM DASH | `-` | `---` | kept |
+| U+2015 | HORIZONTAL BAR | `-` | `---` | kept |
+| U+2212 | MINUS SIGN | `-` | `-` | kept |
+| U+2E17 | DOUBLE OBLIQUE HYPHEN | `-` | `-` | kept |
+| U+2E1A | HYPHEN WITH DIAERESIS | `-` | `-` | kept |
+| U+2E3A | TWO-EM DASH | `-` | `------` | kept |
+| U+2E3B | THREE-EM DASH | `-` | `---------` | kept |
+| U+2E40 | DOUBLE HYPHEN | `-` | `-` | kept |
+| U+2E5D | OBLIQUE HYPHEN | `-` | `-` | kept |
+| U+FE31 | PRESENTATION FORM FOR VERTICAL EM DASH | `-` | `---` | U+2014 |
+| U+FE32 | PRESENTATION FORM FOR VERTICAL EN DASH | `-` | `--` | U+2013 |
+| U+FE58 | SMALL EM DASH | `-` | `---` | U+2014 |
+| U+FE63 | SMALL HYPHEN-MINUS | `-` | `-` | `-` |
+| U+FF0D | FULLWIDTH HYPHEN-MINUS | `-` | `-` | `-` |
+| U+2053 | SWUNG DASH | `~` | `~` | kept |
+| U+301C | WAVE DASH | `~` | `~` | kept |
+| U+3030 | WAVY DASH | `~` | `~` | kept |
+| U+00AD | SOFT HYPHEN | removed | removed | removed |
+
+Script hyphens, folded only with `script_hyphens=True` (to `-`, or to U+2010 in
+`typographic`): U+058A ARMENIAN HYPHEN, U+05BE HEBREW PUNCTUATION MAQAF, U+1400 CANADIAN
+SYLLABICS HYPHEN, U+1806 MONGOLIAN TODO SOFT HYPHEN, U+30A0 KATAKANA-HIRAGANA DOUBLE HYPHEN,
+U+10D6E GARAY HYPHEN and U+10EAD YEZIDI HYPHENATION MARK.
+
+The double hyphens (U+2E40, and U+1400 and U+30A0 among the script hyphens) become `-`
+although UTS #39 lists them as confusable with `=`: they are hyphens in use, and `=` would
+change the meaning of the text.
+
+Never folded: U+002D HYPHEN-MINUS, and U+207B SUPERSCRIPT MINUS and U+208B SUBSCRIPT MINUS,
+which sit next to superscript and subscript digits that stay as they are. Characters that
+only look like a hyphen are not dashes and are left alone: letters in other scripts, the Arabic
+full stop, the hyphen bullet, U+02D7 MODIFIER LETTER MINUS SIGN (a tone mark), and minus-like
+symbols without the Dash property such as U+2052 COMMERCIAL MINUS SIGN, U+2796 HEAVY MINUS
+SIGN (an emoji), U+2E43 DASH WITH LEFT UPTURN and U+10D8F GARAY MINUS SIGN.
+
+## Why
+
+Tools like `unidecode` and `anyascii` transliterate every character, so `café` turns into
+`cafe`. That is the wrong tool when you only want to tidy punctuation and keep the words.
+`smartypants` goes the opposite way, turning ASCII into typographic dashes. `unidash` folds
+punctuation to ASCII, knows the whole Unicode dash family rather than the three most common
+members, and leaves letters alone. It is zero dependency and typed.
+
+## Development
+
+The Unicode data files are not part of the repository. `python scripts/fetch_vectors.py`
+downloads them into `tests/vectors/unicode/` and checks each against the size and SHA-256 in
+`MANIFEST.txt`; without them the data tests skip. The installed-wheel test runs with
+`UNIDASH_E2E=1`.
+
+## License
+
+MIT. See [LICENSE](LICENSE).
+
+## Links
+
+- Website: https://lianes8studio.com
+- GitHub: https://github.com/JaviLianes8
+- LinkedIn: https://www.linkedin.com/in/jlianes/