genderize-offline 0.1.0__tar.gz

genderize_offline-0.1.0/.gitignore

@@ -0,0 +1,21 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+.coverage
+coverage.xml
+htmlcov/
+# Add here any generated outputs or data files that hold personal data (e.g. downloaded pages)
+# Never commit credentials
+.env
+.pypirc
+*apikey*
+# Dictionaries and benchmark inputs (large; names of real people)
+*.tsv.gz
+wgnd*.csv
+test_gender*.csv

genderize_offline-0.1.0/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-11-03
+
+### Added
+
+- `guess(names, country)` and `genderize(names, country_id)` (genderize.io-shaped answer),
+  `country_code`, `tokens`, `p_female`.
+- Command `genderize-offline` with `name`, `csv`, `fetch` and `build`.
+- Compact dictionary built from WGND 2.0, published as the GitHub release `wgnd-2.0-r1`
+  and downloaded on first use with a SHA-256 check; fallback build from Harvard Dataverse.
+- Benchmarks against genderize.io (`benchmarks/benchmark.py`, README).
+
+[Unreleased]: https://github.com/restrepo/genderize-offline/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/restrepo/genderize-offline/releases/tag/v0.1.0

genderize_offline-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego Restrepo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

genderize_offline-0.1.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.5
+Name: genderize-offline
+Version: 0.1.0
+Summary: Offline gender inference from first names and country, a free drop-in for genderize.io, using the World Gender Name Dictionary 2.0.
+Project-URL: Homepage, https://github.com/restrepo/genderize-offline
+Project-URL: Repository, https://github.com/restrepo/genderize-offline
+Project-URL: Issues, https://github.com/restrepo/genderize-offline/issues
+Project-URL: Changelog, https://github.com/restrepo/genderize-offline/blob/main/CHANGELOG.md
+Author-email: Diego Restrepo <restrepo@udea.edu.co>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: WGND,bibliometrics,first names,gender,genderize,names,offline
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff==0.17.3; extra == 'dev'
+Requires-Dist: twine>=5; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# genderize-offline
+
+[![CI](https://github.com/restrepo/genderize-offline/actions/workflows/ci.yml/badge.svg)](https://github.com/restrepo/genderize-offline/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/genderize-offline.svg)](https://pypi.org/project/genderize-offline/)
+[![Python](https://img.shields.io/pypi/pyversions/genderize-offline.svg)](https://pypi.org/project/genderize-offline/)
+
+Infer gender from first names and country **offline**: a free drop-in for
+[genderize.io](https://genderize.io), with no API key, no quota and no network
+after the first use. It uses the
+[World Gender Name Dictionary 2.0](https://doi.org/10.7910/DVN/MSEGSJ) (WGND, Raffo 2021,
+CC0), which gives the share of women and men for 4.9 million name–country pairs.
+Standard library only.
+
+## Install
+
+```bash
+pip install genderize-offline
+genderize-offline fetch          # once: downloads the dictionary (14 MB) into the cache
+```
+
+## Use
+
+```python
+from genderize_offline import guess, genderize
+
+guess("Natalia Andrea", "CO")  # ('female', 1.0, 58)
+guess("Andrea", "Italy")  # ('male', 1.0, 77)  country names or ISO2 codes
+guess("Andrea", "CO")  # ('female', 0.96, 77)
+genderize("Andrea", "IT")  # {'count': 77, 'name': 'Andrea', 'gender': 'male', 'probability': 1.0, 'country_id': 'IT'}
+```
+
+`guess(names, country)` returns `(gender, probability, count)`: gender `male`, `female` or
+`unknown`; probability 0.5–1 (0 for unknown); count, the number of countries that list the
+name (genderize.io's count is people). `genderize()` answers in genderize.io's shape, with
+`gender` `None` when unknown. As with genderize.io, set your own threshold: the benchmarks
+below count a gender only at probability ≥ 0.9.
+
+```bash
+genderize-offline name "María José" ES
+genderize-offline csv authors.csv out.csv --name-col first_names --country-col country
+```
+
+### Method
+
+- Names are lowercased and split on spaces, hyphens and dots; initials are dropped.
+- The first given name found in the dictionary decides (genderize.io also uses the first word).
+- The share of women is WGND's for that name in the given country; if the country is missing
+  or doesn't list the name, the average over every country that lists it is used.
+- Accents are optional: `andres` finds `andrés`, `josé` finds `jose`.
+
+## Benchmarks
+
+Run with `python benchmarks/benchmark.py LABELLED.csv [AUTHORS.json]` (threshold 0.9). The inputs
+hold names of real people and are not in this repository; they are files of
+[restrepo/inspireauth](https://github.com/restrepo/inspireauth).
+
+**Hand-labelled names** (`data/test_gender.csv`: 3,965 names, 3,820 of them Colombian, with
+their true gender and genderize.io's answer):
+
+| | genderize.io | genderize-offline |
+|---|---|---|
+| Names given a gender (coverage) | 98.1% | 93.6% |
+| Accuracy where a gender is given | 99.38% | 98.98% |
+
+**INSPIRE-HEP authors with Latin American affiliations** (`data/inspire_LA.json`: 29,979
+physicists, genders from genderize.io):
+
+| genderize.io ↓ / offline → | male | female | unknown |
+|---|---|---|---|
+| male | 14,479 | 39 | 686 |
+| female | 23 | 2,635 | 155 |
+| unknown | 677 | 176 | 11,109 |
+
+| | genderize.io | genderize-offline |
+|---|---|---|
+| Authors with a gender | 18,017 | 18,029 |
+| Female share | 15.6% | 15.8% |
+
+Where both give a gender (17,176 authors) they agree on **99.64%**.
+
+**Effect on published results.** We recomputed the female fractions of *Statistical Analysis
+of Scientific Metrics in High Energy, Cosmology, and Astroparticle Physics in Latin America*
+(Morales-Alvarado, Quevedo, Ramos-Hamud, Restrepo) with offline genders. Every value moves
+by less than its 95% margin:
+
+| Female share by rank (Table 8) | genderize.io | offline |
+|---|---|---|
+| All ranks | 11.9% | 12.3% |
+| Undergraduate, Master, PhD | 14.3% | 14.7% |
+| Postdoc | 13.4% | 14.0% |
+| Junior or Staff | 12.6% | 12.2% |
+| Senior | 7.9% | 8.5% |
+
+The female share per year (Fig. 21, 45 years with at least 100 authors) differs by −0.04
+points on average and by at most 1.2 points.
+
+These are the genders used to reproduce the paper's results:
+the package gives the same gender to every one of the 29,979 authors.
+
+**Speed** (one core, laptop): the dictionary loads in about 1 s; lookups run at about
+140,000 names/s.
+
+The genders are estimates from names, not self-reported gender. Report them as such, and
+cite WGND 2.0 (Raffo, J. (2021). WGND 2.0. Harvard Dataverse. doi:10.7910/DVN/MSEGSJ).
+
+## The dictionary and this repository
+
+The 91 MB WGND CSV is not stored in git. Instead:
+
+1. **Built reproducibly.** `genderize-offline build wgnd.csv wgnd2_compact.tsv.gz` turns
+   WGND's `wgnd_2_0_name-gender-code.csv` into a compact table (one sorted line per name: average
+   share of women, number of countries, and the countries that differ from the average). The
+   same input gives the same bytes, so anyone can check the SHA-256.
+2. **Published as a release asset.** The manual workflow [Dictionary](.github/workflows/dictionary.yml)
+   downloads WGND from Harvard Dataverse, builds the table and attaches it with its
+   `SHA256SUMS` to a GitHub release of its own, `wgnd-2.0-rN`. These tags don't start
+   with `v`, so they never publish to PyPI, and they aren't marked latest.
+3. **Pinned in the code.** `dictionary.py` names the release and the SHA-256. On first use,
+   `fetch()` downloads the file to `~/.cache/genderize-offline/` (or `$GENDERIZE_OFFLINE_CACHE`)
+   and refuses it if the hash differs. If the release can't be reached, it builds the table
+   from Dataverse instead (`genderize-offline fetch --from-wgnd`).
+
+To change the dictionary (a new WGND version or a new build rule), run the workflow with the
+next tag, update `DICTIONARY_RELEASE` and `DICTIONARY_SHA256`, and release a new version.
+Old releases keep their assets, so older package versions keep working.
+
+## Development
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+ruff check . && ruff format --check .
+```
+
+Tests use a synthetic dictionary and need no network. Releases are published to PyPI from
+GitHub Actions when a `v*` tag is pushed; see [PUBLISHING.md](PUBLISHING.md) and
+[CHANGELOG.md](CHANGELOG.md).
+
+## License
+
+Code: MIT. Dictionary: derived from WGND 2.0, CC0 1.0.

genderize_offline-0.1.0/README.md

@@ -0,0 +1,141 @@
+# genderize-offline
+
+[![CI](https://github.com/restrepo/genderize-offline/actions/workflows/ci.yml/badge.svg)](https://github.com/restrepo/genderize-offline/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/genderize-offline.svg)](https://pypi.org/project/genderize-offline/)
+[![Python](https://img.shields.io/pypi/pyversions/genderize-offline.svg)](https://pypi.org/project/genderize-offline/)
+
+Infer gender from first names and country **offline**: a free drop-in for
+[genderize.io](https://genderize.io), with no API key, no quota and no network
+after the first use. It uses the
+[World Gender Name Dictionary 2.0](https://doi.org/10.7910/DVN/MSEGSJ) (WGND, Raffo 2021,
+CC0), which gives the share of women and men for 4.9 million name–country pairs.
+Standard library only.
+
+## Install
+
+```bash
+pip install genderize-offline
+genderize-offline fetch          # once: downloads the dictionary (14 MB) into the cache
+```
+
+## Use
+
+```python
+from genderize_offline import guess, genderize
+
+guess("Natalia Andrea", "CO")  # ('female', 1.0, 58)
+guess("Andrea", "Italy")  # ('male', 1.0, 77)  country names or ISO2 codes
+guess("Andrea", "CO")  # ('female', 0.96, 77)
+genderize("Andrea", "IT")  # {'count': 77, 'name': 'Andrea', 'gender': 'male', 'probability': 1.0, 'country_id': 'IT'}
+```
+
+`guess(names, country)` returns `(gender, probability, count)`: gender `male`, `female` or
+`unknown`; probability 0.5–1 (0 for unknown); count, the number of countries that list the
+name (genderize.io's count is people). `genderize()` answers in genderize.io's shape, with
+`gender` `None` when unknown. As with genderize.io, set your own threshold: the benchmarks
+below count a gender only at probability ≥ 0.9.
+
+```bash
+genderize-offline name "María José" ES
+genderize-offline csv authors.csv out.csv --name-col first_names --country-col country
+```
+
+### Method
+
+- Names are lowercased and split on spaces, hyphens and dots; initials are dropped.
+- The first given name found in the dictionary decides (genderize.io also uses the first word).
+- The share of women is WGND's for that name in the given country; if the country is missing
+  or doesn't list the name, the average over every country that lists it is used.
+- Accents are optional: `andres` finds `andrés`, `josé` finds `jose`.
+
+## Benchmarks
+
+Run with `python benchmarks/benchmark.py LABELLED.csv [AUTHORS.json]` (threshold 0.9). The inputs
+hold names of real people and are not in this repository; they are files of
+[restrepo/inspireauth](https://github.com/restrepo/inspireauth).
+
+**Hand-labelled names** (`data/test_gender.csv`: 3,965 names, 3,820 of them Colombian, with
+their true gender and genderize.io's answer):
+
+| | genderize.io | genderize-offline |
+|---|---|---|
+| Names given a gender (coverage) | 98.1% | 93.6% |
+| Accuracy where a gender is given | 99.38% | 98.98% |
+
+**INSPIRE-HEP authors with Latin American affiliations** (`data/inspire_LA.json`: 29,979
+physicists, genders from genderize.io):
+
+| genderize.io ↓ / offline → | male | female | unknown |
+|---|---|---|---|
+| male | 14,479 | 39 | 686 |
+| female | 23 | 2,635 | 155 |
+| unknown | 677 | 176 | 11,109 |
+
+| | genderize.io | genderize-offline |
+|---|---|---|
+| Authors with a gender | 18,017 | 18,029 |
+| Female share | 15.6% | 15.8% |
+
+Where both give a gender (17,176 authors) they agree on **99.64%**.
+
+**Effect on published results.** We recomputed the female fractions of *Statistical Analysis
+of Scientific Metrics in High Energy, Cosmology, and Astroparticle Physics in Latin America*
+(Morales-Alvarado, Quevedo, Ramos-Hamud, Restrepo) with offline genders. Every value moves
+by less than its 95% margin:
+
+| Female share by rank (Table 8) | genderize.io | offline |
+|---|---|---|
+| All ranks | 11.9% | 12.3% |
+| Undergraduate, Master, PhD | 14.3% | 14.7% |
+| Postdoc | 13.4% | 14.0% |
+| Junior or Staff | 12.6% | 12.2% |
+| Senior | 7.9% | 8.5% |
+
+The female share per year (Fig. 21, 45 years with at least 100 authors) differs by −0.04
+points on average and by at most 1.2 points.
+
+These are the genders used to reproduce the paper's results:
+the package gives the same gender to every one of the 29,979 authors.
+
+**Speed** (one core, laptop): the dictionary loads in about 1 s; lookups run at about
+140,000 names/s.
+
+The genders are estimates from names, not self-reported gender. Report them as such, and
+cite WGND 2.0 (Raffo, J. (2021). WGND 2.0. Harvard Dataverse. doi:10.7910/DVN/MSEGSJ).
+
+## The dictionary and this repository
+
+The 91 MB WGND CSV is not stored in git. Instead:
+
+1. **Built reproducibly.** `genderize-offline build wgnd.csv wgnd2_compact.tsv.gz` turns
+   WGND's `wgnd_2_0_name-gender-code.csv` into a compact table (one sorted line per name: average
+   share of women, number of countries, and the countries that differ from the average). The
+   same input gives the same bytes, so anyone can check the SHA-256.
+2. **Published as a release asset.** The manual workflow [Dictionary](.github/workflows/dictionary.yml)
+   downloads WGND from Harvard Dataverse, builds the table and attaches it with its
+   `SHA256SUMS` to a GitHub release of its own, `wgnd-2.0-rN`. These tags don't start
+   with `v`, so they never publish to PyPI, and they aren't marked latest.
+3. **Pinned in the code.** `dictionary.py` names the release and the SHA-256. On first use,
+   `fetch()` downloads the file to `~/.cache/genderize-offline/` (or `$GENDERIZE_OFFLINE_CACHE`)
+   and refuses it if the hash differs. If the release can't be reached, it builds the table
+   from Dataverse instead (`genderize-offline fetch --from-wgnd`).
+
+To change the dictionary (a new WGND version or a new build rule), run the workflow with the
+next tag, update `DICTIONARY_RELEASE` and `DICTIONARY_SHA256`, and release a new version.
+Old releases keep their assets, so older package versions keep working.
+
+## Development
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+ruff check . && ruff format --check .
+```
+
+Tests use a synthetic dictionary and need no network. Releases are published to PyPI from
+GitHub Actions when a `v*` tag is pushed; see [PUBLISHING.md](PUBLISHING.md) and
+[CHANGELOG.md](CHANGELOG.md).
+
+## License
+
+Code: MIT. Dictionary: derived from WGND 2.0, CC0 1.0.

genderize_offline-0.1.0/benchmarks/benchmark.py

@@ -0,0 +1,77 @@
+"""Benchmarks of genderize-offline against genderize.io (numbers in README.md).
+
+Usage: python benchmarks/benchmark.py LABELLED.csv [AUTHORS.json] [--threshold 0.9]
+
+LABELLED.csv: hand-labelled names with columns names, gender (the true one),
+country_id, and genderize.io's answer in Gender and Gender Probability
+(as `data/test_gender.csv` of restrepo/inspireauth).
+AUTHORS.json: records with first_names, country, author_id and genderize.io's
+gender and gender_probability (as `data/inspire_LA.json` of restrepo/inspireauth).
+"""
+
+import argparse
+import csv
+import json
+import time
+from collections import Counter
+
+from genderize_offline import core, dictionary, guess
+
+
+def scores(rows, thr):
+    known = [(t, g) for t, g, p in rows if g in ("male", "female") and p >= thr]
+    ok = sum(t == g for t, g in known)
+    return f"coverage {100 * len(known) / len(rows):.1f}%, accuracy {100 * ok / len(known):.2f}% ({len(known)} names)"
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser()
+    ap.add_argument("labelled")
+    ap.add_argument("authors", nargs="?")
+    ap.add_argument("--threshold", type=float, default=0.9)
+    a = ap.parse_args(argv)
+    thr = a.threshold
+
+    t0 = time.perf_counter()
+    core.use_table(dictionary.load())
+    print(f"load: {time.perf_counter() - t0:.1f} s, {len(core.table())} names and aliases")
+
+    with open(a.labelled, newline="", encoding="utf-8") as f:
+        rows = list(csv.DictReader(f))
+    gz = [(r["gender"], r["Gender"] or "unknown", float(r["Gender Probability"] or 0)) for r in rows]
+    t0 = time.perf_counter()
+    og = [(r["gender"], *guess(r["names"], r["country_id"])[:2]) for r in rows]
+    dt = time.perf_counter() - t0
+    print(f"\nhand-labelled names: {len(rows)}, threshold {thr}  ({len(rows) / dt:,.0f} names/s)")
+    print(f"  genderize.io      {scores(gz, thr)}")
+    print(f"  genderize-offline {scores(og, thr)}")
+
+    if not a.authors:
+        return 0
+    with open(a.authors, encoding="utf-8") as f:
+        records = json.load(f)
+    first = {}
+    for r in records:
+        first.setdefault(r["author_id"], r)
+    pairs = Counter()
+    for r in first.values():
+        g0 = r["gender"] if r["gender_probability"] >= thr else "unknown"
+        g, p, _ = guess(r.get("first_names", ""), r.get("country", ""))
+        pairs[g0, g if p >= thr else "unknown"] += 1
+    ks = ["male", "female", "unknown"]
+    print(f"\nauthors: {len(first)}; rows genderize.io, columns genderize-offline")
+    print(" " * 10 + "".join(f"{k:>10}" for k in ks))
+    for x in ks:
+        print(f"{x:>10}" + "".join(f"{pairs[x, y]:>10}" for y in ks))
+    both = sum(pairs[x, y] for x in ks[:2] for y in ks[:2])
+    agree = pairs["male", "male"] + pairs["female", "female"]
+    print(f"agreement where both give a gender: {100 * agree / both:.2f}% of {both}")
+    for name, i in [("genderize.io", 0), ("genderize-offline", 1)]:
+        fe = sum(v for k, v in pairs.items() if k[i] == "female")
+        ma = sum(v for k, v in pairs.items() if k[i] == "male")
+        print(f"  {name:18s} {fe + ma} with a gender, female {100 * fe / (fe + ma):.1f}%")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

genderize_offline-0.1.0/pyproject.toml

@@ -0,0 +1,64 @@
+[build-system]
+requires = ["hatchling>=1.24", "hatch-vcs>=0.4"]
+build-backend = "hatchling.build"
+
+[project]
+name = "genderize-offline"
+dynamic = ["version"]
+description = "Offline gender inference from first names and country, a free drop-in for genderize.io, using the World Gender Name Dictionary 2.0."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+authors = [{ name = "Diego Restrepo", email = "restrepo@udea.edu.co" }]
+keywords = ["gender", "genderize", "names", "first names", "WGND", "bibliometrics", "offline"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Typing :: Typed",
+]
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "pytest-cov>=5", "ruff==0.17.3", "build>=1.2", "twine>=5"]
+
+[project.scripts]
+genderize-offline = "genderize_offline.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/restrepo/genderize-offline"
+Repository = "https://github.com/restrepo/genderize-offline"
+Issues = "https://github.com/restrepo/genderize-offline/issues"
+Changelog = "https://github.com/restrepo/genderize-offline/blob/main/CHANGELOG.md"
+
+# Version from the git tag (v0.1.0 -> 0.1.0); commits after a tag get 0.1.1.devN
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.version.raw-options]
+local_scheme = "no-local-version"   # PyPI and TestPyPI reject local versions (+g1234abc)
+
+[tool.hatch.build.targets.sdist]
+include = ["src/", "tests/", "benchmarks/", "README.md", "CHANGELOG.md", "LICENSE"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/genderize_offline"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra --strict-markers"
+
+[tool.coverage.run]
+source = ["genderize_offline"]
+branch = true
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]

genderize_offline-0.1.0/src/genderize_offline/__init__.py

@@ -0,0 +1,12 @@
+"""Offline gender inference from first names and country, a free drop-in for genderize.io (WGND 2.0)."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from .core import country_code, genderize, guess, p_female, tokens, use_table
+
+try:
+    __version__ = version("genderize-offline")
+except PackageNotFoundError:  # running from a source tree that was never installed
+    __version__ = "0.0.0"
+
+__all__ = ["__version__", "country_code", "genderize", "guess", "p_female", "tokens", "use_table"]

genderize_offline-0.1.0/src/genderize_offline/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

genderize_offline-0.1.0/src/genderize_offline/cli.py

@@ -0,0 +1,58 @@
+"""Command line entry point: `genderize-offline`."""
+
+import argparse
+import csv
+import json
+import sys
+
+from . import __version__, dictionary
+from .core import guess
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(
+        prog="genderize-offline",
+        description="Offline gender inference from first names and country, a free drop-in for genderize.io, "
+        "using the World Gender Name Dictionary 2.0.",
+    )
+    ap.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    p = sub.add_parser("name", help="gender of one name, as JSON")
+    p.add_argument("names")
+    p.add_argument("country", nargs="?", default="", help="ISO2 code or country name")
+    p = sub.add_parser("csv", help="add gender columns to a CSV")
+    p.add_argument("src")
+    p.add_argument("dst")
+    p.add_argument("--name-col", default="name")
+    p.add_argument("--country-col", default="country_id")
+    p.add_argument("--prefix", default="", help="prefix of the added columns gender, probability, count")
+    p = sub.add_parser("fetch", help="download the dictionary into the cache")
+    p.add_argument("--from-wgnd", action="store_true", help="build it from the original WGND CSV instead")
+    p = sub.add_parser("build", help="build the compact dictionary from WGND's name-gender-code CSV")
+    p.add_argument("src")
+    p.add_argument("dst")
+    a = ap.parse_args(argv)
+
+    if a.cmd == "name":
+        g, p_, n = guess(a.names, a.country)
+        print(json.dumps({"name": a.names, "gender": g, "probability": p_, "count": n}, ensure_ascii=False))
+    elif a.cmd == "csv":
+        with open(a.src, newline="", encoding="utf-8") as f:
+            rows = list(csv.DictReader(f))
+        cols = [f"{a.prefix}gender", f"{a.prefix}probability", f"{a.prefix}count"]
+        with open(a.dst, "w", newline="", encoding="utf-8") as f:
+            w = csv.DictWriter(f, list(rows[0]) + cols if rows else cols)
+            w.writeheader()
+            for r in rows:
+                r.update(zip(cols, guess(r.get(a.name_col, ""), r.get(a.country_col, "")), strict=True))
+                w.writerow(r)
+    elif a.cmd == "fetch":
+        print(dictionary.fetch(from_wgnd=a.from_wgnd))
+    elif a.cmd == "build":
+        n = dictionary.build(a.src, a.dst)
+        print(f"{a.dst}: {n} names, sha256 {dictionary.sha256(a.dst)}", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

genderize_offline-0.1.0/src/genderize_offline/core.py

@@ -0,0 +1,87 @@
+"""Gender of a first name in a country, like genderize.io."""
+
+import json
+import re
+from importlib.resources import files
+
+from . import dictionary
+
+COUNTRIES = json.loads(files(__package__).joinpath("country_iso2.json").read_text(encoding="utf-8"))
+
+_table = None
+
+
+def table():
+    """The dictionary, loaded on first use."""
+    global _table
+    if _table is None:
+        _table = dictionary.load()
+    return _table
+
+
+def use_table(t):
+    """Use another dictionary (`dictionary.load(path)` or a dict of the same shape), e.g. in tests."""
+    global _table
+    _table = t
+
+
+def country_code(country):
+    """ISO2 code for an ISO2 code or a country name (INSPIRE/English spelling); '' if unknown."""
+    if not country:
+        return ""
+    c = country.strip()
+    if len(c) == 2 and c.isalpha():
+        return c.upper()
+    return COUNTRIES.get(c) or COUNTRIES.get(c.lower(), "")
+
+
+def tokens(names):
+    """Lowercase given names, without initials, punctuation or hyphens."""
+    names = re.sub(r'[|†•()"]', " ", names or "").replace("-", " ").replace(".", " ")
+    return [t.lower() for t in names.split() if len(t) > 1]
+
+
+def p_female(name, country=""):
+    """(share of women, countries listing the name) for one name, or None if it is not in the dictionary."""
+    t = table()
+    e = t.get(name) or t.get(dictionary.strip_accents(name))
+    if not e:
+        return None
+    world, n, extra = e
+    code = country_code(country)
+    if code and code + ":" in extra:
+        for item in extra.split(","):
+            c, p = item.split(":")
+            if c == code:
+                return float(p), n
+    return world, n
+
+
+def guess(names, country=""):
+    """(gender, probability, count) for given names and a country (ISO2 or name).
+
+    The first given name found in the dictionary decides. gender is 'male',
+    'female' or 'unknown'; probability is 0.5-1 (0 for unknown); count is the
+    number of countries listing the name.
+    """
+    for tok in tokens(names):
+        r = p_female(tok, country)
+        if r is None:
+            continue
+        pf, n = r
+        if pf == 0.5:
+            return "unknown", 0.0, n
+        return ("female", round(pf, 2), n) if pf > 0.5 else ("male", round(1 - pf, 2), n)
+    return "unknown", 0.0, 0
+
+
+def genderize(names, country_id=None):
+    """Answer shaped like genderize.io's: {name, gender, probability, count, country_id}.
+
+    gender is None when unknown, as in genderize.io.
+    """
+    g, p, n = guess(names, country_id or "")
+    out = {"count": n, "name": names, "gender": None if g == "unknown" else g, "probability": p}
+    if country_id:
+        out["country_id"] = country_id
+    return out

genderize_offline-0.1.0/src/genderize_offline/country_iso2.json

@@ -0,0 +1,110 @@
+{
+"Venezuela": "VE",
+"Chile": "CL",
+"Rwanda": "RW",
+"South Africa": "ZA",
+"Spain": "ES",
+"United States": "US",
+"France": "FR",
+"Switzerland": "CH",
+"Italy": "IT",
+"Germany": "DE",
+"Peru": "PE",
+"Argentina": "AR",
+"Uruguay": "UY",
+"Brazil": "BR",
+"Ecuador": "EC",
+"United Kingdom": "GB",
+"Colombia": "CO",
+"Israel": "IL",
+"Mexico": "MX",
+"Netherlands": "NL",
+"Canada": "CA",
+"Sweden": "SE",
+"Iran, Islamic Republic of": "IR",
+"India": "IN",
+"Belarus": "BY",
+"Russian Federation": "RU",
+"Hungary": "HU",
+"Slovakia": "SK",
+"Greece": "GR",
+"Cuba": "CU",
+"Australia": "AU",
+"Kuwait": "KW",
+"Slovenia": "SI",
+"Finland": "FI",
+"Japan": "JP",
+"Ukraine": "UA",
+"Serbia and Montenegro": "RS",
+"Turkey": "TR",
+"Portugal": "PT",
+"Armenia": "AM",
+"Korea, Republic of": "KR",
+"China": "CN",
+"Belgium": "BE",
+"Poland": "PL",
+"Denmark": "DK",
+"Czechia": "CZ",
+"Romania": "RO",
+"Croatia": "HR",
+"Dominican Republic": "DO",
+"Guatemala": "GT",
+"Paraguay": "PY",
+"Panama": "PA",
+"Honduras": "HN",
+"Philippines": "PH",
+"El Salvador": "SV",
+"Pakistan": "PK",
+"Oman": "OM",
+"Costa Rica": "CR",
+"New Zealand": "NZ",
+"Georgia": "GE",
+"Bolivia": "BO",
+"Malta": "MT",
+"Austria": "AT",
+"Norway": "NO",
+"Egypt": "EG",
+"Taiwan": "TW",
+"Puerto Rico": "PR",
+"United Arab Emirates": "AE",
+"Jamaica": "JM",
+"Lithuania": "LT",
+"Indonesia": "ID",
+"Luxembourg": "LU",
+"Estonia": "EE",
+"Ethiopia": "ET",
+"Thailand": "TH",
+"Cyprus": "CY",
+"Bulgaria": "BG",
+"Iceland": "IS",
+"North Macedonia": "MK",
+"Ireland": "IE",
+"Lebanon": "LB",
+"Vietnam": "VN",
+"Bangladesh": "BD",
+"Kazakhstan": "KZ",
+"Morocco": "MA",
+"Uzbekistan": "UZ",
+"Benin": "BJ",
+"Iraq": "IQ",
+"Azerbaijan": "AZ",
+"Nigeria": "NG",
+"Jordan": "JO",
+"Ghana": "GH",
+"Algeria": "DZ",
+"Singapore": "SG",
+"Niger": "NE",
+"Malaysia": "MY",
+"Latvia": "LV",
+"Saudi Arabia": "SA",
+"Madagascar": "MG",
+"Uganda": "UG",
+"colombia": "CO",
+"brazil": "BR",
+"venezuela": "VE",
+"mexico": "MX",
+"cuba": "CU",
+"chile": "CL",
+"peru": "PE",
+"argentina": "AR"
+}

genderize_offline-0.1.0/src/genderize_offline/dictionary.py

@@ -0,0 +1,134 @@
+"""The name dictionary: a compact table derived from WGND 2.0.
+
+WGND 2.0 (World Gender Name Dictionary, Raffo 2021, Harvard Dataverse,
+doi:10.7910/DVN/MSEGSJ, CC0) gives, for each (name, ISO2 country), the share
+of women and men. The compact table keeps one line per name::
+
+    name <TAB> p_female averaged over its countries <TAB> number of countries <TAB> CC:p,CC:p,...
+
+where only countries whose share differs from the average are listed. It is
+built with `build()` from the original CSV and published as a GitHub release
+asset of this repository (tag `wgnd-2.0-rN`); `load()` downloads it once,
+checks its SHA-256 and caches it.
+"""
+
+import bisect
+import csv
+import gzip
+import hashlib
+import os
+import shutil
+import tempfile
+import unicodedata
+import urllib.request
+from collections import defaultdict
+from pathlib import Path
+
+WGND_URL = "https://dataverse.harvard.edu/api/access/datafile/4750348?format=original"
+
+DICTIONARY_RELEASE = "wgnd-2.0-r1"
+DICTIONARY_FILE = "wgnd2_compact.tsv.gz"
+DICTIONARY_URL = (
+    f"https://github.com/restrepo/genderize-offline/releases/download/{DICTIONARY_RELEASE}/{DICTIONARY_FILE}"
+)
+DICTIONARY_SHA256 = "c9020732db57856a45f5bdb469b971a5ad3febf684eef0e6256757b61a07ffaf"
+
+
+def cache_dir():
+    """`$GENDERIZE_OFFLINE_CACHE`, else `~/.cache/genderize-offline`."""
+    return Path(os.environ.get("GENDERIZE_OFFLINE_CACHE") or Path.home() / ".cache" / "genderize-offline")
+
+
+def strip_accents(s):
+    return "".join(c for c in unicodedata.normalize("NFD", s) if unicodedata.category(c) != "Mn")
+
+
+def sha256(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for block in iter(lambda: f.read(1 << 20), b""):
+            h.update(block)
+    return h.hexdigest()
+
+
+def build(src_csv, dst_gz):
+    """Write the compact table from WGND's `wgnd_2_0_name-gender-code.csv`. Returns the number of names."""
+    raw = defaultdict(lambda: defaultdict(lambda: [0.0, 0.0]))
+    with open(src_csv, newline="", encoding="utf-8") as f:
+        for r in csv.DictReader(f):
+            if r["gender"] in ("F", "M"):
+                raw[r["name"].strip().lower()][r["code"]][r["gender"] == "M"] += float(r["wgt"] or 0)
+    rows = {}
+    for name, by_code in raw.items():
+        by = {c: fm[0] / (fm[0] + fm[1]) for c, fm in by_code.items() if fm[0] + fm[1] > 0}
+        if not by or not name or "\t" in name or "\n" in name:
+            continue
+        world = round(sum(by.values()) / len(by), 6)
+        extra = ",".join(f"{c}:{round(p, 6):g}" for c, p in sorted(by.items()) if c and round(p, 6) != world)
+        rows[name] = f"{world:g}\t{sum(1 for c in by if c)}\t{extra}"
+    # an accent-free spelling not in WGND gets the entry of the first accented one (andres -> andrés)
+    for name in sorted(rows):
+        rows.setdefault(strip_accents(name), rows[name])
+    lines = sorted(f"{name}\t{v}" for name, v in rows.items())
+    # mtime=0 so the same input gives the same bytes (and SHA-256)
+    with open(dst_gz, "wb") as raw_out, gzip.GzipFile(filename="", fileobj=raw_out, mode="wb", mtime=0) as gz:
+        gz.write(("\n".join(lines) + "\n").encode("utf-8"))
+    return len(lines)
+
+
+def download(url, dst):
+    dst.parent.mkdir(parents=True, exist_ok=True)
+    with tempfile.NamedTemporaryFile(dir=dst.parent, delete=False) as tmp, urllib.request.urlopen(url) as r:
+        shutil.copyfileobj(r, tmp)
+    os.replace(tmp.name, dst)
+
+
+def fetch(from_wgnd=False):
+    """Make sure the compact table is in the cache and return its path.
+
+    By default it is downloaded from this repository's release and checked
+    against `DICTIONARY_SHA256`; with `from_wgnd=True` (or if the release is
+    unreachable) it is built from the original WGND CSV on Harvard Dataverse.
+    """
+    path = cache_dir() / DICTIONARY_FILE
+    if path.exists() and (from_wgnd or sha256(path) == DICTIONARY_SHA256):
+        return path
+    if not from_wgnd:
+        try:
+            download(DICTIONARY_URL, path)
+            if sha256(path) == DICTIONARY_SHA256:
+                return path
+            path.unlink()
+        except OSError:
+            pass
+    src = cache_dir() / "wgnd_2_0_name-gender-code.csv"
+    if not src.exists():
+        download(WGND_URL, src)
+    build(src, path)
+    return path
+
+
+class Table:
+    """The compact table in memory: sorted lines, looked up by bisection."""
+
+    def __init__(self, lines):
+        self.lines = lines
+
+    def __len__(self):
+        return len(self.lines)
+
+    def get(self, name, default=None):
+        key = name + "\t"
+        i = bisect.bisect_left(self.lines, key)
+        if i < len(self.lines) and self.lines[i].startswith(key):
+            _, world, n, extra = self.lines[i].split("\t")
+            return float(world), int(n), extra
+        return default
+
+
+def load(path=None):
+    """The dictionary as a `Table`: `.get(name)` → (p_female world, n_countries, "CC:p,...") or None."""
+    path = Path(path) if path else fetch()
+    with open(path, "rb") as f:
+        text = gzip.decompress(f.read()).decode("utf-8")
+    return Table(text.splitlines())

genderize_offline-0.1.0/tests/conftest.py

@@ -0,0 +1,35 @@
+import pytest
+
+from genderize_offline import core, dictionary
+
+# A synthetic WGND extract: made-up spellings and shares, not real data.
+WGND = """name,code,gender,wgt
+zorana,AA,F,1
+zorana,BB,F,.8
+zorana,BB,M,.2
+tomislavo,AA,M,1
+kelandre,AA,F,.6
+kelandre,AA,M,.4
+kelandre,BB,M,.9
+kelandre,BB,F,.1
+mirén,AA,F,1
+ambo,AA,F,.5
+ambo,AA,M,.5
+"""
+
+
+@pytest.fixture(scope="session")
+def compact(tmp_path_factory):
+    d = tmp_path_factory.mktemp("wgnd")
+    src = d / "wgnd.csv"
+    src.write_text(WGND, encoding="utf-8")
+    dst = d / "compact.tsv.gz"
+    dictionary.build(src, dst)
+    return dst
+
+
+@pytest.fixture(autouse=True)
+def table(compact):
+    core.use_table(dictionary.load(compact))
+    yield
+    core.use_table(None)

genderize_offline-0.1.0/tests/test_cli.py

@@ -0,0 +1,38 @@
+import csv
+import json
+
+import pytest
+
+from genderize_offline import __version__
+from genderize_offline.cli import main
+
+
+def test_version(capsys):
+    with pytest.raises(SystemExit) as e:
+        main(["--version"])
+    assert e.value.code == 0
+    assert __version__ in capsys.readouterr().out
+
+
+def test_name(capsys):
+    assert main(["name", "Kelandre", "BB"]) == 0
+    assert json.loads(capsys.readouterr().out) == {"name": "Kelandre", "gender": "male", "probability": 0.9, "count": 2}
+
+
+def test_csv(tmp_path):
+    src = tmp_path / "in.csv"
+    src.write_text("name,country_id\nZorana,BB\nXyzzy,AA\n", encoding="utf-8")
+    dst = tmp_path / "out.csv"
+    assert main(["csv", str(src), str(dst)]) == 0
+    with open(dst, newline="", encoding="utf-8") as f:
+        rows = list(csv.DictReader(f))
+    assert [(r["gender"], r["probability"]) for r in rows] == [("female", "0.8"), ("unknown", "0.0")]
+
+
+def test_build(tmp_path, capsys):
+    from conftest import WGND
+
+    src = tmp_path / "w.csv"
+    src.write_text(WGND, encoding="utf-8")
+    assert main(["build", str(src), str(tmp_path / "c.tsv.gz")]) == 0
+    assert "sha256" in capsys.readouterr().err

genderize_offline-0.1.0/tests/test_core.py

@@ -0,0 +1,65 @@
+import gzip
+
+from genderize_offline import country_code, dictionary, genderize, guess, tokens
+
+
+def test_build_is_reproducible(compact, tmp_path):
+    src = tmp_path / "w.csv"
+    from conftest import WGND
+
+    src.write_text(WGND, encoding="utf-8")
+    again = tmp_path / "again.tsv.gz"
+    dictionary.build(src, again)
+    assert dictionary.sha256(again) == dictionary.sha256(compact)
+
+
+def test_compact_lines(compact):
+    lines = gzip.decompress(compact.read_bytes()).decode().splitlines()
+    assert lines == sorted(lines)
+    assert "zorana\t0.9\t2\tAA:1,BB:0.8" in lines
+    assert "miren\t1\t1\t" in lines  # accent-free alias
+
+
+def test_country_first_then_average():
+    assert guess("Zorana", "BB") == ("female", 0.8, 2)
+    assert guess("Zorana", "CC") == ("female", 0.9, 2)
+    assert guess("Zorana") == ("female", 0.9, 2)
+
+
+def test_country_changes_gender():
+    assert guess("Kelandre", "AA") == ("female", 0.6, 2)
+    assert guess("Kelandre", "BB") == ("male", 0.9, 2)
+
+
+def test_first_known_name_decides():
+    assert guess("Q. Xyzzy Tomislavo Zorana", "AA") == ("male", 1.0, 1)
+
+
+def test_unknown_and_unisex():
+    assert guess("Xyzzy", "AA") == ("unknown", 0.0, 0)
+    assert guess("", "AA") == ("unknown", 0.0, 0)
+    assert guess("Ambo", "AA") == ("unknown", 0.0, 1)
+
+
+def test_accents_both_ways():
+    assert guess("Mirén")[0] == "female"
+    assert guess("Miren")[0] == "female"
+
+
+def test_tokens_and_country_code():
+    assert tokens("Jean-Pierre A. B.") == ["jean", "pierre"]
+    assert country_code("co") == "CO"
+    assert country_code("Colombia") == "CO"
+    assert country_code("Russian Federation") == "RU"
+    assert country_code("Atlantis") == ""
+
+
+def test_genderize_shape():
+    assert genderize("Zorana", "BB") == {
+        "count": 2,
+        "name": "Zorana",
+        "gender": "female",
+        "probability": 0.8,
+        "country_id": "BB",
+    }
+    assert genderize("Xyzzy")["gender"] is None