gngram-lookup 0.2.0__tar.gz

gngram_lookup-0.2.0/LICENSE

@@ -0,0 +1,33 @@
+Proprietary License
+
+Copyright (c) 2024 Craig Trim. All rights reserved.
+
+This software and associated documentation files (the "Software") are proprietary
+and confidential. Unauthorized copying, modification, distribution, or use of this
+Software, via any medium, is strictly prohibited.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+Data Attribution:
+
+The frequency data distributed with this package is derived from the Google
+Books Ngram Dataset, which is made available by Google Inc. under the Creative
+Commons Attribution 3.0 Unported License (CC BY 3.0).
+
+Google Books Ngram Viewer: https://books.google.com/ngrams
+Dataset: https://storage.googleapis.com/books/ngrams/books/datasetsv3.html
+
+Citation:
+Jean-Baptiste Michel*, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres,
+Matthew K. Gray, The Google Books Team, Joseph P. Pickett, Dale Hoiberg,
+Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak,
+and Erez Lieberman Aiden*. Quantitative Analysis of Culture Using Millions
+of Digitized Books. Science (Published online ahead of print: 12/16/2010).

gngram_lookup-0.2.0/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.1
+Name: gngram-lookup
+Version: 0.2.0
+Summary: Static Hash-Based Lookup for Google Ngram Frequencies
+Home-page: https://github.com/craigtrim/gngram-lookup
+License: Proprietary
+Keywords: ngram,google-ngram,nlp,natural-language-processing,frequency,linguistics
+Author: Craig Trim
+Author-email: craigtrim@gmail.com
+Maintainer: Craig Trim
+Maintainer-email: craigtrim@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: Other/Proprietary License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Dist: polars (>=1.0,<2.0)
+Requires-Dist: pyarrow (>=18.0,<19.0)
+Project-URL: Repository, https://github.com/craigtrim/gngram-lookup
+Description-Content-Type: text/markdown
+
+# gngram-lookup
+
+[![PyPI version](https://badge.fury.io/py/gngram-lookup.svg)](https://badge.fury.io/py/gngram-lookup)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
+
+Word frequency from 500 years of books. O(1) lookup. 5 million words.
+
+## Install
+
+```bash
+pip install gngram-lookup
+python -m gngram_lookup.download_data
+```
+
+## Python
+
+```python
+import gngram_lookup as ng
+
+ng.exists('computer')       # True
+ng.exists('xyznotaword')    # False
+
+ng.frequency('computer')
+# {'peak_tf': 2000, 'peak_df': 2000, 'sum_tf': 892451, 'sum_df': 312876}
+
+ng.batch_frequency(['the', 'algorithm', 'xyznotaword'])
+# {'the': {...}, 'algorithm': {...}, 'xyznotaword': None}
+```
+
+## CLI
+
+```bash
+gngram-exists computer    # True, exit 0
+gngram-exists xyznotaword # False, exit 1
+
+gngram-freq computer
+# peak_tf_decade: 2000
+# peak_df_decade: 2000
+# sum_tf: 892451
+# sum_df: 312876
+```
+
+## Docs
+
+- [API Reference](docs/api.md)
+- [CLI Reference](docs/cli.md)
+- [Data Format](docs/data-format.md)
+- [Use Cases](docs/use-cases.md)
+- [Development](docs/development.md)
+
+## Attribution
+
+Data derived from the [Google Books Ngram](https://books.google.com/ngrams) dataset.
+
+## License
+
+Proprietary. See [LICENSE](LICENSE).
+

gngram_lookup-0.2.0/README.md

@@ -0,0 +1,57 @@
+# gngram-lookup
+
+[![PyPI version](https://badge.fury.io/py/gngram-lookup.svg)](https://badge.fury.io/py/gngram-lookup)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
+
+Word frequency from 500 years of books. O(1) lookup. 5 million words.
+
+## Install
+
+```bash
+pip install gngram-lookup
+python -m gngram_lookup.download_data
+```
+
+## Python
+
+```python
+import gngram_lookup as ng
+
+ng.exists('computer')       # True
+ng.exists('xyznotaword')    # False
+
+ng.frequency('computer')
+# {'peak_tf': 2000, 'peak_df': 2000, 'sum_tf': 892451, 'sum_df': 312876}
+
+ng.batch_frequency(['the', 'algorithm', 'xyznotaword'])
+# {'the': {...}, 'algorithm': {...}, 'xyznotaword': None}
+```
+
+## CLI
+
+```bash
+gngram-exists computer    # True, exit 0
+gngram-exists xyznotaword # False, exit 1
+
+gngram-freq computer
+# peak_tf_decade: 2000
+# peak_df_decade: 2000
+# sum_tf: 892451
+# sum_df: 312876
+```
+
+## Docs
+
+- [API Reference](docs/api.md)
+- [CLI Reference](docs/cli.md)
+- [Data Format](docs/data-format.md)
+- [Use Cases](docs/use-cases.md)
+- [Development](docs/development.md)
+
+## Attribution
+
+Data derived from the [Google Books Ngram](https://books.google.com/ngrams) dataset.
+
+## License
+
+Proprietary. See [LICENSE](LICENSE).

gngram_lookup-0.2.0/gngram_counter/__init__.py

@@ -0,0 +1,14 @@
+"""gngram-counter: Google Ngram frequency counter."""
+
+from gngram_counter.data import get_data_dir, get_hash_file, is_data_installed
+from gngram_counter.lookup import FrequencyData, batch_frequency, exists, frequency
+
+__all__ = [
+    "get_data_dir",
+    "get_hash_file",
+    "is_data_installed",
+    "exists",
+    "frequency",
+    "batch_frequency",
+    "FrequencyData",
+]

gngram_lookup-0.2.0/gngram_counter/cli.py

@@ -0,0 +1,36 @@
+"""CLI entry points for gngram-counter."""
+
+import sys
+
+from gngram_counter.lookup import exists, frequency
+
+
+def gngram_exists() -> None:
+    """Check if a word exists in the ngram data."""
+    if len(sys.argv) != 2:
+        print("Usage: gngram-exists <word>")
+        sys.exit(1)
+
+    word = sys.argv[1]
+    result = exists(word)
+    print(result)
+    sys.exit(0 if result else 1)
+
+
+def gngram_freq() -> None:
+    """Get frequency data for a word."""
+    if len(sys.argv) != 2:
+        print("Usage: gngram-freq <word>")
+        sys.exit(1)
+
+    word = sys.argv[1]
+    result = frequency(word)
+    if result is None:
+        print("None")
+        sys.exit(1)
+
+    print(f"peak_tf_decade: {result['peak_tf']}")
+    print(f"peak_df_decade: {result['peak_df']}")
+    print(f"sum_tf: {result['sum_tf']}")
+    print(f"sum_df: {result['sum_df']}")
+    sys.exit(0)

gngram_lookup-0.2.0/gngram_counter/data.py

@@ -0,0 +1,43 @@
+"""
+Data path utilities for gngram-counter.
+"""
+
+from pathlib import Path
+
+DATA_DIR = Path.home() / ".gngram-counter" / "data"
+
+
+def get_data_dir() -> Path:
+    """Return the data directory path."""
+    return DATA_DIR
+
+
+def get_hash_file(prefix: str) -> Path:
+    """Return path to a specific hash bucket parquet file.
+
+    Args:
+        prefix: Two hex characters (00-ff)
+
+    Returns:
+        Path to the parquet file (may be in subdirectory from tar extraction)
+    """
+    # Handle both flat structure and nested (from tar extraction)
+    direct = DATA_DIR / f"{prefix}.parquet"
+    if direct.exists():
+        return direct
+
+    nested = DATA_DIR / "parquet-hash" / f"{prefix}.parquet"
+    if nested.exists():
+        return nested
+
+    raise FileNotFoundError(
+        f"Data file not found for prefix '{prefix}'. "
+        "Run 'python -m gngram_counter.download_data' to download the data files."
+    )
+
+
+def is_data_installed() -> bool:
+    """Check if data files are installed."""
+    if not DATA_DIR.exists():
+        return False
+    return any(DATA_DIR.glob("**/*.parquet"))

gngram_lookup-0.2.0/gngram_counter/download_data.py

@@ -0,0 +1,89 @@
+"""
+Download gngram data files from GitHub.
+
+Usage:
+    python -m gngram_counter.download_data
+
+Downloads parquet hash files to ~/.gngram-counter/data/
+"""
+
+import sys
+import tarfile
+import urllib.request
+from io import BytesIO
+from pathlib import Path
+
+GITHUB_REPO = "craigtrim/gngram-counter"
+DATA_VERSION = "v1.0.0"
+DATA_FILENAME = "parquet-hash.tar.gz"
+
+DATA_DIR = Path.home() / ".gngram-counter" / "data"
+
+
+def get_download_url() -> str:
+    return f"https://github.com/{GITHUB_REPO}/releases/download/{DATA_VERSION}/{DATA_FILENAME}"
+
+
+def download_and_extract() -> None:
+    url = get_download_url()
+    print(f"Downloading gngram data from {url}")
+    print(f"Destination: {DATA_DIR}")
+
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+
+    try:
+        with urllib.request.urlopen(url) as response:
+            total_size = response.headers.get("Content-Length")
+            if total_size:
+                total_size = int(total_size)
+                print(f"Size: {total_size / 1024 / 1024:.1f} MB")
+
+            data = BytesIO()
+            downloaded = 0
+            chunk_size = 1024 * 1024  # 1MB chunks
+
+            while True:
+                chunk = response.read(chunk_size)
+                if not chunk:
+                    break
+                data.write(chunk)
+                downloaded += len(chunk)
+                if total_size:
+                    pct = downloaded / total_size * 100
+                    print(f"\r  {downloaded / 1024 / 1024:.1f} MB ({pct:.0f}%)", end="", flush=True)
+                else:
+                    print(f"\r  {downloaded / 1024 / 1024:.1f} MB", end="", flush=True)
+
+            print()
+            data.seek(0)
+
+            print("Extracting...")
+            with tarfile.open(fileobj=data, mode="r:gz") as tar:
+                tar.extractall(DATA_DIR, filter="data")
+
+    except urllib.error.HTTPError as e:
+        print(f"Error: Failed to download ({e.code} {e.reason})")
+        print(f"URL: {url}")
+        print("Make sure the release exists and the file is attached.")
+        sys.exit(1)
+    except urllib.error.URLError as e:
+        print(f"Error: Network error ({e.reason})")
+        sys.exit(1)
+
+    parquet_files = list(DATA_DIR.glob("**/*.parquet"))
+    print(f"Done: {len(parquet_files)} parquet files installed to {DATA_DIR}")
+
+
+def main() -> None:
+    if DATA_DIR.exists() and any(DATA_DIR.glob("**/*.parquet")):
+        print(f"Data already exists at {DATA_DIR}")
+        response = input("Re-download and overwrite? [y/N]: ").strip().lower()
+        if response != "y":
+            print("Cancelled.")
+            return
+
+    download_and_extract()
+
+
+if __name__ == "__main__":
+    main()

gngram_lookup-0.2.0/gngram_counter/lookup.py

@@ -0,0 +1,138 @@
+"""
+High-level lookup API for gngram-counter.
+
+Provides simple functions for word frequency lookups similar to bnc-lookup.
+"""
+
+import hashlib
+from functools import lru_cache
+from typing import TypedDict
+
+import polars as pl
+
+from gngram_counter.data import get_hash_file, is_data_installed
+
+
+class FrequencyData(TypedDict):
+    """Frequency data for a word."""
+
+    peak_tf: int  # Decade with highest term frequency
+    peak_df: int  # Decade with highest document frequency
+    sum_tf: int  # Total term frequency across all decades
+    sum_df: int  # Total document frequency across all decades
+
+
+@lru_cache(maxsize=256)
+def _load_bucket(prefix: str) -> pl.DataFrame:
+    """Load and cache a parquet bucket file."""
+    return pl.read_parquet(get_hash_file(prefix))
+
+
+def _hash_word(word: str) -> tuple[str, str]:
+    """Hash a word and return (prefix, suffix)."""
+    h = hashlib.md5(word.lower().encode("utf-8")).hexdigest()
+    return h[:2], h[2:]
+
+
+def exists(word: str) -> bool:
+    """Check if a word exists in the ngram data.
+
+    Args:
+        word: The word to check (case-insensitive)
+
+    Returns:
+        True if the word exists, False otherwise
+
+    Raises:
+        FileNotFoundError: If data files are not installed
+    """
+    if not is_data_installed():
+        raise FileNotFoundError(
+            "Data files not installed. Run: python -m gngram_counter.download_data"
+        )
+
+    prefix, suffix = _hash_word(word)
+    df = _load_bucket(prefix)
+    return len(df.filter(pl.col("hash") == suffix)) > 0
+
+
+def frequency(word: str) -> FrequencyData | None:
+    """Get frequency data for a word.
+
+    Args:
+        word: The word to look up (case-insensitive)
+
+    Returns:
+        FrequencyData dict with peak_tf, peak_df, sum_tf, sum_df, or None if not found
+
+    Raises:
+        FileNotFoundError: If data files are not installed
+    """
+    if not is_data_installed():
+        raise FileNotFoundError(
+            "Data files not installed. Run: python -m gngram_counter.download_data"
+        )
+
+    prefix, suffix = _hash_word(word)
+    df = _load_bucket(prefix)
+    row = df.filter(pl.col("hash") == suffix)
+
+    if len(row) == 0:
+        return None
+
+    return FrequencyData(
+        peak_tf=row["peak_tf"][0],
+        peak_df=row["peak_df"][0],
+        sum_tf=row["sum_tf"][0],
+        sum_df=row["sum_df"][0],
+    )
+
+
+def batch_frequency(words: list[str]) -> dict[str, FrequencyData | None]:
+    """Get frequency data for multiple words.
+
+    Args:
+        words: List of words to look up (case-insensitive)
+
+    Returns:
+        Dict mapping each word to its FrequencyData or None if not found
+
+    Raises:
+        FileNotFoundError: If data files are not installed
+    """
+    if not is_data_installed():
+        raise FileNotFoundError(
+            "Data files not installed. Run: python -m gngram_counter.download_data"
+        )
+
+    # Group words by bucket prefix for efficient batch lookups
+    by_prefix: dict[str, list[tuple[str, str]]] = {}
+    for word in words:
+        prefix, suffix = _hash_word(word)
+        if prefix not in by_prefix:
+            by_prefix[prefix] = []
+        by_prefix[prefix].append((word, suffix))
+
+    results: dict[str, FrequencyData | None] = {}
+
+    for prefix, word_suffix_pairs in by_prefix.items():
+        df = _load_bucket(prefix)
+        suffixes = [s for _, s in word_suffix_pairs]
+
+        # Filter to all matching suffixes at once
+        matches = df.filter(pl.col("hash").is_in(suffixes))
+        match_dict = {row["hash"]: row for row in matches.iter_rows(named=True)}
+
+        for word, suffix in word_suffix_pairs:
+            if suffix in match_dict:
+                row = match_dict[suffix]
+                results[word] = FrequencyData(
+                    peak_tf=row["peak_tf"],
+                    peak_df=row["peak_df"],
+                    sum_tf=row["sum_tf"],
+                    sum_df=row["sum_df"],
+                )
+            else:
+                results[word] = None
+
+    return results

gngram_lookup-0.2.0/pyproject.toml

@@ -0,0 +1,60 @@
+[tool.poetry]
+name = "gngram-lookup"
+packages = [{include = "gngram_counter"}]
+version = "0.2.0"
+description = "Static Hash-Based Lookup for Google Ngram Frequencies"
+authors = ["Craig Trim <craigtrim@gmail.com>"]
+maintainers = ["Craig Trim <craigtrim@gmail.com>"]
+readme = "README.md"
+license = "Proprietary"
+exclude = ["parquet-hash/", "builder/"]
+
+keywords = [
+  "ngram",
+  "google-ngram",
+  "nlp",
+  "natural-language-processing",
+  "frequency",
+  "linguistics",
+]
+
+repository = "https://github.com/craigtrim/gngram-lookup"
+
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Topic :: Text Processing :: Linguistic",
+  "Natural Language :: English",
+  "Operating System :: OS Independent",
+]
+
+[tool.poetry.scripts]
+gngram-exists = "gngram_counter.cli:gngram_exists"
+gngram-freq = "gngram_counter.cli:gngram_freq"
+
+[tool.poetry.build]
+generate-setup-file = true
+
+[tool.poetry.dependencies]
+python = "^3.11"
+polars = "^1.0"
+pyarrow = "^18.0"
+
+[tool.poetry.dev-dependencies]
+autopep8 = "*"
+ruff = "*"
+pre-commit = "^2.20.0"
+pytest = "*"
+
+[build-system]
+build-backend = "poetry.core.masonry.api"
+requires = ["poetry-core>=1.0.0"]
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+ignore = ["F403", "F405"]

gngram_lookup-0.2.0/setup.py

@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+from setuptools import setup
+
+packages = \
+['gngram_counter']
+
+package_data = \
+{'': ['*']}
+
+install_requires = \
+['polars>=1.0,<2.0', 'pyarrow>=18.0,<19.0']
+
+entry_points = \
+{'console_scripts': ['gngram-exists = gngram_counter.cli:gngram_exists',
+                     'gngram-freq = gngram_counter.cli:gngram_freq']}
+
+setup_kwargs = {
+    'name': 'gngram-lookup',
+    'version': '0.2.0',
+    'description': 'Static Hash-Based Lookup for Google Ngram Frequencies',
+    'long_description': "# gngram-lookup\n\n[![PyPI version](https://badge.fury.io/py/gngram-lookup.svg)](https://badge.fury.io/py/gngram-lookup)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)\n\nWord frequency from 500 years of books. O(1) lookup. 5 million words.\n\n## Install\n\n```bash\npip install gngram-lookup\npython -m gngram_lookup.download_data\n```\n\n## Python\n\n```python\nimport gngram_lookup as ng\n\nng.exists('computer')       # True\nng.exists('xyznotaword')    # False\n\nng.frequency('computer')\n# {'peak_tf': 2000, 'peak_df': 2000, 'sum_tf': 892451, 'sum_df': 312876}\n\nng.batch_frequency(['the', 'algorithm', 'xyznotaword'])\n# {'the': {...}, 'algorithm': {...}, 'xyznotaword': None}\n```\n\n## CLI\n\n```bash\ngngram-exists computer    # True, exit 0\ngngram-exists xyznotaword # False, exit 1\n\ngngram-freq computer\n# peak_tf_decade: 2000\n# peak_df_decade: 2000\n# sum_tf: 892451\n# sum_df: 312876\n```\n\n## Docs\n\n- [API Reference](docs/api.md)\n- [CLI Reference](docs/cli.md)\n- [Data Format](docs/data-format.md)\n- [Use Cases](docs/use-cases.md)\n- [Development](docs/development.md)\n\n## Attribution\n\nData derived from the [Google Books Ngram](https://books.google.com/ngrams) dataset.\n\n## License\n\nProprietary. See [LICENSE](LICENSE).\n",
+    'author': 'Craig Trim',
+    'author_email': 'craigtrim@gmail.com',
+    'maintainer': 'Craig Trim',
+    'maintainer_email': 'craigtrim@gmail.com',
+    'url': 'https://github.com/craigtrim/gngram-lookup',
+    'packages': packages,
+    'package_data': package_data,
+    'install_requires': install_requires,
+    'entry_points': entry_points,
+    'python_requires': '>=3.11,<4.0',
+}
+
+
+setup(**setup_kwargs)