keyflux 0.1.0__tar.gz

keyflux-0.1.0/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.3
+Name: keyflux
+Version: 0.1.0
+Summary: Corpus keyness, rank-turbulence divergence, and allotaxonographs
+Keywords: nlp,keyness,keywords,corpus-linguistics,rank-turbulence-divergence,allotaxonometry,text-analysis,computational-humanities
+Author: Zoltan Varju, Orsolya Putz
+Author-email: Zoltan Varju <zoltan.varju@crowintelligence.org>, Orsolya Putz <orsolya.putz@crowintelligence.org>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Visualization
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Dist: numpy>=1.26
+Requires-Dist: matplotlib>=3.8
+Requires-Dist: pytest>=8 ; extra == 'dev'
+Requires-Dist: pytest-cov ; extra == 'dev'
+Requires-Dist: hypothesis>=6.100 ; extra == 'dev'
+Requires-Dist: ruff>=0.4 ; extra == 'dev'
+Requires-Dist: ty ; extra == 'dev'
+Requires-Dist: mutmut>=3 ; extra == 'dev'
+Requires-Dist: mkdocs-material ; extra == 'docs'
+Requires-Dist: mkdocstrings[python] ; extra == 'docs'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://crowintelligence.org/
+Project-URL: Repository, https://github.com/crow-intelligence/keyflux
+Project-URL: Documentation, https://keyflux.readthedocs.io
+Provides-Extra: dev
+Provides-Extra: docs
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/crow-intelligence/keyflux/main/imgs/logo.svg" alt="keyflux logo" width="440">
+</p>
+
+# keyflux
+
+Corpus keyness, rank-turbulence divergence, and allotaxonographs — in pure Python.
+
+keyflux owns the whole comparison arc that diachronic and comparative discourse
+analysis usually splits across tools and languages. It derives **keywords** and
+**lockwords** from a focus-versus-reference comparison using proper corpus-linguistic
+measures (log-likelihood for significance, log ratio for effect size — not just
+chi-square), compares the resulting ranked lists with **rank-turbulence divergence
+(RTD)**, and renders the **allotaxonograph**: the rank-rank map plus the ranked list
+of which exact words drove the shift. No JavaScript runtime — figures are matplotlib.
+
+It replaces the usual "Jaccard overlap on the top-N keywords" summary — one opaque
+number that throws away rank, everything below the cutoff, and any account of *which*
+words moved — with a transparent, pip-installable pipeline.
+
+## Installation
+
+```bash
+uv add keyflux
+```
+
+## Quickstart
+
+```python
+from collections import Counter
+
+from keyflux import Keyness, RankedList, rtd, allotaxonograph
+
+# 1. Keyness: focus vs reference
+focus = Counter({"climate": 30, "carbon": 12, "the": 80, "policy": 9})
+reference = Counter({"climate": 3, "carbon": 1, "the": 78, "market": 15})
+k = Keyness(focus, reference, measure="log_likelihood")
+keywords = k.keywords(top=20)
+lockwords = k.lockwords()
+
+# 2. Rank-turbulence divergence between two ranked lists
+r1 = RankedList.from_counts(focus, label="2019")
+r2 = RankedList.from_counts(reference, label="2024")
+result = rtd(r1, r2, alpha=1 / 3)
+print(result.divergence)
+
+# 3. Allotaxonograph (returns a matplotlib Figure)
+fig = allotaxonograph(r1, r2, alpha=1 / 3, labels=("2019", "2024"))
+fig.savefig("allotaxonograph.png")
+```
+
+## Features
+
+- **Keyness measures**: log-likelihood (Dunning), log ratio, Simple Maths, %DIFF, and
+  chi-square (for contrast) — significance flagged against the chi-square thresholds
+- **Keywords and lockwords**: positive / negative keywords plus the stable lockword zone
+- **Rank-turbulence divergence**: tunable, rank-sensitive corpus comparison with
+  per-type contributions and an explicit alpha-to-zero log limit
+- **Allotaxonograph**: publication-quality two-panel matplotlib figure, no JS runtime
+- **Reproducibility records**: every keyness result emits its reference, cutoffs, and measure
+
+## Documentation
+
+Full documentation — quickstart, the keyness and allotaxonograph tutorials,
+troubleshooting, and the complete API reference — is at
+[keyflux.readthedocs.io](https://keyflux.readthedocs.io). The sources live in `docs/`.
+
+## Roadmap
+
+Planned for the next iteration. The robustness items are analysed in detail in
+[`PRE-MORTEM.md`](PRE-MORTEM.md), and the open modelling choices are listed in
+[`CHANGES_SUMMARY.md`](CHANGES_SUMMARY.md).
+
+**Robustness / API decisions**
+
+- [ ] Revisit the zero-cell floor default (0.5): it sets the effect size of every exclusive keyword and reorders the top of the list.
+- [ ] Decide whether `min_focus_freq` / `min_reference_freq` should default asymmetrically (keep focus-exclusive keywords while demanding more reference evidence).
+- [ ] Add Cohen's *d* (dispersion-aware effect size) once the corpus input can carry sub-corpus structure.
+
+**Proposed features**
+
+- [ ] `RankedList.from_keyness(..., by="score")` — rank by keyness score, not just frequency, so "compare the distinctive-word lists over time" is a one-liner.
+- [ ] Optional self-contained interactive HTML+JS allotaxonograph export (an alpha slider), gated behind an extra so the core stays pure Python.
+
+**Maintenance**
+
+- [ ] Publish to PyPI and wire up ReadTheDocs.
+
+## Made by
+
+keyflux is made by [Crow Intelligence](https://crowintelligence.org/).
+
+## License
+
+MIT

keyflux-0.1.0/README.md

@@ -0,0 +1,95 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/crow-intelligence/keyflux/main/imgs/logo.svg" alt="keyflux logo" width="440">
+</p>
+
+# keyflux
+
+Corpus keyness, rank-turbulence divergence, and allotaxonographs — in pure Python.
+
+keyflux owns the whole comparison arc that diachronic and comparative discourse
+analysis usually splits across tools and languages. It derives **keywords** and
+**lockwords** from a focus-versus-reference comparison using proper corpus-linguistic
+measures (log-likelihood for significance, log ratio for effect size — not just
+chi-square), compares the resulting ranked lists with **rank-turbulence divergence
+(RTD)**, and renders the **allotaxonograph**: the rank-rank map plus the ranked list
+of which exact words drove the shift. No JavaScript runtime — figures are matplotlib.
+
+It replaces the usual "Jaccard overlap on the top-N keywords" summary — one opaque
+number that throws away rank, everything below the cutoff, and any account of *which*
+words moved — with a transparent, pip-installable pipeline.
+
+## Installation
+
+```bash
+uv add keyflux
+```
+
+## Quickstart
+
+```python
+from collections import Counter
+
+from keyflux import Keyness, RankedList, rtd, allotaxonograph
+
+# 1. Keyness: focus vs reference
+focus = Counter({"climate": 30, "carbon": 12, "the": 80, "policy": 9})
+reference = Counter({"climate": 3, "carbon": 1, "the": 78, "market": 15})
+k = Keyness(focus, reference, measure="log_likelihood")
+keywords = k.keywords(top=20)
+lockwords = k.lockwords()
+
+# 2. Rank-turbulence divergence between two ranked lists
+r1 = RankedList.from_counts(focus, label="2019")
+r2 = RankedList.from_counts(reference, label="2024")
+result = rtd(r1, r2, alpha=1 / 3)
+print(result.divergence)
+
+# 3. Allotaxonograph (returns a matplotlib Figure)
+fig = allotaxonograph(r1, r2, alpha=1 / 3, labels=("2019", "2024"))
+fig.savefig("allotaxonograph.png")
+```
+
+## Features
+
+- **Keyness measures**: log-likelihood (Dunning), log ratio, Simple Maths, %DIFF, and
+  chi-square (for contrast) — significance flagged against the chi-square thresholds
+- **Keywords and lockwords**: positive / negative keywords plus the stable lockword zone
+- **Rank-turbulence divergence**: tunable, rank-sensitive corpus comparison with
+  per-type contributions and an explicit alpha-to-zero log limit
+- **Allotaxonograph**: publication-quality two-panel matplotlib figure, no JS runtime
+- **Reproducibility records**: every keyness result emits its reference, cutoffs, and measure
+
+## Documentation
+
+Full documentation — quickstart, the keyness and allotaxonograph tutorials,
+troubleshooting, and the complete API reference — is at
+[keyflux.readthedocs.io](https://keyflux.readthedocs.io). The sources live in `docs/`.
+
+## Roadmap
+
+Planned for the next iteration. The robustness items are analysed in detail in
+[`PRE-MORTEM.md`](PRE-MORTEM.md), and the open modelling choices are listed in
+[`CHANGES_SUMMARY.md`](CHANGES_SUMMARY.md).
+
+**Robustness / API decisions**
+
+- [ ] Revisit the zero-cell floor default (0.5): it sets the effect size of every exclusive keyword and reorders the top of the list.
+- [ ] Decide whether `min_focus_freq` / `min_reference_freq` should default asymmetrically (keep focus-exclusive keywords while demanding more reference evidence).
+- [ ] Add Cohen's *d* (dispersion-aware effect size) once the corpus input can carry sub-corpus structure.
+
+**Proposed features**
+
+- [ ] `RankedList.from_keyness(..., by="score")` — rank by keyness score, not just frequency, so "compare the distinctive-word lists over time" is a one-liner.
+- [ ] Optional self-contained interactive HTML+JS allotaxonograph export (an alpha slider), gated behind an extra so the core stays pure Python.
+
+**Maintenance**
+
+- [ ] Publish to PyPI and wire up ReadTheDocs.
+
+## Made by
+
+keyflux is made by [Crow Intelligence](https://crowintelligence.org/).
+
+## License
+
+MIT

keyflux-0.1.0/pyproject.toml

@@ -0,0 +1,99 @@
+[project]
+name = "keyflux"
+version = "0.1.0"
+description = "Corpus keyness, rank-turbulence divergence, and allotaxonographs"
+readme = "README.md"
+authors = [
+    { name = "Zoltan Varju", email = "zoltan.varju@crowintelligence.org" },
+    { name = "Orsolya Putz", email = "orsolya.putz@crowintelligence.org" },
+]
+requires-python = ">=3.11"
+license = { text = "MIT" }
+keywords = [
+    "nlp",
+    "keyness",
+    "keywords",
+    "corpus-linguistics",
+    "rank-turbulence-divergence",
+    "allotaxonometry",
+    "text-analysis",
+    "computational-humanities",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Scientific/Engineering :: Visualization",
+    "Topic :: Text Processing :: Linguistic",
+]
+dependencies = [
+    "numpy>=1.26",
+    "matplotlib>=3.8",
+]
+
+[project.urls]
+Homepage = "https://crowintelligence.org/"
+Repository = "https://github.com/crow-intelligence/keyflux"
+Documentation = "https://keyflux.readthedocs.io"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8",
+    "pytest-cov",
+    "hypothesis>=6.100",
+    "ruff>=0.4",
+    "ty",
+    "mutmut>=3",
+]
+docs = [
+    "mkdocs-material",
+    "mkdocstrings[python]",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.22,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 88
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "ANN", "D"]
+ignore = ["D105", "D107"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D100", "D102", "D103", "D104", "ANN"]
+"examples/**" = ["D100", "ANN"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.pytest.ini_options]
+addopts = "--doctest-modules --tb=short"
+testpaths = ["src", "tests"]
+
+# Mutation testing (dev-only, not run in CI). Scoped to the math-heavy core and
+# its fast, plotting-free test files. Run with `uv run mutmut run`.
+[tool.mutmut]
+source_paths = ["src/keyflux"]
+only_mutate = [
+    "src/keyflux/keyness/measures.py",
+    "src/keyflux/keyness/keyness.py",
+    "src/keyflux/ranking/rankedlist.py",
+    "src/keyflux/divergence/rtd.py",
+]
+pytest_add_cli_args_test_selection = [
+    "tests/test_measures.py",
+    "tests/test_keyness.py",
+    "tests/test_rankedlist.py",
+    "tests/test_rtd.py",
+]
+pytest_add_cli_args = ["-p", "no:cacheprovider"]
+
+[tool.ty]

keyflux-0.1.0/src/keyflux/__init__.py

@@ -0,0 +1,30 @@
+"""Keyflux — keyness, rank-turbulence divergence, and allotaxonographs."""
+
+__version__ = "0.1.0"
+
+from keyflux.divergence import Contribution, RTDResult, rtd
+from keyflux.io.corpus import counts_from_text, counts_from_tokens, load_counts
+from keyflux.keyness import (
+    Keyness,
+    KeynessRow,
+    KeywordTable,
+    ReproRecord,
+)
+from keyflux.ranking import RankedList
+from keyflux.viz import allotaxonograph
+
+__all__ = [
+    "Keyness",
+    "KeynessRow",
+    "KeywordTable",
+    "ReproRecord",
+    "RankedList",
+    "rtd",
+    "RTDResult",
+    "Contribution",
+    "allotaxonograph",
+    "counts_from_tokens",
+    "counts_from_text",
+    "load_counts",
+    "__version__",
+]

keyflux-0.1.0/src/keyflux/_types.py

@@ -0,0 +1,31 @@
+"""Shared type aliases for the keyflux package."""
+
+from collections import Counter
+from typing import Literal, TypeAlias
+
+Token: TypeAlias = str
+"""A surface or lemmatised word type."""
+
+Count: TypeAlias = int
+"""A raw frequency count for a single type."""
+
+FreqTable: TypeAlias = Counter[str]
+"""Type-to-count mapping for one corpus (focus or reference)."""
+
+Rank: TypeAlias = float
+"""A 1-based rank; float because tied and tied-last ranks are averaged."""
+
+MeasureName: TypeAlias = Literal[
+    "log_likelihood",
+    "log_ratio",
+    "simple_maths",
+    "percent_diff",
+    "chi_square",
+]
+"""Identifier selecting a keyness scoring function."""
+
+Significance: TypeAlias = Literal["ns", "p05", "p01", "p001", "p0001"]
+"""Significance band from a log-likelihood / chi-square statistic (1 d.f.)."""
+
+Direction: TypeAlias = Literal["positive", "negative", "neutral"]
+"""Keyness polarity: over-represented, under-represented, or neither in focus."""

keyflux-0.1.0/src/keyflux/datasets/__init__.py

@@ -0,0 +1,104 @@
+"""Tiny bundled corpora and fixtures for docs and tests.
+
+The data lives inline as Python dicts so it is always importable in doctests
+with no package-data or ``importlib.resources`` machinery.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from keyflux.ranking.rankedlist import RankedList
+
+# The jkbren/rank-turbulence-divergence reference example. Both systems rank the
+# same seven elements; rtd at alpha=1.0 is 0.45924793111057804.
+_JKBREN_FOCUS: dict[str, int] = {
+    "a": 20,
+    "e": 14,
+    "c": 8,
+    "b": 7,
+    "f": 4,
+    "g": 2,
+    "d": 1,
+}
+_JKBREN_REFERENCE: dict[str, int] = {
+    "b": 24,
+    "a": 16,
+    "e": 5,
+    "d": 4,
+    "c": 3,
+    "f": 2,
+    "g": 1,
+}
+
+_DEMO_FOCUS: dict[str, int] = {
+    "climate": 42,
+    "carbon": 28,
+    "emissions": 19,
+    "warming": 14,
+    "policy": 11,
+    "the": 320,
+    "of": 180,
+    "and": 165,
+    "to": 150,
+    "energy": 22,
+    "renewable": 9,
+    "global": 17,
+}
+_DEMO_REFERENCE: dict[str, int] = {
+    "market": 40,
+    "stock": 26,
+    "trade": 21,
+    "profit": 13,
+    "policy": 12,
+    "the": 318,
+    "of": 176,
+    "and": 170,
+    "to": 148,
+    "energy": 8,
+    "shares": 15,
+    "global": 16,
+}
+
+
+def load_demo_pair() -> tuple[Counter[str], Counter[str]]:
+    """Return the bundled (focus, reference) demo corpus pair.
+
+    A tiny climate-discourse focus corpus versus a finance-discourse reference
+    corpus, with shared function words and a couple of lockword-like overlaps.
+
+    Returns:
+        ``(focus, reference)`` frequency Counters.
+
+    Examples:
+        >>> focus, reference = load_demo_pair()
+        >>> focus["climate"], reference["market"]
+        (42, 40)
+    """
+    return Counter(_DEMO_FOCUS), Counter(_DEMO_REFERENCE)
+
+
+def load_jkbren_example() -> tuple[RankedList, RankedList]:
+    """Return the jkbren rank-turbulence-divergence regression pair.
+
+    Two ranked lists over the same seven elements. Their rank-turbulence
+    divergence at ``alpha=1.0`` is ``0.45924793111057804`` — the regression
+    anchor from the reference implementation.
+
+    Returns:
+        ``(list1, list2)`` as :class:`keyflux.ranking.rankedlist.RankedList`.
+
+    Examples:
+        >>> from keyflux.divergence import rtd
+        >>> r1, r2 = load_jkbren_example()
+        >>> round(rtd(r1, r2, alpha=1.0).divergence, 6)
+        0.459248
+    """
+    from keyflux.ranking.rankedlist import RankedList
+
+    return (
+        RankedList.from_counts(_JKBREN_FOCUS, label="system 1"),
+        RankedList.from_counts(_JKBREN_REFERENCE, label="system 2"),
+    )

keyflux-0.1.0/src/keyflux/divergence/__init__.py

@@ -0,0 +1,5 @@
+"""Divergence: rank-turbulence divergence between two ranked lists."""
+
+from keyflux.divergence.rtd import Contribution, RTDResult, rtd
+
+__all__ = ["rtd", "RTDResult", "Contribution"]