pycorpdiff 0.1.0a9__tar.gz → 0.1.0a11__tar.gz

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to `pycorpdiff` are documented in this file. The format
 follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
 project adheres to [Semantic Versioning](https://semver.org/).
 
-## [0.1.0a9] — first public release
+## [0.1.0a11] — first public release
 
 The first public alpha of `pycorpdiff` — comparative corpus analysis
 for modern Python workflows. Three public verbs (`compare`, `track`,
@@ -20,13 +20,13 @@ and notebook rendering.
 
 - **Keyness**: signed log-likelihood G² with selectable formula
   (`formula="rayson"` 2-cell shortcut, default; matches the UCREL
-  LL Wizard. `formula="dunning"` 4-cell G²; matches NLTK +
-  `quanteda::textstat_keyness(measure="lr")` byte-for-byte.). Pearson
-  χ², Hardie LogRatio, Gabrielatos %DIFF, BIC-approximated Bayes
-  factor (also tracks the `formula=` choice), Juilland D / Gries DP
-  dispersion flagging, Benjamini–Hochberg correction, stop-word
-  filtering, empirical permutation *p*-values, N-way contingency G²
-  via `keyness_multi`.
+  LL Wizard. `formula="dunning"` 4-cell G²; the canonical Dunning
+  1993 form used by NLTK and R's `quanteda::textstat_keyness(measure="lr")`).
+  Pearson χ², Hardie LogRatio, Gabrielatos %DIFF, BIC-approximated
+  Bayes factor (also tracks the `formula=` choice), Juilland D /
+  Gries DP dispersion flagging, Benjamini–Hochberg correction,
+  stop-word filtering, empirical permutation *p*-values, N-way
+  contingency G² via `keyness_multi`.
 - **Collocations**: logDice, PMI, t-score, MI³ with Laplace smoothing;
   cross-corpus `collocation_shift`; co-occurrence networks via
   `cooccurrence_network`.
@@ -49,8 +49,6 @@ The package is checked against standard tools by automated test:
   on every adjacent bigram (slow tier).
 - **Scattertext (Kessler 2017)** — behavioural agreement on the 2012
   US Conventions corpus (slow tier).
-- **quanteda (R)** via `rpy2` — G² agreement to ≤ 1e-6 with
-  `formula="dunning"` (cross-runtime float-arithmetic ceiling; slow tier).
 - **HistWords (Hamilton et al. 2016)** — known-shifter / stable-word
   sanity check on Stanford SNAP COHA decade embeddings; skips
   gracefully when the archive isn't reachable (slow tier).

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/CITATION.cff

@@ -4,7 +4,7 @@ message: >
   entry. GitHub renders a "Cite this repository" widget directly from
   this file.
 title: "pycorpdiff: Comparative Corpus Analysis for Modern Python Workflows"
-version: 0.1.0a9
+version: 0.1.0a11
 date-released: 2026-05-25
 authors:
   - family-names: Turner

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pycorpdiff
-Version: 0.1.0a9
+Version: 0.1.0a11
 Summary: Comparative corpus analysis for Python: keyness, collocations, semantic shift, temporal trajectories with changepoints + causal inference.
 Project-URL: Homepage, https://github.com/jturner-uofl/pycorpdiff
 Project-URL: Documentation, https://github.com/jturner-uofl/pycorpdiff
@@ -131,7 +131,7 @@ points — one-line adapters, no plugin registry. The base install's
 direct runtime dependencies are `numpy`, `pandas`, `scipy`, and
 `pyarrow`; everything else is opt-in via extras.
 
-> **Status: alpha (0.1.0a9).** Public API is stable for the features
+> **Status: alpha (0.1.0a11).** Public API is stable for the features
 > described below; on PyPI as `pip install pycorpdiff`.
 
 ## The three-layer architecture
@@ -252,8 +252,6 @@ Slow tier:
   on every adjacent bigram
 - **Scattertext (Kessler 2017)** — behavioural agreement on the 2012
   US Conventions corpus
-- **quanteda (R)** via `rpy2` — G² agreement to ≤ 1e-6 with
-  `formula="dunning"` (cross-runtime float-arithmetic ceiling)
 - **HistWords (Hamilton et al. 2016)** — known-shifter / stable-word
   sanity check on Stanford SNAP COHA decade embeddings (skips
   gracefully if the archive isn't reachable)

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/README.md

@@ -35,7 +35,7 @@ points — one-line adapters, no plugin registry. The base install's
 direct runtime dependencies are `numpy`, `pandas`, `scipy`, and
 `pyarrow`; everything else is opt-in via extras.
 
-> **Status: alpha (0.1.0a9).** Public API is stable for the features
+> **Status: alpha (0.1.0a11).** Public API is stable for the features
 > described below; on PyPI as `pip install pycorpdiff`.
 
 ## The three-layer architecture
@@ -156,8 +156,6 @@ Slow tier:
   on every adjacent bigram
 - **Scattertext (Kessler 2017)** — behavioural agreement on the 2012
   US Conventions corpus
-- **quanteda (R)** via `rpy2` — G² agreement to ≤ 1e-6 with
-  `formula="dunning"` (cross-runtime float-arithmetic ceiling)
 - **HistWords (Hamilton et al. 2016)** — known-shifter / stable-word
   sanity check on Stanford SNAP COHA decade embeddings (skips
   gracefully if the archive isn't reachable)

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pycorpdiff"
-version = "0.1.0a9"
+version = "0.1.0a11"
 description = "Comparative corpus analysis for Python: keyness, collocations, semantic shift, temporal trajectories with changepoints + causal inference."
 readme = "README.md"
 license = { file = "LICENSE" }

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/src/pycorpdiff/__init__.py

@@ -20,7 +20,7 @@ True
 
 from __future__ import annotations
 
-__version__ = "0.1.0a9"
+__version__ = "0.1.0a11"
 
 from .collocation.network import NetworkResult, cooccurrence_network
 from .compare import Comparison, compare

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/src/pycorpdiff/collocation/measures.py

@@ -16,14 +16,19 @@ NumPy / pandas conventions.
 
 References
 ----------
-Rychlý, P. (2008). A lexicographer-friendly association score. In
-*Proceedings of RASLAN 2008*.
+Church, K. W., & Hanks, P. (1990). Word association norms, mutual
+information, and lexicography. *Computational Linguistics*, 16(1),
+22-29. (Pointwise mutual information for collocation.)
 
 Church, K., Gale, W., Hanks, P., & Hindle, D. (1991). Using statistics in
-lexical analysis. In *Lexical Acquisition*, 115-164.
+lexical analysis. In *Lexical Acquisition*, 115-164. (t-score.)
 
 Daille, B. (1994). *Approche mixte pour l'extraction automatique de
-terminologie*. PhD thesis, Université Paris 7.
+terminologie*. PhD thesis, Université Paris 7. (MI³ — cube weighting
+of PMI to correct rare-pair inflation.)
+
+Rychlý, P. (2008). A lexicographer-friendly association score. In
+*Proceedings of RASLAN 2008*. (logDice.)
 """
 
 from __future__ import annotations

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/src/pycorpdiff/keyness/correction.py

@@ -1,4 +1,17 @@
-"""Multiple-comparison correction for keyness *p*-value vectors."""
+"""Multiple-comparison correction for keyness *p*-value vectors.
+
+References
+----------
+Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery
+rate: A practical and powerful approach to multiple testing. *Journal
+of the Royal Statistical Society: Series B*, 57(1), 289-300.
+(BH-adjusted *p*-values; the FDR control used by default.)
+
+Bonferroni, C. E. (1936). Teoria statistica delle classi e calcolo
+delle probabilità. *Pubblicazioni del R Istituto Superiore di Scienze
+Economiche e Commerciali di Firenze*, 8, 3-62. (Family-wise correction;
+opt-in via ``multiple_comparisons="bonferroni"``.)
+"""
 
 from __future__ import annotations
 

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/src/pycorpdiff/semantic/shift.py

@@ -1,14 +1,29 @@
 """Semantic shift and neighborhood drift between corpora.
 
-The default strategy is *averaged contextual embeddings*: for each
-occurrence of the target term in a corpus, encode its surrounding
-window as a sentence, then average across occurrences. The
-corpus-specific representation that comes out is what we compare.
+The default strategy is *averaged contextual embeddings*
+(Giulianelli, Del Tredici, & Fernández 2020): for each occurrence of
+the target term in a corpus, encode its surrounding window as a
+sentence, then average across occurrences. The corpus-specific
+representation that comes out is what we compare.
 
 This works with any shared-space embedder (SBERT, multilingual SBERT,
 HuggingFace encoders). For Hamilton-style independently-trained
-embeddings, supply ``align="procrustes"`` to rotate the source space
-onto the target space before comparison.
+embeddings (Hamilton, Leskovec, & Jurafsky 2016), supply
+``align="procrustes"`` to rotate the source space onto the target
+space before comparison (Schönemann 1966).
+
+References
+----------
+Giulianelli, M., Del Tredici, M., & Fernández, R. (2020). Analysing
+lexical semantic change with contextualised word representations. In
+*Proceedings of ACL 2020*, 3960-3973.
+
+Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016). Diachronic word
+embeddings reveal statistical laws of semantic change. In *Proceedings
+of ACL 2016*.
+
+Schönemann, P. H. (1966). A generalized solution of the orthogonal
+Procrustes problem. *Psychometrika*, 31(1), 1-10.
 """
 
 from __future__ import annotations

{pycorpdiff-0.1.0a9 → pycorpdiff-0.1.0a11}/src/pycorpdiff/temporal/causal_impact.py

@@ -1,4 +1,4 @@
-"""Bayesian counterfactual causal impact (Brodersen et al. 2015).
+"""Bayesian counterfactual causal impact — no-control variant.
 
 The :func:`interrupted_time_series` module answers "is there a step
 discontinuity at this known event?" via segmented OLS. The harder —
@@ -7,20 +7,35 @@ looked like *without* the event?" That's the counterfactual, and the
 gap between observed reality and counterfactual prediction is the
 causal effect of the event.
 
-Method: Bayesian structural time series (BSTS) — a state-space model
-with a local linear trend fit on the pre-event window, then projected
-forward as the counterfactual for the post-event window. Implemented
-via :class:`statsmodels.tsa.UnobservedComponents`. The credible
-intervals on the pointwise and cumulative effects come from Monte
-Carlo simulation against the joint posterior of the state-space
+Method: a univariate structural time-series counterfactual. A local
+linear trend (Bayesian state-space) model is fit on the pre-event
+window and projected forward as the counterfactual for the post-event
+window. Implemented via :class:`statsmodels.tsa.UnobservedComponents`.
+Credible intervals on the pointwise and cumulative effects come from
+Monte Carlo simulation against the joint posterior of the state-space
 filter — anchored at the end of the pre-event training data and rolled
 forward through the post-event horizon.
 
+**Scope and caveats.** This is the *no-control* variant of the
+Brodersen et al. (2015) framework: pycorpdiff observes only the target
+series itself, not a panel of unaffected control series. The canonical
+Google ``CausalImpact`` implementation uses BSTS with spike-and-slab
+regression on parallel control series; that machinery is not in scope
+here. The univariate version is appropriate when no obvious control
+series exists (the common case in corpus-linguistic event studies),
+but its counterfactual relies entirely on extrapolation of the pre-
+event trend — be wary of distribution shifts unrelated to the event.
+
 Reference
 ---------
 Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L.
 (2015). Inferring causal impact using Bayesian structural time-series
 models. *Annals of Applied Statistics*, 9(1), 247-274.
+(pycorpdiff implements a no-control simplification of this framework.)
+
+Harvey, A. C. (1989). *Forecasting, Structural Time Series Models and
+the Kalman Filter*. Cambridge University Press. (Local linear trend
+state-space specification.)
 """
 
 from __future__ import annotations

pycorpdiff-0.1.0a9/tests/integration/test_crossval_quanteda.py

@@ -1,153 +0,0 @@
-"""Cross-validation against R's quanteda (Benoit et al. 2018).
-
-quanteda's ``textstat_keyness(measure="lr")`` computes the same
-Dunning log-likelihood our :func:`pycorpdiff.keyness.log_likelihood`
-does. With identical inputs they should produce byte-identical G²
-values modulo floating-point representation.
-
-This is the highest-credibility cross-validation we can ship: pycorpdiff
-demonstrably agrees with the R reference implementation on the math,
-on the same fixture, to 6 decimals. It's the receipt that turns
-"the math is correct" into "the math agrees with the standard tool".
-
-Requirements
-------------
-
-- R installed (any 4.x)
-- ``install.packages("quanteda")`` from a CRAN mirror
-- ``pip install rpy2``
-
-Skips silently if rpy2 isn't installed *or* if R doesn't have quanteda.
-"""
-
-from __future__ import annotations
-
-import math
-
-import pandas as pd
-import pytest
-
-import pycorpdiff as pcd
-
-rpy2 = pytest.importorskip("rpy2")
-rpy2_robjects = pytest.importorskip("rpy2.robjects")
-
-pytestmark = pytest.mark.slow
-
-
-def _r_has_quanteda() -> bool:
-    """Probe whether quanteda is installed in this R environment."""
-    try:
-        from rpy2.robjects.packages import importr
-
-        importr("quanteda")
-        importr("quanteda.textstats")
-        return True
-    except Exception:  # pragma: no cover - environment-dependent
-        return False
-
-
-@pytest.fixture(scope="module")
-def fixture_corpus() -> pcd.Corpus:
-    """A small two-class fixture with clean signal."""
-    rows = [
-        {"text": "the migrant worker arrived and settled here peacefully", "frame": "A"},
-        {"text": "the migrant family thrived in our welcoming community", "frame": "A"},
-        {"text": "the migrant community grew through worker organisation", "frame": "A"},
-        {"text": "the migrant family and worker rights advanced together", "frame": "A"},
-        {"text": "the migrant criminal threat grew unchecked at our borders", "frame": "B"},
-        {"text": "the migrant invasion of criminal gangs spread rapidly here", "frame": "B"},
-        {"text": "the migrant criminal element alarmed residents nationwide", "frame": "B"},
-        {"text": "the migrant gangs threaten the border and the criminal risk", "frame": "B"},
-    ]
-    return pcd.from_dataframe(pd.DataFrame(rows), text_col="text", meta_cols=("frame",))
-
-
-def _quanteda_keyness(corpus_df: pd.DataFrame) -> pd.DataFrame:
-    """Run quanteda's textstat_keyness(measure='lr') on the same corpus.
-
-    Returns a DataFrame with columns ``feature`` and ``g2`` (the
-    log-likelihood). Sign matches quanteda's convention.
-    """
-    if not _r_has_quanteda():
-        pytest.skip("quanteda not installed in the R environment")
-
-    from rpy2.robjects import default_converter, pandas2ri, r
-    from rpy2.robjects.conversion import localconverter
-
-    # rpy2 >= 3.5 deprecated pandas2ri.activate() in favour of a
-    # context-managed converter. Within `localconverter(...)` the
-    # `pandas2ri` converter is registered, so `r("out_df")` auto-converts
-    # the R data.frame to a pandas DataFrame on the way out; no explicit
-    # `pandas2ri.rpy2py(...)` call is needed (that's the deprecated
-    # pre-context-manager idiom and now raises NotImplementedError when
-    # the object is already pandas).
-    with localconverter(default_converter + pandas2ri.converter):
-        r("library(quanteda)")
-        r("library(quanteda.textstats)")
-        r.assign("docs_df", pandas2ri.py2rpy(corpus_df))
-
-        r(
-            """
-            cps <- corpus(docs_df, text_field = "text")
-            toks <- tokens(cps, remove_punct = TRUE)
-            dfm_obj <- dfm(toks, tolower = TRUE)
-            keyness <- textstat_keyness(
-                dfm_obj,
-                target = which(docvars(cps, "frame") == "A"),
-                measure = "lr"
-            )
-            out_df <- as.data.frame(keyness)
-            """
-        )
-        out = r("out_df")  # auto-converted to pandas via localconverter
-
-    out = out.rename(columns={"G2": "g2"})
-    return out[["feature", "g2"]]
-
-
-def test_log_likelihood_matches_quanteda_byte_for_byte(
-    fixture_corpus: pcd.Corpus,
-) -> None:
-    """For every term shared with quanteda (using formula='dunning'),
-    our signed G² agrees to ≤ 1e-6.
-
-    quanteda's ``textstat_keyness(measure="lr")`` uses the full 4-cell
-    Dunning G². The Rayson 2-cell shortcut (our default) is a different
-    statistic; comparing like-to-like requires passing ``formula="dunning"``.
-
-    The 1e-6 tolerance reflects the realistic cross-runtime
-    floating-point ceiling between R (BLAS-via-R) and NumPy
-    (BLAS-via-Python); accumulation order in xlogy differs subtly
-    between the two stacks. Tightening below ~1e-7 produces sporadic
-    failures on otherwise-identical math.
-    """
-    a = fixture_corpus.slice(frame="A")
-    b = fixture_corpus.slice(frame="B")
-    ours = (
-        pcd.compare(a, b)
-        .keyness(min_count=1, formula="dunning")
-        .table.set_index("term")["g2"]
-    )
-
-    quanteda_df = _quanteda_keyness(fixture_corpus.docs.copy())
-    theirs = pd.Series(
-        quanteda_df["g2"].to_numpy(), index=quanteda_df["feature"].to_numpy()
-    )
-
-    shared = set(ours.index) & set(theirs.index)
-    assert len(shared) >= 5, (
-        f"too few shared terms for a meaningful comparison ({len(shared)})"
-    )
-
-    for term in shared:
-        ours_v = float(ours[term])
-        theirs_v = float(theirs[term])
-        # quanteda's textstat_keyness uses signed G² with the same
-        # convention we do: positive when overused in the target
-        # group. With matching formulae, the two implementations
-        # agree to cross-runtime float-arithmetic noise (~1e-7 on
-        # this fixture, well below the 1e-6 ceiling we assert here).
-        assert math.isclose(ours_v, theirs_v, abs_tol=1e-6), (
-            f"{term}: pycorpdiff={ours_v}, quanteda={theirs_v}"
-        )