exactcond 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- exactcond-0.2.0/PKG-INFO +86 -0
- exactcond-0.2.0/README.md +76 -0
- exactcond-0.2.0/pyproject.toml +15 -0
- exactcond-0.2.0/setup.cfg +4 -0
- exactcond-0.2.0/setup.py +34 -0
- exactcond-0.2.0/src/exactcond/__init__.py +15 -0
- exactcond-0.2.0/src/exactcond/_core.py +356 -0
- exactcond-0.2.0/src/exactcond/exactcond.c +598 -0
- exactcond-0.2.0/src/exactcond.egg-info/PKG-INFO +86 -0
- exactcond-0.2.0/src/exactcond.egg-info/SOURCES.txt +11 -0
- exactcond-0.2.0/src/exactcond.egg-info/dependency_links.txt +1 -0
- exactcond-0.2.0/src/exactcond.egg-info/top_level.txt +1 -0
- exactcond-0.2.0/tests/test_exactcond.py +142 -0
exactcond-0.2.0/PKG-INFO
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: exactcond
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Exact conditional reference distributions for tables and selected linear models
|
|
5
|
+
Author-email: "M. Theodor Loots" <theo.loots@gmail.com>
|
|
6
|
+
License: GPL-3.0-or-later
|
|
7
|
+
Project-URL: Homepage, https://github.com/mtloots/exactcond
|
|
8
|
+
Requires-Python: >=3.8
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
|
|
11
|
+
# exactcond
|
|
12
|
+
|
|
13
|
+
Exact conditional reference distributions for tables and for linear models whose structure was
|
|
14
|
+
selected from the same data. Pure C back-end, shared byte for byte with the Python package of the
|
|
15
|
+
same name.
|
|
16
|
+
|
|
17
|
+
## The one idea
|
|
18
|
+
|
|
19
|
+
A nuisance parameter is removed by conditioning on a statistic sufficient for it. What survives
|
|
20
|
+
carries no nuisance, so a reference drawn from it is exact at every sample size, with no appeal to
|
|
21
|
+
an asymptotic approximation and no bootstrap.
|
|
22
|
+
|
|
23
|
+
## Two conditionings
|
|
24
|
+
|
|
25
|
+
**Both margins fixed.** In an r by c table the row and column totals are sufficient for the
|
|
26
|
+
marginal parameters, so conditioning on both leaves the Fisher-Yates law. It is sampled exactly,
|
|
27
|
+
and by an argument needing no appeal to Patefield's algorithm: the conditional law given both
|
|
28
|
+
margins is precisely the law of the table formed by pairing a fixed vector of row labels with a
|
|
29
|
+
uniformly random permutation of the column labels.
|
|
30
|
+
|
|
31
|
+
**Direction only.** Under a null in which the tested structure does not enter the data-generating
|
|
32
|
+
process, a statistic invariant to the regression coefficients and to the error scale depends on the
|
|
33
|
+
data only through the residual direction, which is uniform on the sphere of the residual space
|
|
34
|
+
whatever those nuisance parameters are. The reference stays exact when the statistic is maximised
|
|
35
|
+
over candidates, and so remains valid after a spatial weight matrix has been selected by fit.
|
|
36
|
+
|
|
37
|
+
## Why the dispersion statistic
|
|
38
|
+
|
|
39
|
+
For nominal rows against an ordinal column, `stat = "dispersion"` is the between-row dispersion of
|
|
40
|
+
the mean score. Its components are the row mean scores themselves, so where the column score is a
|
|
41
|
+
signed difference the statistic is assembled out of the very contrasts being reported, and a reader
|
|
42
|
+
can follow the arithmetic from the table to the test. A rank test on such a variable answers a
|
|
43
|
+
different question, and answers it on data that are mostly ties.
|
|
44
|
+
|
|
45
|
+
## Example
|
|
46
|
+
|
|
47
|
+
```r
|
|
48
|
+
library(exactcond)
|
|
49
|
+
tab <- matrix(c(3, 228, 72, 10, 548, 67, 7, 276, 131), nrow = 3, byrow = TRUE)
|
|
50
|
+
ec_rc_test(tab, v = c(-1, 0, 1), B = 200000, seed = 4207)
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Notes on honesty
|
|
54
|
+
|
|
55
|
+
`ec_rc_test()` reports the largest null draw alongside the p-value, and says so when the observed
|
|
56
|
+
statistic exceeded every draw: that is a resolution floor, not a p-value, and the two should not be
|
|
57
|
+
written the same way. Every p-value comes with a Wilson interval for the Monte Carlo error in the
|
|
58
|
+
number of draws actually taken.
|
|
59
|
+
|
|
60
|
+
`ec_rc_two()` samples the paired law directly rather than convolving two separately estimated null
|
|
61
|
+
distributions. The convolution needs the observed score to be reachable under no association; for a
|
|
62
|
+
strongly associated table it is not, at any number of draws, and the failure arrives precisely when
|
|
63
|
+
both tables are strongly associated, which is the case in which the correct answer is a large
|
|
64
|
+
p-value.
|
|
65
|
+
|
|
66
|
+
`ec_multi_test()` answers "does ANY of these items differ across the groups" with ONE exact test:
|
|
67
|
+
the sum or the maximum over items of a per-item statistic, referred to the law obtained by permuting
|
|
68
|
+
the group labels across respondents or, when a cluster is given, across clusters that keep their
|
|
69
|
+
members. This is Pesarin's nonparametric combination with the direct and Tippett combining functions: one
|
|
70
|
+
p-value for the whole battery, so nothing to correct for multiplicity, and the per-item statistics
|
|
71
|
+
referred to the same permutations for a closed follow-up in the manner of Westfall and Young. The sum has power against
|
|
72
|
+
many items moving a little, the maximum against one moving a lot, and the cluster permutation is the
|
|
73
|
+
exact treatment of an interviewer or fieldworker effect, at negligible cost when there is none.
|
|
74
|
+
|
|
75
|
+
`ec_diffnet_test()` carries the same idea to a network. A differential network normally rests on a
|
|
76
|
+
Gaussian null; here the edge statistic is unchanged and only its reference is, from a chi-square to
|
|
77
|
+
a permutation of the group labels, which is exact whatever the marginals are and distribution-free
|
|
78
|
+
on ranks. One p-value for the whole network, per-edge p-values from the same permutations, and a
|
|
79
|
+
cluster permutation for interviewer effects. The route is the established one (NetCoMi's `diffnet`,
|
|
80
|
+
the Network Comparison Test, exchangeability blocks); what is here is the implementation.
|
|
81
|
+
|
|
82
|
+
## Reproducibility
|
|
83
|
+
|
|
84
|
+
Every replicate draws from its own stream, seeded from the base seed and the replicate index, so
|
|
85
|
+
results do not depend on the thread count and are identical across the R and Python fronts.
|
|
86
|
+
`parity_exactcond.sh` checks that agreement to fifteen significant figures.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# exactcond
|
|
2
|
+
|
|
3
|
+
Exact conditional reference distributions for tables and for linear models whose structure was
|
|
4
|
+
selected from the same data. Pure C back-end, shared byte for byte with the Python package of the
|
|
5
|
+
same name.
|
|
6
|
+
|
|
7
|
+
## The one idea
|
|
8
|
+
|
|
9
|
+
A nuisance parameter is removed by conditioning on a statistic sufficient for it. What survives
|
|
10
|
+
carries no nuisance, so a reference drawn from it is exact at every sample size, with no appeal to
|
|
11
|
+
an asymptotic approximation and no bootstrap.
|
|
12
|
+
|
|
13
|
+
## Two conditionings
|
|
14
|
+
|
|
15
|
+
**Both margins fixed.** In an r by c table the row and column totals are sufficient for the
|
|
16
|
+
marginal parameters, so conditioning on both leaves the Fisher-Yates law. It is sampled exactly,
|
|
17
|
+
and by an argument needing no appeal to Patefield's algorithm: the conditional law given both
|
|
18
|
+
margins is precisely the law of the table formed by pairing a fixed vector of row labels with a
|
|
19
|
+
uniformly random permutation of the column labels.
|
|
20
|
+
|
|
21
|
+
**Direction only.** Under a null in which the tested structure does not enter the data-generating
|
|
22
|
+
process, a statistic invariant to the regression coefficients and to the error scale depends on the
|
|
23
|
+
data only through the residual direction, which is uniform on the sphere of the residual space
|
|
24
|
+
whatever those nuisance parameters are. The reference stays exact when the statistic is maximised
|
|
25
|
+
over candidates, and so remains valid after a spatial weight matrix has been selected by fit.
|
|
26
|
+
|
|
27
|
+
## Why the dispersion statistic
|
|
28
|
+
|
|
29
|
+
For nominal rows against an ordinal column, `stat = "dispersion"` is the between-row dispersion of
|
|
30
|
+
the mean score. Its components are the row mean scores themselves, so where the column score is a
|
|
31
|
+
signed difference the statistic is assembled out of the very contrasts being reported, and a reader
|
|
32
|
+
can follow the arithmetic from the table to the test. A rank test on such a variable answers a
|
|
33
|
+
different question, and answers it on data that are mostly ties.
|
|
34
|
+
|
|
35
|
+
## Example
|
|
36
|
+
|
|
37
|
+
```r
|
|
38
|
+
library(exactcond)
|
|
39
|
+
tab <- matrix(c(3, 228, 72, 10, 548, 67, 7, 276, 131), nrow = 3, byrow = TRUE)
|
|
40
|
+
ec_rc_test(tab, v = c(-1, 0, 1), B = 200000, seed = 4207)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Notes on honesty
|
|
44
|
+
|
|
45
|
+
`ec_rc_test()` reports the largest null draw alongside the p-value, and says so when the observed
|
|
46
|
+
statistic exceeded every draw: that is a resolution floor, not a p-value, and the two should not be
|
|
47
|
+
written the same way. Every p-value comes with a Wilson interval for the Monte Carlo error in the
|
|
48
|
+
number of draws actually taken.
|
|
49
|
+
|
|
50
|
+
`ec_rc_two()` samples the paired law directly rather than convolving two separately estimated null
|
|
51
|
+
distributions. The convolution needs the observed score to be reachable under no association; for a
|
|
52
|
+
strongly associated table it is not, at any number of draws, and the failure arrives precisely when
|
|
53
|
+
both tables are strongly associated, which is the case in which the correct answer is a large
|
|
54
|
+
p-value.
|
|
55
|
+
|
|
56
|
+
`ec_multi_test()` answers "does ANY of these items differ across the groups" with ONE exact test:
|
|
57
|
+
the sum or the maximum over items of a per-item statistic, referred to the law obtained by permuting
|
|
58
|
+
the group labels across respondents or, when a cluster is given, across clusters that keep their
|
|
59
|
+
members. This is Pesarin's nonparametric combination with the direct and Tippett combining functions: one
|
|
60
|
+
p-value for the whole battery, so nothing to correct for multiplicity, and the per-item statistics
|
|
61
|
+
referred to the same permutations for a closed follow-up in the manner of Westfall and Young. The sum has power against
|
|
62
|
+
many items moving a little, the maximum against one moving a lot, and the cluster permutation is the
|
|
63
|
+
exact treatment of an interviewer or fieldworker effect, at negligible cost when there is none.
|
|
64
|
+
|
|
65
|
+
`ec_diffnet_test()` carries the same idea to a network. A differential network normally rests on a
|
|
66
|
+
Gaussian null; here the edge statistic is unchanged and only its reference is, from a chi-square to
|
|
67
|
+
a permutation of the group labels, which is exact whatever the marginals are and distribution-free
|
|
68
|
+
on ranks. One p-value for the whole network, per-edge p-values from the same permutations, and a
|
|
69
|
+
cluster permutation for interviewer effects. The route is the established one (NetCoMi's `diffnet`,
|
|
70
|
+
the Network Comparison Test, exchangeability blocks); what is here is the implementation.
|
|
71
|
+
|
|
72
|
+
## Reproducibility
|
|
73
|
+
|
|
74
|
+
Every replicate draws from its own stream, seeded from the base seed and the replicate index, so
|
|
75
|
+
results do not depend on the thread count and are identical across the R and Python fronts.
|
|
76
|
+
`parity_exactcond.sh` checks that agreement to fifteen significant figures.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "exactcond"
|
|
7
|
+
version = "0.2.0"
|
|
8
|
+
description = "Exact conditional reference distributions for tables and selected linear models"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.8"
|
|
11
|
+
license = {text = "GPL-3.0-or-later"}
|
|
12
|
+
authors = [{name = "M. Theodor Loots", email = "theo.loots@gmail.com"}]
|
|
13
|
+
|
|
14
|
+
[project.urls]
|
|
15
|
+
Homepage = "https://github.com/mtloots/exactcond"
|
exactcond-0.2.0/setup.py
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
"""Build the shared exact-conditional C back-end at INSTALL time, as an Extension.
|
|
2
|
+
|
|
3
|
+
The earlier setup.py subclassed setuptools' build_py and compiled with distutils' new_compiler.
|
|
4
|
+
distutils is gone from Python 3.12 and current setuptools no longer supports that subclassing, so
|
|
5
|
+
pip failed with "type object 'build_py' has no attribute '__getattr__'" on every platform (12 Oct
|
|
6
|
+
2026). Declaring the sources as an Extension hands the compile to pip, inside build isolation, and
|
|
7
|
+
lets real platform wheels be built, after which users need no compiler at all. Same pattern as
|
|
8
|
+
arcstat-py.
|
|
9
|
+
|
|
10
|
+
init.c is excluded deliberately: it is R's symbol-registration glue and references R headers.
|
|
11
|
+
"""
|
|
12
|
+
import glob
|
|
13
|
+
import os
|
|
14
|
+
|
|
15
|
+
from setuptools import Extension, setup
|
|
16
|
+
|
|
17
|
+
# Paths must be RELATIVE to setup.py: setuptools rejects absolute source paths.
|
|
18
|
+
SRC = os.path.join("src", "exactcond")
|
|
19
|
+
|
|
20
|
+
sources = sorted(
|
|
21
|
+
s for s in glob.glob(os.path.join(SRC, "*.c"))
|
|
22
|
+
if os.path.basename(s) != "init.c"
|
|
23
|
+
)
|
|
24
|
+
|
|
25
|
+
# Leading underscore: it carries no PyInit and is loaded with ctypes.CDLL by _core.py.
|
|
26
|
+
ext = Extension(
|
|
27
|
+
name="exactcond._libexactcond",
|
|
28
|
+
sources=sources,
|
|
29
|
+
include_dirs=[SRC],
|
|
30
|
+
libraries=[] if os.name == "nt" else ["m"],
|
|
31
|
+
optional=False,
|
|
32
|
+
)
|
|
33
|
+
|
|
34
|
+
setup(ext_modules=[ext])
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
"""exactcond: exact conditional reference distributions for tables and selected linear models.
|
|
2
|
+
|
|
3
|
+
Reference distributions exact at every sample size, obtained by conditioning on a statistic
|
|
4
|
+
sufficient for the nuisance parameter rather than by an asymptotic approximation: the Fisher-Yates
|
|
5
|
+
law of a two-way table given both margins, sampled by random permutation of the column labels; a
|
|
6
|
+
two-table form conditioning further on the total score to cancel a common association parameter;
|
|
7
|
+
and the uniform law of the residual direction on the sphere, which stays exact when the statistic
|
|
8
|
+
is maximised over candidate structures and so remains valid after selection by fit. ctypes binding
|
|
9
|
+
to a shared C back end also used by the R package 'exactcond'. Pure standard library.
|
|
10
|
+
"""
|
|
11
|
+
from ._core import (ec_scores, ec_rc_null, ec_rc_test, ec_rc_two, ec_sphere_maxquad, ec_pvalue,
|
|
12
|
+
ec_multi_test, ec_diffnet_test)
|
|
13
|
+
__all__ = ["ec_scores", "ec_rc_null", "ec_rc_test", "ec_rc_two", "ec_sphere_maxquad", "ec_pvalue",
|
|
14
|
+
"ec_multi_test", "ec_diffnet_test"]
|
|
15
|
+
__version__ = "0.2.0"
|
|
@@ -0,0 +1,356 @@
|
|
|
1
|
+
"""exactcond: Python (ctypes) binding to the shared C back-end. Pure standard library.
|
|
2
|
+
|
|
3
|
+
The same C sources back the R package 'exactcond'; the parity harness in the repository requires
|
|
4
|
+
every exported quantity to agree byte for byte between the two fronts, which holds because the
|
|
5
|
+
random stream lives in C and is seeded per replicate rather than drawn from either language's
|
|
6
|
+
generator.
|
|
7
|
+
|
|
8
|
+
LOADING has two paths, as in the other packages in this programme. An INSTALLED package carries a
|
|
9
|
+
library that pip built at install time, so no compiler is needed here. A SOURCE CHECKOUT has the
|
|
10
|
+
.c file but no library, so it is compiled on demand into a writable cache directory, which keeps
|
|
11
|
+
the edit-a-C-file-and-re-import loop working without writing into site-packages.
|
|
12
|
+
"""
|
|
13
|
+
import ctypes as _ct
|
|
14
|
+
import glob as _glob
|
|
15
|
+
import os as _os
|
|
16
|
+
import sys as _sys
|
|
17
|
+
|
|
18
|
+
_here = _os.path.dirname(_os.path.abspath(__file__))
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
def _find_built():
|
|
22
|
+
for pat in ("_libexactcond*.so", "_libexactcond*.dylib", "_libexactcond*.pyd",
|
|
23
|
+
"_libexactcond*.dll"):
|
|
24
|
+
hits = sorted(_glob.glob(_os.path.join(_here, pat)))
|
|
25
|
+
if hits:
|
|
26
|
+
return hits[0]
|
|
27
|
+
return None
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
def _cache_dir():
|
|
31
|
+
if _os.access(_here, _os.W_OK):
|
|
32
|
+
return _here
|
|
33
|
+
base = _os.environ.get("XDG_CACHE_HOME") or _os.path.join(_os.path.expanduser("~"), ".cache")
|
|
34
|
+
d = _os.path.join(base, "exactcond")
|
|
35
|
+
_os.makedirs(d, exist_ok=True)
|
|
36
|
+
return d
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def _ext():
|
|
40
|
+
return "dll" if _sys.platform == "win32" else ("dylib" if _sys.platform == "darwin" else "so")
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def _compile_from_source():
|
|
44
|
+
import subprocess as _sp
|
|
45
|
+
srcs = sorted(f for f in _glob.glob(_os.path.join(_here, "*.c"))
|
|
46
|
+
if _os.path.basename(f) != "init.c")
|
|
47
|
+
if not srcs:
|
|
48
|
+
return None
|
|
49
|
+
out = _os.path.join(_cache_dir(), "libexactcond." + _ext())
|
|
50
|
+
cc = _os.environ.get("CC") or "cc"
|
|
51
|
+
try:
|
|
52
|
+
_sp.check_call([cc, "-O2", "-fPIC", "-shared", "-I", _here] + srcs + ["-lm", "-o", out])
|
|
53
|
+
except (OSError, _sp.CalledProcessError) as exc:
|
|
54
|
+
raise ImportError(
|
|
55
|
+
"exactcond could not build its C back-end from the bundled sources (%s). In an "
|
|
56
|
+
"installed copy this should never happen, because pip builds the library at install "
|
|
57
|
+
"time. In a source checkout a C compiler is required; set CC if it is not 'cc'." % exc)
|
|
58
|
+
return out
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
_libpath = _find_built()
|
|
62
|
+
if _libpath is None:
|
|
63
|
+
_cached = _os.path.join(_cache_dir(), "libexactcond." + _ext())
|
|
64
|
+
_libpath = _cached if _os.path.exists(_cached) else _compile_from_source()
|
|
65
|
+
if _libpath is None:
|
|
66
|
+
raise ImportError("exactcond: no compiled back-end and no C sources to build one from.")
|
|
67
|
+
|
|
68
|
+
_lib = _ct.CDLL(_libpath)
|
|
69
|
+
_dp, _ip = _ct.POINTER(_ct.c_double), _ct.POINTER(_ct.c_int)
|
|
70
|
+
|
|
71
|
+
for _f, _sig in (
|
|
72
|
+
("C_ec_rc_null", [_ip, _ip, _ip, _ip, _ip, _ip, _dp, _dp, _ip, _dp]),
|
|
73
|
+
("C_ec_rc_test", [_ip, _ip, _ip, _ip, _ip, _dp, _dp, _ip, _dp]),
|
|
74
|
+
("C_ec_rc_two", [_ip, _ip, _ip, _ip, _ip, _ip, _ip, _dp, _dp, _dp, _ip, _dp]),
|
|
75
|
+
("C_ec_sphere_maxquad", [_dp, _dp, _ip, _ip, _ip, _ip, _dp]),
|
|
76
|
+
("C_ec_pvalue", [_dp, _dp, _ip, _dp]),
|
|
77
|
+
("C_ec_multi", [_ip, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _dp, _dp, _dp]),
|
|
78
|
+
("C_ec_diffnet", [_dp, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _ip, _dp, _dp, _dp]),
|
|
79
|
+
):
|
|
80
|
+
getattr(_lib, _f).argtypes = _sig
|
|
81
|
+
getattr(_lib, _f).restype = None
|
|
82
|
+
|
|
83
|
+
_STATS = {"linlin": 0, "dispersion": 1, "pearson": 2, "lr": 3}
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def _d(xs):
|
|
87
|
+
xs = list(xs)
|
|
88
|
+
return (_ct.c_double * len(xs))(*[float(x) for x in xs])
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
def _i(xs):
|
|
92
|
+
xs = list(xs)
|
|
93
|
+
return (_ct.c_int * len(xs))(*[int(x) for x in xs])
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def _one(x):
|
|
97
|
+
return _ct.byref(_ct.c_int(int(x)))
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def _oned(x):
|
|
101
|
+
return _ct.byref(_ct.c_double(float(x)))
|
|
102
|
+
|
|
103
|
+
|
|
104
|
+
def _flat(x):
|
|
105
|
+
"""Accept a nested sequence or a flat one and return (rows, cols, flat row-major list)."""
|
|
106
|
+
rows = list(x)
|
|
107
|
+
if not rows:
|
|
108
|
+
raise ValueError("empty table")
|
|
109
|
+
if not hasattr(rows[0], "__len__"):
|
|
110
|
+
raise ValueError("expected a two-dimensional table")
|
|
111
|
+
c = len(rows[0])
|
|
112
|
+
if any(len(r) != c for r in rows):
|
|
113
|
+
raise ValueError("ragged table")
|
|
114
|
+
return len(rows), c, [int(v) for r in rows for v in r]
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def ec_scores(k, centre=False):
|
|
118
|
+
"""Integer scores for an ordinal margin.
|
|
119
|
+
|
|
120
|
+
The dispersion and linear-by-linear statistics are unchanged in distribution by an affine
|
|
121
|
+
transformation of the scores, so centring alters the reported statistic but never the p-value.
|
|
122
|
+
"""
|
|
123
|
+
s = [float(i) for i in range(int(k))]
|
|
124
|
+
if centre:
|
|
125
|
+
m = sum(s) / len(s)
|
|
126
|
+
s = [v - m for v in s]
|
|
127
|
+
return s
|
|
128
|
+
|
|
129
|
+
|
|
130
|
+
def _scores(u, v, r, c):
|
|
131
|
+
u = ec_scores(r) if u is None else [float(t) for t in u]
|
|
132
|
+
v = ec_scores(c) if v is None else [float(t) for t in v]
|
|
133
|
+
if len(u) != r:
|
|
134
|
+
raise ValueError("u must have one score per row, so length %d" % r)
|
|
135
|
+
if len(v) != c:
|
|
136
|
+
raise ValueError("v must have one score per column, so length %d" % c)
|
|
137
|
+
return u, v
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
def ec_rc_null(rows, cols, stat="dispersion", u=None, v=None, B=20000, seed=1):
|
|
141
|
+
"""Draws of a table statistic from the law conditional on BOTH margins."""
|
|
142
|
+
rows = [int(t) for t in rows]
|
|
143
|
+
cols = [int(t) for t in cols]
|
|
144
|
+
if sum(rows) != sum(cols):
|
|
145
|
+
raise ValueError("the row and column totals must agree")
|
|
146
|
+
uu, vv = _scores(u, v, len(rows), len(cols))
|
|
147
|
+
out = (_ct.c_double * int(B))()
|
|
148
|
+
_lib.C_ec_rc_null(_i(rows), _i(cols), _one(len(rows)), _one(len(cols)), _one(B),
|
|
149
|
+
_one(_STATS[stat]), _d(uu), _d(vv), _one(seed), out)
|
|
150
|
+
return list(out)
|
|
151
|
+
|
|
152
|
+
|
|
153
|
+
def ec_rc_test(x, stat="dispersion", u=None, v=None, B=20000, seed=1):
|
|
154
|
+
"""Exact conditional test in a two-way table, conditioning on both margins.
|
|
155
|
+
|
|
156
|
+
Returns a dict with the observed statistic, the exact p-value and its mid-p companion, a
|
|
157
|
+
Wilson interval reflecting the Monte Carlo error in B draws, the largest null draw, and the
|
|
158
|
+
row mean scores, which for the dispersion statistic ARE the contrasts being reported.
|
|
159
|
+
"""
|
|
160
|
+
r, c, flat = _flat(x)
|
|
161
|
+
if any(t < 0 for t in flat):
|
|
162
|
+
raise ValueError("x must hold non-negative counts")
|
|
163
|
+
uu, vv = _scores(u, v, r, c)
|
|
164
|
+
out = (_ct.c_double * 6)()
|
|
165
|
+
_lib.C_ec_rc_test(_i(flat), _one(r), _one(c), _one(B), _one(_STATS[stat]),
|
|
166
|
+
_d(uu), _d(vv), _one(seed), out)
|
|
167
|
+
n = [sum(flat[i * c:(i + 1) * c]) for i in range(r)]
|
|
168
|
+
means = [sum(vv[j] * flat[i * c + j] for j in range(c)) / n[i] if n[i] else float("nan")
|
|
169
|
+
for i in range(r)]
|
|
170
|
+
return {"statistic": out[0], "p_value": out[1], "midp": out[2], "mc_lower": out[3],
|
|
171
|
+
"mc_upper": out[4], "max_null": out[5], "row_means": means, "n": n,
|
|
172
|
+
"B": int(B), "stat": stat}
|
|
173
|
+
|
|
174
|
+
|
|
175
|
+
def ec_rc_two(t1, t2, u=None, v=None, B=20000, burn=20000, thin=20, alpha=0.05, seed=1):
|
|
176
|
+
"""Exact conditional test that two tables share a linear-by-linear association parameter.
|
|
177
|
+
|
|
178
|
+
Under the null of a common parameter the joint density carries it only through the total score,
|
|
179
|
+
which the conditioning fixes, so it leaves the density altogether. The law is sampled by moves
|
|
180
|
+
making a two by two swap in each table with equal and opposite score changes, starting at the
|
|
181
|
+
observed pair -- which is what keeps it usable when both tables are strongly associated, the
|
|
182
|
+
case in which convolving two separately estimated null distributions fails outright.
|
|
183
|
+
"""
|
|
184
|
+
r, c, f1 = _flat(t1)
|
|
185
|
+
r2, c2, f2 = _flat(t2)
|
|
186
|
+
if (r, c) != (r2, c2):
|
|
187
|
+
raise ValueError("the two tables must have the same dimension")
|
|
188
|
+
if r < 2 or c < 2:
|
|
189
|
+
raise ValueError("both tables must have at least two rows and two columns")
|
|
190
|
+
uu, vv = _scores(u, v, r, c)
|
|
191
|
+
if any(abs(t - round(t)) > 1e-9 for t in uu + vv):
|
|
192
|
+
raise ValueError("ec_rc_two needs integer scores, so the total score lies on a lattice")
|
|
193
|
+
out = (_ct.c_double * 5)()
|
|
194
|
+
_lib.C_ec_rc_two(_i(f1), _i(f2), _one(r), _one(c), _one(B), _one(burn), _one(thin),
|
|
195
|
+
_d(uu), _d(vv), _oned(alpha), _one(seed), out)
|
|
196
|
+
if out[0] < 0:
|
|
197
|
+
raise RuntimeError("the sampler could not be initialised on these tables")
|
|
198
|
+
return {"midp": out[0], "size": out[1], "T1": out[2], "S": out[3], "accept": out[4],
|
|
199
|
+
"B": int(B), "burn": int(burn), "thin": int(thin), "alpha": float(alpha)}
|
|
200
|
+
|
|
201
|
+
|
|
202
|
+
def ec_sphere_maxquad(M, A, B=20000, seed=1):
|
|
203
|
+
"""Exact null of a quadratic form maximised over candidate structures.
|
|
204
|
+
|
|
205
|
+
M is the residual maker; A is one symmetric matrix or a list of them. Each draw takes the
|
|
206
|
+
maximum INSIDE the draw, which is what makes the reference valid after a structure has been
|
|
207
|
+
chosen by fit.
|
|
208
|
+
"""
|
|
209
|
+
rows = [list(r) for r in M]
|
|
210
|
+
n = len(rows)
|
|
211
|
+
if any(len(r) != n for r in rows):
|
|
212
|
+
raise ValueError("M must be square")
|
|
213
|
+
if not hasattr(A[0][0], "__len__"):
|
|
214
|
+
A = [A]
|
|
215
|
+
flat = []
|
|
216
|
+
for a in A:
|
|
217
|
+
ar = [list(r) for r in a]
|
|
218
|
+
if len(ar) != n or any(len(r) != n for r in ar):
|
|
219
|
+
raise ValueError("every candidate must be n by n")
|
|
220
|
+
for i in range(n):
|
|
221
|
+
for j in range(n):
|
|
222
|
+
flat.append(float(ar[i][j]))
|
|
223
|
+
Mf = [float(rows[i][j]) for i in range(n) for j in range(n)]
|
|
224
|
+
out = (_ct.c_double * int(B))()
|
|
225
|
+
_lib.C_ec_sphere_maxquad(_d(Mf), _d(flat), _one(n), _one(len(A)), _one(B), _one(seed), out)
|
|
226
|
+
return list(out)
|
|
227
|
+
|
|
228
|
+
|
|
229
|
+
def ec_multi_test(x, group, cluster=None, stat="pearson", B=20000, seed=1):
|
|
230
|
+
"""One exact test for a battery of categorical items across groups.
|
|
231
|
+
|
|
232
|
+
x is a sequence of items (each a sequence of integer codes from zero, or from one, over the
|
|
233
|
+
respondents) or a sequence of respondent rows; group gives each respondent's group; cluster,
|
|
234
|
+
if given, each respondent's cluster, every cluster inside one group, in which case the labels
|
|
235
|
+
are permuted across clusters. The sum and the maximum over items of the per-item statistic
|
|
236
|
+
(Pearson on the group-by-level table, or the dispersion of group mean scores) are each
|
|
237
|
+
referred to the same permutations, giving one p-value for the battery, and the per-item
|
|
238
|
+
statistics are referred to those permutations too, for a closed follow-up.
|
|
239
|
+
"""
|
|
240
|
+
rows = [list(r) for r in x]
|
|
241
|
+
n_g = len(list(group))
|
|
242
|
+
if len(rows) == n_g and (not rows or hasattr(rows[0], "__len__")) and len(rows[0]) != n_g:
|
|
243
|
+
items = [[int(r[j]) for r in rows] for j in range(len(rows[0]))] # respondent rows
|
|
244
|
+
else:
|
|
245
|
+
items = [[int(v) for v in r] for r in rows] # item columns
|
|
246
|
+
n = len(items[0]); m = len(items)
|
|
247
|
+
if any(len(c) != n for c in items) or n_g != n:
|
|
248
|
+
raise ValueError("every item and the group vector must have one entry per respondent")
|
|
249
|
+
items = [[v - 1 for v in c] if min(c) == 1 else c for c in items]
|
|
250
|
+
if any(min(c) < 0 for c in items):
|
|
251
|
+
raise ValueError("items must be coded from zero or one")
|
|
252
|
+
k = [max(c) + 1 for c in items]
|
|
253
|
+
levels = sorted(set(group)); grp = [levels.index(v) for v in group]; g = len(levels)
|
|
254
|
+
if g < 2:
|
|
255
|
+
raise ValueError("group must have at least two levels")
|
|
256
|
+
if cluster is None:
|
|
257
|
+
cl = [0] * n; ncl = 0
|
|
258
|
+
else:
|
|
259
|
+
cluster = list(cluster)
|
|
260
|
+
if len(cluster) != n:
|
|
261
|
+
raise ValueError("cluster must have one entry per respondent")
|
|
262
|
+
clev = sorted(set(cluster)); cl = [clev.index(v) for v in cluster]; ncl = len(clev)
|
|
263
|
+
seen = {}
|
|
264
|
+
for c, a in zip(cl, grp):
|
|
265
|
+
if seen.setdefault(c, a) != a:
|
|
266
|
+
raise ValueError("every cluster must lie inside one group")
|
|
267
|
+
which = {"pearson": 0, "dispersion": 1}[stat]
|
|
268
|
+
out = (_ct.c_double * 12)(); ist = (_ct.c_double * m)(); ip = (_ct.c_double * m)()
|
|
269
|
+
_lib.C_ec_multi(_i([v for c in items for v in c]), _one(n), _one(m), _one(g), _i(grp), _i(k),
|
|
270
|
+
_i(cl), _one(ncl), _one(which), _one(B), _one(seed), out, ist, ip)
|
|
271
|
+
def part(i, j, mx): # obs at i; p, midp, lo, hi from j
|
|
272
|
+
return {"statistic": out[i], "p_value": out[j], "midp": out[j + 1],
|
|
273
|
+
"mc_lower": out[j + 2], "mc_upper": out[j + 3], "max_null": mx}
|
|
274
|
+
return {"sum": part(0, 2, out[10]), "max": part(1, 6, out[11]),
|
|
275
|
+
"items": [{"statistic": ist[j], "p_value": ip[j]} for j in range(m)],
|
|
276
|
+
"n": [grp.count(a) for a in range(g)], "clusters": ncl, "B": int(B), "stat": stat,
|
|
277
|
+
"exchangeable": "clusters" if ncl > 0 else "respondents"}
|
|
278
|
+
|
|
279
|
+
|
|
280
|
+
def _ranks(v):
|
|
281
|
+
"""Average ranks, ties shared, which is what makes the correlation Spearman's."""
|
|
282
|
+
order = sorted(range(len(v)), key=lambda i: v[i])
|
|
283
|
+
out = [0.0] * len(v)
|
|
284
|
+
i = 0
|
|
285
|
+
while i < len(order):
|
|
286
|
+
j = i
|
|
287
|
+
while j + 1 < len(order) and v[order[j + 1]] == v[order[i]]:
|
|
288
|
+
j += 1
|
|
289
|
+
r = (i + j) / 2.0 + 1.0
|
|
290
|
+
for t in range(i, j + 1):
|
|
291
|
+
out[order[t]] = r
|
|
292
|
+
i = j + 1
|
|
293
|
+
return out
|
|
294
|
+
|
|
295
|
+
|
|
296
|
+
def ec_diffnet_test(x, group, cluster=None, rank=True, B=20000, seed=1):
|
|
297
|
+
"""A differential network without a Gaussian assumption.
|
|
298
|
+
|
|
299
|
+
x is a sequence of variables (each a sequence over the respondents) or of respondent rows.
|
|
300
|
+
The edge statistic is the heterogeneity of the Fisher-transformed correlations across groups,
|
|
301
|
+
the same quantity the Gaussian theory uses, referred to the law obtained by permuting the group
|
|
302
|
+
labels rather than to a chi-square; with rank=True the correlations are Spearman's and the
|
|
303
|
+
procedure is distribution-free. Returns the sum and the maximum over edges, each with one
|
|
304
|
+
p-value for the whole network, and the per-edge statistics and p-values from those same
|
|
305
|
+
permutations for a closed follow-up.
|
|
306
|
+
"""
|
|
307
|
+
rows = [list(r) for r in x]
|
|
308
|
+
n_g = len(list(group))
|
|
309
|
+
if len(rows) == n_g and rows and hasattr(rows[0], "__len__") and len(rows[0]) != n_g:
|
|
310
|
+
cols = [[float(r[j]) for r in rows] for j in range(len(rows[0]))]
|
|
311
|
+
else:
|
|
312
|
+
cols = [[float(v) for v in r] for r in rows]
|
|
313
|
+
p = len(cols); n = len(cols[0])
|
|
314
|
+
if p < 2:
|
|
315
|
+
raise ValueError("x must hold at least two variables")
|
|
316
|
+
if any(len(c) != n for c in cols) or n_g != n:
|
|
317
|
+
raise ValueError("every variable and the group vector must have one entry per respondent")
|
|
318
|
+
if rank:
|
|
319
|
+
cols = [_ranks(c) for c in cols]
|
|
320
|
+
levels = sorted(set(group)); grp = [levels.index(v) for v in group]; g = len(levels)
|
|
321
|
+
if g < 2:
|
|
322
|
+
raise ValueError("group must have at least two levels")
|
|
323
|
+
if min(grp.count(a) for a in range(g)) < 4:
|
|
324
|
+
raise ValueError("every group needs at least four respondents")
|
|
325
|
+
if cluster is None:
|
|
326
|
+
cl = [0] * n; ncl = 0
|
|
327
|
+
else:
|
|
328
|
+
cluster = list(cluster)
|
|
329
|
+
if len(cluster) != n:
|
|
330
|
+
raise ValueError("cluster must have one entry per respondent")
|
|
331
|
+
clev = sorted(set(cluster)); cl = [clev.index(v) for v in cluster]; ncl = len(clev)
|
|
332
|
+
seen = {}
|
|
333
|
+
for c, a in zip(cl, grp):
|
|
334
|
+
if seen.setdefault(c, a) != a:
|
|
335
|
+
raise ValueError("every cluster must lie inside one group")
|
|
336
|
+
m = p * (p - 1) // 2
|
|
337
|
+
out = (_ct.c_double * 12)(); est = (_ct.c_double * m)(); ep = (_ct.c_double * m)()
|
|
338
|
+
_lib.C_ec_diffnet(_d([v for c in cols for v in c]), _one(n), _one(p), _one(g), _i(grp),
|
|
339
|
+
_i(cl), _one(ncl), _one(B), _one(seed), out, est, ep)
|
|
340
|
+
def part(i, j, mx):
|
|
341
|
+
return {"statistic": out[i], "p_value": out[j], "midp": out[j + 1],
|
|
342
|
+
"mc_lower": out[j + 2], "mc_upper": out[j + 3], "max_null": mx}
|
|
343
|
+
pairs = [(a, b) for a in range(p) for b in range(a + 1, p)]
|
|
344
|
+
return {"sum": part(0, 2, out[10]), "max": part(1, 6, out[11]),
|
|
345
|
+
"edges": [{"from": a, "to": b, "statistic": est[e], "p_value": ep[e]}
|
|
346
|
+
for e, (a, b) in enumerate(pairs)],
|
|
347
|
+
"n": [grp.count(a) for a in range(g)], "clusters": ncl, "B": int(B),
|
|
348
|
+
"ranked": bool(rank), "exchangeable": "clusters" if ncl > 0 else "respondents"}
|
|
349
|
+
|
|
350
|
+
|
|
351
|
+
def ec_pvalue(obs, null):
|
|
352
|
+
"""Tail probabilities from an observed value and a sample from the null."""
|
|
353
|
+
null = [float(t) for t in null]
|
|
354
|
+
out = (_ct.c_double * 4)()
|
|
355
|
+
_lib.C_ec_pvalue(_oned(obs), _d(null), _one(len(null)), out)
|
|
356
|
+
return {"p_value": out[0], "midp": out[1], "mc_lower": out[2], "mc_upper": out[3]}
|