entropy-invariant 1.1.0__tar.gz

entropy_invariant-1.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: entropy-invariant
+Version: 1.1.0
+Summary: Invariant entropy estimation using nearest neighbor methods
+Author: Felix Truong, Alexandre Giuliani
+License: MIT
+Keywords: entropy,mutual information,information theory,k-nn,nearest neighbor
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: scipy>=1.7.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+
+# entropy-invariant
+
+A Python package implementing an improved nearest neighbor method for estimating differential entropy for continuous variables. This is a port of the Julia [EntropyInvariant](https://github.com/truongfelix/EntropyInvariant-private) package.
+
+## Key Features
+
+- **Invariant under change of variables**: Scale and translation invariant entropy estimation
+- **Always positive**: Solves Edwin Thompson Jaynes' limiting density of discrete points problem
+- **Multiple methods**: Supports invariant (default), k-NN, and histogram methods
+
+## Installation
+
+```bash
+pip install entropy-invariant
+```
+
+Or install from source:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+```python
+import numpy as np
+from entropy_invariant import entropy, mutual_information
+
+# Generate random data
+n = 1000
+x = np.random.rand(n)
+y = 2 * x + np.random.rand(n)
+
+# Compute entropy (invariant method, default)
+h = entropy(x)
+print(f"Entropy: {h}")
+
+# Entropy is invariant under scaling and translation
+h_scaled = entropy(1e5 * x - 123.456)
+print(f"Entropy (scaled): {h_scaled}")  # Same value!
+
+# Mutual information
+mi = mutual_information(x, y)
+print(f"Mutual Information: {mi}")
+
+# Different methods
+h_knn = entropy(x, method="knn")
+h_hist = entropy(x, method="histogram", nbins=20)
+```
+
+## Available Functions
+
+### Core Entropy
+- `entropy(X, method="inv", k=3, base=e, ...)` - Unified entropy interface
+- `entropy_inv(X, ...)` - Invariant method (default)
+- `entropy_knn(X, ...)` - k-NN method
+- `entropy_hist(X, ...)` - Histogram method
+
+### Information Theory
+- `conditional_entropy(X, Y, ...)` - H(Y|X)
+- `mutual_information(X, Y, ...)` - I(X;Y)
+- `conditional_mutual_information(X, Y, Z, ...)` - I(X;Y|Z)
+- `normalized_mutual_information(X, Y, ...)` - NMI
+- `interaction_information(X, Y, Z, ...)` - Three-way interaction
+
+### Partial Information Decomposition
+- `redundancy(X, Y, Z, ...)` - Shared information
+- `unique(X, Y, Z, ...)` - Unique information
+- `synergy(X, Y, Z, ...)` - Synergistic information
+
+### Optimized Matrix Functions
+- `MI(X, ...)` - Pairwise mutual information matrix
+- `CMI(X, Z, ...)` - Pairwise conditional MI matrix
+
+## Authors
+
+- Felix Truong
+- Alexandre Giuliani
+
+## License
+
+MIT

entropy_invariant-1.1.0/README.md

@@ -0,0 +1,82 @@
+# entropy-invariant
+
+A Python package implementing an improved nearest neighbor method for estimating differential entropy for continuous variables. This is a port of the Julia [EntropyInvariant](https://github.com/truongfelix/EntropyInvariant-private) package.
+
+## Key Features
+
+- **Invariant under change of variables**: Scale and translation invariant entropy estimation
+- **Always positive**: Solves Edwin Thompson Jaynes' limiting density of discrete points problem
+- **Multiple methods**: Supports invariant (default), k-NN, and histogram methods
+
+## Installation
+
+```bash
+pip install entropy-invariant
+```
+
+Or install from source:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+```python
+import numpy as np
+from entropy_invariant import entropy, mutual_information
+
+# Generate random data
+n = 1000
+x = np.random.rand(n)
+y = 2 * x + np.random.rand(n)
+
+# Compute entropy (invariant method, default)
+h = entropy(x)
+print(f"Entropy: {h}")
+
+# Entropy is invariant under scaling and translation
+h_scaled = entropy(1e5 * x - 123.456)
+print(f"Entropy (scaled): {h_scaled}")  # Same value!
+
+# Mutual information
+mi = mutual_information(x, y)
+print(f"Mutual Information: {mi}")
+
+# Different methods
+h_knn = entropy(x, method="knn")
+h_hist = entropy(x, method="histogram", nbins=20)
+```
+
+## Available Functions
+
+### Core Entropy
+- `entropy(X, method="inv", k=3, base=e, ...)` - Unified entropy interface
+- `entropy_inv(X, ...)` - Invariant method (default)
+- `entropy_knn(X, ...)` - k-NN method
+- `entropy_hist(X, ...)` - Histogram method
+
+### Information Theory
+- `conditional_entropy(X, Y, ...)` - H(Y|X)
+- `mutual_information(X, Y, ...)` - I(X;Y)
+- `conditional_mutual_information(X, Y, Z, ...)` - I(X;Y|Z)
+- `normalized_mutual_information(X, Y, ...)` - NMI
+- `interaction_information(X, Y, Z, ...)` - Three-way interaction
+
+### Partial Information Decomposition
+- `redundancy(X, Y, Z, ...)` - Shared information
+- `unique(X, Y, Z, ...)` - Unique information
+- `synergy(X, Y, Z, ...)` - Synergistic information
+
+### Optimized Matrix Functions
+- `MI(X, ...)` - Pairwise mutual information matrix
+- `CMI(X, Z, ...)` - Pairwise conditional MI matrix
+
+## Authors
+
+- Felix Truong
+- Alexandre Giuliani
+
+## License
+
+MIT

entropy_invariant-1.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "entropy-invariant"
+version = "1.1.0"
+description = "Invariant entropy estimation using nearest neighbor methods"
+readme = "README.md"
+license = {text = "MIT"}
+authors = [
+    {name = "Felix Truong"},
+    {name = "Alexandre Giuliani"}
+]
+requires-python = ">=3.8"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+keywords = ["entropy", "mutual information", "information theory", "k-nn", "nearest neighbor"]
+dependencies = [
+    "numpy>=1.20.0",
+    "scipy>=1.7.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-cov",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]

entropy_invariant-1.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

entropy_invariant-1.1.0/src/entropy_invariant/__init__.py

@@ -0,0 +1,81 @@
+"""
+EntropyInvariant - Invariant entropy estimation using nearest neighbor methods.
+
+This package implements an improved nearest neighbor method for estimating
+differential entropy for continuous variables, solving Edwin Thompson Jaynes'
+limiting density of discrete points problem.
+
+The main innovation is the invariant measure m(x) based on the median value
+of nearest-neighbor distances, which ensures:
+- Invariance under change of variables (scaling and translation)
+- Always positive entropy values
+
+Example usage:
+    >>> import numpy as np
+    >>> from entropy_invariant import entropy, mutual_information
+    >>>
+    >>> # Generate random data
+    >>> x = np.random.rand(1000)
+    >>> y = 2 * x + np.random.rand(1000)
+    >>>
+    >>> # Compute entropy (invariant method)
+    >>> h = entropy(x)
+    >>>
+    >>> # Entropy is scale-invariant
+    >>> h_scaled = entropy(1e5 * x - 123.456)  # Same value!
+    >>>
+    >>> # Mutual information
+    >>> mi = mutual_information(x, y)
+
+Authors: Felix Truong, Alexandre Giuliani
+"""
+
+from entropy_invariant.entropy import (
+    entropy,
+    entropy_hist,
+    entropy_inv,
+    entropy_knn,
+)
+from entropy_invariant.mutual_information import (
+    conditional_entropy,
+    mutual_information,
+)
+from entropy_invariant.advanced import (
+    conditional_mutual_information,
+    information_quality_ratio,
+    interaction_information,
+    normalized_mutual_information,
+)
+from entropy_invariant.pid import (
+    redundancy,
+    synergy,
+    unique,
+)
+from entropy_invariant.optimized import (
+    MI,
+    CMI,
+)
+
+__version__ = "1.1.0"
+__all__ = [
+    # Core entropy
+    "entropy",
+    "entropy_hist",
+    "entropy_knn",
+    "entropy_inv",
+    # Basic information theory
+    "conditional_entropy",
+    "mutual_information",
+    # Advanced information theory
+    "conditional_mutual_information",
+    "normalized_mutual_information",
+    "interaction_information",
+    "information_quality_ratio",
+    # Partial Information Decomposition
+    "redundancy",
+    "unique",
+    "synergy",
+    # Optimized matrix functions
+    "MI",
+    "CMI",
+]

entropy_invariant-1.1.0/src/entropy_invariant/_constants.py

@@ -0,0 +1,16 @@
+"""Mathematical constants used throughout the package."""
+
+import math
+import numpy as np
+
+# Euler's number (for base conversion and default base)
+E = math.e
+
+# Unit ball volumes for dimensions 1-3
+# V_1 = 2.0 (line segment: [-1, 1])
+# V_2 = pi (circle with radius 1)
+# V_3 = 4*pi/3 (sphere with radius 1)
+UNIT_BALL_VOLUMES = np.array([2.0, math.pi, 4.0 * math.pi / 3.0])
+
+# Precomputed log of unit ball volumes for efficiency
+LOG_UNIT_BALL_VOLUMES = np.log(UNIT_BALL_VOLUMES)

entropy_invariant-1.1.0/src/entropy_invariant/_types.py

@@ -0,0 +1,36 @@
+"""Type definitions for the entropy_invariant package."""
+
+from dataclasses import dataclass
+from typing import List
+import numpy as np
+from numpy.typing import NDArray
+
+
+@dataclass(frozen=True)
+class DataShape:
+    """
+    Shape information for datasets.
+
+    In canonical format (columns as points):
+    - num_points: Number of data points (columns)
+    - num_dimensions: Number of dimensions (rows)
+    """
+
+    num_points: int
+    num_dimensions: int
+
+
+@dataclass
+class KNNResult:
+    """
+    Results from k-nearest neighbor computations.
+
+    Attributes:
+        indices: Indices of k-nearest neighbors for each point, shape (n_points, k+1)
+        all_distances: All k-nearest neighbor distances for each point, shape (n_points, k+1)
+        kth_distances: Distance to the k-th nearest neighbor for each point, shape (n_points,)
+    """
+
+    indices: NDArray[np.intp]
+    all_distances: NDArray[np.float64]
+    kth_distances: NDArray[np.float64]

entropy_invariant-1.1.0/src/entropy_invariant/advanced.py

@@ -0,0 +1,331 @@
+"""Advanced information theory functions."""
+
+from typing import Optional
+import numpy as np
+from numpy.typing import NDArray
+
+from entropy_invariant._constants import E
+from entropy_invariant._types import DataShape
+from entropy_invariant.helpers.data import (
+    ensure_columns_are_points,
+    ensure_2d,
+    get_shape,
+    validate_same_num_points,
+    validate_dimensions_equal_one,
+)
+from entropy_invariant.helpers.utility import log_computation_info
+from entropy_invariant.entropy import entropy
+
+
+def conditional_mutual_information(
+    X: NDArray,
+    Y: NDArray,
+    Z: NDArray,
+    *,
+    method: str = "inv",
+    nbins: int = 10,
+    k: int = 3,
+    base: float = E,
+    verbose: bool = False,
+    degenerate: bool = False,
+    dim: int = 1,
+) -> float:
+    """
+    Compute conditional mutual information I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z).
+
+    Args:
+        X: First variable
+        Y: Second variable
+        Z: Conditioning variable
+        method: Entropy estimation method
+        nbins: Bins for histogram method
+        k: Neighbors for k-NN methods
+        base: Logarithmic base
+        verbose: Print info
+        degenerate: Handle degenerate cases
+        dim: Data layout
+
+    Returns:
+        Conditional mutual information I(X;Y|Z)
+    """
+    mat_x = ensure_2d(X)
+    mat_y = ensure_2d(Y)
+    mat_z = ensure_2d(Z)
+    mat_x = ensure_columns_are_points(mat_x, dim)
+    mat_y = ensure_columns_are_points(mat_y, dim)
+    mat_z = ensure_columns_are_points(mat_z, dim)
+
+    shape_x = get_shape(mat_x)
+    shape_y = get_shape(mat_y)
+    shape_z = get_shape(mat_z)
+
+    validate_same_num_points([shape_x, shape_y, shape_z])
+    validate_dimensions_equal_one([shape_x, shape_y, shape_z])
+
+    if verbose:
+        total_dims = (
+            shape_x.num_dimensions + shape_y.num_dimensions + shape_z.num_dimensions
+        )
+        print(f"Number of points: {shape_x.num_points}")
+        print(f"Dimensions: {total_dims}")
+        print(f"Base: {base}")
+
+    # I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)
+    xz_mat = np.vstack([mat_x, mat_z])
+    yz_mat = np.vstack([mat_y, mat_z])
+    xyz_mat = np.vstack([mat_x, mat_y, mat_z])
+
+    ent_z = entropy(
+        mat_z, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_xz = entropy(
+        xz_mat, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_yz = entropy(
+        yz_mat, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_xyz = entropy(
+        xyz_mat,
+        method=method,
+        nbins=nbins,
+        k=k,
+        base=base,
+        degenerate=degenerate,
+        dim=2,
+    )
+
+    return ent_xz + ent_yz - ent_xyz - ent_z
+
+
+def normalized_mutual_information(
+    X: NDArray,
+    Y: NDArray,
+    *,
+    method: str = "inv",
+    nbins: int = 10,
+    k: int = 3,
+    base: float = E,
+    verbose: bool = False,
+    degenerate: bool = False,
+    dim: int = 1,
+) -> float:
+    """
+    Compute normalized mutual information NMI = I(X;Y) / ((H(X) + H(Y)) / 2).
+
+    Args:
+        X: First variable
+        Y: Second variable
+        method: Entropy estimation method
+        nbins: Bins for histogram method
+        k: Neighbors for k-NN methods
+        base: Logarithmic base
+        verbose: Print info
+        degenerate: Handle degenerate cases
+        dim: Data layout
+
+    Returns:
+        Normalized mutual information in [0, 1]
+    """
+    mat_x = ensure_2d(X)
+    mat_y = ensure_2d(Y)
+    mat_x = ensure_columns_are_points(mat_x, dim)
+    mat_y = ensure_columns_are_points(mat_y, dim)
+
+    shape_x = get_shape(mat_x)
+    shape_y = get_shape(mat_y)
+
+    validate_same_num_points([shape_x, shape_y])
+    validate_dimensions_equal_one([shape_x, shape_y])
+
+    if verbose:
+        total_dims = shape_x.num_dimensions + shape_y.num_dimensions
+        print(f"Number of points: {shape_x.num_points}")
+        print(f"Dimensions: {total_dims}")
+        print(f"Base: {base}")
+
+    # Compute entropies
+    joint_mat = np.vstack([mat_x, mat_y])
+    ent_x = entropy(
+        mat_x, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_y = entropy(
+        mat_y, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_joint = entropy(
+        joint_mat,
+        method=method,
+        nbins=nbins,
+        k=k,
+        base=base,
+        degenerate=degenerate,
+        dim=2,
+    )
+
+    # NMI = I(X;Y) / avg(H(X), H(Y))
+    # I(X;Y) = H(X) + H(Y) - H(X,Y), ensure non-negative
+    mi = max(0.0, ent_x + ent_y - ent_joint)
+    avg_entropy = (ent_x + ent_y) / 2.0
+
+    return mi / avg_entropy
+
+
+def interaction_information(
+    X: NDArray,
+    Y: NDArray,
+    Z: NDArray,
+    *,
+    method: str = "inv",
+    nbins: int = 10,
+    k: int = 3,
+    base: float = E,
+    verbose: bool = False,
+    degenerate: bool = False,
+    dim: int = 1,
+) -> float:
+    """
+    Compute interaction information (three-way interaction).
+
+    II(X;Y;Z) = H(X) + H(Y) + H(Z) - H(X,Y) - H(X,Z) - H(Y,Z) + H(X,Y,Z)
+
+    Args:
+        X: First variable
+        Y: Second variable
+        Z: Third variable
+        method: Entropy estimation method
+        nbins: Bins for histogram method
+        k: Neighbors for k-NN methods
+        base: Logarithmic base
+        verbose: Print info
+        degenerate: Handle degenerate cases
+        dim: Data layout
+
+    Returns:
+        Interaction information II(X;Y;Z)
+    """
+    mat_x = ensure_2d(X)
+    mat_y = ensure_2d(Y)
+    mat_z = ensure_2d(Z)
+    mat_x = ensure_columns_are_points(mat_x, dim)
+    mat_y = ensure_columns_are_points(mat_y, dim)
+    mat_z = ensure_columns_are_points(mat_z, dim)
+
+    shape_x = get_shape(mat_x)
+    shape_y = get_shape(mat_y)
+    shape_z = get_shape(mat_z)
+
+    validate_same_num_points([shape_x, shape_y, shape_z])
+    validate_dimensions_equal_one([shape_x, shape_y, shape_z])
+
+    if verbose:
+        total_dims = (
+            shape_x.num_dimensions + shape_y.num_dimensions + shape_z.num_dimensions
+        )
+        print(f"Number of points: {shape_x.num_points}")
+        print(f"Dimensions: {total_dims}")
+        print(f"Base: {base}")
+
+    # Compute all required entropies
+    xy_mat = np.vstack([mat_x, mat_y])
+    xz_mat = np.vstack([mat_x, mat_z])
+    yz_mat = np.vstack([mat_y, mat_z])
+    xyz_mat = np.vstack([mat_x, mat_y, mat_z])
+
+    ent_x = entropy(
+        mat_x, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_y = entropy(
+        mat_y, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_z = entropy(
+        mat_z, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_xy = entropy(
+        xy_mat, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_xz = entropy(
+        xz_mat, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_yz = entropy(
+        yz_mat, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_xyz = entropy(
+        xyz_mat,
+        method=method,
+        nbins=nbins,
+        k=k,
+        base=base,
+        degenerate=degenerate,
+        dim=2,
+    )
+
+    # II = H(X) + H(Y) + H(Z) - H(X,Y) - H(X,Z) - H(Y,Z) + H(X,Y,Z)
+    return ent_x + ent_y + ent_z - ent_xy - ent_xz - ent_yz + ent_xyz
+
+
+def information_quality_ratio(
+    X: NDArray,
+    Y: NDArray,
+    *,
+    method: str = "inv",
+    nbins: int = 10,
+    k: int = 3,
+    base: float = E,
+    verbose: bool = False,
+    degenerate: bool = False,
+    dim: int = 1,
+) -> float:
+    """
+    Compute information quality ratio IQR = I(X;Y) / H(X).
+
+    Args:
+        X: First variable
+        Y: Second variable
+        method: Entropy estimation method
+        nbins: Bins for histogram method
+        k: Neighbors for k-NN methods
+        base: Logarithmic base
+        verbose: Print info
+        degenerate: Handle degenerate cases
+        dim: Data layout
+
+    Returns:
+        Information quality ratio
+    """
+    mat_x = ensure_2d(X)
+    mat_y = ensure_2d(Y)
+    mat_x = ensure_columns_are_points(mat_x, dim)
+    mat_y = ensure_columns_are_points(mat_y, dim)
+
+    shape_x = get_shape(mat_x)
+    shape_y = get_shape(mat_y)
+
+    validate_same_num_points([shape_x, shape_y])
+    validate_dimensions_equal_one([shape_x, shape_y])
+
+    if verbose:
+        total_dims = shape_x.num_dimensions + shape_y.num_dimensions
+        print(f"Number of points: {shape_x.num_points}")
+        print(f"Dimensions: {total_dims}")
+        print(f"Base: {base}")
+
+    # Compute entropies
+    joint_mat = np.vstack([mat_x, mat_y])
+    ent_x = entropy(
+        mat_x, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_y = entropy(
+        mat_y, method=method, nbins=nbins, k=k, base=base, degenerate=degenerate, dim=2
+    )
+    ent_joint = entropy(
+        joint_mat,
+        method=method,
+        nbins=nbins,
+        k=k,
+        base=base,
+        degenerate=degenerate,
+        dim=2,
+    )
+
+    # IQR = I(X;Y) / H(X)
+    mi = ent_x + ent_y - ent_joint
+    return mi / ent_x