mlquantify 0.1.2__tar.gz → 0.1.4__tar.gz

mlquantify-0.1.4/MANIFEST.in

@@ -0,0 +1 @@
+include VERSION.txt

{mlquantify-0.1.2 → mlquantify-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mlquantify
-Version: 0.1.2
+Version: 0.1.4
 Summary: Quantification Library
 Home-page: https://github.com/luizfernandolj/QuantifyML/tree/master
 Maintainer: Luiz Fernando Luth Junior
@@ -40,9 +40,9 @@ ___
 
 ## Latest Release
 
-- **Version 0.0.11.6**: Inicial beta version. For a detailed list of changes, check the [changelog](#).
-- In case you need any help, refer to the [wiki](https://github.com/luizfernandolj/mlquantify/wiki).
-- Explore the [API documentation](#) for detailed developer information.
+- **Version 0.1.3**: Inicial beta version. For a detailed list of changes, check the [changelog](#).
+- In case you need any help, refer to the [User Guide](https://luizfernandolj.github.io/mlquantify/user_guide.html).
+- Explore the [API documentation](https://luizfernandolj.github.io/mlquantify/api/index.html) for detailed developer information.
 - See also the library in the pypi site in [pypi mlquantify](https://pypi.org/project/mlquantify/)
 
 ___
@@ -70,7 +70,7 @@ ___
 | **21 Quantification Methods** | Methods for quantification, such as classify & Count Correct methods, Threshold Optimization, Mixture Models and more.|
 | **Dynamic class management** | All methods are dynamic, and handles multiclass and binary problems, in case of binary it makes One-Vs-All (OVA) automatically. |
 | **Model Selection** | Criteria and processes used to select the best model, such as grid-search for the case of quantification|
-| **Evaluation Metrics** | Specific metrics used to evaluate quantification performance, (e.g., AE, BIAS, NAE, SE, KLD, etc.). |
+| **Evaluation Metrics** | Specific metrics used to evaluate quantification performance, (e.g., AE, MAE, NAE, SE, KLD, etc.). |
 | **Evaluation Protocols** | Evaluation protocols used, based on sampling generation (e.g., APP, NPP, etc.).. |
 | **Plotting Results** | Tools and techniques used to visualize results, such as the protocol results.|
 | **Comprehensive Documentation** | Complete documentation of the project, including code, data, and results. |
@@ -82,7 +82,10 @@ ___
 This code first loads the breast cancer dataset from _sklearn_, which is then split into training and testing sets. It uses the _Expectation Maximisation Quantifier (EMQ)_ with a RandomForest classifier to predict class prevalence. After training the model, it evaluates performance by calculating and printing the absolute error and bias between the real and predicted prevalences.
 
 ```python
-import mlquantify as mq
+from mlquantify.methods import EMQ
+from mlquantify.evaluation.measures import absolute_error, mean_absolute_error
+from mlquantify.utils import get_real_prev
+
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.datasets import load_breast_cancer
 from sklearn.model_selection import train_test_split
@@ -94,19 +97,19 @@ features, target = load_breast_cancer(return_X_y=True)
 X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.3)
 
 #Create the model, here it is the Expectation Maximisation Quantifier (EMQ) with a classifier
-model = mq.methods.EMQ(RandomForestClassifier())
+model = EMQ(RandomForestClassifier())
 model.fit(X_train, y_train)
 
 #Predict the class prevalence for X_test
 pred_prevalence = model.predict(X_test)
-real_prevalence = mq.utils.get_real_prev(y_test)
+real_prevalence = get_real_prev(y_test)
 
 #Get the error for the prediction
-ae = mq.evaluation.absolute_error(real_prevalence, pred_prevalence)
-bias = mq.evaluation.bias(real_prevalence, pred_prevalence)
+ae = absolute_error(real_prevalence, pred_prevalence)
+mae = mean_absolute_error(real_prevalence, pred_prevalence)
 
-print(f"Mean Squared Error (MSE) -> {ae:.4f}")
-print(f"Bias -> {bias}")
+print(f"Absolute Error -> {ae}")
+print(f"Mean Absolute Error -> {mae}")
 ```
 
 ___
@@ -125,7 +128,7 @@ ___
 
 ## Documentation
 
-##### API is avaliable [here](#)
+##### API is avaliable [here](https://luizfernandolj.github.io/mlquantify/api/index.html)
 
 - [Methods](https://github.com/luizfernandolj/mlquantify/wiki/Methods)
 - [Model Selection](https://github.com/luizfernandolj/mlquantify/wiki/Model-Selection)

{mlquantify-0.1.2 → mlquantify-0.1.4}/README.md

@@ -9,9 +9,9 @@ ___
 
 ## Latest Release
 
-- **Version 0.0.11.6**: Inicial beta version. For a detailed list of changes, check the [changelog](#).
-- In case you need any help, refer to the [wiki](https://github.com/luizfernandolj/mlquantify/wiki).
-- Explore the [API documentation](#) for detailed developer information.
+- **Version 0.1.3**: Inicial beta version. For a detailed list of changes, check the [changelog](#).
+- In case you need any help, refer to the [User Guide](https://luizfernandolj.github.io/mlquantify/user_guide.html).
+- Explore the [API documentation](https://luizfernandolj.github.io/mlquantify/api/index.html) for detailed developer information.
 - See also the library in the pypi site in [pypi mlquantify](https://pypi.org/project/mlquantify/)
 
 ___
@@ -39,7 +39,7 @@ ___
 | **21 Quantification Methods** | Methods for quantification, such as classify & Count Correct methods, Threshold Optimization, Mixture Models and more.|
 | **Dynamic class management** | All methods are dynamic, and handles multiclass and binary problems, in case of binary it makes One-Vs-All (OVA) automatically. |
 | **Model Selection** | Criteria and processes used to select the best model, such as grid-search for the case of quantification|
-| **Evaluation Metrics** | Specific metrics used to evaluate quantification performance, (e.g., AE, BIAS, NAE, SE, KLD, etc.). |
+| **Evaluation Metrics** | Specific metrics used to evaluate quantification performance, (e.g., AE, MAE, NAE, SE, KLD, etc.). |
 | **Evaluation Protocols** | Evaluation protocols used, based on sampling generation (e.g., APP, NPP, etc.).. |
 | **Plotting Results** | Tools and techniques used to visualize results, such as the protocol results.|
 | **Comprehensive Documentation** | Complete documentation of the project, including code, data, and results. |
@@ -51,7 +51,10 @@ ___
 This code first loads the breast cancer dataset from _sklearn_, which is then split into training and testing sets. It uses the _Expectation Maximisation Quantifier (EMQ)_ with a RandomForest classifier to predict class prevalence. After training the model, it evaluates performance by calculating and printing the absolute error and bias between the real and predicted prevalences.
 
 ```python
-import mlquantify as mq
+from mlquantify.methods import EMQ
+from mlquantify.evaluation.measures import absolute_error, mean_absolute_error
+from mlquantify.utils import get_real_prev
+
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.datasets import load_breast_cancer
 from sklearn.model_selection import train_test_split
@@ -63,19 +66,19 @@ features, target = load_breast_cancer(return_X_y=True)
 X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.3)
 
 #Create the model, here it is the Expectation Maximisation Quantifier (EMQ) with a classifier
-model = mq.methods.EMQ(RandomForestClassifier())
+model = EMQ(RandomForestClassifier())
 model.fit(X_train, y_train)
 
 #Predict the class prevalence for X_test
 pred_prevalence = model.predict(X_test)
-real_prevalence = mq.utils.get_real_prev(y_test)
+real_prevalence = get_real_prev(y_test)
 
 #Get the error for the prediction
-ae = mq.evaluation.absolute_error(real_prevalence, pred_prevalence)
-bias = mq.evaluation.bias(real_prevalence, pred_prevalence)
+ae = absolute_error(real_prevalence, pred_prevalence)
+mae = mean_absolute_error(real_prevalence, pred_prevalence)
 
-print(f"Mean Squared Error (MSE) -> {ae:.4f}")
-print(f"Bias -> {bias}")
+print(f"Absolute Error -> {ae}")
+print(f"Mean Absolute Error -> {mae}")
 ```
 
 ___
@@ -94,7 +97,7 @@ ___
 
 ## Documentation
 
-##### API is avaliable [here](#)
+##### API is avaliable [here](https://luizfernandolj.github.io/mlquantify/api/index.html)
 
 - [Methods](https://github.com/luizfernandolj/mlquantify/wiki/Methods)
 - [Model Selection](https://github.com/luizfernandolj/mlquantify/wiki/Model-Selection)

mlquantify-0.1.4/VERSION.txt

@@ -0,0 +1 @@
+0.1.4

mlquantify-0.1.4/mlquantify/evaluation/protocol.py

@@ -0,0 +1,297 @@
+from abc import ABC, abstractmethod
+import numpy as np
+from typing import Generator, Tuple
+from tqdm import tqdm
+
+from ..utils.general import *
+
+class Protocol(ABC):
+    """Base class for evaluation protocols.
+    
+    Parameters
+    ----------
+    batch_size : int or list of int
+        The size of the batches to be used in the evaluation.
+    random_state : int, optional
+        The random seed for reproducibility.
+
+    Attributes
+    ----------
+    n_combinations : int
+
+    Raises
+    ------
+    ValueError
+        If the batch size is not a positive integer or list of positive integers.
+
+    Notes
+    -----
+    This class serves as a base class for different evaluation protocols, each with its own strategy for splitting the data into batches.
+    
+    Examples
+    --------
+    >>> class MyCustomProtocol(Protocol):
+    ...     def _iter_indices(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray]:
+    ...         for batch_size in self.batch_size:
+    ...             yield np.random.choice(X.shape[0], batch_size, replace=True)
+    ...
+    >>> protocol = MyCustomProtocol(batch_size=100, random_state=42)
+    >>> for train_idx, test_idx in protocol.split(X, y):
+    ...     # Train and evaluate model
+    ...     pass
+
+    """
+
+    def __init__(self, batch_size, random_state=None, **kwargs):
+        if isinstance(batch_size, int):
+            self.n_combinations = 1
+        else:
+            self.n_combinations = len(batch_size)
+
+        self.batch_size = [batch_size] if isinstance(batch_size, int) else batch_size
+        self.random_state = random_state
+
+        for name, value in kwargs.items():
+            setattr(self, name, value)
+            if isinstance(value, list):
+                self.n_combinations *= len(value)
+            elif isinstance(value, (int, float)):
+                self.n_combinations *= value
+            else:
+                raise ValueError(f"Invalid argument {name}={value}: must be int/float or list of int/float.")
+                 
+
+    def split(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray, np.ndarray]:
+        """
+        Split the data into samples for evaluation.
+
+        Parameters
+        ----------
+        X : np.ndarray
+            The input features.
+        y : np.ndarray
+            The target labels.
+
+        Yields
+        ------
+        Generator[np.ndarray, np.ndarray]
+            A generator that yields the indices for each split.
+        """
+        indices = np.arange(X.shape[0])
+        for idx in self._split_indices_masks(X, y):
+            indexes = indices[idx]
+            yield indexes
+
+    def _split_indices_masks(self, X: np.ndarray, y: np.ndarray) -> Generator[Tuple[np.ndarray, np.ndarray]]:
+        for idx in self._iter_indices(X, y):
+
+            mask = np.zeros(X.shape[0], dtype=bool)
+            mask[idx] = True 
+
+            yield mask
+
+    @abstractmethod
+    def _iter_indices(self, X, y):
+        """Abstract method to be implemented by subclasses to yield indices for each batch."""
+        pass
+    
+    def get_n_combinations(self) -> int:
+        """
+        Get the number of combinations for the current protocol.
+        """
+        return self.n_combinations
+
+
+class APP(Protocol):
+    """Artificial Prevalence Protocol (APP) for evaluation.
+    This protocol generates artificial prevalence distributions for the evaluation in an exhaustive manner, testing all possible combinations of prevalences.
+    
+    Parameters
+    ----------
+    batch_size : int or list of int
+        The size of the batches to be used in the evaluation.
+    n_prevalences : int
+        The number of artificial prevalences to generate.
+    repeats : int, optional
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int, optional
+        The random seed for reproducibility.
+        
+    Attributes
+    ----------
+    n_prevalences : int
+        The number of artificial prevalences to generate.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int
+        The random seed for reproducibility.
+        
+    Notes
+    -----
+    It is important to note that in case of multiclass problems, the time complexity of this protocol can be significantly higher due to the increased number of combinations to evaluate.
+
+    Examples
+    --------
+    >>> protocol = APP(batch_size=[100, 200], n_prevalences=5, repeats=3, random_state=42)
+    >>> for train_idx, test_idx in protocol.split(X, y):
+    ...     # Train and evaluate model
+    ...     pass
+
+    """
+
+    def __init__(self, batch_size, n_prevalences, repeats=1, random_state=None):
+        super().__init__(batch_size=batch_size, 
+                            random_state=random_state,
+                            n_prevalences=n_prevalences, 
+                            repeats=repeats)
+
+    def _iter_indices(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray]:
+        
+        n_dim = len(np.unique(y))
+        
+        for batch_size in self.batch_size:
+            prevalences = generate_artificial_prevalences(n_dim=n_dim,
+                                                           n_prev=self.n_prevalences,
+                                                           n_iter=self.repeats)
+            for prev in prevalences:
+                indexes = get_indexes_with_prevalence(y, prev, batch_size)
+                yield indexes
+                
+
+            
+
+class NPP(Protocol):
+    """No Prevalence Protocol (NPP) for evaluation.
+    This protocol just samples the data without any consideration for prevalence, with all instances having equal probability of being selected.
+
+    Parameters
+    ----------
+    batch_size : int or list of int
+        The size of the batches to be used in the evaluation.
+    random_state : int, optional
+        The random seed for reproducibility.
+
+    Attributes
+    ----------
+    n_prevalences : int
+        The number of artificial prevalences to generate.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int
+        The random seed for reproducibility.
+
+    Examples
+    --------
+    >>> protocol = NPP(batch_size=100, random_state=42)
+    >>> for train_idx, test_idx in protocol.split(X, y):
+    ...     # Train and evaluate model
+    ...     pass
+    """
+
+    def _iter_indices(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray]:
+        
+        for batch_size in self.batch_size:
+            yield np.random.choice(X.shape[0], batch_size, replace=True)
+            
+
+class UPP(Protocol):
+    """Uniform Prevalence Protocol (UPP) for evaluation.
+    An extension of the APP that generates artificial prevalence distributions uniformly across all classes utilizing the kraemer sampling method.
+
+    Parameters
+    ----------
+    batch_size : int or list of int
+        The size of the batches to be used in the evaluation.
+    n_prevalences : int
+        The number of artificial prevalences to generate.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int, optional
+        The random seed for reproducibility.
+
+    Attributes
+    ----------
+    n_prevalences : int
+        The number of artificial prevalences to generate.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int
+        The random seed for reproducibility.
+
+    Examples
+    --------
+    >>> protocol = UPP(batch_size=100, n_prevalences=5, repeats=3, random_state=42)
+    >>> for train_idx, test_idx in protocol.split(X, y):
+    ...     # Train and evaluate model
+    ...     pass
+    """
+
+    def __init__(self, batch_size, n_prevalences, repeats=1, random_state=None):
+        super().__init__(batch_size=batch_size, 
+                            random_state=random_state,
+                            n_prevalences=n_prevalences, 
+                            repeats=repeats)
+
+    def _iter_indices(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray]:
+        
+        n_dim = len(np.unique(y))
+        
+        for batch_size in self.batch_size:
+            
+            prevalences = kraemer_sampling(n_dim=n_dim,
+                                           n_prev=self.n_prevalences,
+                                           n_iter=self.repeats)
+            
+            for prev in prevalences:
+                indexes = get_indexes_with_prevalence(y, prev, batch_size)
+                yield indexes
+
+
+class PPP(Protocol):
+    """ Personalized Prevalence Protocol (PPP) for evaluation.
+    This protocol generates artificial prevalence distributions personalized for each class.
+
+    Parameters
+    ----------
+    batch_size : int or list of int
+        The size of the batches to be used in the evaluation.
+    prevalences : list of float
+        The list of artificial prevalences to generate for each class.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int, optional
+        The random seed for reproducibility.
+
+    Attributes
+    ----------
+    prevalences : list of float
+        The list of artificial prevalences to generate for each class.
+    repeats : int
+        The number of times to repeat the evaluation with different random seeds.
+    random_state : int
+        The random seed for reproducibility.
+
+    Examples
+    --------
+    >>> protocol = PPP(batch_size=100, prevalences=[0.1, 0.9], repeats=3, random_state=42)
+    >>> for train_idx, test_idx in protocol.split(X, y):
+    ...     # Train and evaluate model
+    ...     pass
+    """
+    
+    def __init__(self, batch_size, prevalences, repeats=1, random_state=None):
+        super().__init__(batch_size=batch_size, 
+                        random_state=random_state,
+                        prevalences=prevalences, 
+                        repeats=repeats)
+    
+    def _iter_indices(self, X: np.ndarray, y: np.ndarray) -> Generator[np.ndarray]:
+        
+        for batch_size in self.batch_size:    
+            for prev in self.prevalences:
+                if isinstance(prev, float):
+                    prev = [1-prev, prev]
+                
+                indexes = get_indexes_with_prevalence(y, prev, batch_size)
+                yield indexes
+        

{mlquantify-0.1.2 → mlquantify-0.1.4}/mlquantify/methods/aggregative.py

@@ -907,10 +907,140 @@ class PCC(AggregativeQuantifier):
 
     
     
+class PACC(AggregativeQuantifier):
+    """
+    Probabilistic Adjusted Classify and Count (PACC). 
+    This method extends the Adjusted Classify and Count (AC) approach 
+    by leveraging the average class-conditional confidences obtained 
+    from a probabilistic classifier instead of relying solely on true 
+    positive and false positive rates.
+
+    Parameters
+    ----------
+    learner : BaseEstimator
+        A scikit-learn compatible classifier to be used for quantification.
+    threshold : float, optional
+        The decision threshold for classification. Default is 0.5.
+
+    Attributes
+    ----------
+    learner : BaseEstimator
+        A scikit-learn compatible classifier.
+    threshold : float
+        Decision threshold for classification. Default is 0.5.
+    tpr : float
+        True positive rate computed during the fitting process.
+    fpr : float
+        False positive rate computed during the fitting process.
+        
+    See Also
+    --------
+    ThresholdOptimization : Base class for threshold-based quantification methods.
+    ACC : Adjusted Classify and Count quantification method.
+    CC : Classify and Count quantification method.
+    
+    References
+    ----------
+    A. Bella, C. Ferri, J. Hernández-Orallo and M. J. Ramírez-Quintana, "Quantification via Probability Estimators," 2010 IEEE International Conference on Data Mining, Sydney, NSW, Australia, 2010, pp. 737-742, doi: 10.1109/ICDM.2010.75. Available at: https://ieeexplore.ieee.org/abstract/document/5694031
     
+    Examples
+    --------
+    >>> from mlquantify.methods.aggregative import PACC
+    >>> from mlquantify.utils.general import get_real_prev
+    >>> from sklearn.datasets import load_breast_cancer
+    >>> from sklearn.svm import SVC
+    >>> from sklearn.model_selection import train_test_split
+    >>>
+    >>> features, target = load_breast_cancer(return_X_y=True)
+    >>>
+    >>> X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2, random_state=42)
+    >>>
+    >>> pacc = PACC(learner=SVC(probability=True))
+    >>> pacc.fit(X_train, y_train)
+    >>> y_pred = pacc.predict(X_test)
+    >>> y_pred
+    {0: 0.4664886119311328, 1: 0.5335113880688672}
+    >>> get_real_prev(y_test)
+    {0: 0.3991228070175439, 1: 0.6008771929824561}
+    """
 
+    def __init__(self, learner: BaseEstimator=None, threshold: float = 0.5):
+        self.learner = learner
+        self.threshold = threshold
+        self.mean_pos = None
+        self.mean_neg = None
+    
+    @property
+    def is_probabilistic(self) -> bool:
+        return True
+    
+    @property
+    def is_multiclass(self) -> bool:
+        return False
 
+    def _fit_method(self, X, y):
+        # Get predicted labels and probabilities
+        if mq.arguments["y_labels"] is not None and mq.arguments["posteriors_train"] is not None:
+            y_labels = mq.arguments["y_labels"]
+            probabilities = mq.arguments["posteriors_train"]
+        else:
+            y_labels, probabilities = get_scores(X, y, self.learner, self.cv_folds, self.learner_fitted)
+        
+        # Adjust thresholds and compute true and false positive rates
+        
+        self.mean_pos = np.mean(probabilities[y_labels == self.classes[1], 1])
+        self.mean_neg = np.mean(probabilities[y_labels != self.classes[1], 1])
+        
+        return self
+
+
+    def _predict_method(self, X):
+        """
+        Predicts the class prevalence using the mean class-conditional 
+        probabilities from a probabilistic classifier.
 
+        Parameters
+        ----------
+        X : array-like or sparse matrix of shape (n_samples, n_features)
+            The input data for prediction.
+
+        Returns
+        -------
+        dict
+            A dictionary with class labels as keys and their respective 
+            prevalence estimates as values.
+
+        Notes
+        -----
+        The prevalence is adjusted using the formula:
+            prevalence = |mean_score - FPR| / (TPR - FPR), 
+        where mean_score is the average probability for the positive class.
+
+        Raises
+        ------
+        ZeroDivisionError
+            If `TPR - FPR` equals zero, indicating that the classifier's 
+            performance does not vary across the threshold range.
+        """
+        prevalences = {}
+
+        # Calculate probabilities for the positive class
+        probabilities = self.predict_learner(X)[:, 1]
+
+        # Compute the mean score for the positive class
+        mean_scores = np.mean(probabilities)
+
+        # Adjust prevalence based on TPR and FPR
+        if self.mean_pos - self.mean_neg == 0:
+            prevalence = mean_scores
+        else:
+            prevalence = np.clip(abs(mean_scores - self.mean_neg) / (self.mean_pos - self.mean_neg), 0, 1)
+
+        # Map the computed prevalence to the class labels
+        prevalences[self.classes[0]] = 1 - prevalence
+        prevalences[self.classes[1]] = prevalence
+
+        return prevalences
 
 
 class PWK(AggregativeQuantifier):
@@ -1012,7 +1142,6 @@ class PWK(AggregativeQuantifier):
 from . import threshold_optimization
 
 ACC = threshold_optimization.ACC
-PACC = threshold_optimization.PACC
 T50 = threshold_optimization.T50
 MAX = threshold_optimization.MAX
 X_method  = threshold_optimization.X_method