ThreeWToolkit 1.0.3__tar.gz → 3.1.0__tar.gz

{threewtoolkit-1.0.3 → threewtoolkit-3.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ThreeWToolkit
-Version: 1.0.3
+Version: 3.1.0
 Summary: A modular and open-source AI toolkit for time-series processing, aimed at fault detection and classification in oil well operation
 Author-email: Ricardo Emanuel Vaz Vargas <ricardo.vargas@petrobras.com.br>
 License: Apache-2.0
@@ -26,6 +26,7 @@ Requires-Dist: ipykernel>=6.29.5
 Requires-Dist: ipywidgets==8.1.7
 Requires-Dist: jupyter-client==8.6.3
 Requires-Dist: jupyter-core==5.8.1
+Requires-Dist: tornado<7,>=6.5.5
 Requires-Dist: Pillow>=11.0.0
 Requires-Dist: scikit-image>=0.25.2
 Requires-Dist: torch>=2.7.0
@@ -38,17 +39,22 @@ Requires-Dist: PyWavelets>=1.8.0
 Requires-Dist: pyarrow>=20.0.0
 Requires-Dist: pylatex>=1.4.2
 Requires-Dist: statsmodels>=0.14.5
-Requires-Dist: requests<=2.35.5
+Requires-Dist: requests<3,>=2.31
+Requires-Dist: black[jupyter]==26.5.0
+Requires-Dist: nbconvert
+Requires-Dist: nbformat
+Requires-Dist: dtaidistance>=2.3
+Requires-Dist: joblib>=1.0
 Provides-Extra: dev
-Requires-Dist: black>=25.9.0; extra == "dev"
-Requires-Dist: mypy>=1.16.0; extra == "dev"
-Requires-Dist: flake8; extra == "dev"
+Requires-Dist: mypy>=1.19.1; extra == "dev"
+Requires-Dist: black==26.5.0; extra == "dev"
+Requires-Dist: ruff>=0.14.13; extra == "dev"
+Requires-Dist: pytest>=9.0.2; extra == "dev"
+Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
 Requires-Dist: jupyter; extra == "dev"
-Requires-Dist: ruff>=0.12.2; extra == "dev"
 Requires-Dist: uv; extra == "dev"
 Requires-Dist: types-requests>=2.32.4.20250611; extra == "dev"
 Requires-Dist: coverage>=7.8.1; extra == "dev"
-Requires-Dist: pytest>=8.3.5; extra == "dev"
 Requires-Dist: pytest-mock>=3.14.1; extra == "dev"
 Provides-Extra: docs
 Requires-Dist: mkdocs; extra == "docs"
@@ -59,6 +65,7 @@ Dynamic: license-file
 [![Apache 2.0][apache-shield]][apache] 
 [![Code style][black-shield]][black]
 [![Versioning][semver-shield]][semver]
+[![Coverage Status][coveralls-shield]][coveralls]
 
 [apache]: https://opensource.org/licenses/Apache-2.0
 [apache-shield]: https://img.shields.io/badge/License-Apache_2.0-blue.svg
@@ -66,6 +73,8 @@ Dynamic: license-file
 [black-shield]: https://img.shields.io/badge/code%20style-black-000000.svg
 [semver]: https://semver.org
 [semver-shield]: https://img.shields.io/badge/semver-2.0.1-blue
+[coveralls]: https://coveralls.io/github/rafaelpadilla/3W?branch=dev
+[coveralls-shield]: https://coveralls.io/repos/github/rafaelpadilla/3W/badge.svg?branch=dev
 
 <h1>
   <img src="https://raw.githubusercontent.com/petrobras/3W/main/images/3w_logo.png" width="45" style="vertical-align: middle; margin-right: 10px;" />
@@ -118,7 +127,7 @@ To better understand how the system is organized, we can divide all classes into
 ## 📊 3W Dataset <a id="dataset"></a>
 The **3W dataset** serves as a **reference dataset** for this project and is hosted on [Figshare](https://figshare.com/projects/3W_Dataset/251195). However, the toolkit is not limited to the 3W dataset and can be adapted for other datasets as well. 
 
-Further details on the 3W dataset’s structure, preprocessing, and usage are available in the [3W_DATASET_STRUCTURE.md](../3W_DATASET_STRUCTURE.md) file.
+Further details on the 3W dataset’s structure, preprocessing, and usage are available in the [3W_DATASET_STRUCTURE.md](../../3W_DATASET_STRUCTURE.md) file.
 
 
 ## ⚙️ Installation & Setup  <a id="installation"></a>
@@ -161,11 +170,11 @@ conda activate 3W
 ## 🪐 Jupyter Notebooks & Examples <a id="notebooks"></a>
 A curated set of ready-to-use jupyter notebooks that demonstrate how to use the toolkit to common fault detection tasks using the 3W dataset. These examples accelerate onboarding and reproducibility.
 
-The set of notebooks can be found in the [notebooks folder](../docs/notebooks/)
+The set of notebooks can be found in the [notebooks folder](../demos/)
 
 
 ## 🤝 Contributing  <a id="contributing"></a>
 
-We welcome contributions to help us improve and expand the functionality of the 3W toolkit. To ensure a smooth collaboration process, please follow our contrubuting guidelines [here](../CONTRIBUTING.md).
+We welcome contributions to help us improve and expand the functionality of the 3W toolkit. To ensure a smooth collaboration process, please follow our contrubuting guidelines [here](../../CONTRIBUTING.md).
 
 

{threewtoolkit-1.0.3 → threewtoolkit-3.1.0}/README.md

@@ -3,6 +3,7 @@
 [![CC BY 4.0][cc-by-shield]][cc-by]
 [![Code style][black-shield]][black]
 [![Versioning][semver-shield]][semver]
+[![Coverage Status][coveralls-shield]][coveralls]
 
 [apache]: https://opensource.org/licenses/Apache-2.0
 [apache-shield]: https://img.shields.io/badge/License-Apache_2.0-blue.svg
@@ -12,6 +13,8 @@
 [black-shield]: https://img.shields.io/badge/code%20style-black-000000.svg
 [semver]: https://semver.org
 [semver-shield]: https://img.shields.io/badge/semver-2.0.0-blue
+[coveralls]: https://coveralls.io/github/rafaelpadilla/3W?branch=dev
+[coveralls-shield]: https://coveralls.io/repos/github/rafaelpadilla/3W/badge.svg?branch=dev
 
 # Table of Content
 
@@ -87,22 +90,24 @@ It is also very important to know, participate and follow the discussions. See t
 
 ## Licenses
 
-All the code of this project is licensed under the [Apache 2.0 License][apache] and all 3W Dataset's data files (Parquet files saved in subdirectories of the [dataset](dataset) directory) are licensed under the [Creative Commons Attribution 4.0 International License][cc-by].
+All the code of this project is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0) and all 3W Dataset's data files (Parquet files saved in subdirectories of the [dataset](dataset) directory) are licensed under the [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).
 
 ## Versioning
 
 In the 3W Project, three types of versions will be managed as follows.
 
-* Version of the 3W Toolkit: specified in the [__init__.py](toolkit/__init__.py) file;
+* Version of the 3W Toolkit: specified in the [pyproject.toml](pyproject.toml) file;
 * Version of the 3W Dataset: specified in the [dataset.ini](dataset/dataset.ini) file;
 * Version of the 3W Project: specified with tags in the git repository;
-* We will exclusively use the semantic versioning defined in https://semver.org;
+* We will exclusively use the semantic versioning defined in [https://semver.org](https://semver.org);
 * Versions will always be updated manually;
 * Versioning of the 3W Toolkit and 3W Dataset are completely independent of each other;
 * The version of the 3W Project will be updated whenever, and only when, there is a new commit in the `main` branch of the repository, regardless of the updated resource: 3W Toolkit, 3W Dataset, 3W Project's documentation, example of use, etc;
 * We will only use annotated tags and for each tag there will be a release in the remote repository (GitHub);
 * Content for each release will be automatically generated with functionality provided by GitHub.
 
+For detailed versioning rules, including SemVer guidelines for the 3W Toolkit, see [VERSIONING.md](VERSIONING.md).
+
 ## Questions
 
 See the discussions section. If you don't get clarification, please open discussions to ask your questions so we can answer them.
@@ -117,7 +122,7 @@ The 3W Dataset consists of multiple Parquet files saved in subdirectories of the
 
 ## Overview
 
-A 3W Dataset's general presentation with some quantities and statistics is available in [this](overviews/_baseline/main.ipynb) Jupyter Notebook.
+A 3W Dataset's general presentation with some quantities and statistics is available in [this](dataset/demos/_basic/main.ipynb) Jupyter Notebook.
 
 # 3W Toolkit
 
@@ -133,23 +138,19 @@ It is important to note that there are arbitrary choices in this toolkit, but th
 
 The 3W Toolkit is implemented in sub-modules as discribed [here](3W_TOOLKIT_STRUCTURE.md).
 
-## Incorporated Problems
-
-Specific problems will be incorporated into this project gradually. At this point, we can work on:
+## Incorporated Benchmarks
 
-* [Binary classifier of Spurious Closure of DHSV](problems/01_binary_classifier_of_spurious_closure_of_dhsv/README.md).
+Benchmarks for specific tasks will be gradually incorporated into this project. They will be located in the `3W/benchmarks/` folder.
 
-All specification is detailed in the [CONTRIBUTING GUIDE](CONTRIBUTING.md).
+We encourage the community to contribute their own benchmarks. All specifications are detailed in the [Contributing Guide](CONTRIBUTING.md).
 
 ## Examples of Use
 
 The list below with examples of how to use the 3W Toolkit will be incremented throughout its development.
 
 * 3W Dataset's overviews:
-  * [Baseline](overviews/_baseline/main.ipynb)
-  * [André Machado's overview](overviews/AndreMachado/main.ipynb)
-* Binary classifier of Spurious Closure of DHSV:
-  * [Baseline](problems/01_binary_classifier_of_spurious_closure_of_dhsv/_baseline/main.ipynb)
+  * [Baseline](dataset/demos/_basic/main.ipynb)
+  * [André Machado's overview](dataset/demos/AndreMachado/main.ipynb)
 
 For a contribution of yours to be listed here, follow the instructions detailed in the [CONTRIBUTING GUIDE](CONTRIBUTING.md).
 
@@ -157,7 +158,7 @@ For a contribution of yours to be listed here, follow the instructions detailed
 
 For all results generated by the 3W Toolkit to be consistent, we recommend you create and use a virtual environment with the packages versions specified in the [environment.yml](environment.yml), which was generated with [conda](https://docs.conda.io). Our current recommendation is to use the conda distributed by [Miniforge](https://conda-forge.org/download/). Download and install Miniforge according to the official instructions. Open a prompt on your operating system (Windows, Linux or MacOS). Make sure the current directory is the directory where you have the 3W. Run the following commands as needed:
 
-* To create a virtual environment from our [environment.yml](environment.yml): 
+* To create a virtual environment from our [environment.yml](environment.yml):
 ```
 $ conda env create -f environment.yml
 ```

{threewtoolkit-1.0.3 → threewtoolkit-3.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ThreeWToolkit"
-version = "1.0.3"
+version = "3.1.0"
 description = "A modular and open-source AI toolkit for time-series processing, aimed at fault detection and classification in oil well operation"
 readme = { file = "toolkit/ThreeWToolkit/README.md", content-type = "text/markdown" }
 license = { text = "Apache-2.0" }
@@ -31,6 +31,7 @@ dependencies = [
     "ipywidgets==8.1.7",
     "jupyter-client==8.6.3",
     "jupyter-core==5.8.1",
+    "tornado>=6.5.5,<7",
     "Pillow>=11.0.0",
     "scikit-image>=0.25.2",
     "torch>=2.7.0",
@@ -43,7 +44,12 @@ dependencies = [
     "pyarrow>=20.0.0",
     "pylatex>=1.4.2",
     "statsmodels>=0.14.5",
-    "requests<=2.35.5",
+    "requests>=2.31,<3",
+    "black[jupyter]==26.5.0",
+    "nbconvert",
+    "nbformat",
+    "dtaidistance>=2.3",
+    "joblib>=1.0",
 ]
 
 [project.urls]
@@ -56,15 +62,15 @@ Changelog = "https://github.com/petrobras/3W/releases"
 
 [project.optional-dependencies]
 dev = [
-    "black>=25.9.0",
-    "mypy>=1.16.0",
-    "flake8",
+    "mypy>=1.19.1",
+    "black==26.5.0",
+    "ruff>=0.14.13",
+    "pytest>=9.0.2",
+    "pytest-cov>=7.0.0",
     "jupyter",
-    "ruff>=0.12.2",
     "uv",
     "types-requests>=2.32.4.20250611",
     "coverage>=7.8.1",
-    "pytest>=8.3.5",
     "pytest-mock>=3.14.1",
 ]
 docs = [
@@ -98,8 +104,11 @@ line-length = 88
 [tool.ruff.format]
 quote-style = "double"
 
+[tool.black]
+line-length = 88
+target-version = ["py310"]
+
 [tool.mypy]
 python_version = "3.10"
 mypy_path = "toolkit/"
 ignore_missing_imports = true
-

threewtoolkit-3.1.0/tests/test_clustering_models.py

@@ -0,0 +1,265 @@
+import pytest
+import numpy as np
+from pydantic import ValidationError
+
+from ThreeWToolkit.core.base_clustering import (
+    HierarchicalClusteringConfig,
+    DivisiveClusteringConfig,
+    MultivariateConsensusConfig,
+)
+from ThreeWToolkit.core.enums import LinkageMethodEnum
+from ThreeWToolkit.clustering import (
+    HierarchicalClusterer,
+    DivisiveRanker,
+    MultivariateConsensus,
+)
+
+
+class TestHierarchicalClusterer:
+    """Test suite for HierarchicalClusterer estimator."""
+
+    @pytest.fixture
+    def distance_matrix(self):
+        """Two tight pairs: (0,1) and (2,3)."""
+        return np.array(
+            [
+                [0.0, 1.0, 5.0, 6.0],
+                [1.0, 0.0, 4.0, 5.0],
+                [5.0, 4.0, 0.0, 2.0],
+                [6.0, 5.0, 2.0, 0.0],
+            ]
+        )
+
+    @pytest.fixture
+    def config(self):
+        return HierarchicalClusteringConfig(
+            linkage_method=LinkageMethodEnum.AVERAGE,
+            default_threshold=0.5,
+        )
+
+    @pytest.fixture
+    def fitted_clusterer(self, config, distance_matrix):
+        clusterer = HierarchicalClusterer(config)
+        clusterer.fit(distance_matrix)
+        return clusterer
+
+    def test_fit_returns_self(self, config, distance_matrix):
+        clusterer = HierarchicalClusterer(config)
+        result = clusterer.fit(distance_matrix)
+        assert result is clusterer
+
+    def test_linkage_matrix_stored(self, fitted_clusterer):
+        assert fitted_clusterer.linkage_matrix_ is not None
+        assert fitted_clusterer.linkage_matrix_.ndim == 2
+        assert fitted_clusterer.linkage_matrix_.shape[1] == 4
+
+    def test_distance_matrix_normalized(self, fitted_clusterer):
+        assert fitted_clusterer.distance_matrix_normalized_ is not None
+        assert fitted_clusterer.distance_matrix_normalized_.max() <= 1.0
+
+    def test_clusters_at_low_threshold(self, fitted_clusterer):
+        """At a very low threshold, each instance is its own cluster."""
+        labels = fitted_clusterer.get_clusters_at_threshold(0.01)
+        assert len(labels) == 4
+        assert len(set(labels)) == 4
+
+    def test_clusters_at_high_threshold(self, fitted_clusterer):
+        """At a high threshold, all instances merge into one cluster."""
+        labels = fitted_clusterer.get_clusters_at_threshold(1.0)
+        assert len(labels) == 4
+        assert len(set(labels)) == 1
+
+    def test_find_main_cluster_indices(self, fitted_clusterer):
+        indices = fitted_clusterer.find_main_cluster_indices(1.0)
+        assert sorted(indices) == [0, 1, 2, 3]
+
+    def test_raises_before_fit(self, config):
+        clusterer = HierarchicalClusterer(config)
+        with pytest.raises(RuntimeError, match="must be fitted"):
+            clusterer.get_clusters_at_threshold(0.5)
+
+    def test_zero_distance_matrix(self, config):
+        """All-zero distance matrix should not crash."""
+        zeros = np.zeros((3, 3))
+        clusterer = HierarchicalClusterer(config)
+        clusterer.fit(zeros)
+        labels = clusterer.get_clusters_at_threshold(0.5)
+        assert len(labels) == 3
+
+
+class TestDivisiveRanker:
+    """Test suite for DivisiveRanker estimator."""
+
+    @pytest.fixture
+    def distance_matrix(self):
+        """Instance 2 is the outlier (highest total distance)."""
+        return np.array(
+            [
+                [0.0, 1.0, 10.0],
+                [1.0, 0.0, 9.0],
+                [10.0, 9.0, 0.0],
+            ]
+        )
+
+    @pytest.fixture
+    def config(self):
+        return DivisiveClusteringConfig()
+
+    @pytest.fixture
+    def fitted_ranker(self, config, distance_matrix):
+        ranker = DivisiveRanker(config)
+        ranker.fit(distance_matrix)
+        return ranker
+
+    def test_fit_returns_self(self, config, distance_matrix):
+        ranker = DivisiveRanker(config)
+        result = ranker.fit(distance_matrix)
+        assert result is ranker
+
+    def test_ranking_length_matches_instances(self, fitted_ranker):
+        ranking = fitted_ranker.get_ranked_indices()
+        assert len(ranking) == 3
+
+    def test_ranking_contains_all_indices(self, fitted_ranker):
+        ranking = fitted_ranker.get_ranked_indices()
+        assert sorted(ranking) == [0, 1, 2]
+
+    def test_outlier_eliminated_first(self, fitted_ranker):
+        """Instance 2 has the largest distance sum and should be eliminated first."""
+        ranking = fitted_ranker.get_ranked_indices()
+        assert ranking[0] == 2
+
+    def test_elimination_distances_length(self, fitted_ranker):
+        assert len(fitted_ranker.elimination_distances_) == 3
+
+    def test_last_elimination_distance_is_zero(self, fitted_ranker):
+        assert fitted_ranker.elimination_distances_[-1] == 0.0
+
+    def test_raises_before_fit(self, config):
+        ranker = DivisiveRanker(config)
+        with pytest.raises(RuntimeError, match="must be fitted"):
+            ranker.get_ranked_indices()
+
+    def test_raises_on_non_square_matrix(self, config):
+        ranker = DivisiveRanker(config)
+        with pytest.raises(ValueError, match="2D square"):
+            ranker.fit(np.array([[1.0, 2.0]]))
+
+
+class TestMultivariateConsensus:
+    """Test suite for MultivariateConsensus estimator."""
+
+    @pytest.fixture
+    def config(self):
+        return MultivariateConsensusConfig(
+            min_threshold=0.1,
+            max_threshold=1.0,
+            threshold_step=0.1,
+        )
+
+    @pytest.fixture
+    def fitted_hierarchical_models(self):
+        """Build two fitted HierarchicalClusterers for two variables."""
+        cfg = HierarchicalClusteringConfig(
+            linkage_method=LinkageMethodEnum.AVERAGE,
+            default_threshold=0.5,
+        )
+        # Variable A: instances 0,1 are close; instance 2 is far
+        dm_a = np.array(
+            [
+                [0.0, 1.0, 10.0],
+                [1.0, 0.0, 9.0],
+                [10.0, 9.0, 0.0],
+            ]
+        )
+        model_a = HierarchicalClusterer(cfg)
+        model_a.fit(dm_a)
+
+        # Variable B: similar structure
+        dm_b = np.array(
+            [
+                [0.0, 2.0, 8.0],
+                [2.0, 0.0, 7.0],
+                [8.0, 7.0, 0.0],
+            ]
+        )
+        model_b = HierarchicalClusterer(cfg)
+        model_b.fit(dm_b)
+
+        return {"var_A": model_a, "var_B": model_b}
+
+    def test_fit_returns_self(self, config, fitted_hierarchical_models):
+        consensus = MultivariateConsensus(config)
+        result = consensus.fit(fitted_hierarchical_models)
+        assert result is consensus
+
+    def test_selection_mask_shape(self, config, fitted_hierarchical_models):
+        consensus = MultivariateConsensus(config)
+        consensus.fit(fitted_hierarchical_models)
+
+        n_thresholds = len(consensus.thresholds_analyzed_)
+        assert consensus.selection_mask_.shape[0] == n_thresholds
+        assert consensus.selection_mask_.shape[1] > 0
+
+    def test_common_counts_populated(self, config, fitted_hierarchical_models):
+        consensus = MultivariateConsensus(config)
+        consensus.fit(fitted_hierarchical_models)
+
+        assert len(consensus.common_counts_) == len(consensus.thresholds_analyzed_)
+
+    def test_high_threshold_selects_all(self, config, fitted_hierarchical_models):
+        consensus = MultivariateConsensus(config)
+        consensus.fit(fitted_hierarchical_models)
+
+        selected = consensus.get_selected_indices_at_threshold(1.0)
+        assert len(selected) == 3
+
+    def test_low_threshold_selects_fewer(self, config, fitted_hierarchical_models):
+        consensus = MultivariateConsensus(config)
+        consensus.fit(fitted_hierarchical_models)
+
+        selected_low = consensus.get_selected_indices_at_threshold(0.1)
+        selected_high = consensus.get_selected_indices_at_threshold(1.0)
+        assert len(selected_low) <= len(selected_high)
+
+    def test_raises_before_fit(self, config):
+        consensus = MultivariateConsensus(config)
+        with pytest.raises(RuntimeError, match="not fitted"):
+            consensus.get_selected_indices_at_threshold(0.5)
+
+    def test_with_valid_indices_map(self, config, fitted_hierarchical_models):
+        valid_indices = {
+            "var_A": [0, 1, 2],
+            "var_B": [0, 1, 2],
+        }
+        consensus = MultivariateConsensus(config)
+        consensus.fit(
+            fitted_hierarchical_models,
+            valid_indices_map=valid_indices,
+        )
+
+        assert consensus._is_fitted
+
+
+class TestMultivariateConsensusConfig:
+    """Test suite for MultivariateConsensusConfig validation."""
+
+    def test_default_values(self):
+        config = MultivariateConsensusConfig()
+        assert config.min_threshold == 0.1
+        assert config.max_threshold == 1.0
+        assert config.threshold_step == 0.1
+
+    def test_max_must_be_greater_than_min(self):
+        with pytest.raises(ValidationError, match="max_threshold"):
+            MultivariateConsensusConfig(
+                min_threshold=0.5,
+                max_threshold=0.3,
+            )
+
+    def test_equal_min_max_rejected(self):
+        with pytest.raises(ValidationError, match="max_threshold"):
+            MultivariateConsensusConfig(
+                min_threshold=0.5,
+                max_threshold=0.5,
+            )