bertrend 0.3.23__tar.gz → 0.4.0__tar.gz

{bertrend-0.3.23 → bertrend-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bertrend
-Version: 0.3.23
+Version: 0.4.0
 Summary: BERTrend analyses topic evolution over time using state-of-the-art transformer models: topic modelling and LLM
 License: LICENSE.md
 License-File: AUTHORS.txt
@@ -12,12 +12,12 @@ Maintainer: Jérôme Picault
 Maintainer-email: jerome.picault@rte-france.com
 Requires-Python: >=3.12,<4.0
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Provides-Extra: apps
 Requires-Dist: APScheduler (>=3.11.0)
 Requires-Dist: PyJWT (>=2.10.1)
 Requires-Dist: Pyqtree (==1.0.0)
 Requires-Dist: SQLAlchemy (>=2.0.40)
-Requires-Dist: arxiv (>=2.2.0) ; extra == "apps"
+Requires-Dist: aio-pika (>=9.5.5)
+Requires-Dist: arxiv (>=2.2.0)
 Requires-Dist: bertopic (==0.16.2)
 Requires-Dist: black (>=24.10.0)
 Requires-Dist: coverage (>=7.6.9)
@@ -27,12 +27,12 @@ Requires-Dist: datamapplot (==0.3.0)
 Requires-Dist: dateparser (>=1.2.0)
 Requires-Dist: dill (>=0.4.0)
 Requires-Dist: fastapi (>=0.120.0)
-Requires-Dist: feedparser (>=6.0.11) ; extra == "apps"
+Requires-Dist: feedparser (>=6.0.11)
 Requires-Dist: gensim (>=4.3.3)
-Requires-Dist: google-api-python-client (>=2.148.0) ; extra == "apps"
-Requires-Dist: google-auth-oauthlib (>=1.2.1) ; extra == "apps"
-Requires-Dist: googlenewsdecoder (>=0.1.7) ; extra == "apps"
-Requires-Dist: goose3 (>=3.1.19) ; extra == "apps"
+Requires-Dist: google-api-python-client (>=2.148.0)
+Requires-Dist: google-auth-oauthlib (>=1.2.1)
+Requires-Dist: googlenewsdecoder (>=0.1.7)
+Requires-Dist: goose3 (>=3.1.19)
 Requires-Dist: hdbscan (>=0.8.40)
 Requires-Dist: ipython (>=8.28.0)
 Requires-Dist: joblib (>=1.4.2)
@@ -42,37 +42,37 @@ Requires-Dist: langdetect (>=1.0.9)
 Requires-Dist: loguru (>=0.7.3)
 Requires-Dist: lxml-html-clean (>=0.2.2)
 Requires-Dist: markdown (>=3.7)
-Requires-Dist: newscatcher-sdk (>=1.4.0) ; extra == "apps"
-Requires-Dist: newspaper4k (>=0.9.3.1) ; extra == "apps"
+Requires-Dist: newscatcher-sdk (>=1.4.0)
+Requires-Dist: newspaper4k (>=0.9.3.1)
 Requires-Dist: nltk (>=3.9.1)
-Requires-Dist: numpy (<2)
 Requires-Dist: openai (>=1.77.0)
 Requires-Dist: openai-agents[litellm] (>=0.4.0)
 Requires-Dist: openpyxl (>=3.1.5)
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc (==1.25.0)
 Requires-Dist: pandas (>=2.2.2)
+Requires-Dist: pika (>=1.3.2)
 Requires-Dist: plotly (>=5.24.1)
 Requires-Dist: plotly-resampler (>=0.10.0)
+Requires-Dist: psutil (>=7.2.1)
 Requires-Dist: pyarrow (>=21.0.0)
-Requires-Dist: pygooglenews (>=0.1.3) ; extra == "apps"
+Requires-Dist: pygooglenews (>=0.1.3)
 Requires-Dist: pylabeladjust (==0.1.13)
 Requires-Dist: pytest (>=8.4.0)
 Requires-Dist: pytest-asyncio
 Requires-Dist: python-dotenv (>=1.1.1)
 Requires-Dist: requests (>=2.32.3)
+Requires-Dist: ruff (>=0.14.14)
 Requires-Dist: scikit-learn (>=1.5.2)
-Requires-Dist: scipy (==1.12)
 Requires-Dist: seaborn (>=0.13.2)
 Requires-Dist: sentence-transformers (==3.0.1)
 Requires-Dist: sentencepiece (>=0.2.0)
 Requires-Dist: starlette (>=0.46.2)
-Requires-Dist: streamlit (>=1.50)
+Requires-Dist: streamlit (>=1.51.0)
 Requires-Dist: streamlit-code-editor (>=0.1.22)
 Requires-Dist: supervisor (>=4.3.0)
 Requires-Dist: thefuzz (>=0.22.1)
 Requires-Dist: tiktoken (>=0.8.0)
 Requires-Dist: tldextract (>=2.0.1)
-Requires-Dist: tokenizers (<0.20.0)
 Requires-Dist: torch (>=2.7.0)
 Requires-Dist: tqdm (>=4.67.1)
 Requires-Dist: typer (>=0.16.0)
@@ -94,42 +94,59 @@ Description-Content-Type: text/markdown
 ![Static Badge](https://img.shields.io/badge/python-%3E%3D3.12-blue)
 ![GitHub License](https://img.shields.io/github/license/rte-france/BERTrend)
 
+## News
+
+2025-11-24: BERTrend has been integrated into
+the [Netminer 5](https://netminer.gitbook.io/netminer5/module-reference/text/bertopic/bertrend) data analysis tool !
+
+## Overview
+
 ![BERTrend pipeline](./BERTrend_overview.png)
 
 ## Paper
 
-The code in this repository is part of the work described in the paper: 
+The code in this repository is part of the work described in the paper:
 [BERTrend: Neural Topic Modeling for Emerging Trends Detection](./BERTrend%20paper.pdf).
 
 ## Description
 
-BERTrend is a novel framework for detecting and monitoring weak signals in large, evolving text corpora. It uses neural topic modeling, specifically [BERTopic](https://github.com/MaartenGr/BERTopic), in an online learning setting to identify and track topic evolution over time. 
-
-BERTrend classifies topics as noise, weak signals, or strong signals based on their popularity trends, using a metric that considers both the number of documents and update frequency. This method allows for real-time monitoring and analysis of emerging trends while filtering out noise.
+BERTrend is a novel framework for detecting and monitoring weak signals in large, evolving text corpora. It uses neural
+topic modeling, specifically [BERTopic](https://github.com/MaartenGr/BERTopic), in an online learning setting to
+identify and track topic evolution over time.
 
+BERTrend classifies topics as noise, weak signals, or strong signals based on their popularity trends, using a metric
+that considers both the number of documents and update frequency. This method allows for real-time monitoring and
+analysis of emerging trends while filtering out noise.
 
 ## Main Features
 
-The BERTrend repository provides a set of metrics, methods, and visualization tools to support the analysis of dynamic topic modeling in large text corpora. The tools are designed to help users detect, analyze, and visualize emerging topics and trends over time.
+The BERTrend repository provides a set of metrics, methods, and visualization tools to support the analysis of dynamic
+topic modeling in large text corpora. The tools are designed to help users detect, analyze, and visualize emerging
+topics and trends over time.
 
 The code base is illustrated via two main demos:
 
 ### Dynamic Topic Modeling Analysis
 
 - [bertrend/demos/topic_analysis](bertrend/demos/topic_analysis): Focuses on dynamic topic modeling analysis using:
-  - Dedicated metrics, called TEMPTopic:
-    - Stability Evaluation: Measures the stability of topics over time, evaluating how consistent and coherent topics remain.
-    - Volatility Assessment: Analyzes the volatility of topics, identifying how much topics change over different time periods.
-    - Two Perspectives:
-      - Temporal Topic Embedding Stability: Evaluates the stability of topic embeddings over time.
-      - Temporal Topic Representation Stability: Assesses the stability of topic representations over time.
-  - Visualization tools to easily assess the results.
-
-####  Topic Modeling Analysis Demonstrator Video
+    - Dedicated metrics, called TEMPTopic:
+        - Stability Evaluation: Measures the stability of topics over time, evaluating how consistent and coherent
+          topics remain.
+        - Volatility Assessment: Analyzes the volatility of topics, identifying how much topics change over different
+          time periods.
+        - Two Perspectives:
+            - Temporal Topic Embedding Stability: Evaluates the stability of topic embeddings over time.
+            - Temporal Topic Representation Stability: Assesses the stability of topic representations over time.
+    - Visualization tools to easily assess the results.
+
+#### Topic Modeling Analysis Demonstrator Video
+
 https://github.com/user-attachments/assets/f600f666-a6da-40be-8b07-5041b3bde1dc
 
 #### Demonstrator Features
-- Data management: selection of single/multiple datasets, filter by time, filter by size, split long documents by paragraphs 
+
+- Data management: selection of single/multiple datasets, filter by time, filter by size, split long documents by
+  paragraphs
 - Customization of BERTopic parameters
 - Exploration of topics by source, generate topic summaries using specific sources
 - Generate and customize newsletters by applying LLMs on BERTopic's output topics
@@ -142,10 +159,11 @@ https://github.com/user-attachments/assets/f600f666-a6da-40be-8b07-5041b3bde1dc
 - [bertrend/demos/weak_signals](bertrend/demos/weak_signals): Identifies and analyzes emerging trends and signals
 
 #### Weak Signal Analysis Demonstrator Video
-https://github.com/user-attachments/assets/d79368d9-d4e0-4324-8a98-a888f0ab3b65
 
+https://github.com/user-attachments/assets/d79368d9-d4e0-4324-8a98-a888f0ab3b65
 
 #### Demonstrator Features
+
 - Exploration of signals in real time at any given timestamp
 - Categorization of signals into three different dataframes: noise, weak and strong signals
   (each dataframe contains useful information about the signals)
@@ -156,70 +174,72 @@ https://github.com/user-attachments/assets/d79368d9-d4e0-4324-8a98-a888f0ab3b65
 
 ### Environment Setup (.env and .venv)
 
-BERTrend now supports configuration via a repository-level `.env` file. On import, the package attempts to load environment variables automatically using `python-dotenv`.
+BERTrend now supports configuration via a repository-level `.env` file. On import, the package attempts to load
+environment variables automatically using `python-dotenv`.
 
 Recommended steps:
 
 1. Create and fill your `.env` at the repo root (a template is provided):
-   - Copy `.env` and set your values (do not commit secrets).
-   - Common keys:
-     - `BERTREND_BASE_DIR`: base directory for BERTrend data/models/logs
-     - OpenAI/LLM:
-       - `OPENAI_API_KEY`
-       - `OPENAI_BASE_URL` (optional for OpenAI-compatible providers such as LiteLLM and Azure)
-       - `OPENAI_DEFAULT_MODEL`
-     - Providers (optional): `SEMANTIC_SCHOLAR_API_KEY`, `NEWSCATCHER_API_KEY`, `DBPEDIA_REST_API_URL`
-     - Embedding service security (optional): `BERTREND_SECRET_KEY`, `DEFAULT_RATE_LIMIT`, `DEFAULT_RATE_WINDOW`, `CLIENT_REGISTRY_FILE`
-     - Email (optional): `BERTREND_EMAIL_BACKEND`, `GMAIL_*`, `SMTP_*`, `SENDGRID_API_KEY`
-     - Misc: `CUDA_VISIBLE_DEVICES`
+    - Copy `.env` and set your values (do not commit secrets).
+    - Common keys:
+        - `BERTREND_BASE_DIR`: base directory for BERTrend data/models/logs
+        - OpenAI/LLM:
+            - `OPENAI_API_KEY`
+            - `OPENAI_BASE_URL` (optional for OpenAI-compatible providers such as LiteLLM and Azure)
+            - `OPENAI_DEFAULT_MODEL`
+        - Providers (optional): `SEMANTIC_SCHOLAR_API_KEY`, `NEWSCATCHER_API_KEY`, `DBPEDIA_REST_API_URL`
+        - Embedding service security (optional): `BERTREND_SECRET_KEY`, `DEFAULT_RATE_LIMIT`, `DEFAULT_RATE_WINDOW`,
+          `CLIENT_REGISTRY_FILE`
+        - Email (optional): `BERTREND_EMAIL_BACKEND`, `GMAIL_*`, `SMTP_*`, `SENDGRID_API_KEY`
+        - Misc: `CUDA_VISIBLE_DEVICES`
 2. Use a virtual environment and install dependencies inside it:
    ```bash
-   python -m venv .venv
+   uv venv --python 3.13
    source .venv/bin/activate  # Windows: .venv\\Scripts\\activate
-   pip install -U pip
-   pip install .  # or pip install ".[apps]"
-   pip install python-dotenv  # ensures .env is auto-loaded
+   uv pip install . 
+   uv pip install python-dotenv  # ensures .env is auto-loaded
    ```
 3. Notes
-   - If `python-dotenv` isn’t installed, `.env` won’t be auto-loaded; you can either install it or export variables via your shell/runner.
-   - Scheduled jobs created from the apps use the exact interpreter of your active environment (`sys.executable`), ensuring `.venv/bin/python` is used. Keep your venv path stable on the machine running cron.
-   - Cron entries source your shell profile and pass through any required variables. You can also rely on `.env` for most settings.
+    - If `python-dotenv` isn’t installed, `.env` won’t be auto-loaded; you can either install it or export variables via
+      your shell/runner.
+    - Scheduled jobs created from the apps use the exact interpreter of your active environment (`sys.executable`),
+      ensuring `.venv/bin/python` is used. Keep your venv path stable on the machine running cron.
+    - Cron entries source your shell profile and pass through any required variables. You can also rely on `.env` for
+      most settings.
 
 ### Hardware and Software Requirements
 
 - Recommended: at least one GPU with 16GB RAM for faster processing
-- Python >= 3.12 is required
+- Python >= 3.13 is required
 
 ### Docker Installation
 
-BERTrend can be run using Docker, which provides an easy way to run the application without installing dependencies directly on your system.
+BERTrend can be run using Docker, which provides an easy way to run the application without installing dependencies
+directly on your system.
 
 For detailed instructions on using BERTrend with Docker, see the [Docker documentation](./docs/docker.md).
 
 ### Package Installation
 
-- (Recommended): create a new python environment using `venv` or `conda`: 
+- (Recommended): create a new python environment using `uv`:
 
-  `python -m venv <path>/bertrend`
+  `uv venv --python 3.13`
 
 
 - Installation from pypi
 
-  `pip install bertrend` 
+  `uv pip install bertrend`
 
 
 - Install using pip or poetry in the BERTrend home directory:
 
   ```bash
   # Basic installation with core dependencies
-  pip install .  # or poetry install
-
-    # Installation with apps dependencies
-  pip install ".[apps]"  # or poetry install --extras apps
+  uv pip install . 
 
   ```
 
-- (Use `pip install -e .` for a local installation)
+- (Use `uv pip install -e .` for a local installation)
 
 NB. Due to the number of dependencies, the installation may take some time on an empty environment.
 
@@ -233,12 +253,8 @@ BERTrend's dependencies are organized into logical groups for easier management:
 - **Visualization and UI**: Tools for data visualization and user interfaces
 - **Utilities**: General utility libraries
 
-Optional dependency groups:
-
-- **test**: Dependencies for running tests and measuring code coverage
-- **apps**: Additional dependencies required for BERTrend applications
-
-This organization simplifies installation and maintenance, allowing users to install only the dependencies they need for their specific use case.
+This organization simplifies installation and maintenance, allowing users to install only the dependencies they need for
+their specific use case.
 
 ### Verifying Installation
 
@@ -248,35 +264,43 @@ To verify that your installation is working correctly, you can run the installat
 python -m bertrend.tests.test_installation
 ```
 
-This script will check that all dependencies are installed correctly and provide a summary of the installation status. If any dependencies are missing, the script will provide instructions on how to install them.
+This script will check that all dependencies are installed correctly and provide a summary of the installation status.
+If any dependencies are missing, the script will provide instructions on how to install them.
 
 ### Datasets
 
 #### Dataset Sources
+
 BERTrend can work with various text datasets such as:
+
 - Scientific publications
 - News articles
 - Social media publications
 
 Recommended public datasets:
+
 - New York Times dataset: https://www.kaggle.com/datasets/aryansingh0909/nyt-articles-21m-2000-present
 - Arxiv dataset: https://www.kaggle.com/datasets/Cornell-University/arxiv
 
 For custom dataset creation, please refer to the documentation page about [data providers](./docs/data_provider.md).
 
-For email configuration and newsletter sending (optional), please refer to the [mail configuration guide](./docs/mail_configuration.md).
-
+For email configuration and newsletter sending (optional), please refer to
+the [mail configuration guide](./docs/mail_configuration.md).
 
 In order to use the provided demos, datasets :
+
 - can be stored on the server in: `$BERTREND_BASE_DIR/data/bertrend/`
 - or can be uploaded from the client device on which are displayed the Streamlit apps.
 
 #### Dataset Format
+
 Required columns:
+
 - `text`
 - `timestamp`
 
 Supported formats:
+
 - csv
 - xlsx
 - parquet
@@ -287,94 +311,105 @@ Supported formats:
 ### 1. Dynamic Topic Modeling Analysis
 
 #### Launch the Demonstrator
+
 ```bash
 cd bertrend/demos/topic_analysis
 CUDA_VISIBLE_DEVICES=<gpu_number> streamlit run app.py
 ```
 
 #### Description of Main Pages
+
 - `Parameters`: Include all BERTopic parameters. Choose the embedding model carefully based on your data language.
-The embedding model can run locally (a GPU is recommended); it is also possible to use a remote embedding service.
+  The embedding model can run locally (a GPU is recommended); it is also possible to use a remote embedding service.
 
 We recommend using a remote embedding service (this is now the default config) as it simplifies the computing.
 
-The change of configuration can be done in: [services_default_config.toml](bertrend/config/services_default_config.toml).
-
-An example of a very simple implementation to deploy a separate embedding service can be found here: `bertrend/services/embedding_server`
+The change of configuration can be done
+in: [services_default_config.toml](bertrend/config/services_default_config.toml).
 
+An example of a very simple implementation to deploy a separate embedding service can be found here:
+`bertrend/services/embedding_server`
 
 ![parameters](docs/images/parameters.png)
 
-Please refer to the [BERTopic documentation](https://maartengr.github.io/BERTopic/algorithm/algorithm.html) for parameter details.
+Please refer to the [BERTopic documentation](https://maartengr.github.io/BERTopic/algorithm/algorithm.html) for
+parameter details.
 
 The application is split into several pages:
 
-- `Data loading & model training`: Allows to 
-  - Select a dataset
-  - Select the range of timestamps
-  - Split paragraphs if needed
-  - Train a topic model
+- `Data loading & model training`: Allows to
+    - Select a dataset
+    - Select the range of timestamps
+    - Split paragraphs if needed
+    - Train a topic model
 
 ![data_selection](docs/images/data_selection.png)
 
 - `Topic exploration`
-  - Computes topics over time
-  ![exploration_over_time](./docs/images/explore1.png)
+    - Computes topics over time
+      ![exploration_over_time](./docs/images/explore1.png)
 
-  - Allow selecting sources and generating short descriptions for each theme
-  ![topic_exploration](./docs/images/explore2.png)
+    - Allow selecting sources and generating short descriptions for each theme
+      ![topic_exploration](./docs/images/explore2.png)
 
 - `Topic visualization`: Several visualization types available:
-  - `Overall Results`: Standard information such as inter-topic distance map
-  - `Topics Treemap`: Block view of topics and their relative importance
-  - `Data Map`: Easy topic and data search interface
+    - `Overall Results`: Standard information such as inter-topic distance map
+    - `Topics Treemap`: Block view of topics and their relative importance
+    - `Data Map`: Easy topic and data search interface
 
-  ![data_map](./docs/images/data_map.png)  
+  ![data_map](./docs/images/data_map.png)
 
 - `Temporal Visualization`: Focus on dynamic topic modeling:
-  - 3D visualization of temporal topic evolution
-  - Metrics to assess topic stability over time
-  - Overall topic stability, temporal topic stability, and temporal representation stability
+    - 3D visualization of temporal topic evolution
+    - Metrics to assess topic stability over time
+    - Overall topic stability, temporal topic stability, and temporal representation stability
 
-  ![temptopic_metrics](./docs/images/temptopic_metrics.png)  
+  ![temptopic_metrics](./docs/images/temptopic_metrics.png)
 
 - `Generate Newsletters`
-  - This is an example of an application combining topic analysis and LLMs
-  - Generates newsletters based on prominent topics
-  - Uses LLMs to generate topic descriptions and summarize representative documents
-
+    - This is an example of an application combining topic analysis and LLMs
+    - Generates newsletters based on prominent topics
+    - Uses LLMs to generate topic descriptions and summarize representative documents
 
 ### 2. Weak Signal Analysis
-**Weak signals** refer to early, small indicators of larger, important trends (like disruptive technologies or public health issues).
 
-Identifying weak signals early is critical for decision-making across domains such as politics, healthcare, and innovation
+**Weak signals** refer to early, small indicators of larger, important trends (like disruptive technologies or public
+health issues).
+
+Identifying weak signals early is critical for decision-making across domains such as politics, healthcare, and
+innovation
 
 Key Features of the demonstrator:
+
 - Dynamic Topic Modeling in an online learning setting
 - Popularity-based Signal Classification
-  * based on their popularity trends: noise, weak signals, and strong signals
-  * **new popularity metric** considers both the number of documents within a topic and the **frequency of updates**
-
+    * based on their popularity trends: noise, weak signals, and strong signals
+    * **new popularity metric** considers both the number of documents within a topic and the **frequency of updates**
 
 #### Launch the Weak Signal Analysis Demonstrator
+
 ```bash
 cd bertrend/demos/weak_signals
 CUDA_VISIBLE_DEVICES=<gpu_number> streamlit run app.py
 ```
 
 #### Data Preparation
+
 - Choose from available datasets
-![bertrend_data](docs/images/bertrend_data.png)
+  ![bertrend_data](docs/images/bertrend_data.png)
 - Click on `Embed Documents`
 
 #### Train Model
+
 - Select the `Model Training` tab
 - Choose time granularity (in days)
 - Click `Train Models` to generate topic models per time slice
 - Click `Merge Models` for the aggregated model
 
 #### Results Analysis
+
 Provides various statistics about topics:
+
 - Number of topics detected
 - Topic size evolution
 - Topic popularity evolution
@@ -385,13 +420,13 @@ Provides various statistics about topics:
 - Lists of noise, weak, and strong signals
 
 #### LLM-based signal interpretation
+
 - As topics are hard to interprete by humans, we showcase how it can be
-done using LLMs.
+  done using LLMs.
 - Choose a specific topic and click on the button to generate an interpretation
 - LLM-based in-depth signal analysis is done using predefined templates, for example:
-  - describe what's happened since the last timestamp (summary of main events)
-  - present through a multi-faceted description info about key developments, foreseen changes, potential impact, etc.
-
+    - describe what's happened since the last timestamp (summary of main events)
+    - present through a multi-faceted description info about key developments, foreseen changes, potential impact, etc.
 
 ![llm_interpretation](docs/images/llm_interpretation.png)
 
@@ -404,6 +439,7 @@ Please use the 'Issues' section to report and discuss any problems with BERTrend
 We welcome contributions. For major changes, please open an issue first to discuss your proposal.
 
 To contribute:
+
 1. Fork the repository at https://github.com/rte-france/BERTrend
 2. Sync your fork with the latest development branch
 3. Implement your changes
@@ -419,7 +455,8 @@ See [AUTHORS](./AUTHORS.txt) for the full list of contributors.
 
 Copyright 2023–2025 RTE France RTE: http://www.rte-france.com
 
-This Source Code is subject to the terms of the Mozilla Public License (MPL) v2 also available here: https://www.mozilla.org/en-US/MPL/2.0/
+This Source Code is subject to the terms of the Mozilla Public License (MPL) v2 also available
+here: https://www.mozilla.org/en-US/MPL/2.0/
 
 ## Citation