data-atlas 0.0.1__tar.gz

data_atlas-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Raffaele Giancotti
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_atlas-0.0.1/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: data-atlas
+Version: 0.0.1
+Summary: Generate clinical data dictionaries using language models
+Author-email: Raffaele Giancotti <raffaele.giancotti@unical.it>
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: ollama>=0.1.0
+Requires-Dist: torch>=2.0.0
+Requires-Dist: transformers>=4.20.0
+Requires-Dist: sentence-transformers>=2.0.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: reportlab>=3.6.0
+Requires-Dist: tqdm>=4.60.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# Data Dictionary Generator
+
+A Python package to automatically generate data dictionaries for clinical datasets using a large language model (LLM) via Ollama. This tool takes in dataset files (CSV format), processes them, and generates descriptions for each column in the dataset, as well as other metadata like data types, sample data, table descriptions and some quality information such as missing values, outliers and redundant values. It is also able to find relationships between tables and columns.
+
+## Features
+### Metadata Generation
+- **Column Descriptions**: AI-generated clinical context for each field
+- **Table Summaries**: Dataset-level documentation
+- **Smart Sampling**: Automatic data type detection with sample values
+
+### Quality Analysis
+- Missing value statistics
+- Outlier detection (numeric fields)
+- Duplicate row/column identification
+
+### Advanced Capabilities
+- **Semantic Relationship Detection**: Finds connected columns across tables
+- **Multi-Format Outputs**: JSON, Markdown
+- **Custom LLM Integration**: Supports any Ollama, OpenAI and Google Gemini models
+
+## Requirements
+
+Make sure you have Python 3.8+ installed. All dependencies are automatically installed when you install the package.
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone https://github.com/rafgia/data-dictionary-generator.git
+cd code
+```
+
+Install the package:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+### Command-line Interface
+
+Once the package is installed, you can use the command line to generate metadata for your dataset(s).
+
+To run the tool, use the following command:
+
+```bash
+python generate_dictionary.py <folder_path> --output-dir <output_dir> --model llama3.1
+```
+
+#### Parameters:
+- `<folder_path>`: The path to the folder containing your CSV files.
+- `<output-dir>`: The name of the path where the metadata will be saved.
+- `--model`: Specify the Ollama or OpenAI model to use for generating metadata.
+
+### Example
+
+1. **Prepare your data files**:
+   Place all your CSV files (representing tables in your dataset) in a folder, e.g., `data/MIMIC`.
+
+2. **Run the generator**:
+
+```bash
+python -m generate_dictionary.py data/MIMIC --output-dir output --model gpt-4o-mini
+```
+
+This will generate metadata for each column in the dataset and save it to a JSON file and a Markdown summary for the dictionary, and a .png plot and a markdown summary for the detected relationships.
+
+### Sample Output
+
+For each **table** in your dataset, the following metadata will be generated:
+- **Table Name**: The name of the table (CSV filename).
+- **Number of Rows**: The total number of rows in the table.
+- **Number of Columns**: The total number of columns in the table.
+- **Table Description**: A generated description of what the table contains.
+
+For each **column** in your dataset:
+- **Column Name**: The name of the column.
+- **Sample Data**: A sample of 5 data points from the column.
+- **Data Type**: The inferred data type (e.g., integer, float, string).
+- **Column Description**: A description generated by the model for the column.
+
+### Example of generated metadata:
+
+
+## Troubleshooting
+
+### 1. If you encounter an error such as `model not found`, make sure you have set up Ollama correctly and the model is available.
+- Ensure that you can manually run the model using `ollama run llama3.1` from the command line before using it in the Python script.
+
+### 2. If the dependencies are not installing, make sure you're using the correct Python version and have all required libraries listed in `pyproject.toml`.
+
+### 3. If the dataset is very large, consider breaking it down into smaller CSV files for more efficient processing.
+
+## Contributing
+
+If you would like to contribute to this project, feel free to fork the repository and submit a pull request. Make sure to add tests and document any new features.
+
+### To contribute:
+1. Fork the repository.
+2. Create a new branch (`git checkout -b feature-name`).
+3. Make your changes and commit them (`git commit -am 'Add new feature'`).
+4. Push to the branch (`git push origin feature-name`).
+5. Open a pull request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## Author
+
+Raffaele Giancotti

data_atlas-0.0.1/README.md

@@ -0,0 +1,115 @@
+# Data Dictionary Generator
+
+A Python package to automatically generate data dictionaries for clinical datasets using a large language model (LLM) via Ollama. This tool takes in dataset files (CSV format), processes them, and generates descriptions for each column in the dataset, as well as other metadata like data types, sample data, table descriptions and some quality information such as missing values, outliers and redundant values. It is also able to find relationships between tables and columns.
+
+## Features
+### Metadata Generation
+- **Column Descriptions**: AI-generated clinical context for each field
+- **Table Summaries**: Dataset-level documentation
+- **Smart Sampling**: Automatic data type detection with sample values
+
+### Quality Analysis
+- Missing value statistics
+- Outlier detection (numeric fields)
+- Duplicate row/column identification
+
+### Advanced Capabilities
+- **Semantic Relationship Detection**: Finds connected columns across tables
+- **Multi-Format Outputs**: JSON, Markdown
+- **Custom LLM Integration**: Supports any Ollama, OpenAI and Google Gemini models
+
+## Requirements
+
+Make sure you have Python 3.8+ installed. All dependencies are automatically installed when you install the package.
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone https://github.com/rafgia/data-dictionary-generator.git
+cd code
+```
+
+Install the package:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+### Command-line Interface
+
+Once the package is installed, you can use the command line to generate metadata for your dataset(s).
+
+To run the tool, use the following command:
+
+```bash
+python generate_dictionary.py <folder_path> --output-dir <output_dir> --model llama3.1
+```
+
+#### Parameters:
+- `<folder_path>`: The path to the folder containing your CSV files.
+- `<output-dir>`: The name of the path where the metadata will be saved.
+- `--model`: Specify the Ollama or OpenAI model to use for generating metadata.
+
+### Example
+
+1. **Prepare your data files**:
+   Place all your CSV files (representing tables in your dataset) in a folder, e.g., `data/MIMIC`.
+
+2. **Run the generator**:
+
+```bash
+python -m generate_dictionary.py data/MIMIC --output-dir output --model gpt-4o-mini
+```
+
+This will generate metadata for each column in the dataset and save it to a JSON file and a Markdown summary for the dictionary, and a .png plot and a markdown summary for the detected relationships.
+
+### Sample Output
+
+For each **table** in your dataset, the following metadata will be generated:
+- **Table Name**: The name of the table (CSV filename).
+- **Number of Rows**: The total number of rows in the table.
+- **Number of Columns**: The total number of columns in the table.
+- **Table Description**: A generated description of what the table contains.
+
+For each **column** in your dataset:
+- **Column Name**: The name of the column.
+- **Sample Data**: A sample of 5 data points from the column.
+- **Data Type**: The inferred data type (e.g., integer, float, string).
+- **Column Description**: A description generated by the model for the column.
+
+### Example of generated metadata:
+
+
+## Troubleshooting
+
+### 1. If you encounter an error such as `model not found`, make sure you have set up Ollama correctly and the model is available.
+- Ensure that you can manually run the model using `ollama run llama3.1` from the command line before using it in the Python script.
+
+### 2. If the dependencies are not installing, make sure you're using the correct Python version and have all required libraries listed in `pyproject.toml`.
+
+### 3. If the dataset is very large, consider breaking it down into smaller CSV files for more efficient processing.
+
+## Contributing
+
+If you would like to contribute to this project, feel free to fork the repository and submit a pull request. Make sure to add tests and document any new features.
+
+### To contribute:
+1. Fork the repository.
+2. Create a new branch (`git checkout -b feature-name`).
+3. Make your changes and commit them (`git commit -am 'Add new feature'`).
+4. Push to the branch (`git push origin feature-name`).
+5. Open a pull request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## Author
+
+Raffaele Giancotti

data_atlas-0.0.1/data_atlas.egg-info/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: data-atlas
+Version: 0.0.1
+Summary: Generate clinical data dictionaries using language models
+Author-email: Raffaele Giancotti <raffaele.giancotti@unical.it>
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: ollama>=0.1.0
+Requires-Dist: torch>=2.0.0
+Requires-Dist: transformers>=4.20.0
+Requires-Dist: sentence-transformers>=2.0.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: reportlab>=3.6.0
+Requires-Dist: tqdm>=4.60.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# Data Dictionary Generator
+
+A Python package to automatically generate data dictionaries for clinical datasets using a large language model (LLM) via Ollama. This tool takes in dataset files (CSV format), processes them, and generates descriptions for each column in the dataset, as well as other metadata like data types, sample data, table descriptions and some quality information such as missing values, outliers and redundant values. It is also able to find relationships between tables and columns.
+
+## Features
+### Metadata Generation
+- **Column Descriptions**: AI-generated clinical context for each field
+- **Table Summaries**: Dataset-level documentation
+- **Smart Sampling**: Automatic data type detection with sample values
+
+### Quality Analysis
+- Missing value statistics
+- Outlier detection (numeric fields)
+- Duplicate row/column identification
+
+### Advanced Capabilities
+- **Semantic Relationship Detection**: Finds connected columns across tables
+- **Multi-Format Outputs**: JSON, Markdown
+- **Custom LLM Integration**: Supports any Ollama, OpenAI and Google Gemini models
+
+## Requirements
+
+Make sure you have Python 3.8+ installed. All dependencies are automatically installed when you install the package.
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone https://github.com/rafgia/data-dictionary-generator.git
+cd code
+```
+
+Install the package:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+### Command-line Interface
+
+Once the package is installed, you can use the command line to generate metadata for your dataset(s).
+
+To run the tool, use the following command:
+
+```bash
+python generate_dictionary.py <folder_path> --output-dir <output_dir> --model llama3.1
+```
+
+#### Parameters:
+- `<folder_path>`: The path to the folder containing your CSV files.
+- `<output-dir>`: The name of the path where the metadata will be saved.
+- `--model`: Specify the Ollama or OpenAI model to use for generating metadata.
+
+### Example
+
+1. **Prepare your data files**:
+   Place all your CSV files (representing tables in your dataset) in a folder, e.g., `data/MIMIC`.
+
+2. **Run the generator**:
+
+```bash
+python -m generate_dictionary.py data/MIMIC --output-dir output --model gpt-4o-mini
+```
+
+This will generate metadata for each column in the dataset and save it to a JSON file and a Markdown summary for the dictionary, and a .png plot and a markdown summary for the detected relationships.
+
+### Sample Output
+
+For each **table** in your dataset, the following metadata will be generated:
+- **Table Name**: The name of the table (CSV filename).
+- **Number of Rows**: The total number of rows in the table.
+- **Number of Columns**: The total number of columns in the table.
+- **Table Description**: A generated description of what the table contains.
+
+For each **column** in your dataset:
+- **Column Name**: The name of the column.
+- **Sample Data**: A sample of 5 data points from the column.
+- **Data Type**: The inferred data type (e.g., integer, float, string).
+- **Column Description**: A description generated by the model for the column.
+
+### Example of generated metadata:
+
+
+## Troubleshooting
+
+### 1. If you encounter an error such as `model not found`, make sure you have set up Ollama correctly and the model is available.
+- Ensure that you can manually run the model using `ollama run llama3.1` from the command line before using it in the Python script.
+
+### 2. If the dependencies are not installing, make sure you're using the correct Python version and have all required libraries listed in `pyproject.toml`.
+
+### 3. If the dataset is very large, consider breaking it down into smaller CSV files for more efficient processing.
+
+## Contributing
+
+If you would like to contribute to this project, feel free to fork the repository and submit a pull request. Make sure to add tests and document any new features.
+
+### To contribute:
+1. Fork the repository.
+2. Create a new branch (`git checkout -b feature-name`).
+3. Make your changes and commit them (`git commit -am 'Add new feature'`).
+4. Push to the branch (`git push origin feature-name`).
+5. Open a pull request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## Author
+
+Raffaele Giancotti

data_atlas-0.0.1/data_atlas.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+README.md
+pyproject.toml
+data_atlas.egg-info/PKG-INFO
+data_atlas.egg-info/SOURCES.txt
+data_atlas.egg-info/dependency_links.txt
+data_atlas.egg-info/entry_points.txt
+data_atlas.egg-info/requires.txt
+data_atlas.egg-info/top_level.txt
+data_dictionary_generator/__init__.py
+data_dictionary_generator/column_stats.py
+data_dictionary_generator/data_type.py
+data_dictionary_generator/description_generator.py
+data_dictionary_generator/generate_dictionary.py
+data_dictionary_generator/load_dataset.py
+data_dictionary_generator/metadata_extraction.py
+data_dictionary_generator/pattern_extraction.py
+data_dictionary_generator/relationships.py
+tests/test_placeholder.py

data_atlas-0.0.1/data_atlas.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

data_atlas-0.0.1/data_atlas.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+generate-dictionary = data_dictionary_generator.generate_dictionary:main

data_atlas-0.0.1/data_atlas.egg-info/requires.txt

@@ -0,0 +1,14 @@
+click>=8.0.0
+pandas>=2.0.0
+numpy>=1.20.0
+requests>=2.25.0
+ollama>=0.1.0
+torch>=2.0.0
+transformers>=4.20.0
+sentence-transformers>=2.0.0
+scikit-learn>=1.0.0
+reportlab>=3.6.0
+tqdm>=4.60.0
+
+[dev]
+pytest>=7.0.0

data_atlas-0.0.1/data_atlas.egg-info/top_level.txt

@@ -0,0 +1 @@
+data_dictionary_generator

data_atlas-0.0.1/data_dictionary_generator/column_stats.py

@@ -0,0 +1,56 @@
+import pandas as pd
+import numpy as np
+from collections import Counter
+from data_type import infer_data_type
+
+def compute_column_stats(series: pd.Series, sample_size: int = 5000) -> dict:
+    """
+    Compute statistics for a column, based on inferred data type.
+    """
+
+    if series.empty:
+        return None
+
+    non_null_series = series.dropna()
+    if non_null_series.empty:
+        return None
+
+    if len(non_null_series) > sample_size:
+        non_null_series = non_null_series.sample(sample_size, random_state=42)
+
+    inferred_types = non_null_series.apply(infer_data_type)
+    col_type = inferred_types.value_counts().idxmax()
+
+    stats = {}
+
+    if col_type in ["integer", "float"]:
+        numeric = pd.to_numeric(non_null_series, errors="coerce").dropna()
+        if not numeric.empty:
+            stats = {
+                "min": float(numeric.min()),
+                "max": float(numeric.max()),
+                "mean": float(numeric.mean()),
+                "median": float(numeric.median()),
+                "std": float(numeric.std())
+            }
+
+    elif col_type == "datetime":
+        dates = pd.to_datetime(non_null_series, errors="coerce", format="mixed").dropna()
+        if not dates.empty:
+            stats = {
+                "min_date": dates.min().isoformat(),
+                "max_date": dates.max().isoformat()
+            }
+
+    elif col_type in ["string", "boolean"]:
+        counts = Counter(non_null_series.astype(str))
+        top_values = counts.most_common(10)
+        stats = {
+            "top_values": top_values,
+            "unique_count": len(counts)
+        }
+
+    else:
+        return None
+
+    return stats if stats else None

data_atlas-0.0.1/data_dictionary_generator/data_type.py

@@ -0,0 +1,66 @@
+from dateutil.parser import parse as dateparse
+import pandas as pd
+import re
+
+_date_pattern = re.compile(
+    r"(\d{4}[-/]\d{2}[-/]\d{2})"           # 2024-01-31
+    r"|(\d{2}[-/]\d{2}[-/]\d{4})"           # 31-01-2024
+    r"|(\d{4}[-/]\d{2}[-/]\d{2}T\d{2}:\d{2})"  # 2024-01-31T12:00
+)
+
+def infer_data_type(value) -> str:
+    if pd.isna(value):
+        return "null"
+    if isinstance(value, bool):
+        return "boolean"
+    if isinstance(value, int):
+        return "integer"
+    if isinstance(value, float):
+        return "float"
+    if not isinstance(value, str):
+        return "string"
+    v = value.strip()
+    if v == "":
+        return "null"
+    lower = v.lower()
+    if lower in ("true", "false", "yes", "no", "y", "n", "t", "f"):
+        return "boolean"
+    temp_v = v.replace(",", "")
+    has_sign = temp_v.startswith(("+", "-"))
+    numeric_part = temp_v[1:] if has_sign else temp_v
+    if "+" in numeric_part or "-" in numeric_part:
+        return "string"
+    if numeric_part.isdigit():
+        return "integer"
+    try:
+        if "." in numeric_part:
+            if numeric_part.replace(".", "", 1).isdigit():
+                return "float"
+        float(temp_v)
+        return "float"
+    except ValueError:
+        pass
+
+    if _date_pattern.search(v):
+        try:
+            dateparse(v, fuzzy=False)
+            return "datetime"
+        except (ValueError, OverflowError):
+            pass
+
+    return "string"
+
+def infer_column_data_type(series: pd.Series, sample_size: int = 100) -> str:
+    """
+    Infer the most likely data type of a column by sampling non-null values
+    and selecting the most frequent inferred type.
+    """
+    non_null_series = series.dropna()
+    if non_null_series.empty:
+        return "null"
+    sample = non_null_series.sample(
+        min(sample_size, len(non_null_series)), random_state=42
+    )
+    inferred_types = sample.apply(infer_data_type)
+    most_common = inferred_types.value_counts().idxmax()
+    return most_common