ai-data-summarizer 0.7.1__py3-none-any.whl

ai_data_summarizer-0.7.1.dist-info/METADATA

@@ -0,0 +1,200 @@
+Metadata-Version: 2.5
+Name: ai-data-summarizer
+Version: 0.7.1
+Summary: A local CLI tool that uses pandas and AI to summarize and interpret datasets.
+Project-URL: Homepage, https://github.com/Bloomy52/ai-data-summarizer
+Author: Louie Bloomberg
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: anthropic
+Requires-Dist: google-genai
+Requires-Dist: openai
+Requires-Dist: openpyxl
+Requires-Dist: pandas
+Requires-Dist: pandas-stubs>=3.0.5.260730
+Requires-Dist: protobuf
+Requires-Dist: pwinput>=1.0.3
+Requires-Dist: readchar>=4.2.2
+Requires-Dist: sentencepiece
+Description-Content-Type: text/markdown
+
+# AI Data Summarization Tool
+A Python Tool that takes a dataset and uses pandas to generate a statistical summary and Artifical Intelligence (AI) to interpret the pandas summary and presents it to the user.
+
+This project is a continuation of my CS178 Final Project. You can find the repo at [Bloomy52/cs178-data-summarizer-py](https://www.github.com/Bloomy52/cs178-data-summarizer-py)
+
+### Why This Exists
+I created this project because I found that it is difficult to understand what an underlying dataset is and what it entails without reading and understanding the full dataset. I found that using a Large Language Model (LLM) to summarize the dataset made the dataset more approachable since I had a general understanding of what the dataset was and some features about said dataset I was analyzing. I specifically crafted the summary templates so they would help the user understand the dataset and its features. It can also give you a heads up if there are any concerns or anomalies before you start fully analyzing the data to prevent issues and to guide the user on the right path to analysis. 
+
+## Repo Structure
+The structure of this git repository is as follows:
+```text
+ai-data-summarizer/
+├── main.py                 # CLI entry point and main application logic
+├── summarizer.py           # Core data summarization functionality
+├── prompt.py               # Prompt templates and management
+├── tokenizer.py            # Token counting and management utilities
+├── .env.sample             # Sample credentials file (rename to .env)
+├── apicheck.py             # Checks API variables to prevent early issues
+├── envvar.py               # Loads environment variables from .env file
+├── fileloader.py           # Loads and parses CSV and Excel files
+├── profiler.py             # Profiles the dataset for pandas summarization
+├── requirements.txt        # Python dependencies
+├── README.md               # Project documentation
+├── INSTALL.md              # Project installation documentation
+├── CONFIGURATION.md        # Project configuration documentation
+├── LICENSE                 # MIT License
+├── .gitignore              # Git ignore rules
+├── pyproject.toml          # Python project application configuration files
+├── uv.lock                 # Python/uv dependency lock file
+├── Dockerfile              # Dockerfile for containerized deployment
+├── .dockerignore           # Docker ignore rules
+├── .vscode/
+│   └── settings.json       # VSCode configuration
+├── summaries/              # Summary output folder -- gets created upon first output summary
+├── .devcontainer/          
+│   └── devcontainer.json   # Dev container configuration for VSCode
+└── examples/
+    ├── CTA_Ridership_RedLine_WrigleyField_DailyTotals.csv                 # Sample dataset
+    └── OverviewPrompt_CTA_Ridership_RedLine_WrigleyField_DailyTotals.txt  # Sample summary
+```
+
+## Example
+The example used here is the number of daily riders from the Addison 'L' Stop on the Chicago Transit Authority's (CTA) Red Line using the Data Overview prompt. More information about the original dataset can be found at the bottom of the README. Other prompts will be added to the example folder as well.
+
+**CSV File Structure**
+```csv
+"date","daytype","rides"
+"01/01/2001","U","1,227"
+"01/02/2001","W","3,937"
+"01/03/2001","W","4,329"
+"01/04/2001","W","4,607"
+"01/05/2001","W","4,666"
+```
+* `date` is when the data was recorded
+* `daytype` is the type of day of which the data was recorded
+   - `W` is a weekday
+   - `A` is a Saturday
+   - `U` is Sunday/Holidays
+* `rides` is the number of riders recorded on a given day
+
+```bash
+sumdata
+```
+```text
+Enter the path to your CSV file: examples/CTA_Ridership_RedLine_WrigleyField_DailyTotals.csv
+
+Please Select a Summary:
+1. Just the Facts (AI is not used)
+2. TL;DR Summary
+3. Data Overview
+4. Deep Dive Analysis
+Or enter 0 to Exit
+Enter the number corresponding to your choice: 3
+Your input text has 195174 tokens, which is within the free tier limit for Gemini.
+Would you like to continue? (Y/n)
+Y
+
+Summary:
+
+Overview:
+
+I reviewed the dataset you shared. It contains daily transit ride tracking data spanning over 25 years, from 
+January 1, 2001, to March 31, 2026. This extensive daily record captures long-term ridership trends and reveals 
+how public transit usage fluctuates across weekdays, weekends, and holidays, alongside the long-term impact 
+of major external disruptions.
+
+```
+The rest of the example can be found in the `examples` folder labeled [OverviewPrompt_CTA_Ridership_RedLine_WrigleyField_DailyTotals.txt](examples/OverviewPrompt_CTA_Ridership_RedLine_WrigleyField_DailyTotals.txt)
+
+Summaries save to the `summaries` folder. It will be created automatically with the subfolder with the prompt and a header in the file.
+
+> [!TIP]
+> The capital `Y` means that it is the default option. You can click the `Enter`/`Return` key as a shortcut.
+
+
+## Project Requirements
+> [!NOTE]
+> This only supports Gemini at the moment. Support for the Anthropic and OpenAI APIs are coming soon.
+
+- Python
+- uv
+- Git
+- Gemini API Key from Google AI Studio
+
+See [INSTALL.md](INSTALL.md) for detailed setup instructions, including how to obtain API keys.
+
+
+## How to Use/Installation
+There are three options for running this program. Please choose one of these options.
+
+### Option A: Install as a package (recommended)
+```bash
+uv tool install git+https://www.github.com/Bloomy52/ai-data-summarizer.git
+```
+   Then run:
+```bash
+sumdata
+```
+
+### Option B: Run without installing using uv
+1. Clone the git repository and `cd` into it
+```bash
+git clone https://www.github.com/Bloomy52/ai-data-summarizer.git
+cd ai-data-summarizer
+```
+
+2. Rename `.env.sample` to `.env` and add your API key.  
+   See [CONFIGURATION.md](CONFIGURATION.md) for details.
+
+3. Run the tool with uv
+```bash
+uv run main.py
+```
+
+### Option C: Run without installing using pip
+1. Clone the git repository and `cd` into it
+```bash
+git clone https://www.github.com/Bloomy52/ai-data-summarizer.git
+cd ai-data-summarizer
+```
+
+2. Rename `.env.sample` to `.env` and add your API key.  
+   See [CONFIGURATION.md](CONFIGURATION.md) for details.
+
+3. Install dependencies and run the tool according to your OS
+
+**Linux & macOS Users**
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip3 install -r requirements.txt
+python3 main.py
+```
+**Windows Users**
+```powershell
+python -m venv .venv
+.\.venv\Scripts\Activate.ps1
+pip install -r requirements.txt
+python main.py
+```
+
+### Option D: Run using a Docker container
+This option requires that you have Docker installed on your computer. Information on how to install Docker is in [INSTALL.md](INSTALL.md#docker).
+```bash
+docker build -t ai-data-summarizer .
+docker run --rm -it --env-file .env ai-data-summarizer
+```
+
+
+
+> [!NOTE]
+> All four options run the same underlying code. Option A installs `sumdata` as a command while Options B & C run it directly via `main.py` inside a virtual environment. 
+
+
+   
+## License
+This project is licensed under the MIT License. The full license text can be found [here](LICENSE)
+
+*The original CTA example dataset came from the City of Chicago's Data Portal. You can find the original dataset link [here](https://data.cityofchicago.org/Transportation/CTA-Ridership-L-Station-Entries-Daily-Totals/5neh-572f/about_data)

ai_data_summarizer-0.7.1.dist-info/RECORD

@@ -0,0 +1,13 @@
+apicheck.py,sha256=nyBXiEeg5BTW8PI78PgVzcmrwAZCw1QDgAGWkLVCeJQ,2842
+envvar.py,sha256=ne2vizD19WupGvoeBaIw_6P26uKJobJndNOFZHxCNnU,3812
+fileloader.py,sha256=y2B1GdwO6VaqqzUNjfOkwG0rKsfysSNbpyuYIquw6IY,4224
+main.py,sha256=qgW6ezqWU47kvgePzs3kLUQLwXGgrO0IA5EhpXxrTfo,5309
+profiler.py,sha256=GWQZKssqplad9LeRuKT9xD-RIcRzi-CRhyTtm9O-tYY,3795
+prompt.py,sha256=W8jSf7K6DPdTyK9HO06fbNocmoHm1nfkpQd8lfN1wrU,7216
+summarizer.py,sha256=71nRxw6aRPf0aA5gg45hQoIUSBa-MMOiuMlACld4amU,3534
+tokenizer.py,sha256=4VLJ0umfy7r9Yj3QxK55B_Xq-SYM_N10PFE0CwQBYlo,1296
+ai_data_summarizer-0.7.1.dist-info/METADATA,sha256=TEEJ2hmAXzHPvj4_8mVUS8tpbWx0_PsBqOItLyNO7os,8101
+ai_data_summarizer-0.7.1.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+ai_data_summarizer-0.7.1.dist-info/entry_points.txt,sha256=q7AKQVFhwm3ZFqGTzj6Se0vOEXSxBApuMA4xParXFWA,38
+ai_data_summarizer-0.7.1.dist-info/licenses/LICENSE,sha256=NRgWTyaRalxqDDAqDxput6odIMzsPSV4RkkQh2cCoN0,1072
+ai_data_summarizer-0.7.1.dist-info/RECORD,,

ai_data_summarizer-0.7.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

ai_data_summarizer-0.7.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+sumdata = main:main

ai_data_summarizer-0.7.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Louie Bloomberg
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

apicheck.py

@@ -0,0 +1,86 @@
+# API Key Checker
+# apicheck.py
+# This file contains functions to check for API keys and token counts for the different model providers.
+# SPDX-License-Identifier: MIT
+
+import os   
+import sys
+
+import anthropic
+from google import genai
+from openai import OpenAI
+
+from envvar import *
+
+def check_api_keys(provider_choice):
+    if provider_choice == 1:
+        check_for_gemini_api_key()
+        check_valid_gemini_api_key()
+    elif provider_choice == 2:
+        check_for_openai_api_key()
+        check_valid_openai_api_key()
+    elif provider_choice == 3:
+        check_for_anthropic_api_key()
+        check_valid_anthropic_api_key()
+
+def check_for_gemini_api_key():
+    if os.getenv("GEMINI_API_KEY") is None or os.getenv("GEMINI_API_KEY") == "":
+        print("Error: Gemini API key not found. Would you like to set it now? (Y/n)")
+        if input().lower() != 'n':
+            set_api_keys(1)
+        else:
+            sys.exit(1)
+    else:
+        return True
+    
+def check_valid_gemini_api_key():
+    try:
+        client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
+        # Attempt to list models to check if the API key is valid
+        models = client.models.list()
+        return True
+    except Exception as e:
+        print(f"Error: Invalid Gemini API key. Please check your .env file. Details: {e}")
+        sys.exit(1)
+
+
+def check_for_openai_api_key():
+    if os.getenv("OPENAI_API_KEY") is None or os.getenv("OPENAI_API_KEY") == "":
+        print("Error: OpenAI API key not found. Would you like to set it now? (Y/n)")
+        if input().lower() != 'n':
+            set_api_keys(2)
+        else:
+            sys.exit(1)
+    else:
+        return True
+    
+def check_valid_openai_api_key():
+    try:
+        client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+        # Attempt to list models to check if the API key is valid
+        # TODO: Implement correct method to validate OpenAI API key
+        return True
+    except Exception as e:
+        print(f"Error: Invalid OpenAI API key. Please check your .env file. Details: {e}")
+        sys.exit(1)
+
+
+def check_for_anthropic_api_key():
+    if os.getenv("ANTHROPIC_API_KEY") is None or os.getenv("ANTHROPIC_API_KEY") == "":
+        print("Error: Anthropic API key not found. Would you like to set it now? (Y/n)")
+        if input().lower() != 'n':
+            set_api_keys(3)
+        else:
+            sys.exit(1)
+    else:
+        return True
+    
+def check_valid_anthropic_api_key():
+    try:
+        client = anthropic.Client(api_key=os.getenv("ANTHROPIC_API_KEY"))
+        # Attempt to list models to check if the API key is valid
+        # TODO: Implement correct method to validate Anthropic API key
+        return True
+    except Exception as e:
+        print(f"Error: Invalid Anthropic API key. Please check your .env file. Details: {e}")
+        sys.exit(1)

envvar.py

@@ -0,0 +1,99 @@
+# Environment Variable Management
+# envvar.py
+# This file contains functions to manage environment variables, including reading and writing to the .env file
+# SPDX-License-Identifier: MIT
+
+import os
+import pwinput
+
+
+def _strip_quotes(value):
+    # Remove a single matching pair of surrounding quotes (single or double), if present.
+    value = value.strip()
+    if len(value) >= 2 and value[0] == value[-1] and value[0] in ('"', "'"):
+        return value[1:-1]
+    return value
+
+
+def write_env_value(env_path, key_name, value):
+    # Update key_name's value in the .env file in place, preserving other lines.
+    # If key_name isn't found in the file, it gets appended.
+    lines = []
+    if os.path.exists(env_path):
+        with open(env_path, "r") as env_file:
+            lines = env_file.readlines()
+
+    found = False
+    for i, line in enumerate(lines):
+        stripped = line.strip()
+        if "=" in stripped and not stripped.startswith("#"):
+            existing_key = stripped.split("=", 1)[0].strip()
+            if existing_key == key_name:
+                lines[i] = f"{key_name}={value}\n"
+                found = True
+                break
+
+    if not found:
+        lines.append(f"{key_name}={value}\n")
+
+    with open(env_path, "w") as env_file:
+        env_file.writelines(lines)
+
+
+def create_env():
+    # Create a .env file if it doesn't exist
+    base_dir = os.path.dirname(os.path.abspath(__file__))
+    env_path = os.path.join(base_dir, ".env")
+    # Check if the .env file exists
+    if not os.path.exists(env_path):
+        # have system create a .env file with default values
+        with open(env_path, "w") as env_file:
+            env_file.write("GEMINI_API_KEY=\n")
+            env_file.write("OPENAI_API_KEY=\n")
+            env_file.write("ANTHROPIC_API_KEY=\n")
+            env_file.write("GEMINI_FREE_TIER=True\n") # TODO: Remove Free Tier Flag and make that hardcoded
+    
+    return None
+
+def read_env():
+    # read full .env and set environment variables
+    base_dir = os.path.dirname(os.path.abspath(__file__))
+    env_path = os.path.join(base_dir, ".env")
+    if not os.path.exists(env_path):
+        create_env()
+    if os.path.exists(env_path):
+        with open(env_path, "r") as env_file:
+            for line in env_file:
+                if "=" in line:
+                    key, value = line.strip().split("=", 1)
+                    value = _strip_quotes(value)
+                    os.environ[key.strip()] = value
+    return None
+
+def set_api_keys(model_provider_choice):
+    base_dir = os.path.dirname(os.path.abspath(__file__))
+    env_path = os.path.join(base_dir, ".env")   
+
+    # Select the appropriate API key based on the model provider choice
+    if model_provider_choice == 1:
+        api_key_name = "GEMINI_API_KEY"
+    elif model_provider_choice == 2:
+        api_key_name = "OPENAI_API_KEY"
+    elif model_provider_choice == 3:
+        api_key_name = "ANTHROPIC_API_KEY"
+
+    if os.getenv(api_key_name):  # Check to see if environment variables are already in RAM
+        return True
+
+    read_env()  # Read the .env file to set environment variables
+
+    # If environment variable is not in .env nor in RAM, prompt the user to enter the API key and save it to .env securely
+    if not os.getenv(api_key_name):
+        api_key = pwinput.pwinput(prompt=f"Enter your {api_key_name} (or leave blank to skip): ", mask="*").strip()
+        api_key = _strip_quotes(api_key)  # Allow the user to paste a key wrapped in quotes
+        if api_key:
+            os.environ[api_key_name] = api_key
+            write_env_value(env_path, api_key_name, api_key)
+            print(f"{api_key_name} has been set and saved to {env_path}.")
+        else:
+            print(f"Warning: {api_key_name} is not set. You will not be able to use the selected model provider.")

fileloader.py

@@ -0,0 +1,142 @@
+# Dataset File Loader
+# fileloader.py
+# This module provides functions to load and read CSV files for processing.
+# SPDX-License-Identifier: MIT
+
+import os
+import sys
+import pandas as pd
+
+
+def detect_file_type(filepath):
+    """
+    Determines the normalized file type ("csv" or "excel") based on the file extension.
+    Raises a ValueError if the extension is not supported.
+
+    Returns: str: "csv" or "excel"
+    """
+    _, ext = os.path.splitext(filepath)
+    ext = ext.lower()
+
+    if ext == ".csv":
+        return "csv"
+    elif ext in [".xlsx", ".xls"]:
+        return "excel"
+    else:
+        print(f"Unsupported file extension '{ext}'. Supported extensions are: .csv, .xlsx, .xls")
+        sys.exit(1)
+
+
+def get_sheet_names(filepath):
+    """
+    Returns a list of sheet names if the file is an Excel workbook.
+    Returns None for CSV files, since they do not have sheets.
+
+    Returns: list[str] | None
+    """
+    file_type = detect_file_type(filepath)
+
+    if file_type == "csv":
+        return None
+
+    try:
+        excel_file = pd.ExcelFile(filepath)
+    except Exception as e:
+        print(f"Could not read Excel file '{filepath}'. Details: {e}")
+        sys.exit(1)
+
+    return excel_file.sheet_names
+
+def choose_sheet(filepath, file_type):
+    if file_type == "excel":
+        sheet_names = get_sheet_names(filepath)
+        if sheet_names is None:
+            print(f"Error: Could not retrieve sheet names from Excel file '{filepath}'.")
+            sys.exit(1)
+
+        print("\nAvailable sheets:")
+        for idx, sheet in enumerate(sheet_names):
+            print(f"{idx + 1}. {sheet}")
+
+        while True:
+            sheet_choice = input("Enter the number of the sheet you want to load: ").strip()
+            try:
+                sheet_index = int(sheet_choice) - 1
+                if 0 <= sheet_index < len(sheet_names):
+                    selected_sheet = sheet_names[sheet_index]
+                    break
+                else:
+                    print("Invalid choice. Please enter a valid number.")
+            except Exception as e:
+                print(f"Invalid input. Please enter a number. Details: {e}")
+                sys.exit(1)
+
+        # Load the selected sheet as CSV text
+        try:
+            dataframe = pd.read_excel(filepath, sheet_name=selected_sheet)
+            csv_text = dataframe.to_csv(index=False)
+            return csv_text
+        except Exception as e:
+            print(f"Error: Could not read the selected sheet '{selected_sheet}' from Excel file '{filepath}'. Details: {e}")
+            sys.exit(1)
+
+
+def load_csv(filepath):
+    """
+    Loads a CSV file and returns it as normalized CSV-formatted text.
+
+    Returns: str
+    """
+    try:
+        dataframe = pd.read_csv(filepath, thousands=',', decimal='.')
+    except Exception as e:
+        print(f"Could not read CSV file '{filepath}'. Details: {e}")
+        sys.exit(1)
+
+    if dataframe.empty:
+        print(f"CSV file '{filepath}' appears to be empty.")
+        sys.exit(1)
+
+    return dataframe
+
+
+def load_excel(filepath, sheet_name=None):
+    """
+    Loads a single sheet from an Excel file (.xlsx or .xls) and returns it as
+    normalized CSV-formatted text. If sheet_name is not provided, the first
+    sheet in the workbook is used.
+
+    Returns: str
+    """
+    try:
+        if sheet_name is None:
+            available_sheets = get_sheet_names(filepath)
+            sheet_name = available_sheets[0]
+
+        dataframe = pd.read_excel(filepath, sheet_name=sheet_name)
+    except Exception as e:
+        print(f"Could not read Excel file '{filepath}'. Details: {e}")
+        sys.exit(1)
+
+    if dataframe.empty:
+        print(f"Sheet '{sheet_name}' in file '{filepath}' appears to be empty.")
+        sys.exit(1)
+
+    return dataframe
+
+
+def load_file(filepath, file_type):
+    """
+    Dispatches to the correct loader based on the file's detected type, and
+    returns normalized CSV-formatted text regardless of the original source format.
+
+    sheet_name is only relevant for Excel files. It is ignored for CSV files.
+
+    Returns: str
+    """
+
+    if file_type == "csv":
+        return load_csv(filepath)
+    elif file_type == "excel":
+        return load_excel(filepath)
+    return None

main.py

@@ -0,0 +1,155 @@
+# Main Code File
+# main.py
+# Code file containing the main function and the main logic for the AI Data Summarizer program.
+# SPDX-License-Identifier: MIT
+
+# Import Statements
+import os
+import sys
+import datetime
+import csv
+from readchar import readkey, key
+
+# Import the functions from the summarizer.py helper file
+from summarizer import *
+from tokenizer import *
+from prompt import *
+from apicheck import *
+from envvar import *
+from fileloader import *
+from profiler import *
+
+# Function Definitions
+
+def check_tokens_gemini(prompt, text):
+    # This function checks the number of tokens to make sure that they are within the Free Tier limits
+    google_tokens = google_tokenizer(prompt, text)
+    if os.getenv("GEMINI_FREE_TIER") == "True" and google_tokens > 250000:
+        print(f"Warning: Your input text has {google_tokens} tokens, which exceeds the free tier limit of 250,000 tokens per minute for Gemini. Consider reducing the input size or upgrading your plan. ")
+        sys.exit(1)
+    elif os.getenv("GEMINI_FREE_TIER") == "True":
+        print(f"Your input text has {google_tokens} tokens, which is within the free tier limit for Gemini.")
+        print("Would you like to continue? (Y/n)")
+        while True:
+            k = readkey()
+            if k == 'y' or k == 'Y' or k == key.ENTER:
+                return None
+            elif k == 'n' or k == 'N':
+                print("Exiting...")
+                sys.exit(1)
+            else:
+                print("Invalid input. Please enter y or n.")
+
+    # TODO (maybe): Add cost functionality to estimate cost of input response
+    return None
+
+def check_tokens_openai(prompt, text):
+    tokens = openai_tokenizer(prompt, text)
+    print(f"Your input text has {tokens} tokens.")
+    return None
+
+def check_tokens_anthropic(prompt, text):
+    tokens = anthropic_tokenizer(prompt, text)
+    print(f"Your input text has {tokens} tokens.")
+    return None
+
+def get_model_provider():
+    while True:
+        print("\nSelect a model provider:")
+        print("1. Google Gemini")
+        print("2. OpenAI")
+        print("3. Anthropic")
+        print("Or press q to quit the program.")
+        k = readkey()
+
+        if k == '1':
+            return 1 # Gemini
+        elif k == '2':
+            return 2 # OpenAI
+        elif k == '3':
+            return 3 # Anthropic
+        elif k == 'q':
+            print("Exiting...")
+            sys.exit(1)
+        else:
+            print("Invalid choice. Please enter 1, 2, or 3.")
+
+# Define Prompt Choosing Function
+def get_prompt_type():
+    """
+    This function allows the user to select a prompt from a list of available prompts.
+    It returns the selected prompt as a string.
+    Returns: str: The selected prompt.
+    """
+    while True:
+        print("\nSelect a Summary:")
+        print("1. Just the Facts (AI is not used)")
+        print("2. TL;DR Summary")
+        print("3. Data Overview")
+        print("4. Deep Dive Analysis")
+        print("Or press q to quit the program.")
+        k = readkey()
+
+        if k == '2':
+            return "tldr"
+        elif k == '3':
+            return "overview"
+        elif k == '4':
+            return "deepdive"
+        elif k == '1':
+            return "facts"
+        elif k == 'q':
+            print("Exiting...")
+            sys.exit(1)
+        else:
+            print("Invalid choice. Please enter 1, 2, 3, 4, or q.")
+
+# Main Function
+def main():
+    read_env()  # Read the .env file to set environment variables
+    # Main Interactive Loop:
+    # Get input file
+    while True:
+        filepath = input("\nEnter the path to your CSV file: ").strip()
+
+        # Remove quotes if user wrapped path in quotes
+        input_file = filepath.strip('"\'')
+        
+
+        if not os.path.exists(input_file):
+            print(f"Error: File '{input_file}' not found.  Please try again.")
+            continue
+
+        if not input_file.lower().endswith('.csv') and not input_file.lower().endswith('.xlsx') and not input_file.lower().endswith('.xls'):
+            print("Warning: file extension is not .csv, .xlsx, or .xls. Continue anyway? (y/N)")
+            if input().lower() != 'y':
+                continue
+
+        break
+
+    file_type = detect_file_type(input_file)
+
+    df = load_file(input_file, file_type)
+    profile = build_base_profile(df)
+    sample = get_sample(df)
+    csv_text = "\nData Profile:" + profile + "\nData Sample:\n" + sample
+
+    model_provider_choice = 1 # get_model_provider()
+                              # Defaults to Gemini since it is the only one provided
+    check_api_keys(model_provider_choice)
+    prompt_type = get_prompt_type()
+    if prompt_type == "facts":
+        summary = "Dataset Statistical Summary:\n" + profile + "\nData Sample:\n" + sample
+        write_outfile(summary, os.path.basename(input_file).split(".")[0], prompt_type, "None")
+    else:
+        prompt = get_prompt(prompt_type)
+        check_tokens_gemini(prompt, csv_text)
+        summary = gemini_summarizer(prompt, csv_text, os.path.basename(input_file).split(".")[0], prompt_type)
+
+    print("\nSummary:\n")
+    print(summary)
+
+    return None
+
+if __name__ == "__main__":
+    main()

profiler.py

@@ -0,0 +1,108 @@
+# Pandas Pre-Summarization Profiler
+# profiler.py
+# This module provides functions to profile a DataFrame before summarization.
+# SPDX-License-Identifier: MIT
+
+import pandas as pd
+
+
+def _parse_dates(df):
+    # Returns a copy of df with any date-like object columns converted to datetime
+    df = df.copy()
+    for col in df.select_dtypes(include="object").columns:
+        try:
+            converted = pd.to_datetime(df[col], format="mixed", errors="coerce")
+            if converted.notna().sum() > len(df) * 0.9:  # 90%+ parsed successfully
+                df[col] = converted
+        except Exception:
+            continue
+    return df
+
+
+def build_base_profile(df):
+    """
+    Returns a plain-text statistical profile of the DataFrame.
+    Includes shape, dtypes, null counts, numeric stats, and top categorical values.
+    Does not mutate the input DataFrame.
+
+    Returns: str
+    """
+    df = _parse_dates(df)
+    lines = []
+
+    # Shape
+    num_rows, num_cols = df.shape
+    lines.append(f"Rows: {num_rows}")
+    lines.append(f"Columns: {num_cols}")
+    lines.append("")
+
+    # Dtypes and null counts
+    lines.append("Column Overview:")
+    for col in df.columns:
+        null_count = df[col].isna().sum()
+        lines.append(f"  {col} ({df[col].dtype}) — {null_count} nulls")
+    lines.append("")
+
+    # Numeric stats
+    numeric_cols = df.select_dtypes(include="number").columns
+    if len(numeric_cols) > 0:
+        lines.append("Numeric Column Stats:")
+        lines.append(df[numeric_cols].describe().to_string())
+        lines.append("")
+
+    # Date column summary
+    date_cols = df.select_dtypes(include=["datetime"]).columns
+    if len(date_cols) > 0:
+        lines.append("Date Column Summary:")
+        for col in date_cols:
+            min_date = df[col].min()
+            max_date = df[col].max()
+            if pd.isna(min_date) or pd.isna(max_date):
+                lines.append(f"  {col}: (no non-null datetimes)")
+                continue
+            span = (max_date - min_date).days + 1
+            duplicate_dates = df[col].duplicated().sum()
+            lines.append(f"  {col}: {min_date.date()} to {max_date.date()} ({span} days)")
+            if duplicate_dates > 0:
+                lines.append(f"  {col} has {duplicate_dates} duplicate date(s)")
+        lines.append("")
+
+    # Standout values with date context
+    if len(date_cols) > 0 and len(numeric_cols) > 0:
+        date_col = date_cols[0]
+        show_all_cols = len(df.columns) <= 5
+        lines.append("Standout Values:")
+        for col in numeric_cols:
+            if show_all_cols:
+                top5 = df.nlargest(5, col).to_string(index=False)
+                bottom5 = df.nsmallest(5, col).to_string(index=False)
+            else:
+                top5 = df[[date_col, col]].nlargest(5, col).to_string(index=False)
+                bottom5 = df[[date_col, col]].nsmallest(5, col).to_string(index=False)
+            lines.append(f"  Top 5 highest {col}:")
+            lines.append(top5)
+            lines.append(f"  Bottom 5 lowest {col}:")
+            lines.append(bottom5)
+            lines.append("")
+
+    # Categorical top values
+    categorical_cols = df.select_dtypes(include=["object", "category"]).columns
+    if len(categorical_cols) > 0:
+        lines.append("Categorical Column Summary:")
+        for col in categorical_cols:
+            unique_count = df[col].nunique()
+            top_values = df[col].value_counts().head(5).to_string()
+            lines.append(f"  {col} ({unique_count} unique values):")
+            lines.append(f"{top_values}")
+            lines.append("")
+
+    return "\n".join(lines)
+
+
+def get_sample(df, n=10):
+    """
+    Returns the first n rows of the DataFrame as CSV-formatted text.
+
+    Returns: str
+    """
+    return df.head(n).to_csv(index=False)

prompt.py

@@ -0,0 +1,144 @@
+# Main Prompt File
+# prompt.py
+# File contains different prompts which can be used to help the user understand the file 
+# SPDX-License-Identifier: MIT
+
+# Define Prompt Choosing Functions
+def get_prompt(prompt_type):
+    """
+    This function retrieves the appropriate prompt based on the user's selection.
+    Returns: str: The selected prompt.
+    """
+    if prompt_type == "tldr":
+        return get_tldr_prompt()
+    elif prompt_type == "overview":
+        return get_overview_prompt()
+    elif prompt_type == "deepdive":
+        return get_deepdive_prompt()
+
+
+# Define Prompts that can be used
+
+def get_tldr_prompt():
+    task_summary = f"""
+    ## Task Summary:
+    {{Produce a 'Too Long; Didn’t Read' (TL;DR) summary of the attached dataset and statistical summary. The TL;DR sentence should describe, in one high-level line, what the dataset is and what it covers. Then provide 2–3 bullets highlighting the most notable patterns, trends, or anomalies visible in the data. Use numeric ranges when helpful, but keep the focus on the biggest takeaways.}}
+    """
+
+    response_style = f"""
+    ## Response style and format requirements:
+    - {{Write in a sharp, punchy, and bottom-line-up-front (BLUF) style}}
+    - {{Format: Start with a single 'TL;DR:' sentence, followed by a short bulleted list}}
+    - {{Bullets must follow these rules:
+        - Start with a short, strong label (e.g., 'Trend shift:', 'Category contrast:', 'Peak anomaly:')
+        - Contain exactly one idea per bullet
+        - Use numeric anchors when possible
+        - Avoid hedging, filler, or multi-clause sentences
+        - Read like headlines, not explanations}}
+    - {{Numbers in the stats block are exact — use them directly, don't estimate from sample rows}}
+    - {{Strictly limit the response to 100 words or less}}
+    - {{Use plain text only — no Markdown. Bullets are to be noted with '-'}}
+    """
+
+    final_prompt = f"""{task_summary}
+    {response_style}"""
+    
+    return final_prompt
+
+
+def get_overview_prompt():
+    # Prompt for the Summarizer:
+    # Use this to clearly define the task and job needed by the model
+    task_summary = f"""
+    ## Task Summary:
+    {{Review the attached CSV and statistical summary and summarize what the data covers, including anything notable or unusual.}}
+    """
+
+    # Use this to provide contextual information related to the task
+    context_information = f"""
+    ## Context Information:
+    - {{Standard CSV format with headers in the first row}}
+    - {{Columns may include numbers, text, or dates}}
+    - {{Treat all dates in the data file as a recorded value and not predictions}}
+    - {{The dataset may cover any domain — do not assume a specific subject area}}
+    - {{The stats block (counts, mean/median, ranges, date span, category counts) is precomputed and exact — treat these numbers as ground truth}}
+    - {{Standout/extreme rows shown are the most unusual in the dataset, not typical examples}}
+    - {{Any raw sample rows are shown only to illustrate formatting and column meaning, not to infer statistics}}
+    """
+
+    # Use this to provide any model instructions that you want model to adhere to
+    model_instructions = f"""
+    ## Model Instructions:
+    - {{Explain what each column represents and flag anything out of the ordinary}}
+    - {{Base all observations only on the data provided}}
+    - {{Only use historical context or external knowledge where it clearly and directly explains a specific data pattern — do not force connections}}
+    - {{If no external context is relevant, rely entirely on what the data shows}}
+    """
+
+    # Use this to provide response style and formatting guidance
+    response_style = f"""
+    ## Response style and format requirements:
+    - {{Write a standalone written summary as if briefing a coworker}}
+    - {{Use three sections: overview, column breakdown, and key takeaways}}
+    - {{Limit the response to 500 words or less}}
+    - {{Use plain text only — no Markdown. Bullets are to be noted with '-'. No Markdown headings.}}
+    """
+
+    # Concatenate to final prompt
+    final_prompt = f"""{task_summary}
+    {context_information}
+    {model_instructions}
+    {response_style}"""
+
+    return final_prompt
+
+
+def get_deepdive_prompt():
+    task_summary = f"""
+    ## Task Summary:
+    {{Perform a detailed deep dive analysis of the attached CSV dataset and statistical summary. Go beyond surface-level description to examine distributions, patterns, relationships, and notable characteristics in the data.}}
+    """
+
+    context_information = f"""
+    ## Context Information:
+    - {{Standard CSV format with headers in the first row}}
+    - {{Columns may include numbers, text, dates, or categorical values}}
+    - {{Treat all dates as recorded historical values}}
+    - {{The dataset may cover any domain — analyze based solely on what is present}}
+    """
+
+    input_composition = f"""
+    ## Input Composition:
+    - {{The stats block (counts, mean, median, std, min/max, date span, category counts) is precomputed directly from the full dataset and is exact — treat it as ground truth, never recompute or estimate these figures yourself}}
+    - {{The standout/extreme rows (top and bottom values) represent the most unusual points in the entire dataset, not a representative sample — use them only to discuss outliers, spikes, or anomalies}}
+    - {{The raw sample rows are a small illustrative slice included only to show formatting, column meaning, and qualitative texture — they are not necessarily representative of the full distribution and should not be used to infer statistics}}
+    """
+
+    model_instructions = f"""
+    ## Model Instructions:
+    - {{For numeric columns: report range (min-max), central tendency (mean/median if relevant), and distribution shape}}
+    - {{For numeric columns: also describe distribution shape based on the relationship between mean, median, and standard deviation}}
+    - {{For categorical/text columns: list top unique values with counts and note any dominant categories}}
+    - {{Identify any clear relationships or correlations between columns that stand out}}
+    - {{Highlight temporal patterns if dates are present, or geographic patterns if location data exists}}
+    - {{Flag outliers, unusual spikes/drops, or data quality concerns}}
+    - {{Base every observation strictly on the data provided — do not speculate beyond visible evidence}}
+    - {{Suggest 2–3 specific follow-up questions or analyses the user could explore next}}
+    """
+
+    response_style = f"""
+    ## Response style and format requirements:
+    - {{Write as if explaining the data in detail to a data-savvy coworker}}
+    - {{Use these four clear sections in order: Overview, Column Analysis, Key Patterns & Relationships, Takeaways & Next Steps}}
+    - {{Use plain text only — no Markdown. Bullets are to be noted with '-'. No Markdown headings in the final output.}}
+    - {{Keep the total response under 800 words}}
+    - {{Be specific and quantitative where possible}}
+    """
+
+    final_prompt = f"""{task_summary}
+    {context_information}
+    {input_composition}
+    {model_instructions}
+    {response_style}"""
+
+    return final_prompt

summarizer.py

@@ -0,0 +1,109 @@
+# Dataset Summarizer
+# summarizer.py
+# Code File for calling the summarization functions -- Mainly a helper file to keep code organized
+# SPDX-License-Identifier: MIT
+
+# Import Statements
+import os
+import sys
+from zoneinfo import ZoneInfo
+import datetime as dt
+import csv
+
+# Importing AI Libraries
+from openai import OpenAI
+import anthropic
+from google import genai
+from google.genai import types
+
+
+# Function Definitions
+# TODO: Implement summarization functions here
+
+# Write File Function
+def write_outfile(output, filename, prompt_type, model_id):
+    # Create summaries directory if it doesn't exist
+    summaries_dir = f"./summaries/{prompt_type}"
+    os.makedirs(summaries_dir, exist_ok=True)
+    
+    now = dt.datetime.now()
+    date_time_string = now.strftime("%Y-%m-%d %H-%M-%S")
+    central_tz = ZoneInfo("America/Chicago")
+    date_written = dt.datetime.now(central_tz).strftime("%A, %B %d, %Y")
+    time_written = dt.datetime.now(central_tz).strftime("%I:%M %p %Z")
+    summary_local_path = f"{summaries_dir}/{date_time_string}.{filename}.{model_id}.txt"
+    with open(summary_local_path, "w", encoding="utf-8") as outfile:
+        # Add Header Section to the output file
+        outfile.write("=" * 10 + "BEGIN HEADER" + "=" * 10 + "\n")
+        outfile.write("Date Written: " + date_written + "\n")
+        outfile.write("Time Written: " + time_written + "\n")
+        outfile.write("Model Used: " + model_id + "\n")
+        outfile.write("Prompt Type: " + prompt_type + "\n")
+        outfile.write("File Name: " + filename + "\n")
+        outfile.write("=" * 10 + "END HEADER" + "=" * 10 + "\n\n")
+        
+        # Add main summary output to the output file
+        outfile.write(output)
+    
+    return None
+
+# Gemini Summarizer
+def gemini_summarizer(prompt, text, filename, prompt_type):
+    MODEL_ID = "gemini-3.5-flash"
+    try:
+        client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
+
+        response = client.models.generate_content(
+            model=MODEL_ID,
+            contents=[
+                text,
+                prompt,
+            ]
+        )
+        output = response.text
+    except Exception as e:
+        print(f"Error generating summary: {e}")
+        sys.exit(1)
+    
+    write_outfile(output, filename, prompt_type, MODEL_ID)
+    return output
+
+
+def openai_summarizer(prompt, text, filename, prompt_type):
+    MODEL_ID = "gpt-5.4-mini"
+    try:
+        client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+
+        response = client.responses.create(
+            model=MODEL_ID,
+            messages=[
+                {"role": "system", "content": "You are a helpful assistant."},
+                {"role": "user", "content": prompt + "\n\n" + text},
+            ],
+        )
+        output = response.output_text
+    except Exception as e:
+        print(f"Error generating summary: {e}")
+        sys.exit(1)
+    
+    write_outfile(output, filename, prompt_type, MODEL_ID)
+    return output
+
+def anthropic_summarizer(prompt, text, filename, prompt_type):
+    MODEL_ID = "claude-haiku-4-5"
+    try:
+        client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
+
+        message = client.messages.create(
+            model=MODEL_ID,
+            messages=[
+                {"role": "user", "content": prompt + "\n\n" + text},
+            ],
+        )
+        output = message.content[0].text
+    except Exception as e:
+        print(f"Error generating summary: {e}")
+        sys.exit(1)
+    
+    write_outfile(output, filename, prompt_type, MODEL_ID)
+    return output

tokenizer.py

@@ -0,0 +1,48 @@
+# Dataset Tokenizer
+# tokenizer.py
+# Code File for calling the tokenization functions -- Mainly a helper file to keep code organized
+# SPDX-License-Identifier: MIT
+
+import os
+import sys
+
+from openai import OpenAI
+import anthropic
+from google import genai
+from google.genai import local_tokenizer
+
+# Tokenization Functions
+# Google GenAI Tokenizer
+def google_tokenizer(prompt, text):
+    MODEL_ID = "gemini-3.5-flash"
+    client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
+
+    response = client.models.count_tokens(
+        model=MODEL_ID,
+        contents=[
+            text,
+            prompt,
+        ]
+    )
+    tokens = response.total_tokens
+    return tokens
+
+# OpenAI Tokenizer
+def openai_tokenizer(prompt, text):
+    client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+    response = client.responses.input_tokens.count(
+        model="gpt-5.4-mini",
+        instructions=prompt,
+        input=text,
+    )
+    return response.input_tokens
+
+# Anthropic Tokenizer
+def anthropic_tokenizer(prompt, text):
+    client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
+    response = client.messages.count_tokens(
+        model="claude-haiku-4-5",
+        system=prompt,
+        messages=[{"role": "user", "content": text}],
+    )
+    return response.get("input_tokens", 0)