universal-database-inspector 0.1.0__tar.gz

universal_database_inspector-0.1.0/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.4
+Name: universal_database_inspector
+Version: 0.1.0
+Summary: Automated MySQL schema decomposition with AI-powered labeling and descriptions.
+Author: nailen1
+License: MIT
+Keywords: mysql,schema,openai,database,inspector,labeling
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: mysql-connector-python>=8.0.0
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: openai>=1.0.0
+
+# universal_database_inspector
+
+MySQL 데이터베이스의 스키마를 자동으로 분석하고, AI를 활용하여 컬럼 라벨링과 테이블 설명을 생성하는 Python 패키지입니다.
+
+## Features
+
+- **스키마 추출** — 데이터베이스의 전체 테이블·컬럼 구조를 JSON으로 저장
+- **AI 컬럼 라벨링** — OpenAI 모델을 활용하여 각 컬럼의 한국어 라벨을 자동 생성
+- **AI 테이블 설명** — 테이블의 목적과 용도를 한국어로 요약
+- **테이블 통계** — 행 수, 컬럼 수, 날짜 범위 자동 수집
+- **Table 객체** — 테이블별 구조·라벨·설명·데이터를 하나의 인터페이스로 접근
+
+## Installation
+
+```bash
+pip install git+https://github.com/nailen1/universal_database_inspector.git
+```
+
+## Setup
+
+`.env.example`을 참고하여 `.env` 파일을 생성합니다.
+
+```
+# MySQL Database Connection
+DB_HOST=your_host
+DB_PORT=3306
+DB_USER=your_user
+DB_PASSWORD=your_password
+DB_NAME=your_database
+
+# OpenAI API Key
+OPENAI_API_KEY=your_openai_api_key
+```
+
+## Quick Start
+
+```python
+from universal_database_inspector import init_structure, inspect_all, describe_all_tables, Table
+```
+
+### 1. 출력 폴더 초기화
+
+```python
+init_structure()
+```
+
+```
+database_structure/
+├── labels/
+└── description/
+```
+
+### 2. 데이터베이스 구조 추출
+
+```python
+structure = inspect_all()
+list(structure.keys())
+# ['bond', 'currency', 'daily_price', 'index', 'index_membership', 'stock_info']
+```
+
+`database_structure/structure.json`에 전체 테이블-컬럼 매핑이 저장됩니다.
+
+### 3. 테이블 설명 생성
+
+```python
+describe_all_tables()
+```
+
+각 테이블에 대해 AI 라벨링과 설명 파일을 일괄 생성합니다.
+
+```
+[1/6] describing: bond
+[2/6] describing: currency
+...
+done: 6 created, 0 skipped (total 6)
+```
+
+### 4. Table 객체로 접근
+
+```python
+table = Table('bond')
+
+table.columns       # ['code', 'date', 'open', 'high', 'low', 'close']
+table.labels        # {'code': '채권 코드', 'date': '거래일자', 'open': '시가', ...}
+table.description   # {'description': '...', 'first_date': '1962-01-02', 'last_date': '2026-02-20', ...}
+table.df            # 원본 데이터 DataFrame
+table.labeled       # 한국어 라벨이 적용된 DataFrame
+```
+
+## Output Structure
+
+```
+database_structure/
+├── structure.json              # 전체 테이블-컬럼 구조
+├── labels/
+│   ├── bond.json               # {"code": "채권 코드", "date": "거래일자", ...}
+│   ├── currency.json
+│   └── ...
+└── description/
+    ├── bond.json               # {"description": "...", "row_count": 85376, ...}
+    ├── currency.json
+    └── ...
+```
+
+## CLI
+
+```bash
+python -m universal_database_inspector
+python -m universal_database_inspector --overwrite
+```
+
+## Dependencies
+
+- `mysql-connector-python` — MySQL 연결
+- `sqlalchemy` — ORM 엔진
+- `pandas` — 데이터 처리
+- `python-dotenv` — 환경변수 로딩
+- `openai` — OpenAI API 호출

universal_database_inspector-0.1.0/README.md

@@ -0,0 +1,118 @@
+# universal_database_inspector
+
+MySQL 데이터베이스의 스키마를 자동으로 분석하고, AI를 활용하여 컬럼 라벨링과 테이블 설명을 생성하는 Python 패키지입니다.
+
+## Features
+
+- **스키마 추출** — 데이터베이스의 전체 테이블·컬럼 구조를 JSON으로 저장
+- **AI 컬럼 라벨링** — OpenAI 모델을 활용하여 각 컬럼의 한국어 라벨을 자동 생성
+- **AI 테이블 설명** — 테이블의 목적과 용도를 한국어로 요약
+- **테이블 통계** — 행 수, 컬럼 수, 날짜 범위 자동 수집
+- **Table 객체** — 테이블별 구조·라벨·설명·데이터를 하나의 인터페이스로 접근
+
+## Installation
+
+```bash
+pip install git+https://github.com/nailen1/universal_database_inspector.git
+```
+
+## Setup
+
+`.env.example`을 참고하여 `.env` 파일을 생성합니다.
+
+```
+# MySQL Database Connection
+DB_HOST=your_host
+DB_PORT=3306
+DB_USER=your_user
+DB_PASSWORD=your_password
+DB_NAME=your_database
+
+# OpenAI API Key
+OPENAI_API_KEY=your_openai_api_key
+```
+
+## Quick Start
+
+```python
+from universal_database_inspector import init_structure, inspect_all, describe_all_tables, Table
+```
+
+### 1. 출력 폴더 초기화
+
+```python
+init_structure()
+```
+
+```
+database_structure/
+├── labels/
+└── description/
+```
+
+### 2. 데이터베이스 구조 추출
+
+```python
+structure = inspect_all()
+list(structure.keys())
+# ['bond', 'currency', 'daily_price', 'index', 'index_membership', 'stock_info']
+```
+
+`database_structure/structure.json`에 전체 테이블-컬럼 매핑이 저장됩니다.
+
+### 3. 테이블 설명 생성
+
+```python
+describe_all_tables()
+```
+
+각 테이블에 대해 AI 라벨링과 설명 파일을 일괄 생성합니다.
+
+```
+[1/6] describing: bond
+[2/6] describing: currency
+...
+done: 6 created, 0 skipped (total 6)
+```
+
+### 4. Table 객체로 접근
+
+```python
+table = Table('bond')
+
+table.columns       # ['code', 'date', 'open', 'high', 'low', 'close']
+table.labels        # {'code': '채권 코드', 'date': '거래일자', 'open': '시가', ...}
+table.description   # {'description': '...', 'first_date': '1962-01-02', 'last_date': '2026-02-20', ...}
+table.df            # 원본 데이터 DataFrame
+table.labeled       # 한국어 라벨이 적용된 DataFrame
+```
+
+## Output Structure
+
+```
+database_structure/
+├── structure.json              # 전체 테이블-컬럼 구조
+├── labels/
+│   ├── bond.json               # {"code": "채권 코드", "date": "거래일자", ...}
+│   ├── currency.json
+│   └── ...
+└── description/
+    ├── bond.json               # {"description": "...", "row_count": 85376, ...}
+    ├── currency.json
+    └── ...
+```
+
+## CLI
+
+```bash
+python -m universal_database_inspector
+python -m universal_database_inspector --overwrite
+```
+
+## Dependencies
+
+- `mysql-connector-python` — MySQL 연결
+- `sqlalchemy` — ORM 엔진
+- `pandas` — 데이터 처리
+- `python-dotenv` — 환경변수 로딩
+- `openai` — OpenAI API 호출

universal_database_inspector-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "universal_database_inspector"
+version = "0.1.0"
+description = "Automated MySQL schema decomposition with AI-powered labeling and descriptions."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "nailen1" },
+]
+keywords = ["mysql", "schema", "openai", "database", "inspector", "labeling"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+]
+dependencies = [
+    "mysql-connector-python>=8.0.0",
+    "sqlalchemy>=2.0.0",
+    "pandas>=2.0.0",
+    "python-dotenv>=1.0.0",
+    "openai>=1.0.0",
+]
+
+[project.scripts]
+database-inspector = "universal_database_inspector.__main__:main"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["universal_database_inspector*"]

universal_database_inspector-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

universal_database_inspector-0.1.0/universal_database_inspector/__init__.py

@@ -0,0 +1,59 @@
+"""universal_database_inspector - Automated MySQL schema decomposition."""
+
+from universal_database_inspector.config import load_db_config
+from universal_database_inspector.connection import (
+    get_engine,
+    get_connection,
+    get_list_tables,
+)
+from universal_database_inspector.inspector import (
+    load_structure,
+    get_columns,
+    get_structure,
+    save_structure,
+    inspect_all,
+)
+from universal_database_inspector.labeler import (
+    fetch_sample_rows,
+    generate_labels,
+    save_labels,
+    label_table,
+)
+from universal_database_inspector.application import (
+    label_all_tables,
+    load_labels,
+    get_labeled_table,
+)
+from universal_database_inspector.describer import (
+    get_table_stats,
+    generate_description,
+    save_description,
+    describe_all_tables,
+)
+from universal_database_inspector.table import Table
+from universal_database_inspector.scaffold import init_structure
+
+__all__ = [
+    "load_db_config",
+    "get_engine",
+    "get_connection",
+    "get_list_tables",
+    "load_structure",
+    "get_columns",
+    "get_structure",
+    "save_structure",
+    "inspect_all",
+    "fetch_sample_rows",
+    "generate_labels",
+    "save_labels",
+    "label_table",
+    "label_all_tables",
+    "load_labels",
+    "get_labeled_table",
+    "get_table_stats",
+    "generate_description",
+    "save_description",
+    "describe_all_tables",
+    "Table",
+    "init_structure",
+]

universal_database_inspector-0.1.0/universal_database_inspector/__main__.py

@@ -0,0 +1,36 @@
+"""CLI entrypoint for universal_database_inspector.
+
+Usage::
+
+    python -m universal_database_inspector
+    python -m universal_database_inspector --overwrite
+"""
+
+import argparse
+
+from universal_database_inspector.inspector import inspect_all
+from universal_database_inspector.describer import describe_all_tables
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Inspect database structure, generate labels and descriptions.",
+    )
+    parser.add_argument(
+        "--overwrite",
+        action="store_true",
+        help="Overwrite existing label/description files.",
+    )
+    args = parser.parse_args()
+
+    print("\n[1/2] structure.json")
+    inspect_all()
+
+    print("\n[2/2] labels + descriptions")
+    describe_all_tables(overwrite=args.overwrite)
+
+    print("\nall done.")
+
+
+if __name__ == "__main__":
+    main()

universal_database_inspector-0.1.0/universal_database_inspector/ai.py

@@ -0,0 +1,69 @@
+"""Minimal OpenAI API wrapper.
+
+Provides a single ``prompt_to_model`` function used by the labeler
+and describer modules.  Replaces the external ``use_ai`` dependency.
+"""
+
+import os
+
+from dotenv import load_dotenv
+from openai import OpenAI, BadRequestError
+
+DEFAULT_MODEL_NAME = "gpt-4.1"
+
+
+def _get_client() -> OpenAI:
+    load_dotenv()
+    api_key = os.getenv("OPENAI_API_KEY")
+    if api_key is None:
+        raise EnvironmentError("Missing environment variable: OPENAI_API_KEY")
+    return OpenAI(api_key=api_key)
+
+
+def _chat_completion(client: OpenAI, **kwargs) -> object:
+    """Call chat completions with automatic parameter fallback.
+
+    Handles two known incompatibilities across model generations:
+      - max_tokens -> max_completion_tokens (newer models)
+      - temperature not supported (reasoning models like o3-mini)
+    """
+    try:
+        return client.chat.completions.create(**kwargs)
+    except BadRequestError as e:
+        body = e.body or {}
+        param = body.get("param", "") if isinstance(body, dict) else ""
+
+        if param == "max_tokens" and "max_tokens" in kwargs:
+            kwargs["max_completion_tokens"] = kwargs.pop("max_tokens")
+            return _chat_completion(client, **kwargs)
+
+        if param == "temperature" and "temperature" in kwargs:
+            del kwargs["temperature"]
+            return _chat_completion(client, **kwargs)
+
+        raise
+
+
+def prompt_to_model(
+    prompt: str,
+    model_name: str = DEFAULT_MODEL_NAME,
+    system_message: str | None = None,
+    max_tokens: int = 1024,
+    temperature: float = 0.7,
+) -> str:
+    """Send a prompt to an OpenAI model and return the response text."""
+    client = _get_client()
+
+    messages = []
+    if system_message:
+        messages.append({"role": "system", "content": system_message})
+    messages.append({"role": "user", "content": prompt})
+
+    response = _chat_completion(
+        client,
+        model=model_name,
+        messages=messages,
+        max_tokens=max_tokens,
+        temperature=temperature,
+    )
+    return response.choices[0].message.content

universal_database_inspector-0.1.0/universal_database_inspector/application.py

@@ -0,0 +1,151 @@
+"""High-level database inspector workflows.
+
+Combines inspector and labeler primitives into batch operations.
+"""
+
+import json
+import os
+
+import pandas as pd
+
+from universal_database_inspector.connection import get_engine
+from universal_database_inspector.inspector import load_structure
+from universal_database_inspector.labeler import save_labels
+
+
+_PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _project_root():
+    return _PROJECT_ROOT
+
+
+def _group_key(table_name: str) -> str | None:
+    """Return the group key if the table belongs to a grouped prefix.
+
+    Only numeric-suffix tables (e.g. ``p_ks_000020``) are grouped.
+    Named variants like ``p_ks_market`` are treated individually.
+    """
+    if table_name.startswith("p_ks_"):
+        suffix = table_name[len("p_ks_"):]
+        if suffix.isdigit():
+            return "p_ks"
+    return None
+
+
+def label_all_tables(
+    output_dir: str = "database_structure",
+    overwrite: bool = False,
+) -> list[str]:
+    """Generate AI labels for every table and save as JSON files.
+
+    Tables sharing a common prefix (e.g. ``p_ks_*``) are labeled once
+    using the union of their columns, saved as ``{prefix}.json``.
+
+    Args:
+        output_dir: Root output directory.
+        overwrite: If False, skip tables whose label file already exists.
+
+    Returns:
+        list[str]: Paths to the saved label files.
+    """
+    structure = load_structure(output_dir=output_dir)
+
+    grouped_columns: dict[str, list[str]] = {}
+    normal_tables: dict[str, list[str]] = {}
+
+    for table, columns in structure.items():
+        gk = _group_key(table)
+        if gk is not None:
+            existing = grouped_columns.get(gk, [])
+            for col in columns:
+                if col not in existing:
+                    existing.append(col)
+            grouped_columns[gk] = existing
+        else:
+            normal_tables[table] = columns
+
+    resolved_dir = output_dir
+    if not os.path.isabs(resolved_dir):
+        resolved_dir = os.path.join(_project_root(), resolved_dir)
+    labels_dir = os.path.join(resolved_dir, "labels")
+
+    targets = {**normal_tables, **grouped_columns}
+    paths = []
+    created = 0
+    skipped = 0
+    total = len(targets)
+    for i, (name, columns) in enumerate(targets.items(), 1):
+        label_file = os.path.join(labels_dir, f"{name}.json")
+        already_exists = os.path.exists(label_file)
+
+        if not overwrite and already_exists:
+            skipped += 1
+            print(f"[{i}/{total}] skip (exists): {name}")
+            paths.append(label_file)
+            continue
+
+        print(f"[{i}/{total}] labeling: {name}")
+        path = save_labels(
+            name, columns,
+            output_dir=output_dir, overwrite=overwrite,
+        )
+        paths.append(path)
+        created += 1
+    print(f"done: {created} created, {skipped} skipped (total {total})")
+    return paths
+
+
+def load_labels(
+    table_name: str,
+    output_dir: str = "database_structure",
+) -> dict:
+    """Load a saved label JSON file for a table.
+
+    For grouped tables (e.g. ``p_ks_000020``), loads the shared label
+    file (``p_ks.json``) instead.
+
+    Args:
+        table_name: Name of the target table.
+        output_dir: Root output directory.
+
+    Returns:
+        dict: ``{column_name: label}`` mapping.
+
+    Raises:
+        FileNotFoundError: If the label file does not exist.
+    """
+    if not os.path.isabs(output_dir):
+        output_dir = os.path.join(_project_root(), output_dir)
+
+    lookup_name = _group_key(table_name) or table_name
+    labels_path = os.path.join(output_dir, "labels", f"{lookup_name}.json")
+    with open(labels_path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def get_labeled_table(
+    table_name: str,
+    option_label: bool = True,
+    output_dir: str = "database_structure",
+) -> pd.DataFrame:
+    """Fetch a table and optionally rename columns to Korean labels.
+
+    Args:
+        table_name: Name of the target table.
+        option_label: If True, rename columns using the label JSON file.
+        output_dir: Root output directory containing label files.
+
+    Returns:
+        pd.DataFrame: Table data with original or labeled column names.
+    """
+    engine = get_engine()
+    query = f"SELECT * FROM `{table_name}`"
+    df = pd.read_sql(query, engine)
+
+    if option_label:
+        labels = load_labels(table_name, output_dir=output_dir)
+        rename_map = {col: labels[col] for col in df.columns if labels.get(col)}
+        df = df.rename(columns=rename_map)
+
+    return df

universal_database_inspector-0.1.0/universal_database_inspector/config.py

@@ -0,0 +1,55 @@
+"""Database configuration loader.
+
+Reads MySQL connection parameters from environment variables:
+DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME.
+"""
+
+import os
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def load_db_config() -> dict:
+    """Load MySQL connection configuration from environment variables.
+
+    Required variables::
+
+        DB_HOST
+        DB_PORT
+        DB_USER
+        DB_PASSWORD
+        DB_NAME
+
+    Returns:
+        dict with keys: host, port, user, password, database.
+
+    Raises:
+        EnvironmentError: If any required variable is missing.
+    """
+    required_keys = {
+        "host": "DB_HOST",
+        "port": "DB_PORT",
+        "user": "DB_USER",
+        "password": "DB_PASSWORD",
+        "database": "DB_NAME",
+    }
+
+    config = {}
+    missing = []
+
+    for key, env_var in required_keys.items():
+        value = os.getenv(env_var)
+        if value is None:
+            missing.append(env_var)
+        else:
+            config[key] = value
+
+    if missing:
+        raise EnvironmentError(
+            f"Missing environment variables: {', '.join(missing)}"
+        )
+
+    config["port"] = int(config["port"])
+    return config