dataprocessing-ai 0.1.1__tar.gz

dataprocessing_ai-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Gbemi Lesi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dataprocessing_ai-0.1.1/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: dataprocessing-ai
+Version: 0.1.1
+Summary: AI-native data processing library — REST API, MCP server, and Python package
+Author: oluwagbemilesi
+License: MIT
+Project-URL: Homepage, https://github.com/Techlon/dataprocessing-ai
+Project-URL: Repository, https://github.com/Techlon/dataprocessing-ai
+Project-URL: Issues, https://github.com/Techlon/dataprocessing-ai/issues
+Keywords: data,processing,AI,REST,MCP,pandas
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.2.0
+Requires-Dist: numpy>=1.26.0
+Provides-Extra: api
+Requires-Dist: fastapi>=0.110.0; extra == "api"
+Requires-Dist: uvicorn>=0.27.0; extra == "api"
+Requires-Dist: pydantic>=2.6.0; extra == "api"
+Requires-Dist: python-multipart>=0.0.9; extra == "api"
+Provides-Extra: mcp
+Requires-Dist: mcp>=0.9.0; extra == "mcp"
+Provides-Extra: formats
+Requires-Dist: openpyxl>=3.1.0; extra == "formats"
+Requires-Dist: pyarrow>=15.0.0; extra == "formats"
+Provides-Extra: all
+Requires-Dist: dataprocessing-ai[api,formats,mcp]; extra == "all"
+Provides-Extra: dev
+Requires-Dist: dataprocessing-ai[all]; extra == "dev"
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: httpx>=0.27.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Dynamic: license-file
+
+# DataProcessing AI
+
+An AI-native Python library for data processing and transformation.
+Built to be called by any AI agent via REST API, MCP server, or direct Python import.
+
+## What it does
+
+- **Ingest** - read CSV, JSON, Excel, Parquet files into a standard format
+- **Clean** - remove nulls, duplicates, outliers, standardise column names
+- **Transform** - filter, sort, group, pivot, merge, reshape datasets
+- **Analyse** - generate statistics, correlations, distributions, outlier reports
+- **Visualise** - produce Vega-Lite chart specs (histogram, bar, scatter, line, correlation heatmap)
+
+## Three ways to use it
+
+### 1. REST API (any AI, any language)
+
+Install with the API extra, then start the server:
+```bash
+pip install "dataprocessing-ai[api]"
+uvicorn dataprocessing.api:app --host 0.0.0.0 --port 8000
+```
+
+Call it:
+```bash
+curl -X POST http://localhost:8000/ingest -F "file=@data.csv"
+curl -X POST http://localhost:8000/clean -H "Content-Type: application/json" -d '{"data": [...]}'
+curl -X POST http://localhost:8000/transform -H "Content-Type: application/json" -d '{"data": [...], "operation": "filter_rows", "params": {"column": "age", "operator": "gt", "value": 25}}'
+curl -X POST http://localhost:8000/analyse -H "Content-Type: application/json" -d '{"data": [...]}'
+curl -X POST http://localhost:8000/visualise -H "Content-Type: application/json" -d '{"data": [...], "chart": "histogram", "params": {"column": "age", "bins": 10}}'
+```
+
+### 2. MCP Server (Claude native tools)
+
+Install with the MCP extra:
+```bash
+pip install "dataprocessing-ai[mcp]"
+```
+
+Then add to your `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "dataprocessing": {
+      "command": "dataprocessing-mcp"
+    }
+  }
+}
+```
+
+No paths, no `PYTHONPATH` — the installed package provides the
+`dataprocessing-mcp` command directly.
+
+### 3. Python package (direct import)
+
+```python
+from dataprocessing.ingest import read_file
+from dataprocessing.clean import clean_all
+from dataprocessing.transform import filter_rows
+from dataprocessing.analyse import full_report
+from dataprocessing.visualise import histogram, correlation_heatmap
+
+df = read_file("data.csv")
+df = clean_all(df)
+report = full_report(df)
+
+# Chart functions return Vega-Lite spec dicts (JSON-serialisable)
+hist_spec = histogram(df, column="age", bins=10)
+heatmap_spec = correlation_heatmap(df)
+```
+
+## Installation
+
+```bash
+pip install dataprocessing-ai
+```
+
+That gives you the core library (`ingest`, `clean`, `transform`, `analyse`,
+`visualise`) with a minimal dependency footprint.
+
+Optional extras add the interfaces and file formats you need:
+
+```bash
+pip install "dataprocessing-ai[mcp]"      # MCP server (for Claude and other agents)
+pip install "dataprocessing-ai[api]"      # REST API server
+pip install "dataprocessing-ai[formats]"  # Excel (.xlsx) and Parquet support
+pip install "dataprocessing-ai[all]"      # everything
+```
+
+### Installing from source
+
+```bash
+git clone https://github.com/Techlon/dataprocessing-ai.git
+cd dataprocessing-ai
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## API Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | /health | API status |
+| POST | /ingest | Upload and read a data file |
+| POST | /clean | Clean a dataset |
+| POST | /transform | Transform a dataset |
+| POST | /analyse | Analyse a dataset |
+| POST | /visualise | Produce a Vega-Lite chart spec from a dataset |
+
+## Charts
+
+The `/visualise` endpoint (and the `visualise_data` MCP tool) accept a `chart`
+name and a `params` object. Each returns a [Vega-Lite](https://vega.github.io/vega-lite/)
+spec dict that any Vega-Lite renderer can display.
+
+| Chart | Params | Notes |
+|-------|--------|-------|
+| `histogram` | `column` (numeric), `bins` (int, default 10) | Distribution of a numeric column |
+| `bar_chart` | `column` (any) | Counts per category |
+| `scatter` | `x` (numeric), `y` (numeric) | Relationship between two numeric columns |
+| `line_chart` | `x` (any), `y` (numeric) | `x` may be a date or category; `y` must be numeric |
+| `correlation_heatmap` | `columns` (list, optional) | Correlations across numeric columns; defaults to all numeric columns |
+
+A bad chart name, a missing column, or a non-numeric column where a numeric one
+is required returns HTTP 400 (or raises `ValueError` when called directly).
+
+### Example response
+
+A call with `{"chart": "histogram", "params": {"column": "age", "bins": 3}}`
+returns a Vega-Lite spec like:
+
+```json
+{
+  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
+  "description": "Histogram of age",
+  "data": {
+    "values": [
+      {"bin_start": 23.0, "bin_end": 29.33, "count": 4},
+      {"bin_start": 29.33, "bin_end": 35.67, "count": 3},
+      {"bin_start": 35.67, "bin_end": 42.0, "count": 1}
+    ]
+  },
+  "mark": "bar",
+  "encoding": {
+    "x": {"field": "bin_start", "type": "quantitative"},
+    "x2": {"field": "bin_end", "type": "quantitative"},
+    "y": {"field": "count", "type": "quantitative"}
+  }
+}
+```
+
+Paste the returned spec into the [Vega-Lite editor](https://vega.github.io/editor/)
+or render it with any Vega-Lite-compatible frontend.
+
+## MCP Tools
+
+The MCP server exposes each capability as a Claude-native tool:
+
+| Tool | Arguments | Description |
+|------|-----------|-------------|
+| `ingest_file` | `file_path` | Read a data file (CSV, JSON, Excel, Parquet) from disk |
+| `clean_data` | `data`, `drop_null_threshold`, `remove_dupes`, `standardise_cols` | Remove nulls, duplicates, and standardise column names |
+| `transform_data` | `data`, `operation`, `params` | Apply a named transform (filter, select, rename, sort, group, add column) |
+| `analyse_data` | `data` | Full statistical report (summary, correlations, missing, outliers) |
+| `visualise_data` | `data`, `chart`, `params` | Produce a Vega-Lite chart spec (see [Charts](#charts) for names and params) |
+
+The `visualise_data` tool uses the same chart names and params as the `/visualise`
+endpoint and returns a Vega-Lite spec dict.
+
+## Supported formats
+CSV, JSON, Excel (.xlsx), Parquet
+
+## License
+MIT

dataprocessing_ai-0.1.1/README.md

@@ -0,0 +1,175 @@
+# DataProcessing AI
+
+An AI-native Python library for data processing and transformation.
+Built to be called by any AI agent via REST API, MCP server, or direct Python import.
+
+## What it does
+
+- **Ingest** - read CSV, JSON, Excel, Parquet files into a standard format
+- **Clean** - remove nulls, duplicates, outliers, standardise column names
+- **Transform** - filter, sort, group, pivot, merge, reshape datasets
+- **Analyse** - generate statistics, correlations, distributions, outlier reports
+- **Visualise** - produce Vega-Lite chart specs (histogram, bar, scatter, line, correlation heatmap)
+
+## Three ways to use it
+
+### 1. REST API (any AI, any language)
+
+Install with the API extra, then start the server:
+```bash
+pip install "dataprocessing-ai[api]"
+uvicorn dataprocessing.api:app --host 0.0.0.0 --port 8000
+```
+
+Call it:
+```bash
+curl -X POST http://localhost:8000/ingest -F "file=@data.csv"
+curl -X POST http://localhost:8000/clean -H "Content-Type: application/json" -d '{"data": [...]}'
+curl -X POST http://localhost:8000/transform -H "Content-Type: application/json" -d '{"data": [...], "operation": "filter_rows", "params": {"column": "age", "operator": "gt", "value": 25}}'
+curl -X POST http://localhost:8000/analyse -H "Content-Type: application/json" -d '{"data": [...]}'
+curl -X POST http://localhost:8000/visualise -H "Content-Type: application/json" -d '{"data": [...], "chart": "histogram", "params": {"column": "age", "bins": 10}}'
+```
+
+### 2. MCP Server (Claude native tools)
+
+Install with the MCP extra:
+```bash
+pip install "dataprocessing-ai[mcp]"
+```
+
+Then add to your `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "dataprocessing": {
+      "command": "dataprocessing-mcp"
+    }
+  }
+}
+```
+
+No paths, no `PYTHONPATH` — the installed package provides the
+`dataprocessing-mcp` command directly.
+
+### 3. Python package (direct import)
+
+```python
+from dataprocessing.ingest import read_file
+from dataprocessing.clean import clean_all
+from dataprocessing.transform import filter_rows
+from dataprocessing.analyse import full_report
+from dataprocessing.visualise import histogram, correlation_heatmap
+
+df = read_file("data.csv")
+df = clean_all(df)
+report = full_report(df)
+
+# Chart functions return Vega-Lite spec dicts (JSON-serialisable)
+hist_spec = histogram(df, column="age", bins=10)
+heatmap_spec = correlation_heatmap(df)
+```
+
+## Installation
+
+```bash
+pip install dataprocessing-ai
+```
+
+That gives you the core library (`ingest`, `clean`, `transform`, `analyse`,
+`visualise`) with a minimal dependency footprint.
+
+Optional extras add the interfaces and file formats you need:
+
+```bash
+pip install "dataprocessing-ai[mcp]"      # MCP server (for Claude and other agents)
+pip install "dataprocessing-ai[api]"      # REST API server
+pip install "dataprocessing-ai[formats]"  # Excel (.xlsx) and Parquet support
+pip install "dataprocessing-ai[all]"      # everything
+```
+
+### Installing from source
+
+```bash
+git clone https://github.com/Techlon/dataprocessing-ai.git
+cd dataprocessing-ai
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## API Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | /health | API status |
+| POST | /ingest | Upload and read a data file |
+| POST | /clean | Clean a dataset |
+| POST | /transform | Transform a dataset |
+| POST | /analyse | Analyse a dataset |
+| POST | /visualise | Produce a Vega-Lite chart spec from a dataset |
+
+## Charts
+
+The `/visualise` endpoint (and the `visualise_data` MCP tool) accept a `chart`
+name and a `params` object. Each returns a [Vega-Lite](https://vega.github.io/vega-lite/)
+spec dict that any Vega-Lite renderer can display.
+
+| Chart | Params | Notes |
+|-------|--------|-------|
+| `histogram` | `column` (numeric), `bins` (int, default 10) | Distribution of a numeric column |
+| `bar_chart` | `column` (any) | Counts per category |
+| `scatter` | `x` (numeric), `y` (numeric) | Relationship between two numeric columns |
+| `line_chart` | `x` (any), `y` (numeric) | `x` may be a date or category; `y` must be numeric |
+| `correlation_heatmap` | `columns` (list, optional) | Correlations across numeric columns; defaults to all numeric columns |
+
+A bad chart name, a missing column, or a non-numeric column where a numeric one
+is required returns HTTP 400 (or raises `ValueError` when called directly).
+
+### Example response
+
+A call with `{"chart": "histogram", "params": {"column": "age", "bins": 3}}`
+returns a Vega-Lite spec like:
+
+```json
+{
+  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
+  "description": "Histogram of age",
+  "data": {
+    "values": [
+      {"bin_start": 23.0, "bin_end": 29.33, "count": 4},
+      {"bin_start": 29.33, "bin_end": 35.67, "count": 3},
+      {"bin_start": 35.67, "bin_end": 42.0, "count": 1}
+    ]
+  },
+  "mark": "bar",
+  "encoding": {
+    "x": {"field": "bin_start", "type": "quantitative"},
+    "x2": {"field": "bin_end", "type": "quantitative"},
+    "y": {"field": "count", "type": "quantitative"}
+  }
+}
+```
+
+Paste the returned spec into the [Vega-Lite editor](https://vega.github.io/editor/)
+or render it with any Vega-Lite-compatible frontend.
+
+## MCP Tools
+
+The MCP server exposes each capability as a Claude-native tool:
+
+| Tool | Arguments | Description |
+|------|-----------|-------------|
+| `ingest_file` | `file_path` | Read a data file (CSV, JSON, Excel, Parquet) from disk |
+| `clean_data` | `data`, `drop_null_threshold`, `remove_dupes`, `standardise_cols` | Remove nulls, duplicates, and standardise column names |
+| `transform_data` | `data`, `operation`, `params` | Apply a named transform (filter, select, rename, sort, group, add column) |
+| `analyse_data` | `data` | Full statistical report (summary, correlations, missing, outliers) |
+| `visualise_data` | `data`, `chart`, `params` | Produce a Vega-Lite chart spec (see [Charts](#charts) for names and params) |
+
+The `visualise_data` tool uses the same chart names and params as the `/visualise`
+endpoint and returns a Vega-Lite spec dict.
+
+## Supported formats
+CSV, JSON, Excel (.xlsx), Parquet
+
+## License
+MIT

dataprocessing_ai-0.1.1/dataprocessing/analyse/__init__.py

@@ -0,0 +1,56 @@
+import pandas as pd
+import numpy as np
+from typing import List, Dict, Any
+
+def summary_stats(df, columns=None):
+    if columns is None:
+        columns = df.select_dtypes(include=[np.number]).columns
+    stats = {}
+    for column in columns:
+        col_data = df[column]
+        stats[column] = {
+            'count': len(col_data), 'mean': col_data.mean(), 'median': col_data.median(),
+            'std': col_data.std(), 'min': col_data.min(), 'max': col_data.max(),
+            'quartiles': {'25%': np.percentile(col_data, 25), '75%': np.percentile(col_data, 75)}
+        }
+    return stats
+
+def correlation_matrix(df, columns=None):
+    if columns is None:
+        columns = df.select_dtypes(include=[np.number]).columns
+    return df[columns].corr().to_dict()
+
+def value_counts(df, column):
+    if not pd.api.types.is_numeric_dtype(df[column]):
+        raise ValueError("Column must be numeric")
+    return df[column].value_counts().to_dict()
+
+def missing_report(df):
+    total = len(df)
+    report = {}
+    for column in df.columns:
+        null_count = df[column].isnull().sum()
+        percentage = (null_count / total) * 100 if total > 0 else 0
+        report[column] = {'missing_count': int(null_count), 'percentage': float(percentage)}
+    return report
+
+def distribution(df, column, bins=10):
+    if not pd.api.types.is_numeric_dtype(df[column]):
+        raise ValueError("Column must be numeric")
+    hist, bin_edges = np.histogram(df[column], bins=bins)
+    return {'bin_edges': list(bin_edges), 'counts': list(hist)}
+
+def detect_outliers(df, columns=None):
+    if columns is None:
+        columns = df.select_dtypes(include=[np.number]).columns
+    outliers = {}
+    for column in columns:
+        q1 = np.percentile(df[column], 25); q3 = np.percentile(df[column], 75)
+        iqr = q3 - q1
+        lo = q1 - 1.5 * iqr; hi = q3 + 1.5 * iqr
+        outliers[column] = list(df[(df[column] < lo) | (df[column] > hi)].index)
+    return outliers
+
+def full_report(df):
+    return {'summary_stats': summary_stats(df), 'correlation_matrix': correlation_matrix(df),
+            'missing_values': missing_report(df), 'outliers': detect_outliers(df)}

dataprocessing_ai-0.1.1/dataprocessing/api.py

@@ -0,0 +1,162 @@
+"""
+DataProcessing REST API
+Exposes ingest, clean, transform and analyse modules over HTTP.
+Any AI agent can call these endpoints.
+"""
+from fastapi import FastAPI, UploadFile, File, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from typing import Optional, List, Dict, Any
+import pandas as pd
+import io
+import json
+
+from dataprocessing.clean import drop_nulls, remove_duplicates, standardise_columns
+from dataprocessing.transform import (
+    filter_rows, select_columns, rename_columns,
+    sort_rows, group_and_aggregate, pivot, add_column
+)
+from dataprocessing.analyse import full_report
+
+from dataprocessing.visualise import (
+    histogram, bar_chart, scatter, line_chart, correlation_heatmap
+)
+
+app = FastAPI(title="DataProcessing API", version="0.1.0")
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+def df_to_json(df):
+    return json.loads(df.where(df.notna(), other=None).to_json(orient="records"))
+
+def to_native(obj):
+    """Recursively convert numpy scalars/arrays to JSON-serialisable Python types.
+
+    full_report() returns dicts containing numpy int64/float64 values and numpy
+    arrays, which FastAPI's encoder cannot serialise. This normalises them.
+    """
+    import numpy as np
+    if isinstance(obj, dict):
+        return {to_native(k): to_native(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [to_native(v) for v in obj]
+    if isinstance(obj, np.generic):
+        return obj.item()
+    if isinstance(obj, np.ndarray):
+        return [to_native(v) for v in obj.tolist()]
+    if isinstance(obj, float) and (obj != obj):  # NaN -> null
+        return None
+    return obj
+
+class CleanRequest(BaseModel):
+    data: List[Dict[str, Any]]
+    drop_null_threshold: float = 0.5
+    remove_outliers: bool = False
+
+class TransformRequest(BaseModel):
+    data: List[Dict[str, Any]]
+    operation: str
+    params: Dict[str, Any] = {}
+
+class AnalyseRequest(BaseModel):
+    data: List[Dict[str, Any]]
+
+@app.get("/health")
+def health():
+    return {"status": "ok", "version": "0.1.0"}
+
+@app.post("/ingest")
+async def ingest(file: UploadFile = File(...)):
+    try:
+        contents = await file.read()
+        ext = file.filename.rsplit(".", 1)[-1].lower()
+        if ext == "csv":
+            df = pd.read_csv(io.BytesIO(contents))
+        elif ext == "json":
+            df = pd.read_json(io.BytesIO(contents))
+        elif ext == "xlsx":
+            df = pd.read_excel(io.BytesIO(contents))
+        elif ext == "parquet":
+            df = pd.read_parquet(io.BytesIO(contents))
+        else:
+            raise HTTPException(status_code=400, detail=f"Unsupported file type: {ext}")
+        return {"data": df_to_json(df), "rows": len(df), "columns": list(df.columns)}
+    except HTTPException:
+        raise
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+@app.post("/clean")
+def clean(req: CleanRequest):
+    try:
+        df = pd.DataFrame(req.data)
+        df = drop_nulls(df, threshold=req.drop_null_threshold)
+        df = remove_duplicates(df)
+        df = standardise_columns(df)
+        return {"data": df_to_json(df), "rows": len(df), "columns": list(df.columns)}
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+@app.post("/transform")
+def transform(req: TransformRequest):
+    try:
+        df = pd.DataFrame(req.data)
+        ops = {
+            "filter_rows": filter_rows,
+            "select_columns": select_columns,
+            "rename_columns": rename_columns,
+            "sort_rows": sort_rows,
+            "group_and_aggregate": group_and_aggregate,
+            "pivot": pivot,
+            "add_column": add_column,
+        }
+        if req.operation not in ops:
+            raise HTTPException(status_code=400, detail=f"Unknown operation: {req.operation}. Valid: {list(ops.keys())}")
+        result = ops[req.operation](df, **req.params)
+        return {"data": df_to_json(result), "rows": len(result), "columns": list(result.columns)}
+    except HTTPException:
+        raise
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+@app.post("/analyse")
+def analyse(req: AnalyseRequest):
+    try:
+        df = pd.DataFrame(req.data)
+        return to_native(full_report(df))
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+        
+class VisualiseRequest(BaseModel):
+    data: List[Dict[str, Any]]
+    chart: str
+    params: Dict[str, Any] = {}
+
+@app.post("/visualise")
+def visualise(req: VisualiseRequest):
+    try:
+        df = pd.DataFrame(req.data)
+        charts = {
+            "histogram": histogram,
+            "bar_chart": bar_chart,
+            "scatter": scatter,
+            "line_chart": line_chart,
+            "correlation_heatmap": correlation_heatmap,
+        }
+        if req.chart not in charts:
+            raise HTTPException(
+                status_code=400,
+                detail=f"Unknown chart: {req.chart}. Valid: {list(charts.keys())}",
+            )
+        result = charts[req.chart](df, **req.params)
+        return to_native(result)
+    except HTTPException:
+        raise
+    except Exception as e:
+        # bad/missing/non-numeric column raises ValueError -> client error
+        raise HTTPException(status_code=400, detail=str(e))

dataprocessing_ai-0.1.1/dataprocessing/clean/__init__.py

@@ -0,0 +1,60 @@
+import pandas as pd
+import numpy as np
+from typing import Optional
+
+def drop_nulls(df, threshold=0.5):
+    cols_to_drop = df.columns[df.isnull().mean() > threshold]
+    df.drop(columns=cols_to_drop, inplace=True)
+    df.dropna(inplace=True)
+    return df
+
+def fix_types(df):
+    for col in df.columns:
+        # Check bool BEFORE numeric: pandas treats bool as a numeric dtype, so
+        # the numeric branch would otherwise swallow boolean columns.
+        if pd.api.types.is_bool_dtype(df[col]):
+            df[col] = df[col].astype(bool)
+        elif pd.api.types.is_numeric_dtype(df[col]):
+            try:
+                df[col] = pd.to_numeric(df[col], errors='coerce')
+            except Exception as e:
+                print(f"Error converting column {col} to numeric: {e}")
+        elif pd.api.types.is_datetime64_any_dtype(df[col]):
+            continue
+        else:
+            try:
+                df[col] = pd.to_datetime(df[col], errors='coerce')
+            except Exception as e:
+                print(f"Error converting column {col} to datetime: {e}")
+    return df
+
+def remove_duplicates(df, subset=None):
+    if subset is not None:
+        df.drop_duplicates(subset=subset, inplace=True)
+    else:
+        df.drop_duplicates(inplace=True)
+    return df
+
+def remove_outliers(df, columns=None, method='iqr'):
+    if method == 'iqr':
+        if columns is None:
+            numeric_df = df.select_dtypes(include=[np.number])
+        else:
+            numeric_df = df[columns]
+        Q1 = numeric_df.quantile(0.25); Q3 = numeric_df.quantile(0.75)
+        IQR = Q3 - Q1
+        lower = Q1 - 1.5 * IQR; upper = Q3 + 1.5 * IQR
+        df = df[~((numeric_df < lower) | (numeric_df > upper)).any(axis=1)]
+    return df
+
+def standardise_columns(df):
+    df.columns = [col.lower().replace(" ", "_") for col in df.columns]
+    return df
+
+def clean_all(df):
+    df = drop_nulls(df)
+    df = fix_types(df)
+    df = remove_duplicates(df)
+    df = remove_outliers(df)
+    df = standardise_columns(df)
+    return df