deepcsv 0.3.0__tar.gz → 0.5.0__tar.gz

deepcsv-0.5.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: deepcsv
+Version: 0.5.0
+Summary: Automatically processes data files in directories, converts array-like strings to NumPy arrays, detects and fixes data type issues, and saves results as optimized Parquet files.
+Home-page: https://github.com/abdubakr77/deepcsv
+Author: Abdullah Bakr
+Author-email: abdubakora1232@gmail.com
+License: MIT
+Project-URL: Source, https://github.com/abdubakr77/deepcsv
+Project-URL: Tracker, https://github.com/abdubakr77/deepcsv/issues
+Keywords: data-processing pandas numpy etl data-cleaning file-conversion automation parquet bulk-conversion
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: pyarrow
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# deepcsv (v0.5.0)
+
+Ever loaded a CSV file and found your carefully structured lists turned into useless strings?
+````python
+"['Action', 'Sci-Fi', 'Thriller']"  # This is a string, not a list
+````
+
+deepcsv fixes this automatically.
+
+---
+
+## The Solution
+
+`deepcsv` handles these cases automatically:
+- Reads CSV/XLSX files or existing DataFrames
+- Converts any string value starting with `[` into real NumPy arrays (fast and lightweight)
+- Detects and fixes mixed-type columns by safely converting them to numeric (float)
+- Recursively processes all CSV/XLSX files in subdirectories
+- Saves results as Parquet format to preserve types and speed up analysis
+
+---
+
+## Installation
+```bash
+pip install deepcsv
+```
+
+## Usage
+
+### Single file processing (process_file)
+```python
+import deepcsv
+
+df = deepcsv.process_file('path/to/file.csv')
+```
+- Accepts `str` (file path) or `pd.DataFrame`
+- Returns `pd.DataFrame` with columns converted to arrays
+
+### Batch directory processing (process_all_files)
+```python
+import deepcsv
+
+deepcsv.process_all_files('path/to/folder')
+```
+- Processes all `.csv` and `.xlsx` files recursively
+- Saves converted files as Parquet in: `All CSV Files is Converted Here`
+
+---
+
+## What it does
+
+- Auto-detects files in directory and subdirectories
+- Converts values like:
+  - `"['item1', 'item2']"` → `array(['item1', 'item2'])` (NumPy array)
+  - Mixed numeric/string columns → single numeric type (float)
+- Handles NaN values without breaking
+- Stores results in Parquet format for type safety and performance
+
+---
+
+## Function Signatures
+
+- `process_file(data_input: Union[str, pd.DataFrame]) -> pd.DataFrame`
+- `process_all_files(directory_path: str) -> None`
+
+Output arrays are NumPy arrays for optimal performance in machine learning workflows.
+
+---
+
+## Key Features
+
+- Fast NumPy array conversion instead of slow Python lists
+- Mixed-type detection with automatic fixes
+- Parquet storage for data integrity
+- Recursive directory traversal
+- Warning messages for transparency
+
+---
+
+## Notes
+
+- Requires `pyarrow` for Parquet support
+- Only saves files that contain converted array columns
+
+---
+
+## Requirements
+
+- Python >= 3.7
+- pandas
+- pyarrow
+

deepcsv-0.5.0/README.md

@@ -0,0 +1,92 @@
+# deepcsv (v0.5.0)
+
+Ever loaded a CSV file and found your carefully structured lists turned into useless strings?
+````python
+"['Action', 'Sci-Fi', 'Thriller']"  # This is a string, not a list
+````
+
+deepcsv fixes this automatically.
+
+---
+
+## The Solution
+
+`deepcsv` handles these cases automatically:
+- Reads CSV/XLSX files or existing DataFrames
+- Converts any string value starting with `[` into real NumPy arrays (fast and lightweight)
+- Detects and fixes mixed-type columns by safely converting them to numeric (float)
+- Recursively processes all CSV/XLSX files in subdirectories
+- Saves results as Parquet format to preserve types and speed up analysis
+
+---
+
+## Installation
+```bash
+pip install deepcsv
+```
+
+## Usage
+
+### Single file processing (process_file)
+```python
+import deepcsv
+
+df = deepcsv.process_file('path/to/file.csv')
+```
+- Accepts `str` (file path) or `pd.DataFrame`
+- Returns `pd.DataFrame` with columns converted to arrays
+
+### Batch directory processing (process_all_files)
+```python
+import deepcsv
+
+deepcsv.process_all_files('path/to/folder')
+```
+- Processes all `.csv` and `.xlsx` files recursively
+- Saves converted files as Parquet in: `All CSV Files is Converted Here`
+
+---
+
+## What it does
+
+- Auto-detects files in directory and subdirectories
+- Converts values like:
+  - `"['item1', 'item2']"` → `array(['item1', 'item2'])` (NumPy array)
+  - Mixed numeric/string columns → single numeric type (float)
+- Handles NaN values without breaking
+- Stores results in Parquet format for type safety and performance
+
+---
+
+## Function Signatures
+
+- `process_file(data_input: Union[str, pd.DataFrame]) -> pd.DataFrame`
+- `process_all_files(directory_path: str) -> None`
+
+Output arrays are NumPy arrays for optimal performance in machine learning workflows.
+
+---
+
+## Key Features
+
+- Fast NumPy array conversion instead of slow Python lists
+- Mixed-type detection with automatic fixes
+- Parquet storage for data integrity
+- Recursive directory traversal
+- Warning messages for transparency
+
+---
+
+## Notes
+
+- Requires `pyarrow` for Parquet support
+- Only saves files that contain converted array columns
+
+---
+
+## Requirements
+
+- Python >= 3.7
+- pandas
+- pyarrow
+

deepcsv-0.5.0/deepcsv/__init__.py

@@ -0,0 +1,15 @@
+from .deepcsv import process_all_files, process_file
+from importlib.metadata import version
+import requests
+
+def _check_for_updates():
+    try:
+        response = requests.get("https://pypi.org/pypi/deepcsv/json")
+        latest = response.json()["info"]["version"]
+        current = version("deepcsv")
+        if latest != current:
+            print(f"DeepCSV: New version {latest} available! — run 'pip install -U deepcsv'")
+    except:
+        pass
+
+_check_for_updates()

deepcsv-0.5.0/deepcsv/deepcsv.py

@@ -0,0 +1,126 @@
+import pyarrow
+import pandas as pd
+from numpy import nan,array
+from os import listdir,makedirs
+from os.path import join,relpath,dirname,isfile,isdir
+from warnings import filterwarnings
+from typing import Union
+filterwarnings("ignore")
+
+def process_file(data_input: Union[str, pd.DataFrame]) -> pd.DataFrame:
+    """
+    Parses string representations of lists in DataFrame columns to actual NumPy arrays.
+
+    (data_input: Union[str, pd.DataFrame]) -> pd.DataFrame
+
+    This function reads a CSV file or takes a DataFrame, scans each column for string values
+    starting with '[', and converts them to NumPy arrays for fast and lightweight processing.
+    It also detects mixed data types in columns and attempts to convert them to numeric (float) for consistency.
+
+    Parameters:
+        data_input (Union[str, pd.DataFrame]): Path to the CSV file or the DataFrame itself.
+
+    Returns:
+        pd.DataFrame: The processed DataFrame with list strings converted to NumPy arrays, mixed types fixed with a warning message, and original columns renamed if arrays are extracted.
+
+    Examples:
+        ### Process a CSV file
+        df = parse_lists('path/to/file.csv')
+
+        ### Process an existing DataFrame
+        df = parse_lists(my_dataframe)
+    """
+
+    print(f"Converting Now On {data_input}")
+    print("-"*50)
+    
+    try:
+        data = pd.read_csv(data_input)
+    except:
+        # print("The Data It's Already Defined")
+        data = data_input
+
+    for ColName in data.columns:
+        
+        First_Value = data[ColName].iloc[0]
+
+        if len(data[ColName].apply(type).unique()) >= 2:
+
+
+            sample = (data[data[ColName].apply(type) == str][ColName].head(2)).values
+
+            if len(sample) > 0 and isinstance(sample[0],str) and sample[0][0].strip().isnumeric():
+
+                print(f"WARNING:\nThis Dataset Name ({data_input.split("\\")[-1]}) Found {len(data[ColName].apply(type).unique())} Mixed DataType in a column called ({ColName})\nPath : {data_input}")
+                print(f"System : This column have These types: {data[ColName].apply(type).unique()}")
+                print(f"System : Trying to fix the column as a Float to be have only one datatype...")
+
+                data[ColName] = pd.to_numeric(data[ColName], errors='coerce')
+                print("System : Done!")
+
+        elif isinstance(First_Value , str) and First_Value.strip().startswith("["):
+            
+            data[f"{ColName.capitalize()}List"] = data[ColName].apply(lambda x : array(x) if pd.notna(x) else nan)
+            data.drop(ColName,inplace=True,axis=1)
+
+    return data
+
+
+def process_all_files(directory_path: str) -> None:
+    """
+    Recursively processes all CSV and XLSX files in a directory, converts array strings to NumPy arrays,
+    and saves as Parquet files if array columns are present.
+
+    (directory_path: str) -> None
+
+    This function traverses the directory tree starting from directory_path, finds all .csv and .xlsx files,
+    applies parse_lists to each for array conversion and type fixing, and saves the result as .parquet files
+    in a subdirectory 'All CSV Files is Converted Here' only if the DataFrame contains new array columns.
+
+    Parameters:
+        directory_path (str): The root directory path to search for CSV/XLSX files.
+
+    Returns:
+        None: Files are saved to a subdirectory 'All CSV Files is Converted Here'.
+
+    Examples:
+        # Process all CSV files in a directory
+        auto_convert('/path/to/directory')
+    """
+
+    base_output = join(directory_path, "All CSV Files is Converted Here")
+    all_folders = [directory_path]
+    
+    makedirs(base_output,exist_ok=True)
+    
+    while True:
+
+        if all_folders:
+
+            Curr_Path = all_folders.pop(0)
+
+            for item_name in listdir(Curr_Path):
+                
+                Sub_Item_Path = join(Curr_Path,item_name)
+                
+                if isfile(Sub_Item_Path) and (Sub_Item_Path.endswith(".csv") or Sub_Item_Path.endswith(".xlsx")):
+                    
+                    df_converted = process_file(Sub_Item_Path)
+                    df_converted.reset_index(drop=True,inplace=True)
+
+                    rel_path = relpath(Sub_Item_Path, directory_path)
+                    output = join(base_output,rel_path)
+                    if "List" in df_converted.columns[-1]:
+                        print(Sub_Item_Path)
+                        makedirs(dirname(output),exist_ok=True)
+                        df_converted.to_parquet(output.replace(".csv", ".parquet"))
+
+                    print("-"*50)
+
+                elif isdir(Sub_Item_Path):
+
+                    all_folders.append(Sub_Item_Path)
+
+        else:
+            break
+

deepcsv-0.5.0/deepcsv.egg-info/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: deepcsv
+Version: 0.5.0
+Summary: Automatically processes data files in directories, converts array-like strings to NumPy arrays, detects and fixes data type issues, and saves results as optimized Parquet files.
+Home-page: https://github.com/abdubakr77/deepcsv
+Author: Abdullah Bakr
+Author-email: abdubakora1232@gmail.com
+License: MIT
+Project-URL: Source, https://github.com/abdubakr77/deepcsv
+Project-URL: Tracker, https://github.com/abdubakr77/deepcsv/issues
+Keywords: data-processing pandas numpy etl data-cleaning file-conversion automation parquet bulk-conversion
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: pyarrow
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# deepcsv (v0.5.0)
+
+Ever loaded a CSV file and found your carefully structured lists turned into useless strings?
+````python
+"['Action', 'Sci-Fi', 'Thriller']"  # This is a string, not a list
+````
+
+deepcsv fixes this automatically.
+
+---
+
+## The Solution
+
+`deepcsv` handles these cases automatically:
+- Reads CSV/XLSX files or existing DataFrames
+- Converts any string value starting with `[` into real NumPy arrays (fast and lightweight)
+- Detects and fixes mixed-type columns by safely converting them to numeric (float)
+- Recursively processes all CSV/XLSX files in subdirectories
+- Saves results as Parquet format to preserve types and speed up analysis
+
+---
+
+## Installation
+```bash
+pip install deepcsv
+```
+
+## Usage
+
+### Single file processing (process_file)
+```python
+import deepcsv
+
+df = deepcsv.process_file('path/to/file.csv')
+```
+- Accepts `str` (file path) or `pd.DataFrame`
+- Returns `pd.DataFrame` with columns converted to arrays
+
+### Batch directory processing (process_all_files)
+```python
+import deepcsv
+
+deepcsv.process_all_files('path/to/folder')
+```
+- Processes all `.csv` and `.xlsx` files recursively
+- Saves converted files as Parquet in: `All CSV Files is Converted Here`
+
+---
+
+## What it does
+
+- Auto-detects files in directory and subdirectories
+- Converts values like:
+  - `"['item1', 'item2']"` → `array(['item1', 'item2'])` (NumPy array)
+  - Mixed numeric/string columns → single numeric type (float)
+- Handles NaN values without breaking
+- Stores results in Parquet format for type safety and performance
+
+---
+
+## Function Signatures
+
+- `process_file(data_input: Union[str, pd.DataFrame]) -> pd.DataFrame`
+- `process_all_files(directory_path: str) -> None`
+
+Output arrays are NumPy arrays for optimal performance in machine learning workflows.
+
+---
+
+## Key Features
+
+- Fast NumPy array conversion instead of slow Python lists
+- Mixed-type detection with automatic fixes
+- Parquet storage for data integrity
+- Recursive directory traversal
+- Warning messages for transparency
+
+---
+
+## Notes
+
+- Requires `pyarrow` for Parquet support
+- Only saves files that contain converted array columns
+
+---
+
+## Requirements
+
+- Python >= 3.7
+- pandas
+- pyarrow
+

deepcsv-0.5.0/setup.py

@@ -0,0 +1,35 @@
+from setuptools import setup, find_packages
+from pathlib import Path
+
+this_directory = Path(__file__).parent
+
+readme = (this_directory / "README.md").read_text(encoding="utf-8")
+
+
+setup(
+    name="deepcsv",
+    version="0.5.0",
+    author="Abdullah Bakr",
+    author_email="abdubakora1232@gmail.com",
+    description="Automatically processes data files in directories, converts array-like strings to NumPy arrays, detects and fixes data type issues, and saves results as optimized Parquet files.",
+    long_description=readme,
+    long_description_content_type="text/markdown",
+    packages=find_packages(),
+    install_requires=["pandas", "pyarrow"],
+    python_requires=">=3.7",
+    url="https://github.com/abdubakr77/deepcsv",
+    license="MIT",
+    keywords="data-processing pandas numpy etl data-cleaning file-conversion automation parquet bulk-conversion",
+    classifiers=[
+        "Development Status :: 5 - Production/Stable",
+        "Intended Audience :: Developers",
+        "Topic :: Software Development :: Libraries :: Python Modules",
+        "Topic :: Scientific/Engineering :: Information Analysis",
+        "License :: OSI Approved :: MIT License",
+        "Programming Language :: Python :: 3",
+    ],    
+    project_urls={
+        "Source": "https://github.com/abdubakr77/deepcsv",
+        "Tracker": "https://github.com/abdubakr77/deepcsv/issues",
+    },
+)

deepcsv-0.3.0/PKG-INFO

@@ -1,74 +0,0 @@
-Metadata-Version: 2.4
-Name: deepcsv
-Version: 0.3.0
-Summary: Automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.
-Author: Abdullah Bakr
-Requires-Python: >=3.7
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pandas
-Requires-Dist: pyarrow
-Dynamic: author
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-# deepcsv
-
-A Python library that automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.
-
-## Installation
-```bash
-pip install deepcsv
-```
-
-## What it does
-
-- Walks through all folders and subfolders automatically
-- Finds every CSV and XLSX file
-- Detects columns that contain list strings like `"['item1', 'item2']"` and converts them into real Python arrays for faster performance
-- Detects columns with mixed data types and tries to fix them automatically
-- Warns you when a column has mixed types so you know what was changed
-- Saves the results as Parquet files to preserve the converted data types
-
-> **Why Parquet?**
-> CSV files cannot store arrays or preserve data types. Parquet solves this by keeping the exact types after conversion.
-
-> **Why arrays instead of Python lists?**
-> Arrays are significantly faster for numerical operations and machine learning workflows.
-
-## Functions
-
-### `ConvertListStrToList(file_path)`
-
-Reads a CSV file, converts list strings to arrays, fixes mixed-type columns, and returns a clean DataFrame.
-```python
-import deepcsv
-
-df = deepcsv.ConvertListStrToList("path/to/file.csv")
-```
-
-### `ReadAllCSVData(path)`
-
-Walks through all folders and subfolders, applies `ConvertListStrToList` on every CSV and XLSX file, and saves the results as Parquet files in a new folder called `All CSV Data is Converted Here`.
-```python
-import deepcsv
-
-deepcsv.ReadAllCSVData("path/to/folder")
-```
-
-## Notes
-
-- Only files that contain list string columns are saved as Parquet
-- Mixed-type columns are converted to float automatically when possible
-- Skips NaN values without breaking
-- Requires `pyarrow` for Parquet support
-
-## Requirements
-
-- Python >= 3.7
-- pandas
-- pyarrow

deepcsv-0.3.0/README.md

@@ -1,56 +0,0 @@
-# deepcsv
-
-A Python library that automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.
-
-## Installation
-```bash
-pip install deepcsv
-```
-
-## What it does
-
-- Walks through all folders and subfolders automatically
-- Finds every CSV and XLSX file
-- Detects columns that contain list strings like `"['item1', 'item2']"` and converts them into real Python arrays for faster performance
-- Detects columns with mixed data types and tries to fix them automatically
-- Warns you when a column has mixed types so you know what was changed
-- Saves the results as Parquet files to preserve the converted data types
-
-> **Why Parquet?**
-> CSV files cannot store arrays or preserve data types. Parquet solves this by keeping the exact types after conversion.
-
-> **Why arrays instead of Python lists?**
-> Arrays are significantly faster for numerical operations and machine learning workflows.
-
-## Functions
-
-### `ConvertListStrToList(file_path)`
-
-Reads a CSV file, converts list strings to arrays, fixes mixed-type columns, and returns a clean DataFrame.
-```python
-import deepcsv
-
-df = deepcsv.ConvertListStrToList("path/to/file.csv")
-```
-
-### `ReadAllCSVData(path)`
-
-Walks through all folders and subfolders, applies `ConvertListStrToList` on every CSV and XLSX file, and saves the results as Parquet files in a new folder called `All CSV Data is Converted Here`.
-```python
-import deepcsv
-
-deepcsv.ReadAllCSVData("path/to/folder")
-```
-
-## Notes
-
-- Only files that contain list string columns are saved as Parquet
-- Mixed-type columns are converted to float automatically when possible
-- Skips NaN values without breaking
-- Requires `pyarrow` for Parquet support
-
-## Requirements
-
-- Python >= 3.7
-- pandas
-- pyarrow

deepcsv-0.3.0/deepcsv/__init__.py

@@ -1 +0,0 @@
-from .deepcsv import ReadAllCSVData, ConvertListStrToList

deepcsv-0.3.0/deepcsv/deepcsv.py

@@ -1,76 +0,0 @@
-import pyarrow
-import os
-import warnings
-import numpy as np
-import pandas as pd
-from ast import literal_eval
-warnings.filterwarnings("ignore")
-
-def ConvertListStrToList(File_Path):
-
-    data = pd.read_csv(File_Path)
-    for ColName in data.columns:
-        
-        First_Value = data[ColName].iloc[0]
-
-        if len(data[ColName].apply(type).unique()) >= 2:
-
-
-            sample = (data[data[ColName].apply(type) == str][ColName].head(2)).values
-
-            if len(sample) > 0 and isinstance(sample[0],str) and sample[0][0].strip().isnumeric():
-
-                print(f"WARNING:\nThis Dataset Name ({File_Path.split("\\")[-1]}) Found {len(data[ColName].apply(type).unique())} Mixed DataType  in a column called ({ColName})\nPath : {File_Path}")
-                print(f"System : This column have These types: {data[ColName].apply(type).unique()}")
-                print(f"System : Trying to fix the column as a Float to be have only one datatype...")
-
-                data[ColName] = pd.to_numeric(data[ColName], errors='coerce')
-                print("System : Done!")
-
-        elif isinstance(First_Value , str) and First_Value.strip().startswith("["):
-            
-            data[f"{ColName.capitalize()}List"] = data[ColName].apply(lambda x : literal_eval(x) if pd.notna(x) else np.nan)
-            data.drop(ColName,inplace=True,axis=1)
-
-    return data
-
-
-def ReadAllCSVData(WorkDirectoryPath):
-
-    base_output = os.path.join(WorkDirectoryPath, "All CSV Data is Converted Here")
-    all_folders = [WorkDirectoryPath]
-    
-    os.makedirs(base_output,exist_ok=True)
-    
-    while True:
-
-        if all_folders:
-
-            Curr_Path = all_folders.pop(0)
-
-            for item_name in os.listdir(Curr_Path):
-                
-                Sub_Item_Path = os.path.join(Curr_Path,item_name)
-                
-                if os.path.isfile(Sub_Item_Path) and (Sub_Item_Path.endswith(".csv") or Sub_Item_Path.endswith(".xlsx")):
-                    
-                        
-
-                    df_converted = ConvertListStrToList(Sub_Item_Path)
-                    df_converted.reset_index(drop=True,inplace=True)
-
-                    rel_path = os.path.relpath(Sub_Item_Path, WorkDirectoryPath)
-                    output = os.path.join(base_output,rel_path)
-                    if "List" in df_converted.columns[-1]:
-                        print(Sub_Item_Path)
-                        os.makedirs(os.path.dirname(output),exist_ok=True)
-                        df_converted.to_parquet(output.replace(".csv", ".parquet"))
-
-                    print("-"*50)
-
-                elif os.path.isdir(Sub_Item_Path):
-
-                    all_folders.append(Sub_Item_Path)
-
-        else:
-            break

deepcsv-0.3.0/deepcsv.egg-info/PKG-INFO

@@ -1,74 +0,0 @@
-Metadata-Version: 2.4
-Name: deepcsv
-Version: 0.3.0
-Summary: Automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.
-Author: Abdullah Bakr
-Requires-Python: >=3.7
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pandas
-Requires-Dist: pyarrow
-Dynamic: author
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-# deepcsv
-
-A Python library that automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.
-
-## Installation
-```bash
-pip install deepcsv
-```
-
-## What it does
-
-- Walks through all folders and subfolders automatically
-- Finds every CSV and XLSX file
-- Detects columns that contain list strings like `"['item1', 'item2']"` and converts them into real Python arrays for faster performance
-- Detects columns with mixed data types and tries to fix them automatically
-- Warns you when a column has mixed types so you know what was changed
-- Saves the results as Parquet files to preserve the converted data types
-
-> **Why Parquet?**
-> CSV files cannot store arrays or preserve data types. Parquet solves this by keeping the exact types after conversion.
-
-> **Why arrays instead of Python lists?**
-> Arrays are significantly faster for numerical operations and machine learning workflows.
-
-## Functions
-
-### `ConvertListStrToList(file_path)`
-
-Reads a CSV file, converts list strings to arrays, fixes mixed-type columns, and returns a clean DataFrame.
-```python
-import deepcsv
-
-df = deepcsv.ConvertListStrToList("path/to/file.csv")
-```
-
-### `ReadAllCSVData(path)`
-
-Walks through all folders and subfolders, applies `ConvertListStrToList` on every CSV and XLSX file, and saves the results as Parquet files in a new folder called `All CSV Data is Converted Here`.
-```python
-import deepcsv
-
-deepcsv.ReadAllCSVData("path/to/folder")
-```
-
-## Notes
-
-- Only files that contain list string columns are saved as Parquet
-- Mixed-type columns are converted to float automatically when possible
-- Skips NaN values without breaking
-- Requires `pyarrow` for Parquet support
-
-## Requirements
-
-- Python >= 3.7
-- pandas
-- pyarrow

deepcsv-0.3.0/setup.py

@@ -1,13 +0,0 @@
-from setuptools import setup, find_packages
-
-setup(
-    name="deepcsv",
-    version="0.3.0",
-    author="Abdullah Bakr",
-    description="Automatically walks through folders and subfolders, finds all CSV and XLSX files, detects and fixes data issues, and saves the results as Parquet files while keeping the exact same folder structure.",
-    long_description=open("README.md").read(),
-    long_description_content_type="text/markdown",
-    packages=find_packages(),
-    install_requires=["pandas","pyarrow"],
-    python_requires=">=3.7",
-)