pydatamax 0.1.5__py3-none-any.whl → 0.1.11__py3-none-any.whl

{pydatamax-0.1.5.dist-info → pydatamax-0.1.11.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (75.6.0)
+Generator: setuptools (80.8.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

{pydatamax-0.1.5.dist-info → pydatamax-0.1.11.dist-info}/top_level.txt

@@ -1 +1,2 @@
 datamax
+tests

tests/test_basic.py

@@ -0,0 +1,20 @@
+"""
+DataMax 基础测试
+"""
+
+from datamax import DataMax
+
+
+def test_import():
+    """测试模块导入"""
+    assert DataMax is not None
+
+
+def test_version():
+    """测试版本号"""
+    import datamax
+
+    assert hasattr(datamax, "__version__") or True  # 版本号检查
+
+
+# 更多测试用例...

pydatamax-0.1.5.dist-info/METADATA

@@ -1,282 +0,0 @@
-Metadata-Version: 2.1
-Name: pydatamax
-Version: 0.1.5
-Summary: A library for parsing and converting various file formats.
-Home-page: https://github.com/cosco/datamax
-Author: hzb | ccy
-Author-email: zhibaohe@hotmail.com | cy.kron@foxmail.com
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: ebooklib
-Requires-Dist: python-docx
-Requires-Dist: beautifulsoup4
-Requires-Dist: python-dotenv
-Requires-Dist: minio
-Requires-Dist: loguru
-Requires-Dist: tqdm
-Requires-Dist: oss2
-Requires-Dist: python-docx
-Requires-Dist: openai
-Requires-Dist: jionlp
-Requires-Dist: chardet
-Requires-Dist: python-pptx
-Requires-Dist: paddlepaddle
-Requires-Dist: paddlepaddle-gpu
-Requires-Dist: openpyxl
-Requires-Dist: paddleocr==2.8.1
-Requires-Dist: pymupdf
-Requires-Dist: langchain_community==0.2.9
-Requires-Dist: premailer
-Requires-Dist: setuptools==75.3.0
-Requires-Dist: docx2markdown
-Requires-Dist: tiktoken
-
-# DataMax
-
-<p align="center">
-  <a href="./README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
-  <a href="./README_zh.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
-</p>
-
-## Project Introduction
-The Text Processing Tool is a multifunctional data processing project designed to provide an efficient and easy-to-use way to handle various formats of text files. The project supports reading, converting, and extracting from common text formats such as PDF, HTML, DOCX/DOC, PPT/PPTX, EPUB, images (.jpg | .png), and TXT.
-
-
-## Installation Guide
-
-## Install
-
-### Linux（Debian/Ubuntu）
-```
-sudo apt-get update
-sudo apt-get install libreoffice
-```
-### Windows
-```
-Windows install LibreOffice：https://www.libreoffice.org/download/download-libreoffice/?spm=5176.28103460.0.0.5b295d275bpHzh
-$env:PATH += ";C:\Program Files\LibreOffice\program"
-```
-### Check LibreOffice Installation
-```
-soffice --version
-```
-### Install Dependencies
-```
-pip install --no-cache-dir -r requirements.txt -i http://repo-reader:6Np2jcK%248Fqp@nexus.csntcorp.com/repository/aliyun_pipy/simple/ --trusted-host nexus.csntcorp.com
-```
-
-1. Clone the repository to local:
-   ```bash
-   git clone 
-   ```
-2. Enter the project directory and create a virtual environment (e.g., uv):
-   ```bash
-   cd DataMax
-   uv init
-   uv add requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
-   uv sync
-   . .venv/bin/activate
-   ```
-
-
-## Features
-- **Multi-format Support**: Capable of handling PDF, HTML, DOCX, and TXT, among other text file types.
-- **Content Extraction**: Provides powerful content extraction capabilities to accurately extract needed information from complex document structures.
-- **Data Conversion**: Supports conversion of processed data to Markdown format for further data analysis.
-- **Batch Processing**: Capable of processing a large number of files at once, improving work efficiency.
-- **Custom Configuration**: Users can adjust processing parameters according to their needs to meet various business requirements.
-- **Cross-platform Compatibility**: This SDK can run on multiple operating systems including Windows, MacOS, and Linux.
-
-
-## Technology Stack
-- **Programming Language**: Python >= 3.9
-- **Dependency Libraries**:
-  - PyMuPDF: For parsing PDF files.
-  - BeautifulSoup: For parsing HTML files.
-  - python-docx: For parsing DOCX files.
-  - pandas: For data processing and conversion.
-  - paddleocr: For parsing PDF scans, forms, and images.
-- **Development Environment**: Visual Studio Code or PyCharm
-- **Version Control**: Git
-
-
-## Usage Instructions
-### Install SDK
-- Installation Commands:
-  ```bash
-  ## Local installation
-  python setup.py sdist bdist_wheel
-  pip install dist/datamax-0.1.3-py3-none-any.whl
-  ```
-- Import Code
-    ```python
-    from datamax import DataMaxParser
-    ## Handling a single file in two ways
-    # 1. List of length 1
-    data = DataMaxParser(file_path=[r"docx_files_example/ShipViewOverview.doc"])
-    data = data.get_data()
-    # 2. String
-    data = DataMaxParser(file_path=r"docx_files_example/ShipViewOverview.doc")
-    data = data.get_data()
-    
-    ## Handling multiple files
-    ## 1. List of length n
-    data = DataMaxParser(file_path=[r"docx_files_example/ShipViewOverview1.doc", r"docx_files_example/ShipViewOverview2.doc"])
-    data = data.get_data()
-    
-    ## 2. Passing a folder string
-    data = DataMaxParser(file_path=r"docx_files_example/")
-    data = data.get_data()
-    ```
-
-## Examples
-```python
-  ## docx | doc | epub | html | txt | jpg | png | ppt | pptx
-  from datamax import DataMaxParser
-  data = DataMaxParser(file_path=r"docx_files_example/ShipViewOverview.doc", to_markdown=True)
-  """
-  Parameters: 
-  file_path: File relative path / File absolute path
-  to_markdown: Whether to convert to markdown (default value False, returns text directly) This parameter only supports Word files (doc | docx)
-  """
-  
-
-  ## pdf
-  from datamax import DataMaxParser
-  data = DataMaxParser(file_path=r"docx_files_example/ShipViewOverview.pdf", use_ocr=True, use_gpu=True, gpu_id=0)
-  """
-  Parameters: 
-  file_path: File relative path / File absolute path
-  use_ocr: Whether to use OCR (True returns markdown, False returns plain text)
-  use_gpu: Whether to use GPU (ensure CUDA environment is available)
-  gpu_id: GPU card (default 6th card)
-  """
-```
-
-## Contribution Guidelines
-We welcome any form of contribution, whether it's reporting bugs, suggesting new features, or directly submitting code improvements. Please read our [Contributor Guidelines](CONTRIBUTING.md) to get started.
-
-## License
-This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
-
-## Contact
-If you encounter any issues during use, or have any suggestions and feedback, please contact us through the following methods:
-- Email: cy.kron@foxmail.com | zhibaohe@hotmail.com
-- Project Homepage: [GitHub Project Link](xxxx)
-
-## RoadMap
-- [x] Implement OSS Data Class
-- [x] Implement OBS Data Class
-- [x] Implement PGSQL Data Class
-- [x] Implement Local Data Class
-- [x] OSS Data Class should be able to obtain bucket information, metadata, and file lists from the OSS data source
-- [x] OBS Data Class should be able to obtain bucket information, metadata, and file lists from the OBS data source
-- [ ] PGSQL Data Class should be able to obtain schema information, metadata, and table structures from the PGSQL data source
-
-
-- (Optional) Data Classes (Access Data Sources) / Direct Local Reading DataSourceClass
-    - MinIO
-    - OSS
-    - OBS
-    - PostgreSQL
-
-DataLoader.load(fp: file_path, ) -> local path 
-
-DataLoader.load(source: DataSourceClass, ) -> Access data source, return metadata of the data source: able to know file size & file path & file download address & storage space occupied by the data source ->
-
-- Data Loading Class (Load and read PDF, Word, Excel. Able to load the following types of data from the data source on demand)
-    - .pdf
-    - .pdf（Image type / Scanned copy）
-    - .docx
-    - .html
-    - .pptx
-    - .epub
-    - .txt
-    - .md
-    ---
-    - .csv
-    - .json
-    - .xlsx
-    ---
-    - http / https
-    --- 
-    - Multimodal
-        - .png
-        - .jpg
-        - .jpeg
-        - .bmp
-        - .gif
-
-
-
-class DataLoaderClass:
-    def read_docx(self, DocxInputVo):  # TODO: ccy
-        ...
-        
-    def read_pdf():
-        ...
-
-
-- Data Parsing Class (Parsing algorithms, parsing logic & output string objects)
-    - .pdf
-    - .pdf (Image type / Scanned copy) PaddleOCR / AI LAB OCR /  xx OCR
-    - .doc
-    - .docx
-    - .html
-    - .ppt
-    - .pptx
-    - .epub
-    - .txt
-    - .md
-    ---
-    - .csv
-    - .json
-    - .xlsx
-    - .xls
-    ---
-    - http / https
-    --- 
-    - Multimodal
-        - .png
-        - .jpg
-        - .jpeg
-        - .bmp
-        - .gif
-
-
-- Data Cleaning Class (Input: String object, according to cleaning rules, get cleaned markdown format text. Output: markdown)
-
-- Large Model Class (Pre-tagging)
-
-- Data Output Class (Unified format)
-
-
-## Structure
-```bash
-├── api            # Various interfaces exposed by the project
-├── datamax        # Core functions and classes of various SDKs
-├── dockerfiles    # Docker configuration files
-├── docs           # Project documentation
-├── example        # Example code
-├── README.md      # Project description file
-├── scripts        # Various scripts
-└── test           # Test code
-```
-
-
-
-## Standards
-
-1. Class names are declared using PascalCase
-2. Function names are declared using lowercase and underscores
-3. Function input and output need to be declared with data types using colons
-4. Remove print logger before pushing code
-5. Rebase:
-    1. git add .
-    2. git commit -m ":boom: new feature"
-    3. git pull --rebase
-    4. git push

pydatamax-0.1.5.dist-info/RECORD

@@ -1,31 +0,0 @@
-datamax/__init__.py,sha256=1rbhISrgmv_MT11nIiajPUxsX5kKUUY1B5Q1A-oscuc,34
-datamax/loader/MinioHandler.py,sha256=0XLvtKayhI2cjPU7S7m91tSzPcaOC924ByJNbabY6So,6386
-datamax/loader/OssHandler.py,sha256=JWBulEPHv3ThA6RUdwA_EGW8ROnB5OGG8P91J1VmP8U,7236
-datamax/loader/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-datamax/loader/core.py,sha256=tSIkOw5D3EVFYme1b7joFt0e_LxJdf-mdUzxpyVt0VI,5098
-datamax/parser/__init__.py,sha256=n4VG9tdHOgnq7KjzD5adQKw6nIQCaB6dLZJWhCtWNiY,117
-datamax/parser/base.py,sha256=TOJHRq7ytQSyHIagChLKHkcuSj_46ivqMiXq9NDlt-w,2525
-datamax/parser/core.py,sha256=acIyW_UXaucDNvaSGj__XDBXp-k_KGW5MiS1OGQzUiU,4521
-datamax/parser/csv_parser.py,sha256=IcyVq8mGE6auIcUInXGYWDnh0H0XJ_3SyQrLVRrS7i0,190
-datamax/parser/doc_parser.py,sha256=U_QCqeXdgJ5KDkO8glPmpmy4lLG-avd4-7wrUrrKd5c,3571
-datamax/parser/docx_parser.py,sha256=yyRW-vPOTn0f_Kk0A1VUX7_sIrOHaJCgjfAAuMUDDG8,1949
-datamax/parser/epub_parser.py,sha256=4TIi6ufQ_7O27z0o2WKxjHkDDI3FT2nvldlfffWycCI,1776
-datamax/parser/html_parser.py,sha256=-y1AtPTZpvtdtvFJtxDoamzI0FAYrWb8Jb4RDt2DX1Q,1608
-datamax/parser/image_parser.py,sha256=XyzZlWvLJL3itOTsy0_eYsljTfgA5O0vacJXjtxeYiU,1260
-datamax/parser/json_parser.py,sha256=MFamKCkP5Ny1kJyJlPkd_vNqk31ngPRf8NoYw8SxMY4,190
-datamax/parser/md_parser.py,sha256=MFamKCkP5Ny1kJyJlPkd_vNqk31ngPRf8NoYw8SxMY4,190
-datamax/parser/pdf_parser.py,sha256=cNXRbpTpncvV56-fMbinslpMuUsXIkv_DqHVtZfUt00,2594
-datamax/parser/ppt_parser.py,sha256=Q_wVl9-PmLbU3ezwltKf4DuVmVQrG_L5vEDEUfKhKEw,3305
-datamax/parser/pptx_parser.py,sha256=y2ea39FCsxO-6gekd4BmjiYzpoG-saBWmd2IonNde2A,2014
-datamax/parser/txt_parser.py,sha256=9NidWdHUOCW17WSMLnDSZqVhlXQwSQhumQ99GxypjEY,1901
-datamax/parser/xlsx_parser.py,sha256=MFamKCkP5Ny1kJyJlPkd_vNqk31ngPRf8NoYw8SxMY4,190
-datamax/utils/__init__.py,sha256=eifKmh0wYX6I7xcaZtZcw3rh1I1US_gPJyr5yKv3VCg,997
-datamax/utils/data_cleaner.py,sha256=XxaDn5QHBOcNKXrLMdmbo5uEvPWCtovZitdrKBqnF4U,7307
-datamax/utils/paddleocr_pdf_operator.py,sha256=-Vqb1L9DhuUWRjM_aDD2Nok8XwUGyuYOVI-hagfb5II,3482
-datamax/utils/ppt_extract.py,sha256=nd6KSqEzxANrPhNPUZY4ogAyxHzKCbdsI5ZfDQCz0Cw,6164
-datamax/utils/tokenizer.py,sha256=5Gr7bfmtr6cfQOrJVf0n8PPTdW0Dj1pN5xO5hjJhOg8,858
-pydatamax-0.1.5.dist-info/LICENSE,sha256=LvCq2Pc7MejIvfNeRl_kAM5l_KXWlQFiC-Sjp7kqFf8,1067
-pydatamax-0.1.5.dist-info/METADATA,sha256=9Fzp3sCiaUaFwBAhWshWdd3Z87ZRmoYZUHlp5Rqol2s,8955
-pydatamax-0.1.5.dist-info/WHEEL,sha256=PZUExdf71Ui_so67QXpySuHtCi3-J3wvF4ORK6k_S8U,91
-pydatamax-0.1.5.dist-info/top_level.txt,sha256=N9TrwI3GKnWW07RRXHr0xX5Bm8dIM_sahfAnf9j8J9M,8
-pydatamax-0.1.5.dist-info/RECORD,,