fundata 1.0.1__tar.gz

fundata-1.0.1/PKG-INFO

@@ -0,0 +1,44 @@
+Metadata-Version: 2.1
+Name: fundata
+Version: 1.0.1
+Summary: Add your description here
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# notedata
+
+在学习和工作过程中，经常用到一些比较通用的数据集，很多是国外的数据集，下载很慢。
+这里整理一些数据，并转存到蓝奏云，如果后续有免费的公有云，也可以再迁移到其他云盘。
+
+
+
+|序号|分类|名称|描述|官网下载|蓝奏下载|
+|:-:|:-:|:-:|:-:|:-:|:-:|
+|0|dataset|iris|iris数据集|[官网链接](https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data)|暂无|
+|1|dataset|electronics-reviews|Amazon评论数据|[官网链接](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Electronics_5.json.gz)|[蓝奏链接](https://wws.lanzous.com/b01hkzfuj)|
+|2|dataset|electronics-meta|Amazon评论数据|[官网链接](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/meta_Electronics.json.gz)|[蓝奏链接](https://wws.lanzous.com/b01hqeora)|
+|3|dataset|movielens-100k|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-100k.zip)|[蓝奏链接](https://wws.lanzous.com/iyykCfbi64j)|
+|4|dataset|movielens-1m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-1m.zip)|[蓝奏链接](https://wws.lanzous.com/ihoSUfbi65a)|
+|5|dataset|movielens-10m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-10m.zip)|[蓝奏链接](https://wws.lanzous.com/iXvEmfbi6di)|
+|6|dataset|movielens-20m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-20m.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkt17g)|
+|7|dataset|movielens-25m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-25m.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkt24j)|
+|8|dataset|adult-train||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/adult.data.txt)|暂无|
+|9|dataset|adult-test||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/adult.test.txt)|暂无|
+|10|dataset|porto-seguro-train||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/porto_seguro_train.csv)|暂无|
+|11|dataset|porto-seguro-test||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/porto_seguro_test.csv)|暂无|
+|12|dataset|bitly-usagov||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/datasets/bitly_usagov/example.txt)|暂无|
+|13|dataset|coco-val2017|大型图像数据集, 用于对象检测、分割、人体关键点检测、语义分割和字幕生成|[官网链接](http://images.cocodataset.org/zips/val2017.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkb8fi)|
+|14|dataset|coco-annotations_trainval2017|大型图像数据集, 用于对象检测、分割、人体关键点检测、语义分割和字幕生成|[官网链接](http://images.cocodataset.org/annotations/annotations_trainval2017.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkb86j)|
+|15|model|yolov3.weight|yolov3模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3ih)|
+|16|model|yolov3.h5|yolov3模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3aj)|
+|17|model|yolov4.weight|yolov4模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3yd)|
+|18|model|yolov4-416.h5|yolov4模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hl9lej)|
+|19|dataset|criteo-sample|criteo_sample数据集|[官网链接](https://raw.githubusercontent.com/shenweichen/DeepCTR/master/examples/criteo_sample.txt)|[蓝奏链接](https://wws.lanzous.com/ihLhrhejkxi)|
+|20|dataset|criteo-kaggle|criteo-kaggle数据集|[官网链接](https://s3-eu-west-1.amazonaws.com/kaggle-display-advertising-challenge-dataset/dac.tar.gz)|[蓝奏链接](https://wws.lanzous.com/b01hqh97i)|
+
+
+注：蓝奏无法上传大于100MB的数据，将一个数据拆分为多个文件上传，必须用[notedrive](https://github.com/notechats/notedrive) 来下载
+
+
+# 感谢
+感谢蓝奏云  

fundata-1.0.1/README.md

@@ -0,0 +1,37 @@
+# notedata
+
+在学习和工作过程中，经常用到一些比较通用的数据集，很多是国外的数据集，下载很慢。
+这里整理一些数据，并转存到蓝奏云，如果后续有免费的公有云，也可以再迁移到其他云盘。
+
+
+
+|序号|分类|名称|描述|官网下载|蓝奏下载|
+|:-:|:-:|:-:|:-:|:-:|:-:|
+|0|dataset|iris|iris数据集|[官网链接](https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data)|暂无|
+|1|dataset|electronics-reviews|Amazon评论数据|[官网链接](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Electronics_5.json.gz)|[蓝奏链接](https://wws.lanzous.com/b01hkzfuj)|
+|2|dataset|electronics-meta|Amazon评论数据|[官网链接](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/meta_Electronics.json.gz)|[蓝奏链接](https://wws.lanzous.com/b01hqeora)|
+|3|dataset|movielens-100k|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-100k.zip)|[蓝奏链接](https://wws.lanzous.com/iyykCfbi64j)|
+|4|dataset|movielens-1m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-1m.zip)|[蓝奏链接](https://wws.lanzous.com/ihoSUfbi65a)|
+|5|dataset|movielens-10m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-10m.zip)|[蓝奏链接](https://wws.lanzous.com/iXvEmfbi6di)|
+|6|dataset|movielens-20m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-20m.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkt17g)|
+|7|dataset|movielens-25m|包含用户对电影的评级数据、电影元数据信息和用户属性信息|[官网链接](http://files.grouplens.org/datasets/movielens/ml-25m.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkt24j)|
+|8|dataset|adult-train||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/adult.data.txt)|暂无|
+|9|dataset|adult-test||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/adult.test.txt)|暂无|
+|10|dataset|porto-seguro-train||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/porto_seguro_train.csv)|暂无|
+|11|dataset|porto-seguro-test||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/recommendation/data/porto_seguro_test.csv)|暂无|
+|12|dataset|bitly-usagov||[官网链接](https://raw.githubusercontent.com/1007530194/data/master/datasets/bitly_usagov/example.txt)|暂无|
+|13|dataset|coco-val2017|大型图像数据集, 用于对象检测、分割、人体关键点检测、语义分割和字幕生成|[官网链接](http://images.cocodataset.org/zips/val2017.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkb8fi)|
+|14|dataset|coco-annotations_trainval2017|大型图像数据集, 用于对象检测、分割、人体关键点检测、语义分割和字幕生成|[官网链接](http://images.cocodataset.org/annotations/annotations_trainval2017.zip)|[蓝奏链接](https://wws.lanzous.com/b01hkb86j)|
+|15|model|yolov3.weight|yolov3模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3ih)|
+|16|model|yolov3.h5|yolov3模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3aj)|
+|17|model|yolov4.weight|yolov4模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hjn3yd)|
+|18|model|yolov4-416.h5|yolov4模型的权重|暂无|[蓝奏链接](https://wws.lanzous.com/b01hl9lej)|
+|19|dataset|criteo-sample|criteo_sample数据集|[官网链接](https://raw.githubusercontent.com/shenweichen/DeepCTR/master/examples/criteo_sample.txt)|[蓝奏链接](https://wws.lanzous.com/ihLhrhejkxi)|
+|20|dataset|criteo-kaggle|criteo-kaggle数据集|[官网链接](https://s3-eu-west-1.amazonaws.com/kaggle-display-advertising-challenge-dataset/dac.tar.gz)|[蓝奏链接](https://wws.lanzous.com/b01hqh97i)|
+
+
+注：蓝奏无法上传大于100MB的数据，将一个数据拆分为多个文件上传，必须用[notedrive](https://github.com/notechats/notedrive) 来下载
+
+
+# 感谢
+感谢蓝奏云  

fundata-1.0.1/pyproject.toml

@@ -0,0 +1,7 @@
+[project]
+name = "fundata"
+version = "1.0.1"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.8"
+dependencies = []

fundata-1.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fundata-1.0.1/src/fundata/__init__.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Time    : 2019/04/02 20:01
+# @Author  : niuliangtao
+# @Site    :
+# @File    : __init__.py.py
+# @Software: PyCharm

fundata-1.0.1/src/fundata/dataset/__init__.py

@@ -0,0 +1,8 @@
+from .core import (
+    get_adult_data,
+    get_bitly_usagov_data,
+    get_electronics,
+    get_movielens,
+    get_porto_seguro_data,
+)
+from .datas import CriteoData, ElectronicsData

fundata-1.0.1/src/fundata/dataset/core.py

@@ -0,0 +1,74 @@
+import pandas as pd
+from .datas import ElectronicsData
+from notedata.manage import DatasetManage
+from notetool.tool import log
+
+logger = log(__name__)
+
+
+def get_electronics(dataset: DatasetManage = None):
+    electronic = ElectronicsData(dataset=dataset, data_path="./download/")
+    electronic.init_data()
+
+
+def get_movielens(dataset: DatasetManage = None):
+    dataset.download("movielens-100k", overwrite=False)
+    dataset.download("movielens-1m", overwrite=False)
+    dataset.download("movielens-10m", overwrite=False)
+    dataset.download("movielens-20m", overwrite=False)
+    dataset.download("movielens-25m", overwrite=False)
+    # os.system('cd ' + file_path(data.path) + ' && unzip ' + file_name(data.path))
+
+
+def get_adult_data(dataset: DatasetManage = None):
+    data_train = dataset.download("adult-train", overwrite=False)
+    data_test = dataset.download("adult-test", overwrite=False)
+
+    print(data_test)
+    train_data = pd.read_table(data_train.path, header=None, delimiter=",")
+    test_data = pd.read_table(
+        data_test.path, header=None, delimiter=",", error_bad_lines=False
+    )
+
+    # all_columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation',
+    #                'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country',
+    #                'label', 'type']
+    #
+    # continus_columns = ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']
+    # dummy_columns = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex',
+    #                  'native-country']
+    #
+    # train_data['type'] = 1
+    # test_data['type'] = 2
+    #
+    # all_data = pd.concat([train_data, test_data], axis=0)
+    # all_data.columns = all_columns
+    #
+    # all_data = pd.get_dummies(all_data, columns=dummy_columns)
+    #
+    # test_data = all_data[all_data['type'] == 2].drop(['type'], axis=1)
+    # train_data = all_data[all_data['type'] == 1].drop(['type'], axis=1)
+    #
+    # train_data['label'] = train_data['label'].map(lambda x: 1 if x.strip() == '>50K' else 0)
+    # test_data['label'] = test_data['label'].map(lambda x: 1 if str(x).strip() == '>50K.' else 0)
+    #
+    # for col in continus_columns:
+    #     ss = StandardScaler()
+    #     train_data[col] = ss.fit_transform(train_data[[col]].astype(np.float64))
+    #     test_data[col] = ss.transform(test_data[[col]].astype(np.float64))
+    #
+    # train_y = train_data['label']
+    # train_x = train_data.drop(['label'], axis=1)
+    # test_y = test_data['label']
+    # test_x = test_data.drop(['label'], axis=1)
+    #
+    # return train_x, train_y, test_x, test_y
+
+
+def get_porto_seguro_data(dataset: DatasetManage = None):
+    dataset.download("porto-seguro-train")
+    dataset.download("porto-seguro-test")
+
+
+def get_bitly_usagov_data(dataset: DatasetManage = None):
+    dataset.download("bitly-usagov")