datablob 0.0.0__tar.gz

datablob-0.0.0/LICENSE

@@ -0,0 +1,121 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

datablob-0.0.0/PKG-INFO

@@ -0,0 +1,45 @@
+Metadata-Version: 2.4
+Name: datablob
+Version: 0.0.0
+Summary: Client for Updating a Simple Data Warehouse on Blob Storage
+Home-page: https://github.com/gocarta/datablob
+Download-URL: https://github.com/gocarta/datablob/tarball/download
+Author: Daniel J. Dufour
+Author-email: daniel.j.dufour@gmail.com
+Keywords: data,python
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: boto3
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary
+
+# datablob
+Client for Updating a Simple Data Warehouse on Blob Storage
+
+## install
+```sh
+pip install datablob
+```
+
+## usage
+More examples coming soon
+```py
+from datablob import DataBlobClient
+
+client = DataBlobClient(bucket_name="example-test-bucket-123", bucket_path="prefix/to/dataportal")
+
+client.update_dataset(name="fleet", version="2", data=rows)
+```

datablob-0.0.0/README.md

@@ -0,0 +1,17 @@
+# datablob
+Client for Updating a Simple Data Warehouse on Blob Storage
+
+## install
+```sh
+pip install datablob
+```
+
+## usage
+More examples coming soon
+```py
+from datablob import DataBlobClient
+
+client = DataBlobClient(bucket_name="example-test-bucket-123", bucket_path="prefix/to/dataportal")
+
+client.update_dataset(name="fleet", version="2", data=rows)
+```

datablob-0.0.0/datablob/__init__.py

@@ -0,0 +1,89 @@
+import boto3
+import csv
+import io
+import json
+
+
+class DataBlobClient:
+    def __init__(self, bucket_name, bucket_path):
+        self.bucket_name = bucket_name
+        self.bucket_path = bucket_path.rstrip("/")
+
+    def _get_unique_keys(self, rows):
+        columns = set()
+        for row in rows:
+            columns.update(row.keys())
+        return list(sorted(list(columns)))
+
+    def upload_csv(self, dataset_name, dataset_version, data):
+        key = (
+            self.bucket_path + "/" + dataset_name + "/v" + dataset_version + "/data.csv"
+        )
+        boto3.client("s3").put_object(
+            Bucket=self.bucket_name,
+            Key=key,
+            Body=data,
+        )
+
+    def get_dataset_as_csv(self, name, version, remove_bom=True):
+        key = self.bucket_path + "/" + name + "/v" + version + "/data.csv"
+        response = boto3.client("s3").get_object(Bucket=self.bucket_name, Key=key)
+        object_content = response["Body"].read().decode("utf-8")
+        if remove_bom:
+            object_content = object_content.lstrip("\ufeff")
+        return object_content
+
+    def get_dataset_as_json(self, name, version):
+        key = self.bucket_path + "/" + name + "/v" + version + "/data.json"
+        response = boto3.client("s3").get_object(Bucket=self.bucket_name, Key=key)
+        object_content = response["Body"].read().decode("utf-8")
+        return json.loads(object_content)
+
+    def upload_json(self, dataset_name, dataset_version, data):
+        key = (
+            self.bucket_path
+            + "/"
+            + dataset_name
+            + "/v"
+            + dataset_version
+            + "/data.json"
+        )
+        boto3.client("s3").put_object(
+            Bucket=self.bucket_name,
+            Key=key,
+            Body=data if isinstance(data, str) else json.dumps(data),
+        )
+
+    def upload_metadata(self, dataset_name, dataset_version, data):
+        key = (
+            self.bucket_path
+            + "/"
+            + dataset_name
+            + "/v"
+            + dataset_version
+            + "/meta.json"
+        )
+        boto3.client("s3").put_object(
+            Bucket=self.bucket_name,
+            Key=key,
+            Body=data if isinstance(data, str) else json.dumps(data),
+        )
+
+    def convert_rows_to_csv(self, rows, fieldnames=None):
+        f = io.StringIO()
+        if fieldnames is None:
+            fieldnames = sorted(list(rows[0].keys()))
+        writer = csv.DictWriter(f, fieldnames=fieldnames)
+        writer.writeheader()
+        writer.writerows(rows)
+        f.seek(0)
+        # read and make sure we don't have a BOM
+        return f.read().lstrip("\ufeff")
+
+    def update_dataset(self, name, version, data, column_names=None):
+        columns = column_names if column_names else self._get_unique_keys(data)
+        meta = {"name": name, "numColumns": len(columns), "numRows": len(data)}
+        data_as_csv = self.convert_rows_to_csv(data, fieldnames=columns)
+        self.upload_csv(name, version, data_as_csv)
+        self.upload_json(name, version, data)
+        self.upload_metadata(name, version, meta)

datablob-0.0.0/datablob.egg-info/PKG-INFO

@@ -0,0 +1,45 @@
+Metadata-Version: 2.4
+Name: datablob
+Version: 0.0.0
+Summary: Client for Updating a Simple Data Warehouse on Blob Storage
+Home-page: https://github.com/gocarta/datablob
+Download-URL: https://github.com/gocarta/datablob/tarball/download
+Author: Daniel J. Dufour
+Author-email: daniel.j.dufour@gmail.com
+Keywords: data,python
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: boto3
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary
+
+# datablob
+Client for Updating a Simple Data Warehouse on Blob Storage
+
+## install
+```sh
+pip install datablob
+```
+
+## usage
+More examples coming soon
+```py
+from datablob import DataBlobClient
+
+client = DataBlobClient(bucket_name="example-test-bucket-123", bucket_path="prefix/to/dataportal")
+
+client.update_dataset(name="fleet", version="2", data=rows)
+```

datablob-0.0.0/datablob.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+setup.cfg
+setup.py
+datablob/__init__.py
+datablob.egg-info/PKG-INFO
+datablob.egg-info/SOURCES.txt
+datablob.egg-info/dependency_links.txt
+datablob.egg-info/requires.txt
+datablob.egg-info/top_level.txt

datablob-0.0.0/datablob.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

datablob-0.0.0/datablob.egg-info/requires.txt

@@ -0,0 +1 @@
+boto3

datablob-0.0.0/datablob.egg-info/top_level.txt

@@ -0,0 +1 @@
+datablob

datablob-0.0.0/setup.cfg

@@ -0,0 +1,7 @@
+[metadata]
+description-file = README.md
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

datablob-0.0.0/setup.py

@@ -0,0 +1,34 @@
+from setuptools import setup
+from os import path
+
+this_directory = path.abspath(path.dirname(__file__))
+
+with open(path.join(this_directory, "README.md")) as f:
+    long_description = f.read()
+
+setup(
+    name="datablob",
+    packages=["datablob"],
+    package_dir={"datablob": "datablob"},
+    package_data={
+        "datablob": [
+            "__init__.py",
+        ]
+    },
+    version="0.0.0",
+    description="Client for Updating a Simple Data Warehouse on Blob Storage",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    author="Daniel J. Dufour",
+    author_email="daniel.j.dufour@gmail.com",
+    url="https://github.com/gocarta/datablob",
+    download_url="https://github.com/gocarta/datablob/tarball/download",
+    keywords=["data", "python"],
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Programming Language :: Python :: 3",
+        "License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication",
+        "Operating System :: OS Independent",
+    ],
+    install_requires=["boto3"],
+)