datatailr 0.1.5__tar.gz → 0.1.79__tar.gz

datatailr-0.1.79/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: datatailr
+Version: 0.1.79
+Summary: Ready-to-Use Platform That Drives Business Insights
+Author-email: Datatailr <info@datatailr.com>
+License-Expression: MIT
+Project-URL: homepage, https://www.datatailr.com/
+Project-URL: documentation, https://docs.datatailr.com/
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: types-setuptools; extra == "dev"
+Requires-Dist: toml; extra == "dev"
+Requires-Dist: coverage; extra == "dev"
+Requires-Dist: sphinx-rtd-theme; extra == "dev"
+Requires-Dist: sphinx; extra == "dev"
+Requires-Dist: sphinx-autodoc-typehints; extra == "dev"
+Requires-Dist: sphinx-autosummary; extra == "dev"
+Requires-Dist: sphinx-design; extra == "dev"
+Requires-Dist: sphinx-copybutton; extra == "dev"
+Requires-Dist: myst-parser; extra == "dev"
+Dynamic: license-file
+
+<div style="text-align: center;">
+  <a href="https://www.datatailr.com/" target="_blank">
+    <img src="https://s3.eu-west-1.amazonaws.com/datatailr.com/assets/datatailr-logo.svg" alt="Datatailr Logo" />
+  </a>
+</div>
+
+---
+
+**Datatailr empowers your team to streamline analytics and data workflows
+from idea to production without infrastructure hurdles.**
+
+# What is Datatailr?
+
+Datatailr is a platform that simplifies the process of building and deploying data applications.
+
+It makes it easier to run and maintain large-scale data processing and analytics workloads.
+
+## What is this package?
+This is the Python package for Datatailr, which allows you to interact with the Datatailr platform.
+
+It provides the tools to build, deploy, and manage batch jobs, data pipelines, services and analytics applications.
+
+Datatailr manages the underlying infrastructure so your applications can be deployed in an easy, secure and scalable way.
+
+## Installation
+
+### Installing the `dt` command line tool
+Before you can use the Datatailr Python package, you need to install the `dt` command line tool.
+**[INSTALLATION INSTRUCTIONS FOR DATATAILR GO HERE]**
+
+### Installing the Python package
+You can install the Datatailr Python package using pip:
+```bash
+pip install datatailr
+```
+
+### Testing the installation
+```python
+import datatailr
+
+print(datatailr.__version__)
+print(datatailr.__provider__)
+```
+
+
+## Quickstart
+The following example shows how to create a simple data pipeline using the Datatailr Python package.
+
+```python
+from datatailr import workflow, task
+
+@task()
+def func_no_args() -> str:
+    return "no_args"
+
+
+@task()
+def func_with_args(a: int, b: float) -> str:
+    return f"args: {a}, {b}"
+
+@workflow(name="MY test DAG")
+def my_workflow():
+    for n in range(2):
+        res1 = func_no_args().alias(f"func_{n}")
+        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")
+my_workflow(local_run=True)
+```
+
+Running this code will create a graph of jobs and execute it.
+Each node on the graph represents a job, which in turn is a call to a function decorated with `@task()`.
+
+Since this is a local run then the execution of each node will happen sequentially in the same process.
+
+To take advantage of the datatailr platform and execute the graph at scale, you can run it using the job scheduler as presented in the next section.
+
+## Execution at Scale
+To execute the graph at scale, you can use the Datatailr job scheduler. This allows you to run your jobs in parallel, taking advantage of the underlying infrastructure.
+
+You will first need to separate your function definitions from the DAG definition. This means you should define your functions as a separate module, which can be imported into the DAG definition.
+
+
+```python
+# my_module.py
+
+from datatailr import task
+
+@task()
+def func_no_args() -> str:
+    return "no_args"
+
+
+@task()
+def func_with_args(a: int, b: float) -> str:
+    return f"args: {a}, {b}"
+```
+
+To use these functions in a batch job, you just need to import them and run in a DAG context:
+
+```python
+from my_module import func_no_args, func_with_args
+from datatailr import workflow
+
+@workflow(name="MY test DAG")
+def my_workflow():
+    for n in range(2):
+        res1 = func_no_args().alias(f"func_{n}")
+        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")
+
+schedule = Schedule(at_hours=0)
+my_workflow(schedule=schedule)
+```
+
+This will submit the entire workflow for execution, and the scheduler will take care of running the jobs in parallel and managing the resources.
+The workflow in the example above will be scheduled to run daily at 00:00.
+
+___
+Visit [our website](https://www.datatailr.com/) for more!

datatailr-0.1.79/README.md

@@ -0,0 +1,117 @@
+<div style="text-align: center;">
+  <a href="https://www.datatailr.com/" target="_blank">
+    <img src="https://s3.eu-west-1.amazonaws.com/datatailr.com/assets/datatailr-logo.svg" alt="Datatailr Logo" />
+  </a>
+</div>
+
+---
+
+**Datatailr empowers your team to streamline analytics and data workflows
+from idea to production without infrastructure hurdles.**
+
+# What is Datatailr?
+
+Datatailr is a platform that simplifies the process of building and deploying data applications.
+
+It makes it easier to run and maintain large-scale data processing and analytics workloads.
+
+## What is this package?
+This is the Python package for Datatailr, which allows you to interact with the Datatailr platform.
+
+It provides the tools to build, deploy, and manage batch jobs, data pipelines, services and analytics applications.
+
+Datatailr manages the underlying infrastructure so your applications can be deployed in an easy, secure and scalable way.
+
+## Installation
+
+### Installing the `dt` command line tool
+Before you can use the Datatailr Python package, you need to install the `dt` command line tool.
+**[INSTALLATION INSTRUCTIONS FOR DATATAILR GO HERE]**
+
+### Installing the Python package
+You can install the Datatailr Python package using pip:
+```bash
+pip install datatailr
+```
+
+### Testing the installation
+```python
+import datatailr
+
+print(datatailr.__version__)
+print(datatailr.__provider__)
+```
+
+
+## Quickstart
+The following example shows how to create a simple data pipeline using the Datatailr Python package.
+
+```python
+from datatailr import workflow, task
+
+@task()
+def func_no_args() -> str:
+    return "no_args"
+
+
+@task()
+def func_with_args(a: int, b: float) -> str:
+    return f"args: {a}, {b}"
+
+@workflow(name="MY test DAG")
+def my_workflow():
+    for n in range(2):
+        res1 = func_no_args().alias(f"func_{n}")
+        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")
+my_workflow(local_run=True)
+```
+
+Running this code will create a graph of jobs and execute it.
+Each node on the graph represents a job, which in turn is a call to a function decorated with `@task()`.
+
+Since this is a local run then the execution of each node will happen sequentially in the same process.
+
+To take advantage of the datatailr platform and execute the graph at scale, you can run it using the job scheduler as presented in the next section.
+
+## Execution at Scale
+To execute the graph at scale, you can use the Datatailr job scheduler. This allows you to run your jobs in parallel, taking advantage of the underlying infrastructure.
+
+You will first need to separate your function definitions from the DAG definition. This means you should define your functions as a separate module, which can be imported into the DAG definition.
+
+
+```python
+# my_module.py
+
+from datatailr import task
+
+@task()
+def func_no_args() -> str:
+    return "no_args"
+
+
+@task()
+def func_with_args(a: int, b: float) -> str:
+    return f"args: {a}, {b}"
+```
+
+To use these functions in a batch job, you just need to import them and run in a DAG context:
+
+```python
+from my_module import func_no_args, func_with_args
+from datatailr import workflow
+
+@workflow(name="MY test DAG")
+def my_workflow():
+    for n in range(2):
+        res1 = func_no_args().alias(f"func_{n}")
+        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")
+
+schedule = Schedule(at_hours=0)
+my_workflow(schedule=schedule)
+```
+
+This will submit the entire workflow for execution, and the scheduler will take care of running the jobs in parallel and managing the resources.
+The workflow in the example above will be scheduled to run daily at 00:00.
+
+___
+Visit [our website](https://www.datatailr.com/) for more!

{datatailr-0.1.5 → datatailr-0.1.79}/pyproject.toml

@@ -1,10 +1,10 @@
 [build-system]
-requires = ["setuptools>=60", "wheel", "toml"]
+requires = ["setuptools", "wheel", "toml"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "datatailr"
-version = "0.1.5"
+version = "0.1.79"
 description = "Ready-to-Use Platform That Drives Business Insights"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -34,7 +34,11 @@ homepage = "https://www.datatailr.com/"
 documentation = "https://docs.datatailr.com/"
 
 [project.scripts]
-run_dt_job = "datatailr.sbin.run_job:main"
+datatailr_run = "datatailr.sbin.datatailr_run:main"
+datatailr_run_batch = "datatailr.sbin.datatailr_run_batch:run"
+datatailr_run_app = "datatailr.sbin.datatailr_run_app:run"
+datatailr_run_excel = "datatailr.sbin.datatailr_run_excel:run"
+datatailr_run_service = "datatailr.sbin.datatailr_run_service:run"
 
 [project.optional-dependencies]
 dev = [
@@ -44,13 +48,25 @@ dev = [
     "types-setuptools",
     "toml",
     "coverage",
+    "sphinx-rtd-theme",
+    "sphinx",
+    "sphinx-autodoc-typehints",
+    "sphinx-autosummary",
+    "sphinx-design",
+    "sphinx-copybutton",
+    "myst-parser"
 ]
 
+[tool.coverage.run]
+branch = true
+source = ["./src/datatailr"]
+
 [tool.ruff]
 src = [
     "src",
     "../../tests/src/python"
 ]
+
 lint.ignore = ["F841"]
 show-fixes = true
 fix = true

datatailr-0.1.79/setup.py

@@ -0,0 +1,21 @@
+from datatailr import __version__  # type: ignore
+from setuptools import find_packages, setup
+
+setup(
+    name="datatailr",
+    version=__version__,
+    packages=find_packages(where="src", exclude=["test_module", "test_module.*"]),
+    package_dir={"": "src"},
+    data_files=[
+        (
+            "/datatailr/sbin",
+            [
+                "src/sbin/datatailr_run.py",
+                "src/sbin/datatailr_run_batch.py",
+                "src/sbin/datatailr_run_app.py",
+                "src/sbin/datatailr_run_excel.py",
+                "src/sbin/datatailr_run_service.py",
+            ],
+        )
+    ],
+)

{datatailr-0.1.5 → datatailr-0.1.79}/src/datatailr/__init__.py

@@ -8,32 +8,22 @@
 #  of this file, in parts or full, via any medium is strictly prohibited.
 # *************************************************************************
 
-from datatailr.wrapper import (
-    dt__Blob,
-    dt__Dns,
-    dt__Email,
-    dt__Group,
-    dt__Job,
-    dt__Kv,
-    dt__Log,
-    dt__Node,
-    dt__Registry,
-    dt__Service,
-    dt__Settings,
-    dt__Sms,
-    dt__System,
-    dt__Tag,
-    dt__User,
-    mock_cli_tool,
-)
+from datatailr.wrapper import dt__System, mock_cli_tool
 from datatailr.group import Group
 from datatailr.user import User
 from datatailr.acl import ACL
 from datatailr.blob import Blob
 from datatailr.build import Image
-from datatailr.dt_json import dt_json
 from datatailr.utils import Environment, is_dt_installed
 from datatailr.version import __version__
+from datatailr.scheduler import (
+    App,
+    Service,
+    ExcelAddin,
+    workflow,
+    task,
+    set_allow_unsafe_scheduling,
+)
 
 system = dt__System()
 if isinstance(system, mock_cli_tool):
@@ -50,21 +40,11 @@ __all__ = [
     "User",
     "__version__",
     "__provider__",
-    "dt__Blob",
-    "dt__Dns",
-    "dt__Email",
-    "dt__Group",
-    "dt__Job",
-    "dt__Kv",
-    "dt__Log",
-    "dt__Node",
-    "dt__Registry",
-    "dt__Service",
-    "dt__Settings",
-    "dt__Sms",
-    "dt__System",
-    "dt__Tag",
-    "dt__User",
-    "dt_json",
     "is_dt_installed",
+    "App",
+    "Service",
+    "ExcelAddin",
+    "workflow",
+    "task",
+    "set_allow_unsafe_scheduling",
 ]

{datatailr-0.1.5 → datatailr-0.1.79}/src/datatailr/acl.py

@@ -8,6 +8,8 @@
 #  of this file, in parts or full, via any medium is strictly prohibited.
 # *************************************************************************
 
+from __future__ import annotations
+
 import json
 from typing import List, Optional, Union
 
@@ -23,15 +25,15 @@ class ACL:
         self,
         user: Union[User, str],
         group: Optional[Union[Group, str]] = None,
-        permissions: Optional[List[str]] = None,
+        permissions: Optional[List[str] | str] = None,
     ):
         if user is None:
             user = User.signed_user()
         self.user = user if isinstance(user, User) else User.get(user)
-        if self.user is not None:
-            self.group = (
-                group if group and isinstance(group, Group) else self.user.primary_group
-            )
+        if group is None:
+            group = self.user.primary_group
+        group = group if isinstance(group, Group) else Group.get(str(group))
+        self.group = group
         self.permissions = permissions or "rwr---"
 
         self.__group_can_read = False
@@ -40,9 +42,9 @@ class ACL:
         self.__user_can_write = False
         self.__world_readable = False
 
-        self.__parse_permissions()
+        self.__parse_permissions_string()
 
-    def __parse_permissions(self):
+    def __parse_permissions_string(self):
         """
         Parse the permissions and set the corresponding flags.
         """
@@ -59,6 +61,19 @@ class ACL:
         self.__world_readable = self.permissions[4] == "r"
         self.__world_writable = self.permissions[5] == "w"
 
+    def _set_permissions_string(self):
+        """
+        Set the permissions string based on the current flags.
+        """
+        self.permissions = (
+            f"{'r' if self.__user_can_read else '-'}"
+            f"{'w' if self.__user_can_write else '-'}"
+            f"{'r' if self.__group_can_read else '-'}"
+            f"{'w' if self.__group_can_write else '-'}"
+            f"{'r' if self.__world_readable else '-'}"
+            f"{'w' if self.__world_writable else '-'}"
+        )
+
     def __repr__(self):
         return (
             f"ACL(user={self.user}, group={self.group}, permissions={self.permissions})"
@@ -67,7 +82,7 @@ class ACL:
     def to_dict(self):
         return {
             "user": self.user.name,
-            "group": self.group.name if self.group else self.user.primary_group.name,
+            "group": self.group.name if self.group else "dtusers",
             "group_can_read": self.__group_can_read,
             "group_can_write": self.__group_can_write,
             "user_can_read": self.__user_can_read,
@@ -76,5 +91,22 @@ class ACL:
             "world_writable": self.__world_writable,
         }
 
+    @classmethod
+    def from_dict(cls, acl_dict: dict) -> ACL:
+        """
+        Create an ACL instance from a dictionary.
+        """
+        user = User(acl_dict["user"])
+        group = Group.get(acl_dict["group"]) if "group" in acl_dict else None
+        acl = cls(user=user, group=group)
+        acl.__group_can_read = acl_dict.get("group_can_read", False)
+        acl.__group_can_write = acl_dict.get("group_can_write", False)
+        acl.__user_can_read = acl_dict.get("user_can_read", False)
+        acl.__user_can_write = acl_dict.get("user_can_write", False)
+        acl.__world_readable = acl_dict.get("world_readable", False)
+        acl.__world_writable = acl_dict.get("world_writable", False)
+        acl._set_permissions_string()
+        return acl
+
     def to_json(self):
         return json.dumps(self.to_dict())

{datatailr-0.1.5 → datatailr-0.1.79}/src/datatailr/blob.py

@@ -10,14 +10,17 @@
 
 from __future__ import annotations
 
+import os
 import tempfile
 
-from datatailr import dt__Blob
+from datatailr.wrapper import dt__Blob
 
+# Datatailr Blob API Client
+__client__ = dt__Blob()
+__user__ = os.getenv("USER", "root")
 
-class Blob:
-    __blob_storage__ = dt__Blob()
 
+class Blob:
     def ls(self, path: str) -> list[str]:
         """
         List files in the specified path.
@@ -25,7 +28,7 @@ class Blob:
         :param path: The path to list files from.
         :return: A list of file names in the specified path.
         """
-        return self.__blob_storage__.ls(path)
+        return __client__.ls(path)
 
     def get_file(self, name: str, path: str):
         """
@@ -35,7 +38,7 @@ class Blob:
             name (str): The name of the blob to retrieve.
             path (str): The path to store the blob as a file.
         """
-        return self.__blob_storage__.cp(f"blob:///{name}", path)
+        return __client__.cp(f"blob://{name}", path)
 
     def put_file(self, name: str, path: str):
         """
@@ -45,7 +48,7 @@ class Blob:
             name (str): The name of the blob to create.
             path (str): The path of the local file to copy.
         """
-        return self.__blob_storage__.cp(path, f"blob:///{name}")
+        return __client__.cp(path, f"blob://{name}")
 
     def exists(self, name: str) -> bool:
         """
@@ -57,7 +60,7 @@ class Blob:
         Returns:
             bool: True if the blob exists, False otherwise.
         """
-        return self.__blob_storage__.exists(name)
+        return __client__.exists(name)
 
     def delete(self, name: str):
         """
@@ -66,7 +69,7 @@ class Blob:
         Args:
             name (str): The name of the blob to delete.
         """
-        return self.__blob_storage__.rm(name)
+        return __client__.rm(name)
 
     def get_blob(self, name: str):
         """
@@ -80,10 +83,15 @@ class Blob:
         """
         # Since direct reading and writting of blobs is not implemented yet, we are using a temporary file.
         # This is a workaround to allow reading the blob content directly from the blob storage.
-
-        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
+        temp_dir = f"/home/{__user__}/tmp"
+        if not os.path.exists(temp_dir):
+            temp_dir = "/tmp"
+        else:
+            temp_dir += "/.dt"
+            os.makedirs(temp_dir, exist_ok=True)
+        with tempfile.NamedTemporaryFile(dir=temp_dir, delete=True) as temp_file:
             self.get_file(name, temp_file.name)
-            with open(temp_file.name, "rb") as f:
+            with open(temp_file.name, "r") as f:
                 return f.read()
 
     def put_blob(self, name: str, blob):
@@ -96,8 +104,11 @@ class Blob:
         """
         # Since direct reading and writting of blobs is not implemented yet, we are using a temporary file.
         # This is a workaround to allow writing the blob content directly to the blob storage.
-
-        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
-            with open(temp_file.name, "wb") as f:
-                f.write(blob)
+        with tempfile.NamedTemporaryFile(delete=True) as temp_file:
+            if isinstance(blob, bytes):
+                with open(temp_file.name, "wb") as f:
+                    f.write(blob)
+            else:
+                with open(temp_file.name, "w") as f:
+                    f.write(blob)
             self.put_file(name, temp_file.name)