lynguine 0.1.0__tar.gz

lynguine-0.1.0/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.1
+Name: lynguine
+Version: 0.1.0
+Summary: Package for linguine data oriented architecture interfaces.
+Home-page: https://github.com/lawrennd/linguine
+License: MIT
+Author: Neil D. Lawrence
+Author-email: lawrennd@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Dist: XlsxWriter
+Requires-Dist: anyascii
+Requires-Dist: bibtexparser
+Requires-Dist: gitpython
+Requires-Dist: google-api-python-client
+Requires-Dist: google-auth-httplib2
+Requires-Dist: gspread-pandas
+Requires-Dist: markdown
+Requires-Dist: markdownify
+Requires-Dist: mimesis (>11)
+Requires-Dist: numpy (>=1.0.0,<2.0.0)
+Requires-Dist: openpyxl
+Requires-Dist: pandas
+Requires-Dist: pylatexenc
+Requires-Dist: pyminizip
+Requires-Dist: pypandoc
+Requires-Dist: python-frontmatter
+Requires-Dist: python-liquid
+Requires-Dist: pyyaml
+Requires-Dist: pyzmq (>=25.1.2,<26.0.0)
+Requires-Dist: wget
+Description-Content-Type: text/markdown
+
+# linguine
+
+
+![Tests](https://github.com/lawrennd/linguine/actions/workflows/python-tests.yml/badge.svg)
+
+
+[![codecov](https://codecov.io/gh/lawrennd/linguine/branch/main/graph/badge.svg?token=YOUR_CODECOV_TOKEN)](https://codecov.io/gh/lawrennd/linguine)
+
+The `linguine` library is a library based on DOA principles for managing data. It provides these capabilities to support other libraries like `lamd` and `referia`.
+To install use
+
+```bash
+%pip install linguine
+```
+
+The softare consists of TK principle parts.
+
+### Config 
+
+First `config` which consists of `interface` and `context`. 
+
+`context` defines the `Context` object which is used to store information about the context, such as machine type etc. On the other hand`interface` defines the `Interface` object that's used for defining inputs and outputs which defines connections to other 'black box processes'. 
+
+#### A short example
+
+The local context can be loaded in using the following command.
+```
+import linguine as ln
+
+ctxt = ln.config.context.Context()
+```
+
+The `interface` module contains the key structure of `linguine`. It specifies incoming and outgoing flows, as well as computational operations. Each flow is specified in the following form.
+
+```yaml
+input:
+  source:
+```  
+
+Preprocessing can be done with a compute field.
+
+```yaml
+input:
+  compute:
+    field: ColumnName0
+    function: computeFunction
+    args:
+      arg1: argument1
+      arg2: argument2
+    row_args:
+      arg3: ColumnName1
+```
+
+Often the data will be stored in another file (csv, excel, yaml etc) but sometimes it's convenient to store it as a `local` in a field calld 'data'. In the next example we do this to illustrate how the `compute` capability can be used to augment the file. Here two fields are added, the full name (used as an index) and today's date as an access date.
+
+```python
+import yaml
+from linguine.config.interface import Interface
+from linguine.assess.data import CustomDataFrame
+
+# Let's assume this is the text stored in the interface file
+yaml_text = """input:
+  type: local
+  index: fullName
+  data:
+  - familyName: Xing
+    givenName: Pei
+  - familyName: Venkatasubramanian
+    givenName: Siva
+  - familyName: Paz Luiz
+    givenName: Miguel
+  compute:
+  - field: fullName
+    function: render_liquid
+    args:
+      template: '{{familyName | replace: " ", "-"}}_{{givenName | replace: " ", "-"}}'
+    row_args:
+      givenName: givenName
+      familyName: familyName
+  - field: accessDate
+    function: today"""
+
+interface = Interface(yaml.safe_load(yaml_text))
+
+data = CustomDataFrame.from_flow(interface)
+print(data)
+```
+
+would create a new field fullname which is then used as the index.
+
+### Access
+
+Secondly the software uses the access, assess, address decomposition. Where `access` is used for accessing data and consists of `io` and `download`. `io` allows for reading from and writing to various different file formats such as `json`, `yaml`, `markdown`, `csv`, `xls`, `bibtex`. 
+
+`download` is for accessing resources from the web, such as downloading a specific url.
+
+#### A short example
+
+Perhaps you would like to create a bibtex file from the PMLR proceedings volume 1, Gaussian processes in practice. In the short example below, we use `linguine` to first download the relevant URL, then we load it in and save as bibtex.
+
+```python
+import linguine
+```
+
+
+### Assess
+
+Assess is about taking the raw data and processing it. Under `assess` `linguine` provides `data` and `compute`. The `data` module provides a `CustomDataFrame` object that provides access to the data and manipulation capabilities. the `compute` module wraps various compute capabilities for preprocessing and processing the data. 
+
+#### A short example
+
+```python
+import linguine
+```
+
+### Util
+
+The `util` module provides various utilities for working with data. They include 
+
+* `dataframe` for operating on data frames. 
+* `fake` for generating fake data.
+* `files` for interacting with files.
+* `html` for working with html.
+* `liquid` for working with the liquid template language.
+* `talk` for working with Neil's talk format.
+* `tex` for working with latex.
+* `text` for working with text.
+* `yaml` for working with yaml.
+* `misc` for other miscellaneous utilities.
+
+### A short example
+
+```python
+import linguine
+```
+
+### Tests
+
+The tests are stored in the `tests` subdirectory. They use `pytest`.
+
+
+#### A short example
+
+If you have poetry installed you can run the tests using
+
+```bash
+poetry run pytest 
+```
+
+### Why linguine?
+
+The name comes from the idea that data oriented architecture is like a set of streams of data, like linguine pasta. In Italian the word also means "little toungues" so there's also a connocation of translation between services.
+

lynguine-0.1.0/README.md

@@ -0,0 +1,152 @@
+# linguine
+
+
+![Tests](https://github.com/lawrennd/linguine/actions/workflows/python-tests.yml/badge.svg)
+
+
+[![codecov](https://codecov.io/gh/lawrennd/linguine/branch/main/graph/badge.svg?token=YOUR_CODECOV_TOKEN)](https://codecov.io/gh/lawrennd/linguine)
+
+The `linguine` library is a library based on DOA principles for managing data. It provides these capabilities to support other libraries like `lamd` and `referia`.
+To install use
+
+```bash
+%pip install linguine
+```
+
+The softare consists of TK principle parts.
+
+### Config 
+
+First `config` which consists of `interface` and `context`. 
+
+`context` defines the `Context` object which is used to store information about the context, such as machine type etc. On the other hand`interface` defines the `Interface` object that's used for defining inputs and outputs which defines connections to other 'black box processes'. 
+
+#### A short example
+
+The local context can be loaded in using the following command.
+```
+import linguine as ln
+
+ctxt = ln.config.context.Context()
+```
+
+The `interface` module contains the key structure of `linguine`. It specifies incoming and outgoing flows, as well as computational operations. Each flow is specified in the following form.
+
+```yaml
+input:
+  source:
+```  
+
+Preprocessing can be done with a compute field.
+
+```yaml
+input:
+  compute:
+    field: ColumnName0
+    function: computeFunction
+    args:
+      arg1: argument1
+      arg2: argument2
+    row_args:
+      arg3: ColumnName1
+```
+
+Often the data will be stored in another file (csv, excel, yaml etc) but sometimes it's convenient to store it as a `local` in a field calld 'data'. In the next example we do this to illustrate how the `compute` capability can be used to augment the file. Here two fields are added, the full name (used as an index) and today's date as an access date.
+
+```python
+import yaml
+from linguine.config.interface import Interface
+from linguine.assess.data import CustomDataFrame
+
+# Let's assume this is the text stored in the interface file
+yaml_text = """input:
+  type: local
+  index: fullName
+  data:
+  - familyName: Xing
+    givenName: Pei
+  - familyName: Venkatasubramanian
+    givenName: Siva
+  - familyName: Paz Luiz
+    givenName: Miguel
+  compute:
+  - field: fullName
+    function: render_liquid
+    args:
+      template: '{{familyName | replace: " ", "-"}}_{{givenName | replace: " ", "-"}}'
+    row_args:
+      givenName: givenName
+      familyName: familyName
+  - field: accessDate
+    function: today"""
+
+interface = Interface(yaml.safe_load(yaml_text))
+
+data = CustomDataFrame.from_flow(interface)
+print(data)
+```
+
+would create a new field fullname which is then used as the index.
+
+### Access
+
+Secondly the software uses the access, assess, address decomposition. Where `access` is used for accessing data and consists of `io` and `download`. `io` allows for reading from and writing to various different file formats such as `json`, `yaml`, `markdown`, `csv`, `xls`, `bibtex`. 
+
+`download` is for accessing resources from the web, such as downloading a specific url.
+
+#### A short example
+
+Perhaps you would like to create a bibtex file from the PMLR proceedings volume 1, Gaussian processes in practice. In the short example below, we use `linguine` to first download the relevant URL, then we load it in and save as bibtex.
+
+```python
+import linguine
+```
+
+
+### Assess
+
+Assess is about taking the raw data and processing it. Under `assess` `linguine` provides `data` and `compute`. The `data` module provides a `CustomDataFrame` object that provides access to the data and manipulation capabilities. the `compute` module wraps various compute capabilities for preprocessing and processing the data. 
+
+#### A short example
+
+```python
+import linguine
+```
+
+### Util
+
+The `util` module provides various utilities for working with data. They include 
+
+* `dataframe` for operating on data frames. 
+* `fake` for generating fake data.
+* `files` for interacting with files.
+* `html` for working with html.
+* `liquid` for working with the liquid template language.
+* `talk` for working with Neil's talk format.
+* `tex` for working with latex.
+* `text` for working with text.
+* `yaml` for working with yaml.
+* `misc` for other miscellaneous utilities.
+
+### A short example
+
+```python
+import linguine
+```
+
+### Tests
+
+The tests are stored in the `tests` subdirectory. They use `pytest`.
+
+
+#### A short example
+
+If you have poetry installed you can run the tests using
+
+```bash
+poetry run pytest 
+```
+
+### Why linguine?
+
+The name comes from the idea that data oriented architecture is like a set of streams of data, like linguine pasta. In Italian the word also means "little toungues" so there's also a connocation of translation between services.

lynguine-0.1.0/lynguine/__init__.py

@@ -0,0 +1,10 @@
+from .util import yaml
+from .util import talk
+from .util import tex
+from .util import misc
+from .util import files
+from .util import html
+from . import log
+from .config import context
+from .config import interface
+from .access import io

lynguine-0.1.0/lynguine/access/download.py

@@ -0,0 +1,311 @@
+import os
+
+from urllib.request import urlopen
+from urllib.error import HTTPError, URLError
+import git
+
+from linguine.util.misc import prompt_stdin
+from ..log import Logger
+from ..config.context import Context
+
+from ..config.interface import Interface
+
+ctxt = Context()
+log = Logger(
+    name=__name__,
+    level=ctxt._data["logging"]["level"],
+    filename=ctxt._data["logging"]["filename"],
+)
+
+
+class FileDownloader:
+    """
+    A class for downloading data files from a url.
+    """
+
+    def __init__(self, interface, data_resources, data_name):
+        """
+        Initialize the FileDownloader class.
+        :param data_resources: The data resources dictionary.
+        :param data_name: The name of the data to download.
+        """
+        self.interface = interface
+        self.data_resources = data_resources
+        self.data_name = data_name
+        if self.data_name not in self.data_resources:
+            raise ValueError(f'Data "{self.data_name}" not found.')
+        self._dr = self.data_resources[self.data_name]
+
+    @property
+    def interface(self):
+        """
+        Return the interface object.
+        :return: The interface object.
+        """
+        return self._interface
+
+    @interface.setter
+    def interface(self, value):
+        """
+        Set the interface object.
+        :param value: The interface object.
+        :return: None
+        """
+        if not isinstance(value, Interface):
+            raise TypeError("interface must be of type Interface.")
+        self._interface = value
+
+    @property
+    def data_name(self):
+        """
+        Return the name of the data to download.
+        :return: The name of the data to download.
+        """
+        return self._data_name
+
+    @data_name.setter
+    def data_name(self, value):
+        """
+        Set the name of the data to download.
+        :param value: The name of the data to download.
+        :return: None
+        """
+        self._data_name = value
+
+    @property
+    def data_resources(self):
+        """
+        Return the data resources dictionary.
+        :return: The data resources dictionary.
+        """
+        return self._data_resources
+
+    @data_resources.setter
+    def data_resources(self, value):
+        """
+        Set the data resources dictionary.
+        :param value: The data resources dictionary.
+        :return: None
+        """
+        self._data_resources = value
+
+    def _authorize_download(self, prompt=None):
+        """
+        Check with the user that they agree to terms and conditions for the data.
+        :param data_name: The name of the data to download.
+        :param prompt: A function that takes a string and returns a boolean.
+        :return: True if the user agrees to the terms and conditions.
+        :raises: ValueError if the data is not found.
+        """
+
+        if prompt is None:
+            prompt = prompt_stdin
+
+        print(f"Acquiring resource: {self.data_name}\n")
+        print("Details of data: ")
+        print(self._dr.get("details", "No details available."))
+
+        if self._dr.get("citation"):
+            print("\nPlease cite:\n" + self._dr["citation"])
+        if self._dr.get("size"):
+            print(
+                f"\nAfter downloading, the data will take up {self._dr['size']} bytes of space."
+            )
+
+        storage_path = os.path.join(self.interface["default_cache_path"], self.data_name)
+        print(f"\nData will be stored in {storage_path}.\n")
+
+        if self.interface["overide_manual_authorize"]:
+            if self._dr.get("license"):
+                print(
+                    "You have agreed to the following license:\n" + self._dr["license"]
+                )
+            return True
+        else:
+            if self._dr.get("license"):
+                print(
+                    "You must also agree to the following license:\n"
+                    + self._dr["license"]
+                )
+            return prompt("Do you wish to proceed with the download? [yes/no] ")
+
+    def _download_url(
+        self,
+        url,
+        dir_name=".",
+        save_name=None,
+        store_directory=None,
+        messages=True,
+        suffix="",
+    ):
+        """
+        Download a file from a url and save it to disk.
+
+        :param url: The url to download from.
+        :param dir_name: The directory to save the file to.
+        :param save_name: The name to save the file as.
+        :param store_directory: The directory to store the file in.
+        :param messages: Whether to print messages to the console.
+        :param suffix: The suffix to add to the url.
+        :return: None
+        """
+        file_name = os.path.basename(url)
+        if store_directory is not None:
+            dir_name = os.path.join(dir_name, store_directory)
+        if save_name is None:
+            save_name = file_name
+        save_path = os.path.join(dir_name, save_name)
+
+        if not os.path.exists(dir_name):
+            os.makedirs(dir_name)
+
+        try:
+            response = urlopen(url + suffix)
+        except HTTPError as e:
+            raise ValueError(f"HTTP error {e.code} when accessing {url}")
+        except URLError as e:
+            raise ValueError(f"URL error {e.reason} when accessing {url}")
+
+        self._save_file(response, save_path)
+
+    def _save_file(self, response, save_path):
+        """
+        Save the file with progress bar.
+
+        :param response: The response object from urlopen.
+        :param save_path: The path to save the file to.
+        :return: None
+        """
+        try:
+            file_size = int(response.info().get("Content-Length"))
+        except (TypeError, ValueError):
+            file_size = None
+
+        with open(save_path, "wb") as f:
+            if file_size:
+                from tqdm import tqdm
+
+                with tqdm(total=file_size, unit="B", unit_scale=True) as bar:
+                    for buff in iter(lambda: response.read(1024), b""):
+                        f.write(buff)
+                        bar.update(len(buff))
+            else:
+                f.write(response.read())
+
+    def download_data(self, prompt=prompt_stdin):
+        """
+        Check with the user that they are happy with terms and conditions for the data, then download it.
+        :param prompt: A function that takes a string and returns a boolean.
+        :return: None
+        :raises: ValueError if the data is not found.
+        """
+        if not self._authorize_download(prompt):
+            raise Exception("Permission to download data denied.")
+
+        self._process_data()
+
+    def _process_data(self):
+        """
+        Process the data for downloading based on its configuration.
+
+        :return: None
+        """
+        if "suffices" in self._dr:
+            self._download_suffices()
+        elif "dirs" in self._dr:
+            self._download_dirs()
+        else:
+            self._download_simple()
+
+    def _download_suffices(self):
+        """
+        Download a data with a suffices.
+
+        :return: None
+        """
+        for url, filenames, suffices in zip(
+            self._dr["urls"], self._dr["files"], self._dr["suffices"]
+        ):
+            for filename, suffix in zip(filenames, suffices):
+                self._download_file(url, filename, suffix=suffix)
+
+    def _download_dirs(self):
+        """
+        Download a data with directories.
+
+        :return: None
+        """
+        for url, dirnames, filenames in zip(
+            self._dr["urls"], self._dr["dirs"], self._dr["files"]
+        ):
+            for filename, dirname in zip(filenames, dirnames):
+                self._download_file(url, filename, dirname=dirname)
+
+    def _download_simple(self):
+        """
+        Download the data.
+
+        :return: None
+        """
+        for url, filenames in zip(self._dr["urls"], self._dr["files"]):
+            for filename in filenames:
+                self._download_file(url, filename)
+
+    def _download_file(self, url, filename, dirname=None, suffix=""):
+        """
+        Download a file from a url and save it to disk.
+
+        :param url: The url to download from.
+        :param filename: The name to save the file as.
+        :param dirname: The directory to store the file in.
+        :param suffix: The suffix to add to the url.
+        :return: None
+        """
+
+        full_url = os.path.join(url, dirname if dirname else "", filename).replace(
+            " ", "%20"
+        )
+        save_name = filename if not suffix else filename + suffix
+        self._download_url(
+            url=full_url,
+            store_directory=self.interface["default_cache_path"],
+            save_name=save_name,
+        )
+
+
+class GitDownloader(FileDownloader):
+    def __init__(self, interface, data_resources, data_name, git_url):
+        super().__init__(interface, data_resources, data_name)
+        self._git_url = git_url
+        self._repo_path = self.interface["default_cache_path"]
+
+    def _process_data(self):
+        """
+        Process the data for downloading based on its configuration.
+
+        :return: None
+        """
+        self._clone_or_pull_repo()
+        
+    def _clone_or_pull_repo(self):
+        """
+        Clone the repository or pull updates if it already exists.
+
+        :return: None
+        """
+        # Ensure the cache path exists
+        if not os.path.exists(self._repo_path):
+            os.makedirs(self._repo_path)
+
+        # Check if the repo already exists
+        if os.path.isdir(os.path.join(self._repo_path, '.git')):
+            try:
+                repo = git.Repo(self._repo_path)
+                repo.git.pull()
+            except Exception as e:
+                raise ValueError(f"Error pulling repository: {e}")
+        else:
+            try:
+                git.Repo.clone_from(self._git_url, self._repo_path)
+            except Exception as e:
+                raise ValueError(f"Error cloning repository: {e}")