crunch-convert 0.1.0__tar.gz

crunch_convert-0.1.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.1
+Name: crunch-convert
+Version: 0.1.0
+Summary: crunch-convert - Conversion module for the CrunchDAO Platform
+Home-page: https://github.com/crunchdao/crunch-convert
+Author: Enzo CACERES
+Author-email: enzo.caceres@crunchdao.com
+Keywords: package development template
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.7
+Requires-Python: >=3
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: libcst
+Requires-Dist: requirements-parser>=0.11.0
+Provides-Extra: test
+Requires-Dist: parameterized; extra == "test"
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+
+# Crunch Convert Tool
+
+[![PyTest](https://github.com/crunchdao/crunch-convert/actions/workflows/pytest.yml/badge.svg)](https://github.com/crunchdao/crunch-convert/actions/workflows/pytest.yml)
+
+This Python library is designed for the [CrunchDAO Platform](https://hub.crunchdao.com/), exposing the conversion tools in a very small CLI.
+
+- [Crunch Convert Tool](#crunch-convert-tool)
+- [Features](#features)
+  - [Automatic line commenting](#automatic-line-commenting)
+  - [Specifying package versions](#specifying-package-versions)
+  - [R imports via rpy2](#r-imports-via-rpy2)
+  - [Embedded Files](#embedded-files)
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Via the CLI](#via-the-cli)
+  - [Via the Code](#via-the-code)
+- [Contributing](#contributing)
+- [License](#license)
+
+# Features
+
+## Automatic line commenting
+
+Only includes the functions, imports, and classes will be kept.
+
+Everything else is commented out to prevent side effects when your code is loaded into the cloud environment. (e.g. when you're exploring the data, debugging your algorithm, or doing visualizating using Matplotlib, etc.)
+
+You can prevent this behavior by using special comments to tell the system to keep part of your code:
+
+- To start a section that you want to keep, write: `@crunch/keep:on`
+- To end the section, write: `@crunch/keep:off`
+
+```python
+# @crunch/keep:on
+
+# keep global initialization
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# keep constants
+TRAIN_DEPTH = 42
+IMPORTANT_FEATURES = [ "a", "b", "c" ]
+
+# @crunch/keep:off
+
+# this will be ignored
+x, y = crunch.load_data()
+
+def train(...):
+    ...
+```
+
+The result will be:
+
+```python
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+TRAIN_DEPTH = 42
+IMPORTANT_FEATURES = [ "a", "b", "c" ]
+
+#x, y = crunch.load_data()
+
+def train(...):
+    ...
+```
+
+> [!TIP]
+> You can put a `@crunch/keep:on` at the top of the cell and never close it to keep everything.
+
+## Specifying package versions
+
+Since submitting a notebook does not include a `requirements.txt`, users can instead specify the version of a package using import-level [requirement specifiers](https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples) in a comment on the same line.
+
+```python
+# Valid statements
+import pandas # == 1.3
+import sklearn # >= 1.2, < 2.0
+import tqdm # [foo, bar]
+import scikit # ~= 1.4.2
+from requests import Session # == 1.5
+```
+
+Specifying multiple times will cause the submission to be rejected if they are different.
+
+```python
+# Inconsistant versions will be rejected
+import pandas # == 1.3
+import pandas # == 1.5
+```
+
+Specifying versions on standard libraries does nothing (but they will still be rejected if there is an inconsistent version).
+
+```python
+# Will be ignored
+import os # == 1.3
+import sys # == 1.5
+```
+
+If an optional dependency is required for the code to work properly, an import statement must be added, even if the code does not use it directly.
+
+```python
+import castle.algorithms
+
+# Keep me, I am needed by castle
+import torch
+```
+
+It is possible for multiple import names to resolve to different libraries on PyPI. If this happens, you must specify which one you want. If you do not want a specific version, you can use `@latest`, as without this, we cannot distinguish between commented code and version specifiers.
+
+```python
+# Prefer https://pypi.org/project/EMD-signal/
+import pyemd # EMD-signal @latest
+
+# Prefer https://pypi.org/project/pyemd/
+import pyemd # pyemd @latest
+```
+
+## R imports via rpy2
+
+For notebook users, the packages are automatically extracted from the `importr("<name>")` calls, which is provided by [rpy2](https://rpy2.github.io/).
+
+```python
+# Import the `importr` function
+from rpy2.robjects.packages import importr
+
+# Import the "base" R package
+base = importr("base")
+```
+
+The following format must be followed:
+- The import must be declared at the root level.
+- The result must be assigned to a variable; the variable's name will not matter.
+- The function name must be `importr`, and it must be imported as shown in the example above.
+- The first argument must be a string constant, variables or other will be ignored.
+- The other arguments are ignored; this allows for [custom import mapping](https://rpy2.github.io/doc/latest/html/robjects_rpackages.html#importing-r-packages) if necessary.
+
+The line will not be commented, [read more about line commenting here](#automatic-line-commenting).
+
+## Embedded Files
+
+Additional files can be embedded in cells to be submitted with the Notebook. In order for the system to recognize a cell as an Embed File, the following syntax must be followed:
+
+```
+---
+file: <file_name>.md
+---
+
+<!-- File content goes here -->
+Lorem ipsum dolor sit amet, consectetur adipiscing elit.
+Aenean rutrum condimentum ornare.
+```
+
+Submitting multiple cells with the same file name will be rejected.
+
+While the focus is on Markdown files, any text file will be accepted. Including but not limited to: `.txt`, `.yaml`, `.json`, ...
+
+# Installation
+
+Use [pip](https://pypi.org/project/crunch-convert/) to install the `crunch-convert`.
+
+```bash
+pip install --upgrade crunch-convert
+```
+
+# Usage
+
+## Via the CLI
+
+```bash
+crunch-convert notebook my-notebook.ipynb
+```
+
+## Via the Code
+
+```python
+from crunch_convert.notebook import extract_from_file
+
+flatten = extract_from_file("notebook.ipynb")
+
+with open("main.py", "w") as fd:
+  fd.write(flatten.source_code)
+
+with open("requirements.txt", "w") as fd:
+  for requirement in flatten.requirements:
+    fd.write(str(requirement) + "\n")
+
+for embedded_file in flatten.embedded_files:
+  with open(embedded_file.normalized_path, "w") as fd:
+    fd.write(embedded_file.content)
+```
+
+# Contributing
+
+Pull requests are always welcome! If you find any issues or have suggestions for improvements, please feel free to submit a pull request or open an issue in the GitHub repository.
+
+# License
+
+[MIT](https://choosealicense.com/licenses/mit/)

crunch_convert-0.1.0/README.md

@@ -0,0 +1,197 @@
+# Crunch Convert Tool
+
+[![PyTest](https://github.com/crunchdao/crunch-convert/actions/workflows/pytest.yml/badge.svg)](https://github.com/crunchdao/crunch-convert/actions/workflows/pytest.yml)
+
+This Python library is designed for the [CrunchDAO Platform](https://hub.crunchdao.com/), exposing the conversion tools in a very small CLI.
+
+- [Crunch Convert Tool](#crunch-convert-tool)
+- [Features](#features)
+  - [Automatic line commenting](#automatic-line-commenting)
+  - [Specifying package versions](#specifying-package-versions)
+  - [R imports via rpy2](#r-imports-via-rpy2)
+  - [Embedded Files](#embedded-files)
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Via the CLI](#via-the-cli)
+  - [Via the Code](#via-the-code)
+- [Contributing](#contributing)
+- [License](#license)
+
+# Features
+
+## Automatic line commenting
+
+Only includes the functions, imports, and classes will be kept.
+
+Everything else is commented out to prevent side effects when your code is loaded into the cloud environment. (e.g. when you're exploring the data, debugging your algorithm, or doing visualizating using Matplotlib, etc.)
+
+You can prevent this behavior by using special comments to tell the system to keep part of your code:
+
+- To start a section that you want to keep, write: `@crunch/keep:on`
+- To end the section, write: `@crunch/keep:off`
+
+```python
+# @crunch/keep:on
+
+# keep global initialization
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# keep constants
+TRAIN_DEPTH = 42
+IMPORTANT_FEATURES = [ "a", "b", "c" ]
+
+# @crunch/keep:off
+
+# this will be ignored
+x, y = crunch.load_data()
+
+def train(...):
+    ...
+```
+
+The result will be:
+
+```python
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+TRAIN_DEPTH = 42
+IMPORTANT_FEATURES = [ "a", "b", "c" ]
+
+#x, y = crunch.load_data()
+
+def train(...):
+    ...
+```
+
+> [!TIP]
+> You can put a `@crunch/keep:on` at the top of the cell and never close it to keep everything.
+
+## Specifying package versions
+
+Since submitting a notebook does not include a `requirements.txt`, users can instead specify the version of a package using import-level [requirement specifiers](https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples) in a comment on the same line.
+
+```python
+# Valid statements
+import pandas # == 1.3
+import sklearn # >= 1.2, < 2.0
+import tqdm # [foo, bar]
+import scikit # ~= 1.4.2
+from requests import Session # == 1.5
+```
+
+Specifying multiple times will cause the submission to be rejected if they are different.
+
+```python
+# Inconsistant versions will be rejected
+import pandas # == 1.3
+import pandas # == 1.5
+```
+
+Specifying versions on standard libraries does nothing (but they will still be rejected if there is an inconsistent version).
+
+```python
+# Will be ignored
+import os # == 1.3
+import sys # == 1.5
+```
+
+If an optional dependency is required for the code to work properly, an import statement must be added, even if the code does not use it directly.
+
+```python
+import castle.algorithms
+
+# Keep me, I am needed by castle
+import torch
+```
+
+It is possible for multiple import names to resolve to different libraries on PyPI. If this happens, you must specify which one you want. If you do not want a specific version, you can use `@latest`, as without this, we cannot distinguish between commented code and version specifiers.
+
+```python
+# Prefer https://pypi.org/project/EMD-signal/
+import pyemd # EMD-signal @latest
+
+# Prefer https://pypi.org/project/pyemd/
+import pyemd # pyemd @latest
+```
+
+## R imports via rpy2
+
+For notebook users, the packages are automatically extracted from the `importr("<name>")` calls, which is provided by [rpy2](https://rpy2.github.io/).
+
+```python
+# Import the `importr` function
+from rpy2.robjects.packages import importr
+
+# Import the "base" R package
+base = importr("base")
+```
+
+The following format must be followed:
+- The import must be declared at the root level.
+- The result must be assigned to a variable; the variable's name will not matter.
+- The function name must be `importr`, and it must be imported as shown in the example above.
+- The first argument must be a string constant, variables or other will be ignored.
+- The other arguments are ignored; this allows for [custom import mapping](https://rpy2.github.io/doc/latest/html/robjects_rpackages.html#importing-r-packages) if necessary.
+
+The line will not be commented, [read more about line commenting here](#automatic-line-commenting).
+
+## Embedded Files
+
+Additional files can be embedded in cells to be submitted with the Notebook. In order for the system to recognize a cell as an Embed File, the following syntax must be followed:
+
+```
+---
+file: <file_name>.md
+---
+
+<!-- File content goes here -->
+Lorem ipsum dolor sit amet, consectetur adipiscing elit.
+Aenean rutrum condimentum ornare.
+```
+
+Submitting multiple cells with the same file name will be rejected.
+
+While the focus is on Markdown files, any text file will be accepted. Including but not limited to: `.txt`, `.yaml`, `.json`, ...
+
+# Installation
+
+Use [pip](https://pypi.org/project/crunch-convert/) to install the `crunch-convert`.
+
+```bash
+pip install --upgrade crunch-convert
+```
+
+# Usage
+
+## Via the CLI
+
+```bash
+crunch-convert notebook my-notebook.ipynb
+```
+
+## Via the Code
+
+```python
+from crunch_convert.notebook import extract_from_file
+
+flatten = extract_from_file("notebook.ipynb")
+
+with open("main.py", "w") as fd:
+  fd.write(flatten.source_code)
+
+with open("requirements.txt", "w") as fd:
+  for requirement in flatten.requirements:
+    fd.write(str(requirement) + "\n")
+
+for embedded_file in flatten.embedded_files:
+  with open(embedded_file.normalized_path, "w") as fd:
+    fd.write(embedded_file.content)
+```
+
+# Contributing
+
+Pull requests are always welcome! If you find any issues or have suggestions for improvements, please feel free to submit a pull request or open an issue in the GitHub repository.
+
+# License
+
+[MIT](https://choosealicense.com/licenses/mit/)

crunch_convert-0.1.0/crunch_convert/__init__.py

@@ -0,0 +1,7 @@
+"""
+crunch-convert
+~~~~~~
+The crunch module to convert a notebook into multiple components!
+"""
+
+from crunch_convert import notebook as notebook

crunch_convert-0.1.0/crunch_convert/__main__.py

@@ -0,0 +1,3 @@
+from crunch_convert.cli import cli
+
+cli()

crunch_convert-0.1.0/crunch_convert/__version__.py

@@ -0,0 +1,6 @@
+__title__ = 'crunch-convert'
+__description__ = 'crunch-convert - Conversion module for the CrunchDAO Platform'
+__version__ = '0.1.0'
+__author__ = 'Enzo CACERES'
+__author_email__ = 'enzo.caceres@crunchdao.com'
+__url__ = 'https://github.com/crunchdao/crunch-convert'

crunch_convert-0.1.0/crunch_convert/cli.py

@@ -0,0 +1,69 @@
+import json
+import os
+
+import click
+
+from crunch_convert.__version__ import __version__
+from crunch_convert.notebook import (ConverterError,
+                                     InconsistantLibraryVersionError,
+                                     NotebookCellParseError, extract_from_file)
+from crunch_convert.notebook._utils import print_indented
+
+
+@click.group()
+@click.version_option(__version__, package_name="__version__.__title__")
+def cli():
+    pass
+
+
+@cli.command(help="Convert a notebook to a python script.")
+@click.option("--override", is_flag=True, help="Force overwrite of the python file.")
+@click.argument("notebook-file-path", required=True)
+@click.argument("python-file-path", default="main.py")
+def notebook(
+    override: bool,
+    notebook_file_path: str,
+    python_file_path: str,
+):
+    try:
+        flatten = extract_from_file(
+            notebook_file_path,
+            print=print,
+            validate=True,
+        )
+    except IOError as error:
+        print(f"{notebook_file_path}: cannot read notebook file: {error}")
+        raise click.Abort()
+    except json.JSONDecodeError as error:
+        print(f"{notebook_file_path}: cannot parse notebook file: {error}")
+        raise click.Abort()
+    except ConverterError as error:
+        print(f"{notebook_file_path}: convert failed: {error}")
+
+        if isinstance(error, NotebookCellParseError):
+            print(f"  cell: {error.cell_id} ({error.cell_index})")
+            print(f"  source:")
+            print_indented(error.cell_source)
+            print(f"  parser error:")
+            print_indented(error.parser_error or "None")
+
+        elif isinstance(error, InconsistantLibraryVersionError):
+            print(f"  package name: {error.package_name}")
+            print(f"  first version: {error.old}")
+            print(f"  other version: {error.new}")
+
+        raise click.Abort()
+
+    if not override and os.path.exists(python_file_path):
+        override = click.prompt(
+            f"file {python_file_path} already exists, override?",
+            type=bool,
+            default=False,
+            prompt_suffix=" "
+        )
+
+        if not override:
+            raise click.Abort()
+
+    with open(python_file_path, "w") as fd:
+        fd.write(flatten.source_code)

crunch_convert-0.1.0/crunch_convert/notebook/__init__.py

@@ -0,0 +1,27 @@
+"""
+crunch-convert
+~~~~~~
+The crunch module to convert a notebook into multiple components!
+"""
+
+from crunch_convert.notebook._embedded import EmbeddedFile as EmbeddedFile
+from crunch_convert.notebook._notebook import ConverterError as ConverterError
+from crunch_convert.notebook._notebook import \
+    FlattenNotebook as FlattenNotebook
+from crunch_convert.notebook._notebook import \
+    InconsistantLibraryVersionError as InconsistantLibraryVersionError
+from crunch_convert.notebook._notebook import \
+    NotebookCellParseError as NotebookCellParseError
+from crunch_convert.notebook._notebook import \
+    RequirementVersionParseError as RequirementVersionParseError
+from crunch_convert.notebook._notebook import \
+    extract_from_cells as extract_from_cells
+from crunch_convert.notebook._notebook import \
+    extract_from_file as extract_from_file
+from crunch_convert.notebook._requirement import \
+    ImportedRequirement as ImportedRequirement
+from crunch_convert.notebook._requirement import \
+    ImportedRequirementLanguage as ImportedRequirementLanguage
+
+# alias for compatibility with previous versions
+EmbedFile = EmbeddedFile

crunch_convert-0.1.0/crunch_convert/notebook/_embedded.py

@@ -0,0 +1,8 @@
+from dataclasses import dataclass
+
+
+@dataclass()
+class EmbeddedFile:
+    path: str
+    normalized_path: str
+    content: str