mlcroissant 0.0.2__tar.gz

mlcroissant-0.0.2/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.1
+Name: mlcroissant
+Version: 0.0.2
+Summary: MLCommons datasets format.
+Author: Joaquin Vanschoren, Jos van der Velde, Omar Benjelloun, Peter Mattson, Pieter Gijsbers, Pierre Marcenac, Pierre Ruyssen, Prabhant Singh
+Description-Content-Type: text/markdown
+Requires-Dist: absl-py
+Requires-Dist: etils[epath]
+Requires-Dist: jsonpath-rw
+Requires-Dist: networkx
+Requires-Dist: pandas
+Requires-Dist: rdflib
+Requires-Dist: requests
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: datasets; extra == "dev"
+Requires-Dist: flake8-docstrings; extra == "dev"
+Requires-Dist: mlcroissant[git]; extra == "dev"
+Requires-Dist: mlcroissant[image]; extra == "dev"
+Requires-Dist: mlcroissant[parquet]; extra == "dev"
+Requires-Dist: pyflakes; extra == "dev"
+Requires-Dist: pylint; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytype; extra == "dev"
+Provides-Extra: git
+Requires-Dist: GitPython; extra == "git"
+Provides-Extra: image
+Requires-Dist: Pillow; extra == "image"
+Provides-Extra: parquet
+Requires-Dist: pyarrow; extra == "parquet"
+
+# mlcroissant 🥐
+
+Discover `mlcroissant 🥐` with this
+[introduction tutorial in Google Colab](https://colab.sandbox.google.com/github/mlcommons/croissant/blob/main/python/mlcroissant/recipes/introduction.ipynb).
+
+## Python requirements
+
+Python version >= 3.10.
+
+If you do not have a Python environment:
+
+```bash
+python3 -m venv ~/py3
+source ~/py3/bin/activate
+```
+
+## Install
+
+```bash
+python -m pip install ".[dev]"
+```
+
+## Verify/load a Croissant dataset
+
+```bash
+python scripts/validate.py --file ../../datasets/titanic/metadata.json
+```
+
+The command:
+
+- Exits with 0, prints `Done` and displays encountered warnings, when no error was found in the file.
+- Exits with 1 and displays all encountered errors/warnings, otherwise.
+
+Similarly, you can generate a dataset by launching:
+
+```bash
+python scripts/load.py \
+    --file ../../datasets/titanic/metadata.json \
+    --record_set passengers \
+    --num_records 10
+```
+
+## Programmatically build JSON-LD files
+
+You can programmatically build Croissant JSON-LD files using the Python API.
+
+```python
+import mlcroissant as mlc
+metadata=mlc.nodes.Metadata(
+  name="...",
+)
+metadata.to_json()  # this returns the JSON-LD file.
+```
+
+For a full working example, refer to
+[the script to convert Hugging Face datasets to Croissant files](./scripts/from_huggingface_to_croissant.py).
+This script uses the Python API to programmatically build JSON-LD files.
+
+## Run tests
+
+All tests can be run from the Makefile:
+
+```bash
+make tests
+```
+
+## Design
+
+The most important modules in the library are:
+
+- [`mlcroissant/_src/structure_graph`](./mlcroissant/_src/structure_graph/graph.py) is responsible for the **static analysis** of the Croissant files. We convert Croissant files to a Python representation called "**structure graph**" (using [NetworkX](https://networkx.org/)). In the process, we catch any static analysis issues (e.g., a missing mandatory field or a logic problem in the file).
+- [`mlcroissant/_src/operation_graph`](./mlcroissant/_src/operation_graph/graph.py) is responsible for the **dynamic analysis** of the Croissant files (i.e., actually loading the dataset by yielding examples). We convert the structure graph into an "**operation graph**". Operations are the unit transformations that allow to build the dataset (like [`Download`](./mlcroissant/_src/operation_graph/operations/download.py), [`Extract`](./mlcroissant/_src/operation_graph/operations/extract.py), etc).
+
+Other important modules are:
+
+- [`mlcroissant/_src/core`](./mlcroissant/_src/core) defines all needed core internals. For instance, [`Issues`](./mlcroissant/_src/core/issues.py) are a way to track errors and warning during the analysis of Croissant files.
+- [`mlcroissant/__init__.py`](./mlcroissant/__init__.py) declares the public API with [`mlcroissant.Dataset`](./mlcroissant/_src/datasets.py).
+
+For the full design, refer to the [design doc](https://docs.google.com/document/d/1zYQIUX9ae1sZOOBq9OCsJ8JW8-Ejy3NLSeqaI5LtOEM/edit?resourcekey=0-CK78DfFvF7fnufyZqF3h3Q) for an overview of the implementation.
+
+## Contribute
+
+All contributions are welcome! We even have [good first issues](https://github.com/mlcommons/croissant/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) to start in the project. Refer to the [GitHub project](https://github.com/orgs/mlcommons/projects/26) for more detailed user stories and read above how the repo is [designed](#design).
+
+An easy way to contribute to `mlcroissant` is using Croissant's configured [codespaces](https://docs.github.com/en/codespaces/overview).
+To start a codespace:
+
+- On Croissant's main [repo page](https://github.com/mlcommons/croissant), click on the `<Code>` button and select the `Codespaces` tab. You can start a new codespace by clicking on the `+` sign on the left side of the tab. By default, the codespace will start on Croissant's `main` branch, unless you select otherwise from the branches drop-down menu on the left side.
+- While building the environment, your codespaces will install all `mlcroissant`'s required dependencies - so that you can start coding right away! Of course, you can [further personalize](https://docs.github.com/en/codespaces/customizing-your-codespace/personalizing-github-codespaces-for-your-account) your codespace.
+- To start contributing to Croissant:
+  - Create a new branch from the `Terminal` tab in the bottom panel of your codespace with `git checkout -b feature/my-awesome-new-feature`
+  - You can create new commits, and run most git commands from the `Source Control` tab in the left panel of your codespace. Alternatively, use the `Terminal` in the bottom panel of your codespace.
+  - Iterate on your code until all tests are green (you can run tests with `make pytest` or form the `Tests` tab in the left panel of your codespace).
+  - Open a pull request (PR) with the main branch of https://github.com/mlcommons/croissant, and ask for feedback!
+
+Alternatively, you can contribute to `mlcroissant` using the "classic" GitHub workflow:
+
+
+## Debug
+
+You can debug the validation of the file using the `--debug` flag:
+
+```bash
+python scripts/validate.py --file ../../datasets/titanic/metadata.json --debug
+```
+
+This will:
+1. print extra information, like the generated nodes;
+2. save the generated structure graph to a folder indicated in the logs.
+
+## Publishing wheels
+
+Publishing is done manually.
+We are in the process of setting up an automatic deployment with GitHub Actions.
+
+1. Bump the version in `croissant/python/mlcroissant/pyproject.toml`.
+1. Build locally:
+```bash
+python -m build
+```
+1. Upload to pypi.org.

mlcroissant-0.0.2/README.md

@@ -0,0 +1,121 @@
+# mlcroissant 🥐
+
+Discover `mlcroissant 🥐` with this
+[introduction tutorial in Google Colab](https://colab.sandbox.google.com/github/mlcommons/croissant/blob/main/python/mlcroissant/recipes/introduction.ipynb).
+
+## Python requirements
+
+Python version >= 3.10.
+
+If you do not have a Python environment:
+
+```bash
+python3 -m venv ~/py3
+source ~/py3/bin/activate
+```
+
+## Install
+
+```bash
+python -m pip install ".[dev]"
+```
+
+## Verify/load a Croissant dataset
+
+```bash
+python scripts/validate.py --file ../../datasets/titanic/metadata.json
+```
+
+The command:
+
+- Exits with 0, prints `Done` and displays encountered warnings, when no error was found in the file.
+- Exits with 1 and displays all encountered errors/warnings, otherwise.
+
+Similarly, you can generate a dataset by launching:
+
+```bash
+python scripts/load.py \
+    --file ../../datasets/titanic/metadata.json \
+    --record_set passengers \
+    --num_records 10
+```
+
+## Programmatically build JSON-LD files
+
+You can programmatically build Croissant JSON-LD files using the Python API.
+
+```python
+import mlcroissant as mlc
+metadata=mlc.nodes.Metadata(
+  name="...",
+)
+metadata.to_json()  # this returns the JSON-LD file.
+```
+
+For a full working example, refer to
+[the script to convert Hugging Face datasets to Croissant files](./scripts/from_huggingface_to_croissant.py).
+This script uses the Python API to programmatically build JSON-LD files.
+
+## Run tests
+
+All tests can be run from the Makefile:
+
+```bash
+make tests
+```
+
+## Design
+
+The most important modules in the library are:
+
+- [`mlcroissant/_src/structure_graph`](./mlcroissant/_src/structure_graph/graph.py) is responsible for the **static analysis** of the Croissant files. We convert Croissant files to a Python representation called "**structure graph**" (using [NetworkX](https://networkx.org/)). In the process, we catch any static analysis issues (e.g., a missing mandatory field or a logic problem in the file).
+- [`mlcroissant/_src/operation_graph`](./mlcroissant/_src/operation_graph/graph.py) is responsible for the **dynamic analysis** of the Croissant files (i.e., actually loading the dataset by yielding examples). We convert the structure graph into an "**operation graph**". Operations are the unit transformations that allow to build the dataset (like [`Download`](./mlcroissant/_src/operation_graph/operations/download.py), [`Extract`](./mlcroissant/_src/operation_graph/operations/extract.py), etc).
+
+Other important modules are:
+
+- [`mlcroissant/_src/core`](./mlcroissant/_src/core) defines all needed core internals. For instance, [`Issues`](./mlcroissant/_src/core/issues.py) are a way to track errors and warning during the analysis of Croissant files.
+- [`mlcroissant/__init__.py`](./mlcroissant/__init__.py) declares the public API with [`mlcroissant.Dataset`](./mlcroissant/_src/datasets.py).
+
+For the full design, refer to the [design doc](https://docs.google.com/document/d/1zYQIUX9ae1sZOOBq9OCsJ8JW8-Ejy3NLSeqaI5LtOEM/edit?resourcekey=0-CK78DfFvF7fnufyZqF3h3Q) for an overview of the implementation.
+
+## Contribute
+
+All contributions are welcome! We even have [good first issues](https://github.com/mlcommons/croissant/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) to start in the project. Refer to the [GitHub project](https://github.com/orgs/mlcommons/projects/26) for more detailed user stories and read above how the repo is [designed](#design).
+
+An easy way to contribute to `mlcroissant` is using Croissant's configured [codespaces](https://docs.github.com/en/codespaces/overview).
+To start a codespace:
+
+- On Croissant's main [repo page](https://github.com/mlcommons/croissant), click on the `<Code>` button and select the `Codespaces` tab. You can start a new codespace by clicking on the `+` sign on the left side of the tab. By default, the codespace will start on Croissant's `main` branch, unless you select otherwise from the branches drop-down menu on the left side.
+- While building the environment, your codespaces will install all `mlcroissant`'s required dependencies - so that you can start coding right away! Of course, you can [further personalize](https://docs.github.com/en/codespaces/customizing-your-codespace/personalizing-github-codespaces-for-your-account) your codespace.
+- To start contributing to Croissant:
+  - Create a new branch from the `Terminal` tab in the bottom panel of your codespace with `git checkout -b feature/my-awesome-new-feature`
+  - You can create new commits, and run most git commands from the `Source Control` tab in the left panel of your codespace. Alternatively, use the `Terminal` in the bottom panel of your codespace.
+  - Iterate on your code until all tests are green (you can run tests with `make pytest` or form the `Tests` tab in the left panel of your codespace).
+  - Open a pull request (PR) with the main branch of https://github.com/mlcommons/croissant, and ask for feedback!
+
+Alternatively, you can contribute to `mlcroissant` using the "classic" GitHub workflow:
+
+
+## Debug
+
+You can debug the validation of the file using the `--debug` flag:
+
+```bash
+python scripts/validate.py --file ../../datasets/titanic/metadata.json --debug
+```
+
+This will:
+1. print extra information, like the generated nodes;
+2. save the generated structure graph to a folder indicated in the logs.
+
+## Publishing wheels
+
+Publishing is done manually.
+We are in the process of setting up an automatic deployment with GitHub Actions.
+
+1. Bump the version in `croissant/python/mlcroissant/pyproject.toml`.
+1. Build locally:
+```bash
+python -m build
+```
+1. Upload to pypi.org.

mlcroissant-0.0.2/mlcroissant/__init__.py

@@ -0,0 +1,16 @@
+"""Defines the public interface to the `mlcroissant` package."""
+from mlcroissant._src import nodes
+from mlcroissant._src.core import constants
+from mlcroissant._src.core.issues import ValidationError
+from mlcroissant._src.datasets import Dataset
+from mlcroissant._src.datasets import Records
+from mlcroissant._src.structure_graph.nodes.field import Field
+
+__all__ = [
+    "constants",
+    "Dataset",
+    "Field",
+    "nodes",
+    "Records",
+    "ValidationError",
+]

mlcroissant-0.0.2/mlcroissant/_src/core/constants.py

@@ -0,0 +1,112 @@
+"""constants module."""
+
+from etils import epath
+import rdflib
+from rdflib import namespace
+from rdflib import term
+
+# MLCommons-defined URIs (still draft).
+ML_COMMONS = rdflib.Namespace("http://mlcommons.org/schema/")
+ML_COMMONS_COLUMN = ML_COMMONS.column
+ML_COMMONS_DATA = ML_COMMONS.data
+ML_COMMONS_DATA_TYPE = ML_COMMONS.dataType
+ML_COMMONS_DATA_TYPE_BOUNDING_BOX = ML_COMMONS.BoundingBox
+ML_COMMONS_EXTRACT = ML_COMMONS.extract
+ML_COMMONS_FILE_PROPERTY = ML_COMMONS.fileProperty
+ML_COMMONS_FIELD = ML_COMMONS.field
+ML_COMMONS_FIELD_TYPE = ML_COMMONS.Field
+# ML_COMMONS.format is understood as the `format` method on the class Namespace.
+ML_COMMONS_FORMAT = term.URIRef("http://mlcommons.org/schema/format")
+ML_COMMONS_INCLUDES = ML_COMMONS.includes
+ML_COMMONS_IS_ENUMERATION = ML_COMMONS.isEnumeration
+ML_COMMONS_JSON_PATH = ML_COMMONS.jsonPath
+ML_COMMONS_PARENT_FIELD = ML_COMMONS.parentField
+ML_COMMONS_PATH = ML_COMMONS.path
+ML_COMMONS_RECORD_SET = ML_COMMONS.recordSet
+ML_COMMONS_RECORD_SET_TYPE = ML_COMMONS.RecordSet
+ML_COMMONS_REFERENCES = ML_COMMONS.references
+ML_COMMONS_REGEX = ML_COMMONS.regex
+ML_COMMONS_REPEATED = ML_COMMONS.repeated
+# ML_COMMONS.replace is understood as the `replace` method on the class Namespace.
+ML_COMMONS_REPLACE = term.URIRef("http://mlcommons.org/schema/replace")
+ML_COMMONS_SEPARATOR = ML_COMMONS.separator
+ML_COMMONS_SOURCE = ML_COMMONS.source
+ML_COMMONS_SUB_FIELD = ML_COMMONS.subField
+ML_COMMONS_SUB_FIELD_TYPE = ML_COMMONS.SubField
+ML_COMMONS_TRANSFORM = ML_COMMONS.transform
+
+
+# RDF standard URIs.
+# For "@type" key:
+RDF_TYPE = namespace.RDF.type
+
+# Schema.org standard URIs.
+SCHEMA_ORG_CITATION = namespace.SDO.citation
+SCHEMA_ORG_CONTAINED_IN = namespace.SDO.containedIn
+SCHEMA_ORG_CONTENT_SIZE = namespace.SDO.contentSize
+SCHEMA_ORG_CONTENT_URL = namespace.SDO.contentUrl
+SCHEMA_ORG_DATASET = namespace.SDO.Dataset
+SCHEMA_ORG_DATA_TYPE_BOOL = namespace.SDO.Boolean
+SCHEMA_ORG_DATA_TYPE_DATE = namespace.SDO.Date
+SCHEMA_ORG_DATA_TYPE_FLOAT = namespace.SDO.Float
+SCHEMA_ORG_DATA_TYPE_IMAGE_OBJECT = namespace.SDO.ImageObject
+SCHEMA_ORG_DATA_TYPE_INTEGER = namespace.SDO.Integer
+SCHEMA_ORG_DATA_TYPE_TEXT = namespace.SDO.Text
+SCHEMA_ORG_DATA_TYPE_URL = namespace.SDO.URL
+SCHEMA_ORG_DESCRIPTION = namespace.SDO.description
+SCHEMA_ORG_DISTRIBUTION = namespace.SDO.distribution
+SCHEMA_ORG_EMAIL = namespace.SDO.email
+SCHEMA_ORG_ENCODING_FORMAT = namespace.SDO.encodingFormat
+SCHEMA_ORG_LICENSE = namespace.SDO.license
+SCHEMA_ORG_NAME = namespace.SDO.name
+SCHEMA_ORG_SHA256 = namespace.SDO.sha256
+SCHEMA_ORG_URL = namespace.SDO.url
+
+# Schema.org URIs that do not exist yet in the standard.
+SCHEMA_ORG = rdflib.Namespace("https://schema.org/")
+SCHEMA_ORG_KEY = SCHEMA_ORG.key
+SCHEMA_ORG_FILE_OBJECT = SCHEMA_ORG.FileObject
+SCHEMA_ORG_FILE_SET = SCHEMA_ORG.FileSet
+SCHEMA_ORG_MD5 = SCHEMA_ORG.md5
+
+TO_CROISSANT = {
+    ML_COMMONS_TRANSFORM: "transforms",
+    ML_COMMONS_COLUMN: "csv_column",
+    ML_COMMONS_DATA_TYPE: "data_type",
+    ML_COMMONS_DATA: "data",
+    ML_COMMONS_EXTRACT: "extract",
+    ML_COMMONS_FIELD: "field",
+    ML_COMMONS_FILE_PROPERTY: "file_property",
+    ML_COMMONS_FORMAT: "format",
+    ML_COMMONS_INCLUDES: "includes",
+    ML_COMMONS_JSON_PATH: "json_path",
+    ML_COMMONS_REFERENCES: "references",
+    ML_COMMONS_REGEX: "regex",
+    ML_COMMONS_REPLACE: "replace",
+    ML_COMMONS_SEPARATOR: "separator",
+    ML_COMMONS_SOURCE: "source",
+    SCHEMA_ORG_CITATION: "citation",
+    SCHEMA_ORG_CONTAINED_IN: "contained_in",
+    SCHEMA_ORG_CONTENT_SIZE: "content_size",
+    SCHEMA_ORG_CONTENT_URL: "content_url",
+    SCHEMA_ORG_DESCRIPTION: "description",
+    SCHEMA_ORG_DISTRIBUTION: "distribution",
+    SCHEMA_ORG_ENCODING_FORMAT: "encoding_format",
+    SCHEMA_ORG_LICENSE: "license",
+    SCHEMA_ORG_MD5: "md5",
+    SCHEMA_ORG_NAME: "name",
+    SCHEMA_ORG_SHA256: "sha256",
+    SCHEMA_ORG_URL: "url",
+}
+
+FROM_CROISSANT = {v: k for k, v in TO_CROISSANT.items()}
+
+# Environment variables
+CROISSANT_CACHE = epath.Path("~/.cache/croissant").expanduser()
+DOWNLOAD_PATH = CROISSANT_CACHE / "download"
+EXTRACT_PATH = CROISSANT_CACHE / "extract"
+CROISSANT_GIT_USERNAME = "CROISSANT_GIT_USERNAME"
+CROISSANT_GIT_PASSWORD = "CROISSANT_GIT_PASSWORD"
+
+# Encoding formats
+GIT_HTTPS_ENCODING_FORMAT = "git+https"

mlcroissant-0.0.2/mlcroissant/_src/core/constants_test.py

@@ -0,0 +1,14 @@
+"""constants_test module."""
+
+from mlcroissant._src.core.constants import TO_CROISSANT
+
+
+def test_to_croissant_values_are_unique():
+    deja_vu = {}
+    for key, value in TO_CROISSANT.items():
+        if value in deja_vu:
+            raise ValueError(
+                f"Keys {key} and {deja_vu[value]} define the same Croissant value:"
+                f" {value}."
+            )
+        deja_vu[value] = key

mlcroissant-0.0.2/mlcroissant/_src/core/data_types.py

@@ -0,0 +1,34 @@
+"""data_types module."""
+
+import pandas as pd
+
+from mlcroissant._src.core import constants
+from mlcroissant._src.core.issues import Issues
+from mlcroissant._src.core.types import Json
+
+
+def check_expected_type(issues: Issues, jsonld: Json, expected_type: str):
+    """Checks that JSON-LD `jsonld` has "@type" == expected_type."""
+    node_name = jsonld.get(constants.SCHEMA_ORG_NAME, "<unknown node>")
+    node_type = jsonld.get("@type")
+    if node_type != expected_type:
+        issues.add_error(
+            f'"{node_name}" should have an attribute "@type": "{expected_type}". Got'
+            f" {node_type} instead."
+        )
+
+
+EXPECTED_DATA_TYPES: dict[str, type] = {
+    constants.ML_COMMONS_DATA_TYPE_BOUNDING_BOX: (
+        constants.ML_COMMONS_DATA_TYPE_BOUNDING_BOX
+    ),
+    constants.SCHEMA_ORG_DATA_TYPE_BOOL: bool,
+    constants.SCHEMA_ORG_DATA_TYPE_DATE: pd.Timestamp,
+    constants.SCHEMA_ORG_DATA_TYPE_FLOAT: float,
+    constants.SCHEMA_ORG_DATA_TYPE_IMAGE_OBJECT: (
+        constants.SCHEMA_ORG_DATA_TYPE_IMAGE_OBJECT
+    ),
+    constants.SCHEMA_ORG_DATA_TYPE_INTEGER: int,
+    constants.SCHEMA_ORG_DATA_TYPE_TEXT: str,
+    constants.SCHEMA_ORG_DATA_TYPE_URL: str,
+}

mlcroissant-0.0.2/mlcroissant/_src/core/git.py

@@ -0,0 +1,37 @@
+"""git module."""
+
+import os
+
+from absl import logging
+from etils import epath
+
+from mlcroissant._src.core.optional import deps
+from mlcroissant._src.core.path import Path
+
+
+def is_git_lfs_file(filepath: epath.Path) -> bool:
+    """Returns whether a file a non-downloaded git-lfs file by checking its header.
+
+    An optimization of this function would be to only read the file if there is a git
+    repository in the distribution.
+    """
+    with open(filepath, "rb") as file:
+        # Only read the first line of the file. In the future, this could be a problem,
+        # e.g. if we accept *.txt files and the file starts with the same header.
+        first_line = file.readline()
+        if first_line.startswith(b"version https://git-lfs.github.com/spec"):
+            return True
+    return False
+
+
+def download_git_lfs_file(file: Path):
+    """Downloads a specific git-lfs file within its repo."""
+    # Path(filepath="/tmp/full/path.json", fullpath="path.json")
+    # => working_dir = "/tmp/full"
+    fullpath = os.fspath(file.fullpath)
+    working_dir = os.fspath(file.filepath).rsplit(fullpath)[0]
+    repo = deps.git.Git(working_dir)
+    logging.info(
+        "Downloading git-lfs file: %s in working dir: %s", fullpath, working_dir
+    )
+    repo.execute(["git", "lfs", "pull", "--include", fullpath])

mlcroissant-0.0.2/mlcroissant/_src/core/git_test.py

@@ -0,0 +1,54 @@
+"""Tests for git."""
+
+import pathlib
+import tempfile
+from unittest import mock
+
+from etils import epath
+import git
+
+from mlcroissant._src.core.git import download_git_lfs_file
+from mlcroissant._src.core.git import is_git_lfs_file
+from mlcroissant._src.core.path import Path
+
+_GIT_LFS_CONTENT = lambda: """version https://git-lfs.github.com/spec/v1
+oid sha256:5e2785fcd9098567a49d6e62e328923d955b307b6dcd0492f6234e96e670772a
+size 309207547
+"""
+
+_NON_GIT_LFS_CONTENT = lambda: """name,age
+a,1
+b,2
+c,3"""
+
+_NON_ASCII_CONTENT = lambda: (255).to_bytes(1, byteorder="big")
+
+
+def test_is_git_lfs_file():
+    with tempfile.TemporaryDirectory() as tempdir:
+        tempdir = epath.Path(tempdir)
+
+        gitlfs_file = tempdir / "gitlfs"
+        gitlfs_file.write_text(_GIT_LFS_CONTENT())
+        assert is_git_lfs_file(gitlfs_file)
+
+        no_gitlfs_file = tempdir / "no-gitlfs"
+        no_gitlfs_file.write_text(_NON_GIT_LFS_CONTENT())
+        assert not is_git_lfs_file(no_gitlfs_file)
+
+        no_ascii_file = tempdir / "no-ascii"
+        no_ascii_file.write_bytes(_NON_ASCII_CONTENT())
+        assert not is_git_lfs_file(no_ascii_file)
+
+
+def test_download_git_lfs_file():
+    file = Path(
+        filepath=epath.Path("/tmp/full/path.json"),
+        fullpath=pathlib.PurePath("path.json"),
+    )
+    with mock.patch.object(git, "Git", autospec=True) as git_mock:
+        download_git_lfs_file(file)
+        git_mock.assert_called_once_with("/tmp/full/")
+        git_mock.return_value.execute.assert_called_once_with(
+            ["git", "lfs", "pull", "--include", "path.json"]
+        )

mlcroissant-0.0.2/mlcroissant/_src/core/graphs/utils.py

@@ -0,0 +1,46 @@
+"""Utils for graphs."""
+
+import time
+
+import networkx as nx
+
+
+def pretty_print_graph(graph: nx.Graph, simplify=False):
+    """Pretty prints a NetworkX graph.
+
+    Warning: this function is for debugging purposes only.
+
+    Args:
+        graph: Any NetworkX graph.
+        simplify: Whether to print a simplified version of nodes.
+    """
+    if simplify:
+        simple_graph = nx.Graph()
+        for x, y in graph.edges():
+            x = getattr(x, "uid", x)
+            y = getattr(y, "uid", y)
+            simple_graph.add_edge(x, y)
+        graph = simple_graph
+    agraph = nx.nx_agraph.to_agraph(graph)
+    agraph.layout(prog="dot")
+    temporary_file = f"/tmp/graph_{time.time()}.png"
+    agraph.draw(temporary_file, args="-Gnodesep=0.01 -Gfont_size=1", prog="dot")
+    print(f"Generated a graph and saved it in: {temporary_file}")
+
+
+def print_graph_traversal(graph: nx.Graph):
+    """Pretty prints a NetworkX graph.
+
+    Warning: this function is for debugging purposes only.
+
+    Args:
+        graph: Any NetworkX graph.
+    """
+    visited = {}
+    print("--- Graph traversal ---")
+    for start, end, _ in nx.edge_bfs(graph):
+        for node in [start, end]:
+            if node.name not in visited:
+                print(f"Visited: {node.name}")
+                visited[node.name] = True
+    print("Done traversing the graph.")