siliconcompiler 0.31.0__tar.gz → 0.31.1__tar.gz

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/Changes

@@ -8,6 +8,20 @@ The changes in each SiliconCompiler release version are described below. Commit
 version shown in (). Where applicable, the contributors that suggested a given
 feature are shown in [].
 
+SiliconCompiler 0.31.1 (2025-03-06)
+=========================================
+
+**Minor:**
+
+ * Fixed tcl manifest to correctly reflect the data types from the schema.
+ * Added end of install status message to `sc-install`.
+
+
+ * Tools:
+
+  * vpr: updated to v9.0 and added timing metrics extraction.
+
+
 SiliconCompiler 0.31.0 (2025-03-03)
 =========================================
 

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: siliconcompiler
-Version: 0.31.0
+Version: 0.31.1
 Summary: A compiler framework that automates translation from source code to silicon.
 Author-email: Andreas Olofsson <andreas.d.olofsson@gmail.com>
 License: Apache License 2.0
@@ -99,19 +99,19 @@ SiliconCompiler is a modular hardware build system ("make for silicon"). The pro
 
 | Type | Supported|
 |------|----------|
-|**Design Languages**| C, Verilog, SV, VHDL, Chisel, Migen/Amaranth, Bluespec
+|**Design Languages**| C, Verilog, SV, VHDL, Chisel, Migen/Amaranth, Bluespec, [MLIR](https://en.wikipedia.org/wiki/MLIR_(software))
 |**Simulation Tools**| Verilator, Icarus, GHDL, Xyce
 |**Synthesis**| Yosys, Vivado, Synopsys, Cadence
-|**ASIC APR**| OpenRoad, Synopsys, Cadence
+|**ASIC APR**| OpenROAD, Synopsys, Cadence
 |**FPGA APR**| VPR, nextpnr, Vivado
-|**Layout Viewer**| Klayout, OpenRoad, Cadence, Synopsys
-|**DRC/LVS**| Magic, Synopsys, Siemens
-|**PDKs**| sky130, gf180, asap7, freepdk45, gf12lp, gf22fdx, intel16, ihp130
+|**Layout Viewer**| Klayout, OpenROAD, Cadence, Synopsys
+|**DRC/LVS**| Klayout, Magic, Synopsys, Siemens
+|**PDKs**| sky130, ihp130, gf180, asap7, freepdk45, gf12lp, gf22fdx, intel16
 
 # Getting Started
 
 SiliconCompiler is available as wheel packages on PyPI for macOS, Windows and
-Linux platforms. For working Python 3.8-3.12 environment, just use pip.
+Linux platforms. For working Python 3.8-3.13 environment, just use pip.
 
 ```sh
 python3 -m pip install --upgrade siliconcompiler
@@ -166,17 +166,15 @@ A. Olofsson, W. Ransohoff, N. Moroze, "[Invited: A Distributed Approach to Silic
 Bibtex:
 ```
 @inproceedings{10.1145/3489517.3530673,
-author = {Olofsson, Andreas and Ransohoff, William and Moroze, Noah},
-title = {A Distributed Approach to Silicon Compilation: Invited},
-year = {2022},
-booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
-pages = {1343–1346},
-location = {San Francisco, California}
+  author = {Olofsson, Andreas and Ransohoff, William and Moroze, Noah},
+  title = {A Distributed Approach to Silicon Compilation: Invited},
+  year = {2022},
+  booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
+  pages = {1343–1346},
+  location = {San Francisco, California}
 }
 ```
 
-
-
 # Installation
 
 Complete installation instructions are available in the [Installation Guide](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html).
@@ -194,14 +192,14 @@ python3 -m pip install -e .[docs,test]  # Optional install step for generating d
 
 Installation instructions for all external tools can be found in the
 [External Tools](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html#external-tools) section
-of the user guide. We have included shell setup scripts (Ubuntu) for most of the supported tools.
+of the user guide. We have included shell setup scripts (Ubuntu) for most of the supported tools, which can be accessed via [sc-install](https://docs.siliconcompiler.com/en/latest/reference_manual/apps.html#apps-sc-install-ref).
 See the [./siliconcompiler/toolscripts](./siliconcompiler/toolscripts) directory for a complete set of scripts and [./siliconcompiler/toolscripts/_tools.json](./siliconcompiler/toolscripts/_tools.json) for the currently recommended tool versions.
 
 # Contributing
 
 SiliconCompiler is an open-source project and welcomes contributions. To find out
 how to contribute to the project, see our
-[Contributing Guidelines.](./CONTRIBUTING.md)
+[Contributing Guidelines](./CONTRIBUTING.md).
 
 # Issues / Bugs
 

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/README.md

@@ -17,19 +17,19 @@ SiliconCompiler is a modular hardware build system ("make for silicon"). The pro
 
 | Type | Supported|
 |------|----------|
-|**Design Languages**| C, Verilog, SV, VHDL, Chisel, Migen/Amaranth, Bluespec
+|**Design Languages**| C, Verilog, SV, VHDL, Chisel, Migen/Amaranth, Bluespec, [MLIR](https://en.wikipedia.org/wiki/MLIR_(software))
 |**Simulation Tools**| Verilator, Icarus, GHDL, Xyce
 |**Synthesis**| Yosys, Vivado, Synopsys, Cadence
-|**ASIC APR**| OpenRoad, Synopsys, Cadence
+|**ASIC APR**| OpenROAD, Synopsys, Cadence
 |**FPGA APR**| VPR, nextpnr, Vivado
-|**Layout Viewer**| Klayout, OpenRoad, Cadence, Synopsys
-|**DRC/LVS**| Magic, Synopsys, Siemens
-|**PDKs**| sky130, gf180, asap7, freepdk45, gf12lp, gf22fdx, intel16, ihp130
+|**Layout Viewer**| Klayout, OpenROAD, Cadence, Synopsys
+|**DRC/LVS**| Klayout, Magic, Synopsys, Siemens
+|**PDKs**| sky130, ihp130, gf180, asap7, freepdk45, gf12lp, gf22fdx, intel16
 
 # Getting Started
 
 SiliconCompiler is available as wheel packages on PyPI for macOS, Windows and
-Linux platforms. For working Python 3.8-3.12 environment, just use pip.
+Linux platforms. For working Python 3.8-3.13 environment, just use pip.
 
 ```sh
 python3 -m pip install --upgrade siliconcompiler
@@ -84,17 +84,15 @@ A. Olofsson, W. Ransohoff, N. Moroze, "[Invited: A Distributed Approach to Silic
 Bibtex:
 ```
 @inproceedings{10.1145/3489517.3530673,
-author = {Olofsson, Andreas and Ransohoff, William and Moroze, Noah},
-title = {A Distributed Approach to Silicon Compilation: Invited},
-year = {2022},
-booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
-pages = {1343–1346},
-location = {San Francisco, California}
+  author = {Olofsson, Andreas and Ransohoff, William and Moroze, Noah},
+  title = {A Distributed Approach to Silicon Compilation: Invited},
+  year = {2022},
+  booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
+  pages = {1343–1346},
+  location = {San Francisco, California}
 }
 ```
 
-
-
 # Installation
 
 Complete installation instructions are available in the [Installation Guide](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html).
@@ -112,14 +110,14 @@ python3 -m pip install -e .[docs,test]  # Optional install step for generating d
 
 Installation instructions for all external tools can be found in the
 [External Tools](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html#external-tools) section
-of the user guide. We have included shell setup scripts (Ubuntu) for most of the supported tools.
+of the user guide. We have included shell setup scripts (Ubuntu) for most of the supported tools, which can be accessed via [sc-install](https://docs.siliconcompiler.com/en/latest/reference_manual/apps.html#apps-sc-install-ref).
 See the [./siliconcompiler/toolscripts](./siliconcompiler/toolscripts) directory for a complete set of scripts and [./siliconcompiler/toolscripts/_tools.json](./siliconcompiler/toolscripts/_tools.json) for the currently recommended tool versions.
 
 # Contributing
 
 SiliconCompiler is an open-source project and welcomes contributions. To find out
 how to contribute to the project, see our
-[Contributing Guidelines.](./CONTRIBUTING.md)
+[Contributing Guidelines](./CONTRIBUTING.md).
 
 # Issues / Bugs
 

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/pyproject.toml

@@ -90,6 +90,10 @@ scsetup = "siliconcompiler.utils.showtools:setup"
 [project.entry-points."siliconcompiler.target"]
 targets = "siliconcompiler.targets:get_targets"
 
+[project.entry-points."siliconcompiler.path_resolver"]
+https = "siliconcompiler.package.https:get_resolver"
+git = "siliconcompiler.package.git:get_resolver"
+
 [project.entry-points."siliconcompiler.docs"]
 linkcode = "siliconcompiler.sphinx_ext:get_codeurl"
 targets = "siliconcompiler.sphinx_ext:targets"

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/siliconcompiler/_metadata.py

@@ -1,5 +1,5 @@
 # Version number following semver standard.
-version = '0.31.0'
+version = '0.31.1'
 
 # Default server address for remote runs, if unspecified.
 default_server = 'https://server.siliconcompiler.com'

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/siliconcompiler/apps/sc_install.py

@@ -101,6 +101,19 @@ def print_machine_info():
     print("Scripts:  ", _get_tool_script_dir())
 
 
+def __print_summary(successful, failed):
+    max_len = 64
+    print("#"*max_len)
+    if successful:
+        msg = f"Installed: {', '.join(sorted(successful))}"
+        print(f"# {msg}")
+
+    if failed:
+        msg = f"Failed to install: {failed}"
+        print(f"# {msg}")
+    print("#"*max_len)
+
+
 def _get_tool_script_dir():
     return Path(siliconcompiler.__file__).parent / "toolscripts"
 
@@ -242,6 +255,7 @@ To system debugging information (this should only be used to debug):
                 args.tool.extend(tool_groups[group])
 
     tools_handled = set()
+    tools_completed = set()
     for tool in args.tool:
         if tool in tools_handled:
             continue
@@ -250,9 +264,14 @@ To system debugging information (this should only be used to debug):
             show_tool(tool, tools[tool])
         else:
             if not install_tool(tool, tools[tool], args.build_dir, args.prefix):
+                __print_summary(tools_completed, tool)
                 return 1
+            else:
+                tools_completed.add(tool)
 
     if not args.show:
+        __print_summary(tools_completed, None)
+
         msgs = []
         for env, path in (
                 ("PATH", "bin"),
@@ -262,7 +281,6 @@ To system debugging information (this should only be used to debug):
                 os.path.expandvars(os.path.expanduser(p))
                 for p in os.getenv(env, "").split(":")
             ]
-            print(envs)
             if check_path not in envs:
                 msgs.extend([
                     "",

{siliconcompiler-0.31.0 → siliconcompiler-0.31.1}/siliconcompiler/core.py

@@ -2950,7 +2950,7 @@ class Chip:
                 elif os.path.isfile(results_html):
                     _open_html_report(self, results_html)
                 else:
-                    self._dashboard(wait=False)
+                    self.dashboard(wait=False)
 
     ###########################################################################
     def clock(self, pin, period, jitter=0, mode='global'):

siliconcompiler-0.31.0/siliconcompiler/package.py → siliconcompiler-0.31.1/siliconcompiler/package/__init__.py

@@ -1,25 +1,20 @@
 import os
-import requests
-import tarfile
-import zipfile
-from git import Repo, GitCommandError
 from urllib.parse import urlparse
 import importlib
-import shutil
 import re
 from siliconcompiler import SiliconCompilerError
 from siliconcompiler.utils import default_cache_dir, _resolve_env_vars
 import json
 from importlib.metadata import distributions, distribution
 import functools
-import fasteners
 import time
 from pathlib import Path
-from io import BytesIO
 
 from github import Github
 import github.Auth
 
+from siliconcompiler.utils import get_plugins
+
 
 def get_cache_path(chip):
     cache_path = chip.get('option', 'cachedir')
@@ -33,10 +28,45 @@ def get_cache_path(chip):
     return cache_path
 
 
-def _path(chip, package, download_handler):
-    if package in chip._packages:
-        return chip._packages[package]
+def get_download_cache_path(chip, package, ref):
+    cache_path = get_cache_path(chip)
+    if not os.path.exists(cache_path):
+        os.makedirs(cache_path, exist_ok=True)
+
+    if ref is None:
+        raise SiliconCompilerError(f'Reference is required for cached data: {package}', chip=chip)
+
+    return \
+        os.path.join(cache_path, f'{package}-{ref}'), \
+        os.path.join(cache_path, f'{package}-{ref}.lock')
+
+
+def _file_path_resolver(chip, package, path, ref, url):
+    return os.path.abspath(path.replace('file://', ''))
+
+
+def _python_path_resolver(chip, package, path, ref, url):
+    return path_from_python(chip, url.netloc)
 
+
+def _get_path_resolver(path):
+    url = urlparse(path)
+
+    for resolver in get_plugins("path_resolver"):
+        func = resolver(url)
+        if func:
+            return func, url
+
+    if url.scheme == "file":
+        return _file_path_resolver, url
+
+    if url.scheme == "python":
+        return _python_path_resolver, url
+
+    raise ValueError(f"{path} is not supported")
+
+
+def _path(chip, package):
     # Initially try retrieving data source from schema
     data = {}
     data['path'] = chip.get('package', 'source', package, 'path')
@@ -48,48 +78,13 @@ def _path(chip, package, download_handler):
 
     data['path'] = _resolve_env_vars(chip, data['path'], None, None)
 
-    url = urlparse(data['path'])
-
-    # check network drive for package data source
-    if data['path'].startswith('file://') or os.path.exists(data['path']):
-        path = os.path.abspath(data['path'].replace('file://', ''))
-        chip.logger.info(f'Found {package} data at {path}')
-        chip._packages[package] = path
-        return path
-    elif data['path'].startswith('python://'):
-        path = path_from_python(chip, url.netloc)
-        chip.logger.info(f'Found {package} data at {path}')
-        chip._packages[package] = path
-        return path
-
-    # location of the python package
-    cache_path = get_cache_path(chip)
-    if not os.path.exists(cache_path):
-        os.makedirs(cache_path, exist_ok=True)
-    project_id = f'{package}-{data.get("ref")}'
-    if url.scheme not in ['git', 'git+https', 'https', 'git+ssh', 'ssh'] or not project_id:
-        raise SiliconCompilerError(
-            f'Could not find data path in package {package}: {data["path"]}',
-            chip=chip)
-
-    data_path = os.path.join(cache_path, project_id)
-
-    if download_handler:
-        download_handler(chip,
-                         package,
-                         data,
-                         url,
-                         data_path,
-                         os.path.join(cache_path, f'{project_id}.lock'))
-
-    if os.path.exists(data_path):
-        if package not in chip._packages:
-            chip.logger.info(f'Saved {package} data to {data_path}')
-            chip._packages[package] = data_path
-        return data_path
+    if os.path.exists(data['path']):
+        # Path is already a path
+        return os.path.abspath(data['path'])
 
-    raise SiliconCompilerError(f'Extracting {package} data to {data_path} failed',
-                               chip=chip)
+    path_resolver, url = _get_path_resolver(data['path'])
+
+    return path_resolver(chip, package, data['path'], data['ref'], url)
 
 
 def path(chip, package):
@@ -102,40 +97,25 @@ def path(chip, package):
         path: Location of data source on the local system
     """
 
-    return _path(chip, package, __download_data)
-
+    if package not in chip._packages:
+        changed = False
+        data_path = _path(chip, package)
 
-def __download_data(chip, package, data, url, data_path, data_path_lock):
-    data_lock = fasteners.InterProcessLock(data_path_lock)
+        if isinstance(data_path, tuple) and len(data_path) == 2:
+            data_path, changed = data_path
 
-    _aquire_data_lock(data_path, data_lock)
+        if os.path.exists(data_path):
+            if package not in chip._packages and changed:
+                chip.logger.info(f'Saved {package} data to {data_path}')
+            else:
+                chip.logger.info(f'Found {package} data at {data_path}')
 
-    # check cached package data source
-    if os.path.exists(data_path):
-        chip.logger.info(f'Found cached {package} data at {data_path}')
-        if url.scheme in ['git', 'git+https', 'ssh', 'git+ssh']:
-            try:
-                repo = Repo(data_path)
-                if repo.untracked_files or repo.index.diff("HEAD"):
-                    chip.logger.warning('The repo of the cached data is dirty.')
-                _release_data_lock(data_lock)
-                chip._packages[package] = data_path
-                return
-            except GitCommandError:
-                chip.logger.warning('Deleting corrupted cache data.')
-                shutil.rmtree(path)
-        else:
-            _release_data_lock(data_lock)
             chip._packages[package] = data_path
-            return
-
-    # download package data source
-    if url.scheme in ['git', 'git+https', 'ssh', 'git+ssh']:
-        clone_synchronized(chip, package, data, data_path)
-    elif url.scheme == 'https':
-        extract_from_url(chip, package, data, data_path)
+        else:
+            raise SiliconCompilerError(f'Unable to locate {package} data in {data_path}',
+                                       chip=chip)
 
-    _release_data_lock(data_lock)
+    return chip._packages[package]
 
 
 def __get_filebased_lock(data_lock):
@@ -143,7 +123,7 @@ def __get_filebased_lock(data_lock):
     return Path(f'{base}.sc_lock')
 
 
-def _aquire_data_lock(data_path, data_lock):
+def aquire_data_lock(data_path, data_lock):
     # Wait a maximum of 10 minutes for other processes to finish
     max_seconds = 10 * 60
     try:
@@ -168,7 +148,7 @@ def _aquire_data_lock(data_path, data_lock):
                                'please delete it.')
 
 
-def _release_data_lock(data_lock):
+def release_data_lock(data_lock):
     # Check if file based locking method was used
     lock_file = __get_filebased_lock(data_lock)
     if lock_file.exists():
@@ -178,100 +158,6 @@ def _release_data_lock(data_lock):
     data_lock.release()
 
 
-def clone_synchronized(chip, package, data, data_path):
-    url = urlparse(data['path'])
-    try:
-        clone_from_git(chip, package, data, data_path)
-    except GitCommandError as e:
-        if 'Permission denied' in repr(e):
-            if url.scheme in ['ssh', 'git+ssh']:
-                chip.logger.error('Failed to authenticate. Please setup your git ssh.')
-            elif url.scheme in ['git', 'git+https']:
-                chip.logger.error('Failed to authenticate. Please use a token or ssh.')
-        else:
-            chip.logger.error(str(e))
-
-
-def clone_from_git(chip, package, data, repo_path):
-    url = urlparse(data['path'])
-    if url.scheme in ['git', 'git+https'] and url.username:
-        chip.logger.warning('Your token is in the data source path and will be stored in the '
-                            'schema. If you do not want this set the env variable GIT_TOKEN '
-                            'or use ssh for authentication.')
-    if url.scheme in ['git+ssh', 'ssh']:
-        chip.logger.info(f'Cloning {package} data from {url.netloc}:{url.path[1:]}')
-        # Git requires the format git@github.com:org/repo instead of git@github.com/org/repo
-        repo = Repo.clone_from(f'{url.netloc}:{url.path[1:]}',
-                               repo_path,
-                               recurse_submodules=True)
-    else:
-        if os.environ.get('GIT_TOKEN') and not url.username:
-            url = url._replace(netloc=f'{os.environ.get("GIT_TOKEN")}@{url.hostname}')
-        url = url._replace(scheme='https')
-        chip.logger.info(f'Cloning {package} data from {url.geturl()}')
-        repo = Repo.clone_from(url.geturl(), repo_path, recurse_submodules=True)
-    chip.logger.info(f'Checking out {data["ref"]}')
-    repo.git.checkout(data["ref"])
-    for submodule in repo.submodules:
-        submodule.update(init=True)
-
-
-def extract_from_url(chip, package, data, data_path):
-    url = urlparse(data['path'])
-    data_url = data.get('path')
-    headers = {}
-    if os.environ.get('GIT_TOKEN') or url.username:
-        headers['Authorization'] = f'token {os.environ.get("GIT_TOKEN") or url.username}'
-    if "github" in data_url:
-        headers['Accept'] = 'application/octet-stream'
-    data_url = data['path']
-    if data_url.endswith('/'):
-        data_url = f"{data_url}{data['ref']}.tar.gz"
-    chip.logger.info(f'Downloading {package} data from {data_url}')
-    response = requests.get(data_url, stream=True, headers=headers)
-    if not response.ok:
-        raise SiliconCompilerError(f'Failed to download {package} data source.', chip=chip)
-
-    fileobj = BytesIO(response.content)
-    try:
-        with tarfile.open(fileobj=fileobj, mode='r|gz') as tar_ref:
-            tar_ref.extractall(path=data_path)
-    except tarfile.ReadError:
-        fileobj.seek(0)
-        # Try as zip
-        with zipfile.ZipFile(fileobj) as zip_ref:
-            zip_ref.extractall(path=data_path)
-
-    if 'github' in url.netloc and len(os.listdir(data_path)) == 1:
-        # Github inserts one folder at the highest level of the tar file
-        # this compensates for this behavior
-        gh_url = urlparse(data_url)
-
-        repo = gh_url.path.split('/')[2]
-
-        ref = gh_url.path.split('/')[-1]
-        if repo.endswith('.git'):
-            ref = data['ref']
-        elif ref.endswith('.tar.gz'):
-            ref = ref[0:-7]
-        elif ref.endswith('.tgz'):
-            ref = ref[0:-4]
-        else:
-            ref = ref.split('.')[0]
-
-        if ref.startswith('v'):
-            ref = ref[1:]
-
-        github_folder = f"{repo}-{ref}"
-
-        if github_folder in os.listdir(data_path):
-            # This moves all files one level up
-            git_path = os.path.join(data_path, github_folder)
-            for data_file in os.listdir(git_path):
-                shutil.move(os.path.join(git_path, data_file), data_path)
-            os.removedirs(git_path)
-
-
 def path_from_python(chip, python_package, append_path=None):
     try:
         module = importlib.import_module(python_package)

siliconcompiler-0.31.1/siliconcompiler/package/git.py

@@ -0,0 +1,81 @@
+import shutil
+
+import os.path
+
+from git import Repo, GitCommandError
+from fasteners import InterProcessLock
+
+from siliconcompiler.package import get_download_cache_path
+from siliconcompiler.package import aquire_data_lock, release_data_lock
+
+
+def get_resolver(url):
+    if url.scheme in ("git", "git+https", "git+ssh", "ssh"):
+        return git_resolver
+    return None
+
+
+def git_resolver(chip, package, path, ref, url):
+    data_path, data_path_lock = get_download_cache_path(chip, package, ref)
+
+    # Acquire lock
+    data_lock = InterProcessLock(data_path_lock)
+    aquire_data_lock(data_path, data_lock)
+
+    if os.path.exists(data_path):
+        try:
+            repo = Repo(data_path)
+            if repo.untracked_files or repo.index.diff("HEAD"):
+                chip.logger.warning('The repo of the cached data is dirty.')
+            release_data_lock(data_lock)
+            return data_path, False
+        except GitCommandError:
+            chip.logger.warning('Deleting corrupted cache data.')
+            shutil.rmtree(data_path)
+
+    clone_synchronized(chip, package, path, ref, url, data_path)
+
+    release_data_lock(data_lock)
+
+    return data_path, True
+
+
+def clone_synchronized(chip, package, path, ref, url, data_path):
+    try:
+        clone_from_git(chip, package, path, ref, url, data_path)
+    except GitCommandError as e:
+        if 'Permission denied' in repr(e):
+            if url.scheme in ['ssh', 'git+ssh']:
+                chip.logger.error('Failed to authenticate. Please setup your git ssh.')
+            elif url.scheme in ['git', 'git+https']:
+                chip.logger.error('Failed to authenticate. Please use a token or ssh.')
+        else:
+            chip.logger.error(str(e))
+
+
+def clone_from_git(chip, package, path, ref, url, data_path):
+    if url.scheme in ['git', 'git+https'] and url.username:
+        chip.logger.warning('Your token is in the data source path and will be stored in the '
+                            'schema. If you do not want this set the env variable GIT_TOKEN '
+                            'or use ssh for authentication.')
+    if url.scheme in ['git+ssh']:
+        chip.logger.info(f'Cloning {package} data from {url.netloc}:{url.path[1:]}')
+        # Git requires the format git@github.com:org/repo instead of git@github.com/org/repo
+        repo = Repo.clone_from(f'{url.netloc}/{url.path[1:]}',
+                               data_path,
+                               recurse_submodules=True)
+    elif url.scheme in ['ssh']:
+        chip.logger.info(f'Cloning {package} data from {path}')
+        repo = Repo.clone_from(path,
+                               data_path,
+                               recurse_submodules=True)
+    else:
+        if os.environ.get('GIT_TOKEN') and not url.username:
+            url = url._replace(netloc=f'{os.environ.get("GIT_TOKEN")}@{url.hostname}')
+        url = url._replace(scheme='https')
+        chip.logger.info(f'Cloning {package} data from {url.geturl()}')
+        repo = Repo.clone_from(url.geturl(), data_path, recurse_submodules=True)
+    chip.logger.info(f'Checking out {ref}')
+    repo.git.checkout(ref)
+    for submodule in repo.submodules:
+        submodule.update(init=True)