dvc-utils 0.0.2__tar.gz → 0.0.4__tar.gz

{dvc-utils-0.0.2 → dvc-utils-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dvc-utils
-Version: 0.0.2
+Version: 0.0.4
 Summary: CLI for diffing DVC files at two commits (or one commit vs. current worktree), optionally passing both through another command first
 Home-page: https://github.com/runsascoded/dvc-utils
 Author: Ryan Williams
@@ -10,14 +10,26 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 
 # dvc-utils
-CLI for diffing [DVC] files at two commits (or one commit vs. current worktree), optionally passing both through another command first
-
-## Installation
+CLI for diffing [DVC] files, optionally passing both through another command first
+
+<!-- toc -->
+- [Installation](#installation)
+- [Usage](#usage)
+    - [`dvc-utils diff`](#dvc-utils-diff)
+- [Examples](#examples)
+    - [Parquet file](#parquet-diff)
+        - [Schema diff](#parquet-schema-diff)
+        - [Row diff](#parquet-row-diff)
+        - [Row count diff](#parquet-row-count-diff)
+<!-- /toc -->
+
+## Installation <a id="installation"></a>
 ```bash
 pip install dvc-utils
 ```
 
-## Usage
+## Usage <a id="usage"></a>
+<!-- `bmdf -- dvc-utils --help` -->
 ```bash
 dvc-utils --help
 # Usage: dvc-utils [OPTIONS] COMMAND [ARGS]...
@@ -30,7 +42,8 @@ dvc-utils --help
 #         worktree), optionally passing both through another command first
 ```
 
-### `dvc-utils diff`
+### `dvc-utils diff` <a id="dvc-utils-diff"></a>
+<!-- `bmdf -- dvc-utils diff --help` -->
 ```bash
 dvc-utils diff --help
 # Usage: dvc-utils diff [OPTIONS] [cmd...] <path>
@@ -48,24 +61,31 @@ dvc-utils diff --help
 #   optional) at HEAD (last committed value) vs. the current worktree content.
 #
 # Options:
-#   -r, --refspec TEXT  <commit 1>..<commit 2> (compare two commits) or <commit>
-#                       (compare <commit> to the worktree)
-#   -S, --no-shell      Don't pass `shell=True` to Python `subprocess`es
-#   -v, --verbose       Log intermediate commands to stderr
-#   --help              Show this message and exit.
+#   -c, --color              Colorize the output
+#   -r, --refspec TEXT       <commit 1>..<commit 2> (compare two commits) or
+#                            <commit> (compare <commit> to the worktree)
+#   -S, --no-shell           Don't pass `shell=True` to Python `subprocess`es
+#   -U, --unified INTEGER    Number of lines of context to show (passes through
+#                            to `diff`)
+#   -v, --verbose            Log intermediate commands to stderr
+#   -w, --ignore-whitespace  Ignore whitespace differences (pass `-w` to `diff`)
+#   --help                   Show this message and exit.
 ```
 
-## Examples
+## Examples <a id="examples"></a>
+
+### Parquet file <a id="parquet-diff"></a>
 See sample commands and output below for inspecting changes to [a DVC-tracked Parquet file][commit path] in [a given commit][commit].
 
+Setup:
 ```bash
-git clone https://github.com/neighbor-ryan/nj-crashes
-commit=c8ae28e
-path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc
+git clone https://github.com/hudcostreets/nj-crashes && cd nj-crashes # Clone + enter example repo
+commit=c8ae28e  # Example commit that changed some DVC-tracked Parquet files
+path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc  # One of the changed files
 ```
 
-### Parquet schema diff
-Use [`parquet2json`] to observe schema changes to a Parquet file, in [a given commit][commit] from [neighbor-ryan/nj-crashes]:
+#### Schema diff <a id="parquet-schema-diff"></a>
+Use [`parquet2json`] to observe schema changes to a Parquet file:
 ```bash
 parquet_schema() {
     parquet2json "$1" schema
@@ -120,11 +140,12 @@ Here we can see that various date/time columns were consolidated, and several st
 
 </details>
 
-### Parquet row diff
-Diff the first row of the Parquet file above (pretty-printed as JSON), before and after the given commit:
+#### Row diff <a id="parquet-row-diff"></a>
+Diff the first row of the Parquet file above (pretty-printed as JSON using [`jq`]), before and after the given commit:
 
 ```bash
 pretty_print_first_row() {
+    # Print first row of Parquet file as JSON, pretty-print with jq
     parquet2json "$1" cat -l 1 | jq .
 }
 export -f pretty_print_first_row
@@ -181,7 +202,7 @@ This reflects the schema changes above.
 
 </details>
 
-### Parquet row count diff
+#### Row count diff <a id="parquet-row-count-diff"></a>
 ```bash
 parquet_row_count() {
     parquet2json "$1" rowcount
@@ -194,8 +215,9 @@ This time we get no output; [the given `$commit`][commit] didn't change the row
 
 [DVC]: https://dvc.org/
 [`parquet2json`]: https://github.com/jupiter/parquet2json
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
 [Parquet]: https://parquet.apache.org/
-[commit]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
-[commit path]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[commit]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
+[commit path]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
+[`jq`]: https://jqlang.github.io/jq/

{dvc-utils-0.0.2 → dvc-utils-0.0.4}/README.md

@@ -1,12 +1,24 @@
 # dvc-utils
-CLI for diffing [DVC] files at two commits (or one commit vs. current worktree), optionally passing both through another command first
-
-## Installation
+CLI for diffing [DVC] files, optionally passing both through another command first
+
+<!-- toc -->
+- [Installation](#installation)
+- [Usage](#usage)
+    - [`dvc-utils diff`](#dvc-utils-diff)
+- [Examples](#examples)
+    - [Parquet file](#parquet-diff)
+        - [Schema diff](#parquet-schema-diff)
+        - [Row diff](#parquet-row-diff)
+        - [Row count diff](#parquet-row-count-diff)
+<!-- /toc -->
+
+## Installation <a id="installation"></a>
 ```bash
 pip install dvc-utils
 ```
 
-## Usage
+## Usage <a id="usage"></a>
+<!-- `bmdf -- dvc-utils --help` -->
 ```bash
 dvc-utils --help
 # Usage: dvc-utils [OPTIONS] COMMAND [ARGS]...
@@ -19,7 +31,8 @@ dvc-utils --help
 #         worktree), optionally passing both through another command first
 ```
 
-### `dvc-utils diff`
+### `dvc-utils diff` <a id="dvc-utils-diff"></a>
+<!-- `bmdf -- dvc-utils diff --help` -->
 ```bash
 dvc-utils diff --help
 # Usage: dvc-utils diff [OPTIONS] [cmd...] <path>
@@ -37,24 +50,31 @@ dvc-utils diff --help
 #   optional) at HEAD (last committed value) vs. the current worktree content.
 #
 # Options:
-#   -r, --refspec TEXT  <commit 1>..<commit 2> (compare two commits) or <commit>
-#                       (compare <commit> to the worktree)
-#   -S, --no-shell      Don't pass `shell=True` to Python `subprocess`es
-#   -v, --verbose       Log intermediate commands to stderr
-#   --help              Show this message and exit.
+#   -c, --color              Colorize the output
+#   -r, --refspec TEXT       <commit 1>..<commit 2> (compare two commits) or
+#                            <commit> (compare <commit> to the worktree)
+#   -S, --no-shell           Don't pass `shell=True` to Python `subprocess`es
+#   -U, --unified INTEGER    Number of lines of context to show (passes through
+#                            to `diff`)
+#   -v, --verbose            Log intermediate commands to stderr
+#   -w, --ignore-whitespace  Ignore whitespace differences (pass `-w` to `diff`)
+#   --help                   Show this message and exit.
 ```
 
-## Examples
+## Examples <a id="examples"></a>
+
+### Parquet file <a id="parquet-diff"></a>
 See sample commands and output below for inspecting changes to [a DVC-tracked Parquet file][commit path] in [a given commit][commit].
 
+Setup:
 ```bash
-git clone https://github.com/neighbor-ryan/nj-crashes
-commit=c8ae28e
-path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc
+git clone https://github.com/hudcostreets/nj-crashes && cd nj-crashes # Clone + enter example repo
+commit=c8ae28e  # Example commit that changed some DVC-tracked Parquet files
+path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc  # One of the changed files
 ```
 
-### Parquet schema diff
-Use [`parquet2json`] to observe schema changes to a Parquet file, in [a given commit][commit] from [neighbor-ryan/nj-crashes]:
+#### Schema diff <a id="parquet-schema-diff"></a>
+Use [`parquet2json`] to observe schema changes to a Parquet file:
 ```bash
 parquet_schema() {
     parquet2json "$1" schema
@@ -109,11 +129,12 @@ Here we can see that various date/time columns were consolidated, and several st
 
 </details>
 
-### Parquet row diff
-Diff the first row of the Parquet file above (pretty-printed as JSON), before and after the given commit:
+#### Row diff <a id="parquet-row-diff"></a>
+Diff the first row of the Parquet file above (pretty-printed as JSON using [`jq`]), before and after the given commit:
 
 ```bash
 pretty_print_first_row() {
+    # Print first row of Parquet file as JSON, pretty-print with jq
     parquet2json "$1" cat -l 1 | jq .
 }
 export -f pretty_print_first_row
@@ -170,7 +191,7 @@ This reflects the schema changes above.
 
 </details>
 
-### Parquet row count diff
+#### Row count diff <a id="parquet-row-count-diff"></a>
 ```bash
 parquet_row_count() {
     parquet2json "$1" rowcount
@@ -183,8 +204,9 @@ This time we get no output; [the given `$commit`][commit] didn't change the row
 
 [DVC]: https://dvc.org/
 [`parquet2json`]: https://github.com/jupiter/parquet2json
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
 [Parquet]: https://parquet.apache.org/
-[commit]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
-[commit path]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[commit]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
+[commit path]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
+[`jq`]: https://jqlang.github.io/jq/

dvc-utils-0.0.4/dvc_utils/main.py

@@ -0,0 +1,183 @@
+from functools import cache
+from os import environ as env, getcwd
+
+from typing import Optional, Tuple
+
+import shlex
+from os.path import join, relpath
+
+from click import option, argument, group
+from subprocess import Popen
+
+import click
+import yaml
+from utz import process, singleton, err
+
+from dvc_utils.named_pipes import named_pipes
+
+
+@group()
+def cli():
+    pass
+
+
+def dvc_paths(path: str) -> Tuple[str, str]:
+    if path.endswith('.dvc'):
+        dvc_path = path
+        path = dvc_path[:-len('.dvc')]
+    else:
+        dvc_path = f'{path}.dvc'
+    return path, dvc_path
+
+
+@cache
+def get_git_root() -> str:
+    return process.line('git', 'rev-parse', '--show-toplevel', log=False)
+
+
+@cache
+def get_dir_path() -> str:
+    return relpath(getcwd(), get_git_root())
+
+
+@cache
+def dvc_cache_dir(log: bool = False) -> str:
+    dvc_cache_relpath = env.get('DVC_UTILS_CACHE_DIR')
+    if dvc_cache_relpath:
+        return join(get_git_root(), dvc_cache_relpath)
+    else:
+        return process.line('dvc', 'cache', 'dir', log=log)
+
+
+def dvc_md5(git_ref: str, dvc_path: str, log: bool = False) -> str:
+    dir_path = get_dir_path()
+    dir_path = '' if dir_path == '.' else f'{dir_path}/'
+    dvc_spec = process.output('git', 'show', f'{git_ref}:{dir_path}{dvc_path}', log=log)
+    dvc_obj = yaml.safe_load(dvc_spec)
+    out = singleton(dvc_obj['outs'], dedupe=False)
+    md5 = out['md5']
+    return md5
+
+
+def dvc_cache_path(ref: str, dvc_path: Optional[str] = None, log: bool = False) -> str:
+    if dvc_path:
+        md5 = dvc_md5(ref, dvc_path, log=log)
+    elif ':' in ref:
+        git_ref, dvc_path = ref.split(':', 1)
+        md5 = dvc_md5(git_ref, dvc_path, log=log)
+    else:
+        md5 = ref
+    dirname = md5[:2]
+    basename = md5[2:]
+    return join(dvc_cache_dir(log=log), 'files', 'md5', dirname, basename)
+
+
+def diff_cmds(
+    cmd1: str,
+    cmd2: str,
+    verbose: bool = False,
+    color: bool = False,
+    unified: int | None = None,
+    ignore_whitespace: bool = False,
+    **kwargs,
+):
+    """Run two commands and diff their output.
+
+    Adapted from https://stackoverflow.com/a/28840955"""
+    with named_pipes(n=2) as pipes:
+        (pipe1, pipe2) = pipes
+        diff_cmd = [
+            'diff',
+            *(['-w'] if ignore_whitespace else []),
+            *(['-U', str(unified)] if unified is not None else []),
+            *(['--color=always'] if color else []),
+            pipe1,
+            pipe2,
+        ]
+        diff = Popen(diff_cmd)
+        processes = []
+        for path, cmd in ((pipe1, cmd1), (pipe2, cmd2)):
+            with open(path, 'wb', 0) as pipe:
+                if verbose:
+                    err(f"Running: {cmd}")
+                processes.append(Popen(cmd, stdout=pipe, close_fds=True, **kwargs))
+        for p in [diff] + processes:
+            p.wait()
+
+
+@cli.command('diff', short_help='Diff a DVC-tracked file at two commits (or one commit vs. current worktree), optionally passing both through another command first')
+@option('-c', '--color', is_flag=True, help='Colorize the output')
+@option('-r', '--refspec', default='HEAD', help='<commit 1>..<commit 2> (compare two commits) or <commit> (compare <commit> to the worktree)')
+@option('-S', '--no-shell', is_flag=True, help="Don't pass `shell=True` to Python `subprocess`es")
+@option('-U', '--unified', type=int, help='Number of lines of context to show (passes through to `diff`)')
+@option('-v', '--verbose', is_flag=True, help="Log intermediate commands to stderr")
+@option('-w', '--ignore-whitespace', is_flag=True, help="Ignore whitespace differences (pass `-w` to `diff`)")
+@argument('args', metavar='[cmd...] <path>', nargs=-1)
+def dvc_utils_diff(
+    color: bool,
+    refspec: str | None,
+    no_shell: bool,
+    unified: int | None,
+    verbose: bool,
+    ignore_whitespace: bool,
+    args: Tuple[str, ...],
+):
+    """Diff a file at two commits (or one commit vs. current worktree), optionally passing both through `cmd` first
+
+    Examples:
+
+    dvc-utils diff -r HEAD^..HEAD wc -l foo.dvc  # Compare the number of lines (`wc -l`) in `foo` (the file referenced by `foo.dvc`) at the previous vs. current commit (`HEAD^..HEAD`).
+
+    dvc-utils diff md5sum foo  # Diff the `md5sum` of `foo` (".dvc" extension is optional) at HEAD (last committed value) vs. the current worktree content.
+    """
+    if not args:
+        raise click.UsageError('Must specify [cmd...] <path>')
+
+    shell = not no_shell
+    if len(args) == 2:
+        cmd, path = args
+        cmd = shlex.split(cmd)
+    elif len(args) == 1:
+        cmd = None
+        path, = args
+    else:
+        raise click.UsageError('Maximum 2 positional args: [cmd] <path>')
+
+    path, dvc_path = dvc_paths(path)
+
+    pcs = refspec.split('..', 1)
+    if len(pcs) == 1:
+        before = pcs[0]
+        after = None
+    elif len(pcs) == 2:
+        before, after = pcs
+    else:
+        raise ValueError(f"Invalid refspec: {refspec}")
+
+    log = err if verbose else False
+    before_path = dvc_cache_path(before, dvc_path, log=log)
+    after_path = path if after is None else dvc_cache_path(after, dvc_path, log=log)
+
+    if cmd:
+        def args(path: str):
+            arr = cmd + [path]
+            return shlex.join(arr) if shell else arr
+
+        shell_kwargs = dict(shell=shell) if shell else {}
+        before_cmd = args(before_path)
+        after_cmd = args(after_path)
+        diff_cmds(
+            before_cmd,
+            after_cmd,
+            verbose=verbose,
+            color=color,
+            unified=unified,
+            ignore_whitespace=ignore_whitespace,
+            **shell_kwargs,
+        )
+    else:
+        process.run('diff', before_path, after_path, log=log)
+
+
+if __name__ == '__main__':
+    cli()

{dvc-utils-0.0.2 → dvc-utils-0.0.4}/dvc_utils/named_pipes.py

@@ -5,7 +5,7 @@ from contextlib import contextmanager
 
 
 @contextmanager
-def named_pipes(n=1):
+def named_pipes(n: int = 1):
     """Yield a list of paths to named pipes that are created and destroyed
 
     From https://stackoverflow.com/a/28840955"""

{dvc-utils-0.0.2 → dvc-utils-0.0.4}/dvc_utils.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dvc-utils
-Version: 0.0.2
+Version: 0.0.4
 Summary: CLI for diffing DVC files at two commits (or one commit vs. current worktree), optionally passing both through another command first
 Home-page: https://github.com/runsascoded/dvc-utils
 Author: Ryan Williams
@@ -10,14 +10,26 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 
 # dvc-utils
-CLI for diffing [DVC] files at two commits (or one commit vs. current worktree), optionally passing both through another command first
-
-## Installation
+CLI for diffing [DVC] files, optionally passing both through another command first
+
+<!-- toc -->
+- [Installation](#installation)
+- [Usage](#usage)
+    - [`dvc-utils diff`](#dvc-utils-diff)
+- [Examples](#examples)
+    - [Parquet file](#parquet-diff)
+        - [Schema diff](#parquet-schema-diff)
+        - [Row diff](#parquet-row-diff)
+        - [Row count diff](#parquet-row-count-diff)
+<!-- /toc -->
+
+## Installation <a id="installation"></a>
 ```bash
 pip install dvc-utils
 ```
 
-## Usage
+## Usage <a id="usage"></a>
+<!-- `bmdf -- dvc-utils --help` -->
 ```bash
 dvc-utils --help
 # Usage: dvc-utils [OPTIONS] COMMAND [ARGS]...
@@ -30,7 +42,8 @@ dvc-utils --help
 #         worktree), optionally passing both through another command first
 ```
 
-### `dvc-utils diff`
+### `dvc-utils diff` <a id="dvc-utils-diff"></a>
+<!-- `bmdf -- dvc-utils diff --help` -->
 ```bash
 dvc-utils diff --help
 # Usage: dvc-utils diff [OPTIONS] [cmd...] <path>
@@ -48,24 +61,31 @@ dvc-utils diff --help
 #   optional) at HEAD (last committed value) vs. the current worktree content.
 #
 # Options:
-#   -r, --refspec TEXT  <commit 1>..<commit 2> (compare two commits) or <commit>
-#                       (compare <commit> to the worktree)
-#   -S, --no-shell      Don't pass `shell=True` to Python `subprocess`es
-#   -v, --verbose       Log intermediate commands to stderr
-#   --help              Show this message and exit.
+#   -c, --color              Colorize the output
+#   -r, --refspec TEXT       <commit 1>..<commit 2> (compare two commits) or
+#                            <commit> (compare <commit> to the worktree)
+#   -S, --no-shell           Don't pass `shell=True` to Python `subprocess`es
+#   -U, --unified INTEGER    Number of lines of context to show (passes through
+#                            to `diff`)
+#   -v, --verbose            Log intermediate commands to stderr
+#   -w, --ignore-whitespace  Ignore whitespace differences (pass `-w` to `diff`)
+#   --help                   Show this message and exit.
 ```
 
-## Examples
+## Examples <a id="examples"></a>
+
+### Parquet file <a id="parquet-diff"></a>
 See sample commands and output below for inspecting changes to [a DVC-tracked Parquet file][commit path] in [a given commit][commit].
 
+Setup:
 ```bash
-git clone https://github.com/neighbor-ryan/nj-crashes
-commit=c8ae28e
-path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc
+git clone https://github.com/hudcostreets/nj-crashes && cd nj-crashes # Clone + enter example repo
+commit=c8ae28e  # Example commit that changed some DVC-tracked Parquet files
+path=njdot/data/2001/NewJersey2001Accidents.pqt.dvc  # One of the changed files
 ```
 
-### Parquet schema diff
-Use [`parquet2json`] to observe schema changes to a Parquet file, in [a given commit][commit] from [neighbor-ryan/nj-crashes]:
+#### Schema diff <a id="parquet-schema-diff"></a>
+Use [`parquet2json`] to observe schema changes to a Parquet file:
 ```bash
 parquet_schema() {
     parquet2json "$1" schema
@@ -120,11 +140,12 @@ Here we can see that various date/time columns were consolidated, and several st
 
 </details>
 
-### Parquet row diff
-Diff the first row of the Parquet file above (pretty-printed as JSON), before and after the given commit:
+#### Row diff <a id="parquet-row-diff"></a>
+Diff the first row of the Parquet file above (pretty-printed as JSON using [`jq`]), before and after the given commit:
 
 ```bash
 pretty_print_first_row() {
+    # Print first row of Parquet file as JSON, pretty-print with jq
     parquet2json "$1" cat -l 1 | jq .
 }
 export -f pretty_print_first_row
@@ -181,7 +202,7 @@ This reflects the schema changes above.
 
 </details>
 
-### Parquet row count diff
+#### Row count diff <a id="parquet-row-count-diff"></a>
 ```bash
 parquet_row_count() {
     parquet2json "$1" rowcount
@@ -194,8 +215,9 @@ This time we get no output; [the given `$commit`][commit] didn't change the row
 
 [DVC]: https://dvc.org/
 [`parquet2json`]: https://github.com/jupiter/parquet2json
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
 [Parquet]: https://parquet.apache.org/
-[commit]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
-[commit path]: https://github.com/neighbor-ryan/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
-[neighbor-ryan/nj-crashes]: https://github.com/neighbor-ryan/nj-crashes
+[commit]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7
+[commit path]: https://github.com/hudcostreets/nj-crashes/commit/c8ae28e64f4917895d84074913f48e0a7afbc3d7#diff-7f812dce61e0996354f4af414203e0933ccdfe9613cb406c40c1c41a14b9769c
+[hudcostreets/nj-crashes]: https://github.com/hudcostreets/nj-crashes
+[`jq`]: https://jqlang.github.io/jq/

{dvc-utils-0.0.2 → dvc-utils-0.0.4}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup
 
 setup(
     name='dvc-utils',
-    version="0.0.2",
+    version="0.0.4",
     description="CLI for diffing DVC files at two commits (or one commit vs. current worktree), optionally passing both through another command first",
     long_description=open("README.md").read(),
     long_description_content_type="text/markdown",

dvc-utils-0.0.2/dvc_utils/main.py

@@ -1,119 +0,0 @@
-import shlex
-from os.path import join
-from subprocess import Popen
-
-import click
-import yaml
-from utz import process, singleton, err
-
-from dvc_utils.named_pipes import named_pipes
-
-
-@click.group()
-def cli():
-    pass
-
-
-def dvc_paths(path):
-    if path.endswith('.dvc'):
-        dvc_path = path
-        path = dvc_path[:-len('.dvc')]
-    else:
-        dvc_path = f'{path}.dvc'
-    return path, dvc_path
-
-
-def dvc_md5(git_ref, dvc_path, log=False):
-    dvc_spec = process.output('git', 'show', f'{git_ref}:{dvc_path}', log=log)
-    dvc_obj = yaml.safe_load(dvc_spec)
-    out = singleton(dvc_obj['outs'], dedupe=False)
-    md5 = out['md5']
-    return md5
-
-
-_dvc_cache_dir = None
-def dvc_cache_dir(log=False):
-    global _dvc_cache_dir
-    if _dvc_cache_dir is None:
-        _dvc_cache_dir = process.line('dvc', 'cache', 'dir', log=log)
-    return _dvc_cache_dir
-
-
-def dvc_cache_path(spec, dvc_path=None, log=False):
-    if dvc_path:
-        md5 = dvc_md5(spec, dvc_path, log=log)
-    elif ':' in spec:
-        git_ref, dvc_path = spec.split(':', 1)
-        md5 = dvc_md5(git_ref, dvc_path, log=log)
-    else:
-        md5 = spec
-    dirname = md5[:2]
-    basename = md5[2:]
-    return join(dvc_cache_dir(log=log), 'files', 'md5', dirname, basename)
-
-
-def diff_cmds(cmd1, cmd2, **kwargs):
-    """Run two commands and diff their output.
-
-    Adapted from https://stackoverflow.com/a/28840955"""
-    with named_pipes(n=2) as paths:
-        someprogram = Popen(['diff'] + paths)
-        processes = []
-        for path, cmd in zip(paths, [ cmd1, cmd2 ]):
-            with open(path, 'wb', 0) as pipe:
-                processes.append(Popen(cmd, stdout=pipe, close_fds=True, **kwargs))
-        for p in [someprogram] + processes:
-            p.wait()
-
-
-@cli.command('diff', short_help='Diff a DVC-tracked file at two commits (or one commit vs. current worktree), optionally passing both through another command first')
-@click.option('-r', '--refspec', default='HEAD', help='<commit 1>..<commit 2> (compare two commits) or <commit> (compare <commit> to the worktree)')
-@click.option('-S', '--no-shell', is_flag=True, help="Don't pass `shell=True` to Python `subprocess`es")
-@click.option('-v', '--verbose', is_flag=True, help="Log intermediate commands to stderr")
-@click.argument('args', metavar='[cmd...] <path>', nargs=-1)
-def dvc_utils_diff(refspec, no_shell, verbose, args):
-    """Diff a file at two commits (or one commit vs. current worktree), optionally passing both through `cmd` first
-
-    Examples:
-
-    dvc-utils diff -r HEAD^..HEAD wc -l foo.dvc  # Compare the number of lines (`wc -l`) in `foo` (the file referenced by `foo.dvc`) at the previous vs. current commit (`HEAD^..HEAD`).
-
-    dvc-utils diff md5sum foo  # Diff the `md5sum` of `foo` (".dvc" extension is optional) at HEAD (last committed value) vs. the current worktree content.
-    """
-    if not args:
-        raise click.UsageError('Must specify [cmd...] <path>')
-
-    shell = not no_shell
-    (*cmd, path) = args
-    if path.endswith('.dvc'):
-        dvc_path = path
-        path = dvc_path[:-len('.dvc')]
-    else:
-        dvc_path = f'{path}.dvc'
-
-    pcs = refspec.split('..', 1)
-    if len(pcs) == 1:
-        before = pcs[0]
-        after = None
-    else:
-        before, after = pcs
-
-    log = err if verbose else False
-    before_path = dvc_cache_path(before, dvc_path, log=log)
-    after_path = path if after is None else dvc_cache_path(after, dvc_path, log=log)
-
-    if cmd:
-        def args(path):
-            arr = cmd + [path]
-            return shlex.join(arr) if shell else arr
-
-        shell_kwargs = dict(shell=shell) if shell else {}
-        before_cmd = args(before_path)
-        after_cmd = args(after_path)
-        diff_cmds(before_cmd, after_cmd, **shell_kwargs)
-    else:
-        process.run('diff', before_path, after_path, log=log)
-
-
-if __name__ == '__main__':
-    cli()