datatrail-cli 1.0.0__tar.gz → 1.0.2__tar.gz

datatrail_cli-1.0.2/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "1.0.2"
+}

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## [1.0.2](https://github.com/CHIMEFRB/datatrail-cli/compare/datatrail-cli-v1.0.1...datatrail-cli-v1.0.2) (2026-09-29)
+
+
+### Documentation
+
+* cover readiness, verification, and recursive discovery ([#201](https://github.com/CHIMEFRB/datatrail-cli/issues/201)) ([b7b4808](https://github.com/CHIMEFRB/datatrail-cli/commit/b7b48084b094f89ce5812182ac85b487478057d7))
+* restore standalone GitHub Pages build ([#199](https://github.com/CHIMEFRB/datatrail-cli/issues/199)) ([476935e](https://github.com/CHIMEFRB/datatrail-cli/commit/476935e9530f196f64ea2551deb89f65f0fece55))
+
+## [1.0.1](https://github.com/CHIMEFRB/datatrail-cli/compare/datatrail-cli-v1.0.0...datatrail-cli-v1.0.1) (2026-09-29)
+
+
+### Bug Fixes
+
+* **pull:** continue when optional size query is unavailable ([#195](https://github.com/CHIMEFRB/datatrail-cli/issues/195)) ([b36bbbf](https://github.com/CHIMEFRB/datatrail-cli/commit/b36bbbf90d47eb22f20eaa7a12cfea8f4ae703bb))
+* **scout:** restore discrepancy detection and confirmed repairs ([#197](https://github.com/CHIMEFRB/datatrail-cli/issues/197)) ([54dfbf4](https://github.com/CHIMEFRB/datatrail-cli/commit/54dfbf4b3abf8ff0f7f9db13e5eb7ccdb28ccce1))
+
+
+### Performance Improvements
+
+* **pull:** apply CANFAR permissions without subprocesses ([#157](https://github.com/CHIMEFRB/datatrail-cli/issues/157)) ([b8c2230](https://github.com/CHIMEFRB/datatrail-cli/commit/b8c22306432a6b24b435b5ddb421ac5cdcd9e1eb))
+
 ## [1.0.0](https://github.com/CHIMEFRB/datatrail-cli/compare/datatrail-cli-v0.13.0...datatrail-cli-v1.0.0) (2026-09-28)
 
 

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: datatrail-cli
-Version: 1.0.0
+Version: 1.0.2
 Summary: CHIME/FRB Datatrail CLI
 Author: CHIME FRB Project Office
 Author-email: Tarik Zegmott <tarik.zegmott@mcgill.ca>
@@ -113,3 +113,19 @@ datatrail config init --site chime
 # Ensure valid CADC Certificate exists
 cadc-get-cert -u [username]
 ```
+
+## Building the documentation
+
+From the repository root, install the locked documentation dependencies with
+[uv](https://docs.astral.sh/uv/), then build the standalone public site:
+
+```shell
+uv sync --locked --group docs
+uv run --no-sync mkdocs build --strict --config-file mkdocs-site.yml
+```
+
+The site is written to `site/`. To preview it locally, run
+`uv run --no-sync mkdocs serve --config-file mkdocs-site.yml`.
+The installed checkout supplies the generated CLI reference. `mkdocs-site.yml`
+adds the public site's theme and rendering extensions while inheriting the
+navigation from `mkdocs.yml`, which remains the handbook configuration.

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/README.md

@@ -85,3 +85,19 @@ datatrail config init --site chime
 # Ensure valid CADC Certificate exists
 cadc-get-cert -u [username]
 ```
+
+## Building the documentation
+
+From the repository root, install the locked documentation dependencies with
+[uv](https://docs.astral.sh/uv/), then build the standalone public site:
+
+```shell
+uv sync --locked --group docs
+uv run --no-sync mkdocs build --strict --config-file mkdocs-site.yml
+```
+
+The site is written to `site/`. To preview it locally, run
+`uv run --no-sync mkdocs serve --config-file mkdocs-site.yml`.
+The installed checkout supplies the generated CLI reference. `mkdocs-site.yml`
+adds the public site's theme and rendering extensions while inheriting the
+navigation from `mkdocs.yml`, which remains the handbook configuration.

datatrail_cli-1.0.2/docs/commands.md

@@ -0,0 +1,34 @@
+<div align="center">
+    <img src="../images/Datatrail-logo.png" width="110", height="100">
+</div>
+
+<h1 align="center">Commands</h1>
+
+The commands available to you are:
+
+- [`clear`](clear.md): Remove a dataset's files at the local or CANFAR site.
+- [`config`](cli.md#datatrail-config): Initialize, inspect, or edit the CLI
+  configuration.
+- [`doctor`](doctor.md): Check configuration, server health, certificate validity,
+  and authentication to Minoc and Luskan.
+- [`inventory`](inventory.md): Recursively discover datasets and write their
+  file replica URIs to a resumable JSON manifest.
+- [`list` / `ls`](list.md): Browse scopes and datasets, filter with `--match`,
+  or discover children with `--expand` and `--recursive`.
+- [`ps`](ps.md): Inspect a dataset's files and storage locations. JSON output
+  includes common paths for each storage element.
+- [`pull`](pull.md): Download a dataset's files.
+- [`pull-manifest`](pull-manifest.md): Download Minoc files from an inventory
+  manifest with resumable transfer state.
+- [`scout`](scout.md): Compare registered and observed file counts and, after
+  confirmation, register missing Minoc replicas.
+- [`unregistered`](unregistered.md): Summarize registration failures or search
+  for records of a specific event.
+- [`verify`](verify.md): Compare registered Minoc files with Minoc and Luskan
+  metadata, reporting missing files, size or checksum mismatches, and
+  unavailable metadata.
+- [`version`](cli.md#datatrail-version): Show CLI and server version information.
+
+For practical examples, start with the [User Guide](user_guide.md) or the
+[recursive discovery walkthrough](discovery.md). The
+[Reference](cli.md) lists every command and its options.

datatrail_cli-1.0.2/docs/discovery.md

@@ -0,0 +1,73 @@
+# Discover datasets recursively
+
+Use `datatrail list` (or its alias `datatrail ls`) when you know part of a
+dataset name but not where it sits in the hierarchy. Choose how far to explore:
+
+- `--match`: Select larger datasets using comma-separated, case-insensitive
+  terms. Every term must appear in the combined scope and dataset name.
+- `--expand`: Open each selected larger dataset one level and list its children.
+- `--recursive`: Follow descendants to terminal datasets and record the path
+  to each result.
+
+## Start with a bounded search
+
+For example, find gain datasets within one scope:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains
+```
+
+To inspect one level of children, add `--expand`:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --expand
+```
+
+To continue through nested datasets, use `--recursive` instead:
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --recursive
+```
+
+You can omit the scope to search across scopes, but `--expand` and `--recursive`
+then require `--match` to bound the search. Matching selects the starting larger
+datasets; descendants are followed even if their own names do not contain the
+search terms.
+
+The recursive walk visits each dataset once, so shared descendants are not
+repeated and cycles cannot loop forever. It retains the first path found in
+sorted order. A dataset with an answered empty child list is terminal.
+
+## Save discovery results for a script
+
+```shell
+datatrail ls gbo.acquisition.processed --match gains --recursive --json > datasets.json
+```
+
+The JSON map contains `results` and `failed`. Each recursive result records its
+`scope`, `dataset`, `parent`, and `path`. See the [list guide](list.md) for output
+examples and ordinary scope or child-dataset queries. A top-level failure can
+return an `error` object instead of a map and exits `1`, so check the process
+status before reading the result fields.
+
+!!! warning "Check for incomplete discovery"
+
+    An unanswered branch is retained as a partial row and recorded in `failed`.
+    A partial map with rows still exits `0`; no rows plus unanswered queries exits
+    `1`. Scripts that require complete results must also check that `failed` is
+    empty.
+
+## Turn discovery into a resumable download
+
+`list --recursive` maps datasets. Use [`inventory`](inventory.md) when you also
+need their file replica URIs saved in a durable manifest:
+
+```shell
+datatrail inventory gbo.acquisition.processed --match gains --output gains-inventory.json
+datatrail pull-manifest gains-inventory.json --directory ./gains --cores 4
+```
+
+An inventory can resume unfinished queries. [`pull-manifest`](pull-manifest.md)
+tracks completed transfers in a separate state file so a later run can continue
+without repeating completed downloads. Read both guides for incomplete results,
+state-file ownership, and transfer behavior.

datatrail_cli-1.0.2/docs/doctor.md

@@ -0,0 +1,55 @@
+# Check readiness with `doctor`
+
+Use `doctor` when a command cannot connect, authentication fails, or you want to
+check your setup before downloading data:
+
+```bash
+datatrail doctor
+```
+
+The command reads your existing CLI configuration and certificate, then checks
+Datatrail, Minoc, and Luskan. See [initial setup](initialising.md) if you have not
+configured the CLI or obtained a CADC certificate.
+
+## Understand the checks
+
+The text report prints one line per check, labeled `OK` or `FAILED`:
+
+| Check | What must pass |
+| --- | --- |
+| `config` | The configuration loads and contains a server URL, certificate path, site, and root mount for that site. |
+| `server` | The configured Datatrail server returns a valid, healthy `/health/check` report, including database and API status. |
+| `certificate` | The configured certificate is readable PEM and its validity period includes the current time. |
+| `minoc` | Minoc's capabilities endpoint responds successfully and confirms authentication with the certificate. |
+| `luskan` | Luskan's capabilities endpoint responds successfully and confirms authentication with the certificate. |
+
+If configuration fails, the remaining readiness checks are skipped. If the
+certificate fails, the Minoc and Luskan checks are skipped. Skipped checks appear
+as `FAILED` with a message explaining why they were not checked; they do not
+independently establish that a service is down.
+
+A successful report confirms these readiness checks at that moment. It does not
+prove permission to read every dataset, verify file contents, or continuously
+monitor registration workers. The command does not change configuration, renew
+certificates, or download datasets.
+
+## Use the result in a script
+
+```bash
+datatrail doctor --json > doctor.json
+```
+
+`--json` replaces the text report with a JSON object on standard output. It has a
+top-level `ok` boolean and a `checks` object containing `config`, `server`,
+`certificate`, `minoc`, and `luskan`. Each check contains its own `ok` boolean and
+`message`. The report is still written when a check fails.
+
+| Exit status | Meaning |
+| --- | --- |
+| `0` | Every readiness check passed. |
+| `1` | At least one readiness check failed or was skipped. |
+
+Address the first relevant failure: review [configuration and certificate setup](initialising.md)
+for local problems, or share the failed check and message with the service operator.
+Run `doctor` again after addressing the cause. For dataset metadata checks, use
+[`verify`](verify.md).

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/docs/index.md

@@ -9,7 +9,7 @@ comments: true
 <h1 align="center">Datatrail CLI</h1>
 
 <h4 align="center">
-  <a src="https://github.com/CHIMEFRB/datatrail-cli/install/">Install</a>
+  <a href="install/">Install</a>
   ·
   <a href="https://chimefrb.github.io/datatrail-cli/">Docs</a>
 </h4>
@@ -41,6 +41,10 @@ In order to fully utilise this CLI, you must have an account with
 [CANFAR](https://www.canfar.net) and access to the either of the following
 groups: `chime-frb-ro` or `chime-frb-rw`.
 
+Check your setup with [`doctor`](doctor.md). The [User Guide](user_guide.md)
+covers [recursive dataset discovery](discovery.md), resumable inventory and downloads,
+[`verify`](verify.md), and registration troubleshooting.
+
 ## 🛠️ [Installation](install.md)
 
 ## ⚙️  [Initialise](initialising.md)

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/docs/initialising.md

@@ -56,13 +56,13 @@ one of the outrigger sites. See below for a guide for each of the sites.
         of 30 days. You must refresh the certificate periodically.
 
     If you do not keep your CADC certificate in the default location, you must
-    update the configuration file to pointt to the correct location.
+    update the configuration file to point to the correct location.
 
     ```shell
     # Updating CADC Certificate location
-    $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem
-    Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem
-    Set vospace_certificate to /non/standard/location/cadcproxy.pem
+    $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem
+    Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem
+    Set vospace_certfile to /non/standard/location/cadcproxy.pem
     ```
 
 === "CANFAR"
@@ -123,13 +123,13 @@ one of the outrigger sites. See below for a guide for each of the sites.
         of 30 days. You must refresh the certificate periodically.
 
     If you do not keep your CADC certificate in the default location, you must
-    update the configuration file to pointt to the correct location.
+    update the configuration file to point to the correct location.
 
     ```shell
     # Updating CADC Certificate location
-    $> datatrail config set vospace_certificate /non/standard/location/cadcproxy.pem
-    Attempting to set vospace_certificate to /non/standard/location/cadcproxy.pem
-    Set vospace_certificate to /non/standard/location/cadcproxy.pem
+    $> datatrail config set vospace_certfile /non/standard/location/cadcproxy.pem
+    Attempting to set vospace_certfile to /non/standard/location/cadcproxy.pem
+    Set vospace_certfile to /non/standard/location/cadcproxy.pem
     ```
 
 === "CHIME"
@@ -171,3 +171,15 @@ one of the outrigger sites. See below for a guide for each of the sites.
     # Ensure valid CADC Certificate exists
     cadc-get-cert -u [username]
     ```
+
+## Check your setup
+
+After initializing the configuration and obtaining your certificate, run
+[`doctor`](doctor.md) to check configuration, server health, certificate validity,
+and authentication to Minoc and Luskan:
+
+```shell
+datatrail doctor
+```
+
+The [doctor guide](doctor.md) explains failed checks and JSON output for scripts.

datatrail_cli-1.0.2/docs/user_guide.md

@@ -0,0 +1,58 @@
+<div align="center">
+    <img src="../images/Datatrail-logo.png" width="110", height="100">
+</div>
+
+<h1 align="center">User Guide</h1>
+
+## Set up and check your connection
+
+[Install the CLI](install.md), then [initialize its configuration](initialising.md).
+Use [`doctor`](doctor.md) to check your configuration, Datatrail server health,
+CANFAR certificate, and authentication to Minoc and Luskan before working with
+files. The [`config` reference](cli.md#datatrail-config) covers inspecting and
+editing configuration values.
+
+## Find and inspect datasets
+
+Start with [Recursive discovery](discovery.md) for a walkthrough of `--match`,
+`--expand`, and `--recursive`, followed by a resumable inventory and download.
+
+- [`list` / `ls`](list.md): Browse scopes and datasets. Use `--match` to narrow
+  a search, `--expand` to see one level of children, and `--recursive` to follow
+  descendants to terminal datasets.
+- [`ps`](ps.md): Inspect a dataset's files and storage locations, including
+  per-storage-element common paths in JSON output.
+- [`inventory`](inventory.md): Save recursive discovery and file replica URIs
+  in a JSON manifest that can be resumed after an interruption.
+
+## Download and check files
+
+For a large collection, use [`inventory`](inventory.md) to save the discovered
+files, then pass that manifest to [`pull-manifest`](pull-manifest.md) to download
+them with resumable progress.
+
+- [`pull`](pull.md): Download the files in a dataset.
+- [`pull-manifest`](pull-manifest.md): Download Minoc files from an inventory
+  with saved transfer progress and bounded concurrency.
+- [`verify`](verify.md): Compare registered Minoc files with Minoc and Luskan
+  metadata to identify missing files, size or checksum mismatches, and
+  unavailable metadata.
+- [`clear`](clear.md): Remove a dataset's local or CANFAR files.
+
+## Investigate registration problems
+
+- [`scout`](scout.md): Compare registered and observed file counts and review
+  discrepancies before confirming repairs.
+- [`unregistered`](unregistered.md): Summarize registration failures or inspect
+  records for a specific event, with optional scope and partial-name filters.
+
+## Use results in scripts
+
+`list`, `ps`, `doctor`, `verify`, and `unregistered search` support `--json`.
+Their guides explain each output format. `inventory` writes a JSON manifest,
+while `pull-manifest` saves a separate JSON transfer-state file. Consult the
+relevant command's guide for exit status and error handling.
+
+The [command overview](commands.md) links to every command, including
+[`version`](cli.md#datatrail-version). The [CLI reference](cli.md) lists all
+arguments and options.

datatrail_cli-1.0.2/docs/verify.md

@@ -0,0 +1,66 @@
+# Compare registered Minoc files with `verify`
+
+Use `verify` to check whether the Minoc files recorded by Datatrail have consistent
+metadata in Minoc and the Luskan inventory. Datatrail supplies the file list;
+`verify` compares the sizes and MD5 checksums reported by the two CADC services.
+
+```bash
+datatrail verify chime.event.baseband.raw EVENT_ID
+```
+
+Replace the scope and `EVENT_ID` with your dataset's actual identifiers. The CLI
+needs its normal configuration and a valid CADC certificate with service access.
+Use [`doctor`](doctor.md) to troubleshoot configuration or authentication first.
+
+## Read the results
+
+The report starts with the scope, dataset, and `registered` count: the number of
+unique Minoc file URIs returned by Datatrail. It then shows these result counts
+and lists the URIs with problems:
+
+| Result | Meaning |
+| --- | --- |
+| `present` | Both services returned complete, matching size and checksum metadata. |
+| `missing` | A successful metadata lookup did not return the registered URI from Minoc, Luskan, or both. |
+| `size-mismatch` | Both sizes are available, but their byte counts differ. |
+| `checksum-mismatch` | Both MD5 values are available, but they differ. |
+| `unavailable` | A service request failed, a response was invalid, or required metadata was incomplete. The comparison could not be completed. |
+
+One file can have more than one problem, so problem counts need not add up to
+`registered`. An unavailable lookup is not reported as proof that a file is
+missing.
+
+!!! note "An empty check can succeed"
+
+    If Datatrail returns no registered Minoc files, the command reports
+    `registered: 0` and succeeds. This does not establish that the dataset has
+    been replicated. Inspect its recorded locations with [`ps`](ps.md).
+
+This command reads metadata; it does not download files, recompute checksums from
+their contents, or repair registrations. It checks only the registered Minoc
+URIs supplied by Datatrail, so it does not discover unregistered files or verify
+other storage elements. Matching service metadata is not a fresh integrity check
+of the stored bytes.
+
+## Save a detailed report
+
+```bash
+datatrail verify chime.event.baseband.raw EVENT_ID --json > verification.json
+```
+
+`--json` writes a JSON object on standard output, including on verification
+failure. It contains `scope`, `dataset`, `registered`, `ok`, `summary`, and
+`results`. The category keys use underscores (`size_mismatch` and
+`checksum_mismatch`). Detailed results identify the URI, affected services, or
+the differing Minoc and Luskan values as appropriate.
+
+| Exit status | Meaning |
+| --- | --- |
+| `0` | No missing files, mismatches, or unavailable metadata were reported; this includes an empty registered file list. |
+| `1` | Missing files or metadata mismatches were found, with no unavailable results. |
+| `2` | At least one result was unavailable, even if other files also had mismatches. |
+
+For unavailable results, check service readiness and retry after the underlying
+problem is resolved. For missing files or mismatches, retain the detailed report
+and investigate the dataset with [`ps`](ps.md) and [`scout`](scout.md) before
+requesting corrective work.

{datatrail_cli-1.0.0 → datatrail_cli-1.0.2}/dtcli/pull.py

@@ -4,7 +4,13 @@ import logging
 from os import cpu_count, path
 
 import click
-from requests.exceptions import ConnectionError, SSLError
+from cadcutils.exceptions import (
+    HttpException,
+    InternalServerException,
+    TransferException,
+    UnexpectedException,
+)
+from requests.exceptions import ConnectionError, SSLError, Timeout
 from rich.console import Console
 from rich.prompt import Confirm
 
@@ -149,6 +155,18 @@ Create one using 'cadc-get-cert -u <USERNAME>'.
 """
             )
             return None
+        except (ConnectionError, Timeout, HttpException) as error:
+            # CADC also uses the base HttpException for failed connections/retries.
+            # Preserve its authentication, certificate, and invalid-request errors.
+            if isinstance(error, HttpException) and type(error) not in (
+                HttpException,
+                InternalServerException,
+                TransferException,
+                UnexpectedException,
+            ):
+                raise
+            logger.debug("Unable to query Luskan for download size", exc_info=True)
+            to_download_size = -1
     elif not luskan_up:
         to_download_size = -1
     else:
@@ -161,7 +179,7 @@ Create one using 'cadc-get-cert -u <USERNAME>'.
         f" - {len(files['missing'])} files can be downloaded from minoc.",
         style="yellow",
     )
-    if luskan_up:
+    if to_download_size >= 0:
         console.print(
             f"     - Size to download: {to_download_size:.2f} GB.\n",
             style="yellow",