mastodown 0.1.0__tar.gz

mastodown-0.1.0/.github/copilot-instructions.md

@@ -0,0 +1,28 @@
+# Mastodown Copilot Instructions
+
+## Commands
+
+- Set up the development environment with `uv sync`.
+- Run all tests with `uv run pytest`.
+- Run one test with `uv run pytest tests/test_query.py::QueryObservationTests::test_query_uses_date_range_filters_and_max_entries`.
+- Build documentation, including generated API and CLI reference pages, with `uv run make -C docs html`. The build treats warnings as errors.
+- Check that the committed CLI reference matches `mastodown --help` with `uv run python docs/generate_cli_reference.py --check`.
+
+Python 3.11 or later is required. When using a pip-installed environment rather than uv, omit `uv run`.
+
+## Architecture
+
+- The package uses a `src/` layout. The `mastodown` console command is registered in `pyproject.toml` and calls `mastodown.cli:main`.
+- `cli.py` owns argparse setup and orchestration. Both `query` and `download` use the same query arguments; `run_query()` delegates to `query_obs()`, prints its dataframe, and optionally writes CSV. `download` then asks for confirmation unless `--yes` is supplied.
+- `query.py` translates CLI/library filters into `astroquery.mast.Observations` calls for JWST. It fetches observations, gets and filters their products, joins `target_name` back from the observation table, and returns a pandas dataframe with `target_name` as its first column. Target-acquisition filtering uses the JWST mission metadata interface unless `keep_ta=True`.
+- `download.py` consumes that dataframe and requires its product, proposal, target, and data URI columns. It derives `local_path`, compares it with `<download-dir>/manifest.csv`, downloads only required products, and updates the manifest only with successful downloads. Proposal directories are enabled by default; target directories use `target_directory_name()` to produce filesystem-safe names.
+- Sphinx documentation is in `docs/`. `docs/Makefile` runs `generate_api_reference.py` before HTML builds, while `docs/conf.py` regenerates `cli-reference.md` from `cli.py` at Sphinx startup. `docs/cli-reference.md` is generated output; change the parser or generator rather than editing its content manually.
+
+## Repository Conventions
+
+- Keep the CLI parser, `COMMAND_NAMES`, generated CLI reference, and CLI reference tests aligned whenever commands or options change. Run the freshness check after parser changes.
+- Preserve the shared argument forwarding contract between `query` and `download`: `add_query_arguments()` defines it once, and `run_query()` is the only CLI-to-`query_obs()` bridge.
+- Authentication is intentionally centralized in `authenticate()`: use `MAST_API_TOKEN` when present; only trigger Astroquery's interactive login when `--auth` is set and no environment token exists.
+- Query validation is library-level: date bounds must be supplied together in exact `YYYY-MM-DD` form, end dates are inclusive, and `max_entries` must be positive. Tests mock Astroquery network methods, so new query behavior should remain testable without live MAST access.
+- Treat `manifest.csv` as the download state source of truth. Reconcile missing files already recorded in it, add already-present unrecorded files, and do not record failed downloads. `dry_run=True` must avoid filesystem writes, including the manifest.
+- Tests use `unittest.TestCase` with `unittest.mock` and `pytest` as the runner. Patch dependencies at their use site (for example, `mastodown.query.Observations` or `mastodown.cli.query_obs`) and use temporary directories for download filesystem behavior.

mastodown-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+src/mastho/_version.py
+build/
+dist/
+wheels/
+*.egg-info
+_version.py
+
+# Virtual environments
+.venv
+docs/_build/

mastodown-0.1.0/.python-version

@@ -0,0 +1 @@
+3.14

mastodown-0.1.0/.readthedocs.yaml

@@ -0,0 +1,18 @@
+# Read the Docs configuration file for Mastodown.
+version: 2
+
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+
+python:
+  install:
+    - method: uv
+      command: sync
+      groups:
+        - docs
+
+sphinx:
+  configuration: docs/conf.py
+  fail_on_warning: true

mastodown-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thomas Vandal
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mastodown-0.1.0/PKG-INFO

@@ -0,0 +1,28 @@
+Metadata-Version: 2.5
+Name: mastodown
+Version: 0.1.0
+Summary: Tiny Python MAST Client
+Author-email: Thomas Vandal <thomas.vandal@umontreal.ca>
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: astroquery
+Requires-Dist: pandas
+Description-Content-Type: text/markdown
+
+# Mastodown
+
+Mastodown is a small Python client for querying and downloading data from the
+[MAST](https://mast.stsci.edu/) archive on the command line or from Python.
+
+Read the [documentation](https://mastodown.readthedocs.io/) for installation,
+command-line usage, authentication, and the Python API reference.
+
+## Quick start
+
+```bash
+python -m pip install git+https://github.com/vandalt/mastodown.git
+mastodown query --programs 01200 02473 --calib-level 1 --product-type SCIENCE --extension fits -o products.csv
+```
+
+See the [CLI guide](https://mastodown.readthedocs.io/en/latest/cli.html) for all
+commands and options.

mastodown-0.1.0/README.md

@@ -0,0 +1,17 @@
+# Mastodown
+
+Mastodown is a small Python client for querying and downloading data from the
+[MAST](https://mast.stsci.edu/) archive on the command line or from Python.
+
+Read the [documentation](https://mastodown.readthedocs.io/) for installation,
+command-line usage, authentication, and the Python API reference.
+
+## Quick start
+
+```bash
+python -m pip install git+https://github.com/vandalt/mastodown.git
+mastodown query --programs 01200 02473 --calib-level 1 --product-type SCIENCE --extension fits -o products.csv
+```
+
+See the [CLI guide](https://mastodown.readthedocs.io/en/latest/cli.html) for all
+commands and options.

mastodown-0.1.0/docs/Makefile

@@ -0,0 +1,19 @@
+SPHINXOPTS ?=
+SPHINXBUILD ?= sphinx-build
+PYTHON = python
+SOURCEDIR = .
+BUILDDIR = _build
+
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+api:
+	@$(PYTHON) generate_api_reference.py
+
+html: api
+	@$(SPHINXBUILD) -M html "$(SOURCEDIR)" "$(BUILDDIR)" -W $(SPHINXOPTS) $(O)
+
+.PHONY: help api html Makefile
+
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" -W $(SPHINXOPTS) $(O)

mastodown-0.1.0/docs/api/index.rst

@@ -0,0 +1,9 @@
+API Reference
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   mastodown.cli
+   mastodown.download
+   mastodown.query

mastodown-0.1.0/docs/api/mastodown.cli.rst

@@ -0,0 +1,8 @@
+mastodown.cli module
+====================
+
+.. automodule:: mastodown.cli
+   :members:
+   :exclude-members: ArgumentParser, DataFrame, InvalidQueryError, MastMissions, Namespace, Observations, Path, Series, Time, concat, datetime, environ, read_csv, sub, timedelta
+   :show-inheritance:
+   :undoc-members:

mastodown-0.1.0/docs/api/mastodown.download.rst

@@ -0,0 +1,8 @@
+mastodown.download module
+=========================
+
+.. automodule:: mastodown.download
+   :members:
+   :exclude-members: ArgumentParser, DataFrame, InvalidQueryError, MastMissions, Namespace, Observations, Path, Series, Time, concat, datetime, environ, read_csv, sub, timedelta
+   :show-inheritance:
+   :undoc-members:

mastodown-0.1.0/docs/api/mastodown.query.rst

@@ -0,0 +1,8 @@
+mastodown.query module
+======================
+
+.. automodule:: mastodown.query
+   :members:
+   :exclude-members: ArgumentParser, DataFrame, InvalidQueryError, MastMissions, Namespace, Observations, Path, Series, Time, concat, datetime, environ, read_csv, sub, timedelta
+   :show-inheritance:
+   :undoc-members:

mastodown-0.1.0/docs/cli-recipes.md

@@ -0,0 +1,55 @@
+# Command line recipes
+
+For complete option details, see the {doc}`command reference <cli-reference>`.
+
+## Viewing MAST metadata columns
+
+List metadata for all available MAST observation columns:
+
+```bash
+mastodown get-meta
+```
+
+## Querying products
+
+Query JWST products and save the resulting table to CSV:
+
+```bash
+mastodown query --programs 01200 02473 --calib-level 1 --product-type SCIENCE --extension fits -o products.csv
+```
+
+Filter options accept one or more space-separated values. Use `--keep-ta` to
+retain target-acquisition observations and `--verbose` to print observation
+details. Use `--filters` for MAST instrument filters,
+`--date-range START_DATE END_DATE` for inclusive `YYYY-MM-DD` observation
+dates, and `--max-entries` to limit returned observations.
+
+## Downloading products
+
+Query products and download them after confirmation:
+
+```bash
+mastodown download --programs 01200 02473 --download-dir data --dry-run
+```
+
+The download command prompts after showing the query result; press Enter to
+continue, enter `n` to cancel, or pass `-y` / `--yes` to skip the prompt. Use
+`--query-output products.csv` to save the query result, and
+`--no-proposal-subdir` to save files directly in the download directory. Use
+`--target-subdir` to place each product under its target name, after the
+optional proposal directory.
+
+(proprietary-data-without-an-environment-variable)=
+## Accessing proprietary data without an environment variable
+
+As described in the {ref}`installation docs <accessing-proprietary-data>`,
+the simplest way to access proprietary data is by storing your MAST API token in an environment variable.
+
+Alternatively, you can pass `--auth` to securely enter a token through Astroquery's prompt:
+
+```bash
+mastodown query --programs 01200 --auth
+```
+
+The environment token takes precedence over `--auth`.
+Do not pass tokens directly on the command line.

mastodown-0.1.0/docs/cli-reference.md

@@ -0,0 +1,127 @@
+# Command reference
+
+This reference is generated from the CLI's `--help` output. Do not edit it manually.
+
+## `mastodown`
+
+```console
+$ mastodown --help
+usage: mastodown [-h] {get-meta,query,download} ...
+
+Tiny Python MAST client.
+
+positional arguments:
+  {get-meta,query,download}
+    get-meta            List metadata for all MAST observation columns. Should
+                        match the web portal interface.
+    query               Query the MAST observations portal.
+    download            Query the MAST observations portal and download the
+                        matching products.
+
+options:
+  -h, --help            show this help message and exit
+```
+
+## `mastodown get-meta`
+
+```console
+$ mastodown get-meta --help
+usage: mastodown get-meta [-h]
+
+List metadata for all MAST observation columns. Should match the web portal
+interface.
+
+options:
+  -h, --help  show this help message and exit
+```
+
+## `mastodown query`
+
+```console
+$ mastodown query --help
+usage: mastodown query [-h] [--programs PROGRAMS [PROGRAMS ...]]
+                       [--calib-level CALIB_LEVEL [CALIB_LEVEL ...]]
+                       [--product-type PRODUCT_TYPE [PRODUCT_TYPE ...]]
+                       [--extension EXTENSION [EXTENSION ...]]
+                       [--filters FILTERS [FILTERS ...]]
+                       [--date-range START_DATE END_DATE]
+                       [--max-entries MAX_ENTRIES] [--keep-ta] [--verbose]
+                       [--auth] [-o QUERY_OUTPUT]
+
+Query the MAST observations portal.
+
+options:
+  -h, --help            show this help message and exit
+  --programs PROGRAMS [PROGRAMS ...]
+                        Proposal IDs to query.
+  --calib-level CALIB_LEVEL [CALIB_LEVEL ...]
+                        Calibration levels of products to include.
+  --product-type PRODUCT_TYPE [PRODUCT_TYPE ...]
+                        Product types to include.
+  --extension EXTENSION [EXTENSION ...]
+                        File extensions to include.
+  --filters FILTERS [FILTERS ...]
+                        MAST instrument filters to include.
+  --date-range START_DATE END_DATE
+                        Inclusive observation dates in YYYY-MM-DD format.
+  --max-entries MAX_ENTRIES
+                        Maximum number of observations to return.
+  --keep-ta             Keep target-acquisition observations.
+  --verbose             Print matching observations and product summary
+                        details.
+  --auth                Prompt securely for a MAST API token when
+                        MAST_API_TOKEN is not set.
+  -o, --output QUERY_OUTPUT
+                        Write the resulting dataframe to this CSV path.
+```
+
+## `mastodown download`
+
+```console
+$ mastodown download --help
+usage: mastodown download [-h] [--programs PROGRAMS [PROGRAMS ...]]
+                          [--calib-level CALIB_LEVEL [CALIB_LEVEL ...]]
+                          [--product-type PRODUCT_TYPE [PRODUCT_TYPE ...]]
+                          [--extension EXTENSION [EXTENSION ...]]
+                          [--filters FILTERS [FILTERS ...]]
+                          [--date-range START_DATE END_DATE]
+                          [--max-entries MAX_ENTRIES] [--keep-ta] [--verbose]
+                          [--auth] [--query-output QUERY_OUTPUT]
+                          [--download-dir DOWNLOAD_DIR] [--overwrite]
+                          [--dry-run] [--no-proposal-subdir] [--target-subdir]
+                          [-y]
+
+Query the MAST observations portal and download the matching products.
+
+options:
+  -h, --help            show this help message and exit
+  --programs PROGRAMS [PROGRAMS ...]
+                        Proposal IDs to query.
+  --calib-level CALIB_LEVEL [CALIB_LEVEL ...]
+                        Calibration levels of products to include.
+  --product-type PRODUCT_TYPE [PRODUCT_TYPE ...]
+                        Product types to include.
+  --extension EXTENSION [EXTENSION ...]
+                        File extensions to include.
+  --filters FILTERS [FILTERS ...]
+                        MAST instrument filters to include.
+  --date-range START_DATE END_DATE
+                        Inclusive observation dates in YYYY-MM-DD format.
+  --max-entries MAX_ENTRIES
+                        Maximum number of observations to return.
+  --keep-ta             Keep target-acquisition observations.
+  --verbose             Print matching observations and product summary
+                        details.
+  --auth                Prompt securely for a MAST API token when
+                        MAST_API_TOKEN is not set.
+  --query-output QUERY_OUTPUT
+                        Write the resulting dataframe to this CSV path.
+  --download-dir DOWNLOAD_DIR
+                        Directory in which to save downloaded products.
+  --overwrite           Download products even when they already exist.
+  --dry-run             List planned downloads without writing files.
+  --no-proposal-subdir  Save products directly in the download directory.
+  --target-subdir       Group downloaded products into target-name
+                        subdirectories.
+  -y, --yes             Download without prompting for confirmation.
+```

mastodown-0.1.0/docs/conf.py

@@ -0,0 +1,39 @@
+"""Configuration for the Mastodown Sphinx documentation."""
+
+from pathlib import Path
+import sys
+
+DOCS_DIR = Path(__file__).resolve().parent
+sys.path.insert(0, str(DOCS_DIR))
+sys.path.insert(0, str(DOCS_DIR.parent / "src"))
+
+project = "Mastodown"
+copyright = "2026, Thomas Vandal"
+author = "Thomas Vandal"
+
+extensions = [
+    "myst_parser",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.viewcode",
+    "sphinx_autodoc_typehints",
+]
+
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".md": "markdown",
+}
+
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+html_theme = "furo"
+
+
+def generate_cli_reference(_: object) -> None:
+    """Refresh generated CLI help before Sphinx reads source files."""
+    from generate_cli_reference import write_reference
+
+    write_reference()
+
+
+def setup(app: object) -> None:
+    """Register documentation build hooks."""
+    app.connect("builder-inited", generate_cli_reference)

mastodown-0.1.0/docs/generate_api_reference.py

@@ -0,0 +1,71 @@
+"""Generate API reference pages from the Mastodown package."""
+
+from pathlib import Path
+import subprocess
+import sys
+
+DOCS_DIR = Path(__file__).resolve().parent
+PROJECT_DIR = DOCS_DIR.parent
+API_DIR = DOCS_DIR / "api"
+PACKAGE_DIR = PROJECT_DIR / "src" / "mastodown"
+INDEX_PATH = API_DIR / "index.rst"
+
+
+def main() -> None:
+    """Generate module pages and an API-reference toctree."""
+    API_DIR.mkdir(exist_ok=True)
+
+    for path in API_DIR.glob("mastodown*.rst"):
+        path.unlink()
+
+    subprocess.run(
+        [
+            sys.executable,
+            "-m",
+            "sphinx.ext.apidoc",
+            "--separate",
+            "--no-toc",
+            "--force",
+            "-o",
+            str(API_DIR),
+            str(PACKAGE_DIR),
+        ],
+        check=True,
+    )
+
+    # The package overview duplicates the flat API index's role and introduces
+    # duplicate toctree references for each module.
+    (API_DIR / "mastodown.rst").unlink()
+    modules = sorted(path.stem for path in API_DIR.glob("mastodown.*.rst"))
+    for module in modules:
+        exclude_imported_members(API_DIR / f"{module}.rst")
+    INDEX_PATH.write_text(
+        "API Reference\n"
+        "=============\n"
+        "\n"
+        ".. toctree::\n"
+        "   :maxdepth: 1\n"
+        "\n"
+        + "".join(f"   {module}\n" for module in modules),
+        encoding="utf-8",
+    )
+
+
+def exclude_imported_members(path: Path) -> None:
+    """Keep autodoc focused on Mastodown's public objects."""
+    content = path.read_text(encoding="utf-8")
+    content = content.replace(
+        "   :members:\n",
+        (
+            "   :members:\n"
+            "   :exclude-members: ArgumentParser, DataFrame, InvalidQueryError, "
+            "MastMissions, Namespace, Observations, Path, Series, Time, concat, "
+            "datetime, environ, read_csv, sub, timedelta\n"
+        ),
+        1,
+    )
+    path.write_text(content, encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

mastodown-0.1.0/docs/generate_cli_reference.py

@@ -0,0 +1,76 @@
+"""Generate the command-line reference from Mastodown's argument parser."""
+
+from argparse import ArgumentParser
+from pathlib import Path
+import re
+import sys
+
+DOCS_DIR = Path(__file__).resolve().parent
+PROJECT_DIR = DOCS_DIR.parent
+REFERENCE_PATH = DOCS_DIR / "cli-reference.md"
+ANSI_ESCAPE_PATTERN = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]")
+
+sys.path.insert(0, str(PROJECT_DIR / "src"))
+
+from mastodown.cli import COMMAND_NAMES, create_parser
+
+
+def format_command_help(parser: ArgumentParser, command: str | None = None) -> str:
+    """Return formatted help for the root parser or one subcommand."""
+    if command is None:
+        return ANSI_ESCAPE_PATTERN.sub("", parser.format_help()).rstrip()
+
+    for action in parser._actions:
+        choices = getattr(action, "choices", None)
+        if choices is not None and command in choices:
+            return ANSI_ESCAPE_PATTERN.sub("", choices[command].format_help()).rstrip()
+
+    msg = f"Command {command!r} is not registered with the argument parser."
+    raise ValueError(msg)
+
+
+def render_reference() -> str:
+    """Render the complete command reference as MyST Markdown."""
+    parser = create_parser()
+    sections = [
+        "# Command reference",
+        "",
+        "This reference is generated from the CLI's `--help` output. Do not edit it manually.",
+    ]
+    for command in (None, *COMMAND_NAMES):
+        invocation = "mastodown" if command is None else f"mastodown {command}"
+        sections.extend(
+            [
+                "",
+                f"## `{invocation}`",
+                "",
+                "```console",
+                f"$ {invocation} --help",
+                format_command_help(parser, command),
+                "```",
+            ]
+        )
+    return "\n".join(sections) + "\n"
+
+
+def write_reference(*, check: bool = False) -> None:
+    """Write the reference, or fail when the committed version is stale."""
+    content = render_reference()
+    if check:
+        if not REFERENCE_PATH.is_file() or REFERENCE_PATH.read_text(
+            encoding="utf-8"
+        ) != content:
+            msg = f"{REFERENCE_PATH.relative_to(PROJECT_DIR)} is out of date."
+            raise SystemExit(msg)
+        return
+
+    REFERENCE_PATH.write_text(content, encoding="utf-8")
+
+
+def main() -> None:
+    """Generate the reference, accepting a freshness-check mode."""
+    write_reference(check="--check" in sys.argv[1:])
+
+
+if __name__ == "__main__":
+    main()