rubin-sim 2.6.1.dev2__py3-none-any.whl

docs/vseqarchive.rst

@@ -0,0 +1,225 @@
+.. py:currentmodule:: rubin_sim.sim_archive.vseqarchive
+
+.. _vseqarchivedemo:
+
+===========================================================================
+Example of running a simulation and adding it to the visit sequence archive
+===========================================================================
+
+Introduction
+============
+
+The precess used in this demonstration closely follows the bash script used to run prenight simuluations.
+It has three major sections:
+
+1. Prepare the environment.
+2. Run the simulation, saving files to local disk as we go.
+3. Add any date we want to keep to the archive.
+
+Preparing the environment
+=========================
+
+In this demonstration, we will start with the lates tagged LSST stack available from ``cvmfs``.
+(It is tested at the USDF, but should work anywhere ``cvmfs`` is installed.)
+Find the version we want, and activate the ``conda`` environment::
+
+    LATEST_TAGGED_STACK=$(
+        find /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib \
+            -maxdepth 1 \
+            -regex '.*/v[0-9]+\.[0-9]+\.[0-9]+' \
+            -printf "%f\n" |
+        sort -V |
+        tail -1
+    )
+    source /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib/${LATEST_TAGGED_STACK}/loadLSST-ext.sh
+
+Next we install specific versions of packages in a separate location so we have full control for just this execution::
+
+    SIM_PACKAGE_DIR=$(mktemp -d --suffix=.opsim_packages)
+    export PYTHONPATH=${SIM_PACKAGE_DIR}:${PYTHONPATH}
+    export PATH=${SIM_PACKAGE_DIR}/bin:${PATH}
+
+Set the actual versions we want. These will usually be tags, but can be any github reference (e.g. tags, branches, commits)::
+
+
+    RUBIN_SCHEDULER_REFERENCE="v3.18.1"
+    RUBIN_SIM_REFERENCE="tickets/SP-2709"
+    SCHEDVIEW_REFERENCE="tickets/SP-2167"
+    TS_FBS_UTILS_REFERENCE="v0.18.0"
+    LSST_SURVEY_SIM_REFERENCE="v0.2.0"
+    RUBIN_NIGHTS_REFERENCE="v0.7.0"
+
+Note that the ``obs_version_at_time`` command provided by ``schedview`` will query the EFD for the latest version being used (for some of these packages).
+You can also use the github API and bash tools to get the highest tag for a repository, for example::
+
+    SOMEREPO_LATEST_REFERENCE=$(
+        curl -s https://api.github.com/repos/lsst/somerepo/tags |
+        jq -r '.[].name' |
+        egrep '^v[0-9]+\.[0-9]+\.[0-9]+$' |
+        sort -V |
+        tail -1
+    )
+
+Now actually do the install of all requested versions.
+Pick up the current version of ``lsst-resources`` along the way.::
+
+    pip install --no-deps --target=${SIM_PACKAGE_DIR} \
+        git+https://github.com/lsst/rubin_sim.git@${RUBIN_SIM_REFERENCE} \
+        git+https://github.com/lsst/schedview.git@${SCHEDVIEW_REFERENCE} \
+        git+https://github.com/lsst-ts/ts_fbs_utils.git@${TS_FBS_UTILS_REFERENCE} \
+        git+https://github.com/lsst-sims/lsst_survey_sim.git@${LSST_SURVEY_SIM_REFERENCE} \
+        git+https://github.com/lsst-sims/rubin_nights.git@${RUBIN_NIGHTS_REFERENCE} \
+        git+https://github.com/lsst/rubin_scheduler.git@${RUBIN_SCHEDULER_REFERENCE} \
+        lsst-resources
+
+Running the simulation
+======================
+
+Begin by creating a directory in which to work::
+
+    WORK_DIR=${HOME}/devel/my_sample_opsim
+    mkdir ${WORK_DIR}
+    cd $WORK_DIR
+
+Next we need is the scheduler configuration we're going to use.
+We want the archive to track this, so we want it to be at a URL somewhere that it would be useful for the archive to reference.
+Let's use something from github::
+
+    TS_CONFIG_SCHEDULER_REFERENCE="develop"
+    SCHED_CONFIG_FNAME="ts_config_scheduler/Scheduler/feature_scheduler/maintel/fbs_config_lsst_survey.py"
+    echo "Using ts_config_scheduler ${SCHED_CONFIG_FNAME} from ${TS_CONFIG_SCHEDULER_REFERENCE}"
+    git clone https://github.com/lsst-ts/ts_config_scheduler
+    cd ts_config_scheduler
+    git checkout "${TS_CONFIG_SCHEDULER_REFERENCE}"
+    cd ${WORK_DIR}
+
+We need to set the ``dayobs`` on which the simulation should start::
+
+    DAYOBS="20260101"
+
+Get the pre-existing visits from consdb::
+
+    USDF_ACCESS_TOKEN_PATH=~/.lsst/usdf_access_token
+    fetch_lsst_visits ${DAYOBS} completed_visits.db ${USDF_ACCESS_TOKEN_PATH}
+
+Note that you will need a USDF access token.
+
+Create a pickle of the scheduler to run, with completed visits pre-loaded, and write it to a pickle, ``scheduler.p``::
+
+    make_lsst_scheduler scheduler.p --opsim completed_visits.db --config-script ${SCHED_CONFIG_FNAME}
+
+Create a model observatory and write it to pickle file, ``observatory.p``::
+
+    make_model_observatory observatory.p
+
+Run the simulation, writing the result to files in the local directory::
+
+    RESULTS_DIR="."
+    OPSIMRUN="prenight_nominal_$(date --iso=s)"
+    run_lsst_sim scheduler.p observatory.p "" ${DAYOBS} 1 "${OPSIMRUN}" --keep_rewards --results ${RESULTS_DIR}
+
+There will now be an assortment of output files in the current working directory.
+
+Adding entries to the visit sequence archive
+============================================
+
+Setting up the environment for the archive
+------------------------------------------
+
+The visit sequence archive has two components: a ``postgresql`` dataabase that tracks metadata, and a resource (directory or S3 bucket) in which the visits and other file content can be saved.
+
+Begin by configuring the environment variables that the tools use to find the metadata database::
+
+    export VSARCHIVE_PGDATABASE="opsim_log"
+    export VSARCHIVE_PGHOST="usdf-maf-visit-seq-archive-tx.sdf.slac.stanford.edu"
+    export VSARCHIVE_PGUSER="tester"
+    export VSARCHIVE_PGSCHEMA="test"
+
+Note that we have set ``VSARCHIVE_PGSCHEMA`` to ``test``, so metadata will be saved in a test schema.
+The production schema is ``vsmd``.
+
+Now, create a root for a demonstration resource in which to save the data itself::
+
+    mkdir ${HOME}/devel/test_visitseq_archive
+    export ARCHIVE_URL="file:///${HOME}/devel/test_visitseq_archive"
+
+Make a simple utility shell function
+------------------------------------
+
+For the demonstration, it will be useful to query the metadata databas, but specifying all the connection parameters will be inconvenient.
+One option would be to set the environment variables ``psql`` uses (``PGDATABASE``, ``PGHOST``, ``PGUSER``), but setting these might confuse other utilities that query different postgresql databases (like ``consdb``), so lets make a utility that just sets them for one command::
+
+    vseq-psql() {
+        PGDATABASE=${VSARCHIVE_PGDATABASE} \
+        PGHOST=${VSARCHIVE_PGHOST} \
+        PGUSER=${VSARCHIVE_PGUSER} \
+        psql "$@"
+    }
+
+This results in a now shell command, ``vseq-psql``, that works just like ``psql``, but where the user does not need to specify connection parameters.
+
+Adding an entry for pre-existing visits to the archive
+------------------------------------------------------
+
+We need to add two entries to the visit sequence archive, one for the pre-existing sequences of visits queried from consdb, and the other for the sequence generated by the simulation.
+
+Begin by creating an entry for the pre-existing visits::
+
+    COMPLETED=$(vseqarchive record-visitseq-metadata \
+        completed \
+        completed_visits.db \
+        "Consdb query through 2025-10-31" \
+        --first_day_obs 20250620 \
+        --last_day_obs 20251031)
+
+The ``COMPLETED`` UUID will now contain a reference for the sequence of visits returned from the consdb.
+This command only adds an entry to the metadata, it does not save the visits themselves in the archive.
+We can skip saving the visits themselves, if we are okay with relying on using consdb to recreate it.
+(If you want to be sure, you can save them in the same way as simulated visits are saved below.)
+
+Adding the simulation
+---------------------
+
+Now create an entry for the simulated visits::
+
+    SIM_UUID=$(vseqarchive record-visitseq-metadata \
+        simulations \
+        opsim.db \
+        "Test pre-night simulation 1" \
+        --first_day_obs 20261201 \
+        --last_day_obs 20261202
+        )
+
+This command only stored the bare minimum of metadata, and did not save the visits or any of the files in the archive.
+We can now add additional metadata to the database::
+
+    vseqarchive update-visitseq-metadata ${SIM_UUID} parent_visitseq_uuid ${COMPLETED}
+    vseqarchive update-visitseq-metadata ${SIM_UUID} parent_last_day_obs 2025-12-31
+
+    SCHEDULER_VERSION=$(python -c "import rubin_scheduler; print(rubin_scheduler.__version__)")
+    vseqarchive update-visitseq-metadata ${SIM_UUID} scheduler_version "${SCHEDULER_VERSION}"
+
+and the visits and rewards to the resource::
+
+    vseqarchive archive-file ${SIM_UUID} opsim.db visits --archive-base ${ARCHIVE_URL}
+    vseqarchive archive-file ${SIM_UUID} rewards.h5 rewards --archive-base ${ARCHIVE_URL}
+
+We can alse add tags and comments to the metadata database::
+
+    vseqarchive tag ${SIM_UUID} test prenight nominal
+    vseqarchive comment ${SIM_UUID} "Just a test prenight"
+
+Another option is to save specification for the ``conda`` environment::
+
+    CONDA_HASH=$(vseqarchive record-conda-env)
+    vseqarchive update-visitseq-metadata ${SIM_UUID} conda_env_sha256 ${CONDA_HASH}
+
+Finaly, we can save statistics.
+For the basic statistics tools currently available, the visits are needed in an HDF5 file, but in the above instructions we just have an sqlite3 file.
+We can get the HDF5 by asking for the visits from the archive and giving it a destination filename with an ``.h5`` extension::
+
+    vseqarchive get-file ${SIM_UUID} visits visits.h5
+
+and then we can compute the statistics on our columns of interest and add them to the metadata database::
+
+    vseqarchive add-nightly-stats ${SIM_UUID} visits.h5 fieldRA fieldDec azimuth altitude

rubin_sim/__init__.py

@@ -0,0 +1,7 @@
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("rubin_sim")
+except PackageNotFoundError:
+    # package is not installed
+    pass

rubin_sim/data/__init__.py

@@ -0,0 +1 @@
+from .rs_download_data import *  # noqa: F403

rubin_sim/data/rs_download_data.py

@@ -0,0 +1,155 @@
+__all__ = ("get_data_dir", "get_baseline", "data_dict", "rs_download_testing", "rs_download_data")
+
+import argparse
+import glob
+import os
+
+from rubin_scheduler.data import DEFAULT_DATA_URL, download_rubin_data
+from rubin_scheduler.data import get_data_dir as gdd
+
+
+def get_data_dir() -> str:
+    """Wraps rubin_scheduler.data.get_data_dir().
+    Provided here for backwards compatibility.
+
+    Returns
+    -------
+    $RUBIN_SIM_DATA_DIR : `str`
+        Directory containing the necessary data for rubin_sim_data.
+    """
+    return gdd()
+
+
+def get_baseline() -> str:
+    """Get the path to the baseline cadence simulation sqlite file.
+
+    Returns
+    -------
+    file : `str`
+        Path to the baseline cadence simulation sqlite file.
+    """
+    dd = get_data_dir()
+    path = os.path.join(dd, "sim_baseline")
+    file = glob.glob(path + "/*10yrs.db")[0]
+    return file
+
+
+def data_dict() -> dict:
+    """
+    Dictionary containing expected version information for rubin_sim_data
+    data sets, for this version of rubin_sim.
+
+    Returns
+    -------
+    file_dict : `dict`
+        Data bucket filenames dictionary with keys:
+        ``"name"``
+        Data bucket name (`str`).
+        ``"version"``
+        Versioned file name (`str`).
+    """
+    # Note for developers:
+    # to create tar files and follow any sym links, run: e.g.
+    #  ``tar -chvzf maf_may_2021.tgz maf``
+    file_dict = {
+        "maf": "maf_2024_06_13.tgz",
+        "maps": "maps_2022_2_28.tgz",
+        "movingObjects": "movingObjects_2025_07_15.tgz",
+        "orbits": "orbits_2022_3_1.tgz",
+        "orbits_precompute": "orbits_precompute_2023_05_23.tgz",
+        "sim_baseline": "sim_baseline_2025_09_02.tgz",
+        "skybrightness": "skybrightness_2023_09_11.tgz",
+        "throughputs": "throughputs_2025_08_25.tgz",
+        "tests": "tests_2024_04_23.tgz",
+    }
+    return file_dict
+
+
+def rs_download_testing() -> None:
+    """Convenience function for github actions, to download only a subset
+    of data.
+
+    Contains maf, maps, movingObjects, sim_baseline, skybrightness,
+    throughputs and tests.
+    For users who don't care about moving objects, this is likely sufficient.
+    """
+    all_files = data_dict()
+    keys = ["maf", "maps", "movingObjects", "sim_baseline", "skybrightness", "throughputs", "tests"]
+    subset = {}
+    for k in keys:
+        subset[k] = all_files[k]
+
+    download_rubin_data(
+        subset,
+        dirs=None,
+        print_versions_only=False,
+        update=True,
+        force=False,
+        url_base=DEFAULT_DATA_URL,
+        tdqm_disable=True,
+    )
+
+
+def rs_download_data() -> None:
+    """Utility to download necessary data for rubin_sim.
+
+    Wrapper around rubin_scheduler.scheduler_download_data,
+    but downloading the data files specified by rubin_sim.
+    """
+
+    files = data_dict()
+    parser = argparse.ArgumentParser(description="Download data files for rubin_sim package")
+    parser.add_argument(
+        "--versions",
+        dest="versions",
+        default=False,
+        action="store_true",
+        help="Report expected versions, then quit",
+    )
+    parser.add_argument(
+        "-d",
+        "--dirs",
+        type=str,
+        default=None,
+        help="Comma-separated list of directories to download",
+    )
+    parser.add_argument(
+        "-f",
+        "--force",
+        dest="force",
+        default=False,
+        action="store_true",
+        help="Force re-download of data directory(ies)",
+    )
+    parser.add_argument(
+        "--url_base",
+        type=str,
+        default=DEFAULT_DATA_URL,
+        help="Root URL of download location",
+    )
+    parser.add_argument(
+        "--tdqm_disable",
+        dest="tdqm_disable",
+        default=False,
+        action="store_true",
+        help="Turn off tdqm progress bar",
+    )
+    parser.add_argument(
+        "--update",
+        dest="update",
+        default=False,
+        action="store_true",
+        help="Update versions of data on disk to match current",
+    )
+
+    args = parser.parse_args()
+
+    download_rubin_data(
+        files,
+        dirs=args.dirs,
+        print_versions_only=args.versions,
+        update=args.update,
+        force=args.force,
+        url_base=args.url_base,
+        tdqm_disable=args.tdqm_disable,
+    )

rubin_sim/maf/__init__.py

@@ -0,0 +1,34 @@
+#
+# LSST Data Management System
+# Copyright 2008, 2009, 2010 LSST Corporation.
+#
+# This product includes software developed by the
+# LSST Project (http://www.lsst.org/).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the LSST License Statement and
+# the GNU General Public License along with this program.  If not,
+# see <http://www.lsstcorp.org/LegalNotices/>.
+#
+"""Python interface to the metrics analysis framework."""
+from .batches import *
+from .db import *
+from .maf_contrib import *
+from .maps import *
+from .metric_bundles import *
+from .metrics import *
+from .plots import *
+from .run_comparison import *
+from .slicers import *
+from .stackers import *
+from .utils import *
+from .web import *

rubin_sim/maf/batches/__init__.py

@@ -0,0 +1,18 @@
+from .altaz_batch import *
+from .col_map_dict import *
+from .common import *
+from .ddf_batch import *
+from .filterchange_batch import *
+from .glance_batch import *
+from .hourglass_batch import *
+from .info_batch import *
+from .metadata_batch import *
+from .moving_objects_batch import *
+from .openshutter_batch import *
+from .radar_limited import *
+from .science_radar_batch import *
+from .skycoverage import *
+from .slew_batch import *
+from .srd_batch import *
+from .time_batch import *
+from .visitdepth_batch import *

rubin_sim/maf/batches/altaz_batch.py

@@ -0,0 +1,161 @@
+__all__ = ("altazHealpix", "altazLambert")
+
+import rubin_sim.maf.metric_bundles as mb
+import rubin_sim.maf.metrics as metrics
+import rubin_sim.maf.plots as plots
+import rubin_sim.maf.slicers as slicers
+
+from .col_map_dict import col_map_dict
+from .common import filter_list
+
+
+def basicSetup(metric_name, colmap=None, nside=64):
+    if colmap is None:
+        colmap = col_map_dict()
+
+    slicer = slicers.HealpixSlicer(
+        nside=nside,
+        lat_col=colmap["alt"],
+        lon_col=colmap["az"],
+        lat_lon_deg=colmap["raDecDeg"],
+        use_cache=False,
+    )
+    metric = metrics.CountMetric(colmap["mjd"], metric_name=metric_name)
+
+    return colmap, slicer, metric
+
+
+def altazHealpix(
+    colmap=None,
+    run_name="run name",
+    extraSql=None,
+    extraInfoLabel=None,
+    metric_name="NVisits Alt/Az",
+):
+    """
+    Generate a set of metrics measuring the number visits as a function
+    of alt/az plotted on a HealpixSkyMap.
+
+    Parameters
+    ----------
+    colmap : `dict`, optional
+        A dictionary with a mapping of column names.
+    run_name : `str`, optional
+        The name of the simulated survey.
+    extraSql : `str`, optional
+        Additional constraint to add to any sql constraints.
+    extraInfoLabel : `str`, optional
+        Additional info_label to add before any below (i.e. "WFD").
+    metric_name : `str`, optional
+        Unique name to assign to metric
+
+    Returns
+    -------
+    metric_bundleDict : `dict` of {`str`: `maf.MetricBundle`}
+    """
+
+    colmap, slicer, metric = basicSetup(metric_name=metric_name, colmap=colmap)
+
+    # Set up basic all and per filter sql constraints.
+    filterlist, colors, orders, sqls, info_label = filter_list(
+        all=True, extra_sql=extraSql, extra_info_label=extraInfoLabel
+    )
+
+    bundleList = []
+
+    plotDict = {"rot": (90, 90, 90), "flip": "geo"}
+    plotFunc = plots.HealpixSkyMap()
+
+    for f in filterlist:
+        if f == "all":
+            subgroup = "All Observations"
+        else:
+            subgroup = "Per filter"
+        displayDict = {
+            "group": "Alt/Az",
+            "order": orders[f],
+            "subgroup": subgroup,
+            "caption": "Pointing History on the alt-az sky (zenith center) for filter %s" % f,
+        }
+        bundle = mb.MetricBundle(
+            metric,
+            slicer,
+            sqls[f],
+            plot_dict=plotDict,
+            run_name=run_name,
+            info_label=info_label[f],
+            plot_funcs=[plotFunc],
+            display_dict=displayDict,
+        )
+        bundleList.append(bundle)
+
+    for b in bundleList:
+        b.set_run_name(run_name)
+    return mb.make_bundles_dict_from_list(bundleList)
+
+
+def altazLambert(
+    colmap=None,
+    runName="opsim",
+    extraSql=None,
+    extraInfoLabel=None,
+    metric_name="Nvisits as function of Alt/Az",
+):
+    """
+    Generate a set of metrics measuring the number visits as a
+    function of alt/az plotted on a LambertSkyMap.
+
+    Parameters
+    ----------
+    colmap : `dict`, optional
+        A dictionary with a mapping of column names.
+    runName : `str`, optional
+        The name of the simulated survey.
+    extraSql : `str`, optional
+        Additional constraint to add to any sql constraints.
+    extraInfoLabel : `str`, optional
+        Additional info_label to add before any below (i.e. "WFD").
+    metric_name : `str`, optional
+        Unique name to assign to metric
+
+    Returns
+    -------
+    metric_bundleDict : `dict` of {`str`: `maf.MetricBundle`}
+    """
+
+    colmap, slicer, metric = basicSetup(metric_name=metric_name, colmap=colmap)
+
+    # Set up basic all and per filter sql constraints.
+    filterlist, colors, orders, sqls, info_label = filter_list(
+        all=True, extra_sql=extraSql, extra_info_label=extraInfoLabel
+    )
+
+    bundleList = []
+
+    plotFunc = plots.LambertSkyMap()
+
+    for f in filterlist:
+        if f == "all":
+            subgroup = "All Observations"
+        else:
+            subgroup = "Per filter"
+        displayDict = {
+            "group": "Alt/Az",
+            "order": orders[f],
+            "subgroup": subgroup,
+            "caption": "Alt/Az pointing distribution for filter %s" % f,
+        }
+        bundle = mb.MetricBundle(
+            metric,
+            slicer,
+            sqls[f],
+            run_name=runName,
+            info_label=info_label[f],
+            plot_funcs=[plotFunc],
+            display_dict=displayDict,
+        )
+        bundleList.append(bundle)
+
+    for b in bundleList:
+        b.set_run_name(runName)
+    return mb.make_bundles_dict_from_list(bundleList)