lstosa 0.10.4__tar.gz → 0.10.6__tar.gz

{lstosa-0.10.4 → lstosa-0.10.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lstosa
-Version: 0.10.4
+Version: 0.10.6
 Summary: Onsite analysis pipeline for the CTA LST-1
 Author: María Láinez, José Enrique Ruiz, Lab Saha, Andrés Baquero, José Luis Contreras, Maximilian Linhoff
 Author-email: Daniel Morcuende <dmorcuen@ucm.es>
@@ -54,20 +54,20 @@ Requires-Dist: lstosa[dev,doc,test]; extra == "all"
   [![ci](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml)
   [![Documentation Status](https://readthedocs.org/projects/lstosa/badge/?version=latest)](https://lstosa.readthedocs.io/en/latest/?badge=latest)
   [![coverage](https://codecov.io/gh/cta-observatory/lstosa/branch/main/graph/badge.svg?token=Zjk1U1ytaG)](https://codecov.io/gh/cta-observatory/lstosa)
-  [![quality](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://www.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=cta-observatory/lstosa&amp;utm_campaign=Badge_Grade)
+  [![Codacy Badge](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://app.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
   [![pypi](https://img.shields.io/pypi/v/lstosa)](https://pypi.org/project/lstosa/)
   [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6567234.svg)](https://doi.org/10.5281/zenodo.6567234)
 
 
-Prototype onsite processing pipeline for the Large Size Telescope prototype (LST-1) of [CTA](https://www.cta-observatory.org/) (Cherenkov Telescope Array) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite center at ORM (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance.
-Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release:
+Onsite processing pipeline for the Large-Sized Telescope prototype (LST-1) of [CTAO](https://www.cta-observatory.org/) (Cherenkov Telescope Array Observatory) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite data center at Observatorio Roque de los Muchachos (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance. Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release.
+
  - Code: <https://github.com/cta-observatory/lstosa>
  - Docs: <https://lstosa.readthedocs.io/>
  - License: [BSD-3-Clause](https://github.com/cta-observatory/lstosa/blob/main/LICENSE)
 
 ## Install
 We recommend using an isolated conda environment.
- - Install miniconda first.
+ - Install mamba/miniconda first.
  - Clone the repository, create and activate the conda environment using the `environment.yml` file:
 
     ```bash
@@ -77,7 +77,7 @@ We recommend using an isolated conda environment.
     conda activate osa
     ```
 
-Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`.
+Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`. To install test, docs dependencies use `pip install -e .[test]`, `pip install -e .[doc]` or simply `pip install -e .[all]`
 
 In case you want to install the lstchain development version instead of a fixed tag, you can run inside the `osa` environment:
 

{lstosa-0.10.4 → lstosa-0.10.6}/README.md

@@ -3,20 +3,20 @@
   [![ci](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml)
   [![Documentation Status](https://readthedocs.org/projects/lstosa/badge/?version=latest)](https://lstosa.readthedocs.io/en/latest/?badge=latest)
   [![coverage](https://codecov.io/gh/cta-observatory/lstosa/branch/main/graph/badge.svg?token=Zjk1U1ytaG)](https://codecov.io/gh/cta-observatory/lstosa)
-  [![quality](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://www.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=github.com&utm_medium=referral&utm_content=cta-observatory/lstosa&utm_campaign=Badge_Grade)
+  [![Codacy Badge](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://app.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
   [![pypi](https://img.shields.io/pypi/v/lstosa)](https://pypi.org/project/lstosa/)
   [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6567234.svg)](https://doi.org/10.5281/zenodo.6567234)
 
 
-Prototype onsite processing pipeline for the Large Size Telescope prototype (LST-1) of [CTA](https://www.cta-observatory.org/) (Cherenkov Telescope Array) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite center at ORM (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance.
-Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release:
+Onsite processing pipeline for the Large-Sized Telescope prototype (LST-1) of [CTAO](https://www.cta-observatory.org/) (Cherenkov Telescope Array Observatory) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite data center at Observatorio Roque de los Muchachos (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance. Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release.
+
  - Code: <https://github.com/cta-observatory/lstosa>
  - Docs: <https://lstosa.readthedocs.io/>
  - License: [BSD-3-Clause](https://github.com/cta-observatory/lstosa/blob/main/LICENSE)
 
 ## Install
 We recommend using an isolated conda environment.
- - Install miniconda first.
+ - Install mamba/miniconda first.
  - Clone the repository, create and activate the conda environment using the `environment.yml` file:
 
     ```bash
@@ -26,7 +26,7 @@ We recommend using an isolated conda environment.
     conda activate osa
     ```
 
-Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`.
+Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`. To install test, docs dependencies use `pip install -e .[test]`, `pip install -e .[doc]` or simply `pip install -e .[all]`
 
 In case you want to install the lstchain development version instead of a fixed tag, you can run inside the `osa` environment:
 

{lstosa-0.10.4 → lstosa-0.10.6}/src/lstosa.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lstosa
-Version: 0.10.4
+Version: 0.10.6
 Summary: Onsite analysis pipeline for the CTA LST-1
 Author: María Láinez, José Enrique Ruiz, Lab Saha, Andrés Baquero, José Luis Contreras, Maximilian Linhoff
 Author-email: Daniel Morcuende <dmorcuen@ucm.es>
@@ -54,20 +54,20 @@ Requires-Dist: lstosa[dev,doc,test]; extra == "all"
   [![ci](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml)
   [![Documentation Status](https://readthedocs.org/projects/lstosa/badge/?version=latest)](https://lstosa.readthedocs.io/en/latest/?badge=latest)
   [![coverage](https://codecov.io/gh/cta-observatory/lstosa/branch/main/graph/badge.svg?token=Zjk1U1ytaG)](https://codecov.io/gh/cta-observatory/lstosa)
-  [![quality](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://www.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=cta-observatory/lstosa&amp;utm_campaign=Badge_Grade)
+  [![Codacy Badge](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://app.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
   [![pypi](https://img.shields.io/pypi/v/lstosa)](https://pypi.org/project/lstosa/)
   [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6567234.svg)](https://doi.org/10.5281/zenodo.6567234)
 
 
-Prototype onsite processing pipeline for the Large Size Telescope prototype (LST-1) of [CTA](https://www.cta-observatory.org/) (Cherenkov Telescope Array) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite center at ORM (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance.
-Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release:
+Onsite processing pipeline for the Large-Sized Telescope prototype (LST-1) of [CTAO](https://www.cta-observatory.org/) (Cherenkov Telescope Array Observatory) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite data center at Observatorio Roque de los Muchachos (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance. Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release.
+
  - Code: <https://github.com/cta-observatory/lstosa>
  - Docs: <https://lstosa.readthedocs.io/>
  - License: [BSD-3-Clause](https://github.com/cta-observatory/lstosa/blob/main/LICENSE)
 
 ## Install
 We recommend using an isolated conda environment.
- - Install miniconda first.
+ - Install mamba/miniconda first.
  - Clone the repository, create and activate the conda environment using the `environment.yml` file:
 
     ```bash
@@ -77,7 +77,7 @@ We recommend using an isolated conda environment.
     conda activate osa
     ```
 
-Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`.
+Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`. To install test, docs dependencies use `pip install -e .[test]`, `pip install -e .[doc]` or simply `pip install -e .[all]`
 
 In case you want to install the lstchain development version instead of a fixed tag, you can run inside the `osa` environment:
 

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.10.4'
-__version_tuple__ = version_tuple = (0, 10, 4)
+__version__ = version = '0.10.6'
+__version_tuple__ = version_tuple = (0, 10, 6)

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/configs/options.py

@@ -16,7 +16,6 @@ warning = None
 nocheck = None
 no_dl2 = None
 prod_id = None
-calib_prod_id = None
 dl1_prod_id = None
 dl2_prod_id = None
 append = None

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/configs/sequencer.cfg

@@ -32,7 +32,6 @@ SEQUENCER_WEB_DIR: %(OSA_DIR)s/SequencerWeb
 # To be set by the user. Using PROD-ID will overcome the automatic
 # fetching of lstchain version. Otherwise leave it empty (and without the colon symbol).
 PROD_ID: v0.1.0
-CALIB_PROD_ID: v01
 # Change this to produce a different DL1b or DL2 sub-productions.
 # Otherwise, keep it empty to use the common PROD-ID
 DL1_PROD_ID: tailcut84

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/conftest.py

@@ -26,6 +26,7 @@ from osa.nightsummary.nightsummary import run_summary_table
 from osa.scripts.tests.test_osa_scripts import run_program
 from osa.utils.utils import date_to_dir
 from datetime import datetime
+import lstchain
 
 date = datetime.fromisoformat("2020-01-17")
 nightdir = date_to_dir(date)
@@ -68,16 +69,25 @@ def calibration_base_dir(monitoring_dir):
     return base_dir
 
 
+@pytest.fixture(scope="session")
+def drive_log(monitoring_dir):
+    drive_dir = monitoring_dir / "DrivePositioning"
+    drive_file = drive_dir / "DrivePosition_log_20200117.txt"
+    drive_dir.mkdir(parents=True, exist_ok=True)
+    drive_file.touch()
+    return drive_file
+
+
 @pytest.fixture(scope="session")
 def calibration_dir(calibration_base_dir):
-    directory = calibration_base_dir / "calibration" / nightdir / "pro"
+    directory = calibration_base_dir / "calibration" / nightdir / f"v{lstchain.__version__}"
     directory.mkdir(parents=True, exist_ok=True)
     return directory
 
 
 @pytest.fixture(scope="session")
 def drs4_baseline_dir(calibration_base_dir):
-    directory = calibration_base_dir / "drs4_baseline" / nightdir / "pro"
+    directory = calibration_base_dir / "drs4_baseline" / nightdir / f"v{lstchain.__version__}"
     directory.mkdir(parents=True, exist_ok=True)
     return directory
 

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/paths.py

@@ -1,17 +1,19 @@
 """Handle the paths of the analysis products."""
 
 import logging
+import re
+from datetime import datetime
 from pathlib import Path
 from typing import List
-from datetime import datetime
 
+import lstchain
 from astropy.table import Table
-from lstchain.onsite import find_systematics_correction_file, find_time_calibration_file
-from lstchain.scripts.onsite.onsite_create_calibration_file import search_filter
+from lstchain.onsite import (find_systematics_correction_file,
+                             find_time_calibration_file,
+                             find_filter_wheels)
 
 from osa.configs import options
-from osa.configs.config import DEFAULT_CFG
-from osa.configs.config import cfg
+from osa.configs.config import DEFAULT_CFG, cfg
 from osa.configs.datamodel import Sequence
 from osa.utils import utils
 from osa.utils.logging import myLogger
@@ -19,8 +21,8 @@ from osa.utils.logging import myLogger
 log = myLogger(logging.getLogger(__name__))
 
 __all__ = [
-    "get_calibration_file",
-    "get_drs4_pedestal_file",
+    "get_calibration_filename",
+    "get_drs4_pedestal_filename",
     "pedestal_ids_file_exists",
     "get_run_date",
     "drs4_pedestal_exists",
@@ -40,6 +42,8 @@ __all__ = [
 
 
 DATACHECK_WEB_BASEDIR = Path(cfg.get("WEBSERVER", "DATACHECK"))
+CALIB_BASEDIR = Path(cfg.get("LST1", "CALIB_DIR"))
+DRS4_PEDESTAL_BASEDIR = Path(cfg.get("LST1", "PEDESTAL_DIR"))
 
 
 def analysis_path(tel) -> Path:
@@ -86,18 +90,25 @@ def get_run_date(run_id: int) -> datetime:
     return datetime.strptime(date_string, "%Y-%m-%d")
 
 
-def get_drs4_pedestal_file(run_id: int) -> Path:
+def get_drs4_pedestal_filename(run_id: int, prod_id: str) -> Path:
     """
     Return the drs4 pedestal file corresponding to a given run id
     regardless of the date when the run was taken.
     """
-    drs4_pedestal_dir = Path(cfg.get("LST1", "PEDESTAL_DIR"))
+    if drs4_pedestal_exists(run_id, prod_id):
+        files = search_drs4_files(run_id, prod_id)
+        return files[-1]  # Get the latest production among the major lstchain version
+
     date = utils.date_to_dir(get_run_date(run_id))
-    file = drs4_pedestal_dir / date / f"pro/drs4_pedestal.Run{run_id:05d}.0000.h5"
-    return file.resolve()
+    return (
+        DRS4_PEDESTAL_BASEDIR
+        / date
+        / f"v{lstchain.__version__}/drs4_pedestal.Run{run_id:05d}.0000.h5"
+    ).resolve()
 
 
-def get_calibration_file(run_id: int) -> Path:
+def get_calibration_filename(run_id: int, prod_id: str) -> Path:
+    # sourcery skip: remove-unnecessary-cast
     """
     Return the calibration file corresponding to a given run_id.
 
@@ -109,14 +120,18 @@ def get_calibration_file(run_id: int) -> Path:
     Notes
     -----
     The file path will be built regardless of the date when the run was taken.
-    We follow the naming convention of the calibration files produced by the lstchain script
-    which depends on the filter wheels position. Therefore, we need to try to fetch the filter
-    position from the CaCo database. If the filter position is not found, we assume the default
-    filter position 5-2. Filter information is not available in the database for runs taken before
+    We follow the naming convention of the calibration files produced by the
+    lstchain script which depends on the filter wheels position. Therefore, we
+    need to try to fetch the filter position from the CaCo database. If the
+    filter position is not found, we assume the default filter position 5-2.
+    Filter information is not available in the database for runs taken before
     mid 2021 approx.
     """
 
-    calib_dir = Path(cfg.get("LST1", "CALIB_DIR"))
+    if calibration_file_exists(run_id, prod_id):
+        files = search_calibration_files(run_id, prod_id)
+        return files[-1]  # Get the latest production among the major lstchain version
+
     date = utils.date_to_dir(get_run_date(run_id))
 
     if options.test:  # Run tests avoiding the access to the database
@@ -126,13 +141,16 @@ def get_calibration_file(run_id: int) -> Path:
         mongodb = cfg.get("database", "caco_db")
         try:
             # Cast run_id to int to avoid problems with numpy int64 encoding in MongoDB
-            options.filters = search_filter(int(run_id), mongodb)
+            options.filters = find_filter_wheels(int(run_id), mongodb)
         except IOError:
             log.warning("No filter information found in database. Assuming positions 52.")
             options.filters = 52
 
-    file = calib_dir / date / f"pro/calibration_filters_{options.filters}.Run{run_id:05d}.0000.h5"
-    return file.resolve()
+    return (
+        CALIB_BASEDIR
+        / date
+        / f"v{lstchain.__version__}/calibration_filters_{options.filters}.Run{run_id:05d}.0000.h5"
+    ).resolve()
 
 
 def pedestal_ids_file_exists(run_id: int) -> bool:
@@ -142,16 +160,51 @@ def pedestal_ids_file_exists(run_id: int) -> bool:
     return bool(file_list)
 
 
-def drs4_pedestal_exists(run_id: int) -> bool:
+def drs4_pedestal_exists(run_id: int, prod_id: str) -> bool:
     """Return true if drs4 pedestal file was already produced."""
-    file = get_drs4_pedestal_file(run_id)
-    return file.exists()
+    files = search_drs4_files(run_id, prod_id)
+
+    return len(files) != 0
 
 
-def calibration_file_exists(run_id: int) -> bool:
+def calibration_file_exists(run_id: int, prod_id: str) -> bool:
     """Return true if calibration file was already produced."""
-    file = get_calibration_file(run_id)
-    return file.exists()
+    files = search_calibration_files(run_id, prod_id)
+
+    return len(files) != 0
+
+
+def search_drs4_files(run_id: int, prod_id: str) -> list:
+    """
+    Find DRS4 baseline correction files corresponding to a run ID
+    and major lstchain production version
+    """
+    date = utils.date_to_dir(get_run_date(run_id))
+    version = get_major_version(prod_id)
+    drs4_dir = DRS4_PEDESTAL_BASEDIR / date
+    return sorted(
+        drs4_dir.glob(f"{version}*/drs4_pedestal.Run{run_id:05d}.0000.h5")
+    )
+
+
+def get_major_version(prod_id):
+    """Given a version as vX.Y.Z return vX.Y"""
+    # First check that the given version is in the correct format
+    if prod_id.startswith("v") and len(prod_id.split(".")) >= 2:
+        return re.search(r"\D\d+\.\d+", prod_id)[0]
+
+    raise ValueError("Format of the version is not in the form vW.X.Y.Z")
+
+
+def search_calibration_files(run_id: int, prod_id: str) -> list:
+    """
+    Search charge calibration files corresponding to a run ID and major lstchain production version
+    """
+    date = utils.date_to_dir(get_run_date(run_id))
+    version = get_major_version(prod_id)
+    return sorted(
+        (CALIB_BASEDIR / date).glob(f"{version}*/calibration_filters_*.Run{run_id:05d}.0000.h5")
+    )
 
 
 def get_drive_file(date: str) -> Path:
@@ -180,11 +233,16 @@ def sequence_calibration_files(sequence_list: List[Sequence]) -> None:
     """Build names of the calibration files for each sequence in the list."""
     flat_date = utils.date_to_dir(options.date)
     base_dir = Path(cfg.get("LST1", "BASE"))
+    prod_id = options.prod_id
 
     for sequence in sequence_list:
         # Assign the calibration files to the sequence object
-        sequence.drs4_file = get_drs4_pedestal_file(sequence.drs4_run)
-        sequence.calibration_file = get_calibration_file(sequence.pedcal_run)
+        sequence.drs4_file = get_drs4_pedestal_filename(
+            sequence.drs4_run, prod_id
+        )
+        sequence.calibration_file = get_calibration_filename(
+            sequence.pedcal_run, prod_id
+        )
         sequence.time_calibration_file = find_time_calibration_file(
             "pro", sequence.pedcal_run, base_dir=base_dir
         )
@@ -235,10 +293,7 @@ def destination_dir(concept: str, create_dir: bool = True) -> Path:
         directory = Path(cfg.get(options.tel_id, "DL1_DIR")) / nightdir / options.prod_id / "muons"
     elif concept == "INTERLEAVED":
         directory = (
-            Path(cfg.get(options.tel_id, "DL1_DIR"))
-            / nightdir
-            / options.prod_id
-            / "interleaved"
+            Path(cfg.get(options.tel_id, "DL1_DIR")) / nightdir / options.prod_id / "interleaved"
         )
     elif concept == "DATACHECK":
         directory = (
@@ -257,17 +312,21 @@ def destination_dir(concept: str, create_dir: bool = True) -> Path:
         )
     elif concept in {"DL2", "DL3"}:
         directory = (
-            Path(cfg.get(options.tel_id, concept + "_DIR"))
-            / nightdir
+            (Path(cfg.get(options.tel_id, f"{concept}_DIR")) / nightdir)
             / options.prod_id
-            / options.dl2_prod_id
-        )
+        ) / options.dl2_prod_id
     elif concept in {"PEDESTAL", "CALIB", "TIMECALIB"}:
         directory = (
-            Path(cfg.get(options.tel_id, concept + "_DIR")) / nightdir / options.calib_prod_id
+            Path(cfg.get(options.tel_id, f"{concept}_DIR"))
+            / nightdir
+            / options.prod_id
         )
     elif concept == "HIGH_LEVEL":
-        directory = Path(cfg.get(options.tel_id, concept + "_DIR")) / nightdir / options.prod_id
+        directory = (
+            Path(cfg.get(options.tel_id, f"{concept}_DIR"))
+            / nightdir
+            / options.prod_id
+        )
     else:
         log.warning(f"Concept {concept} not known")
         directory = None

lstosa-0.10.6/src/osa/provenance/utils.py

@@ -0,0 +1,201 @@
+"""Utility functions for OSA pipeline provenance."""
+
+
+from importlib.resources import files
+from pathlib import Path
+
+from osa.configs import options
+from osa.configs.config import cfg
+from osa.utils.utils import date_to_dir, get_lstchain_version
+
+__all__ = ["parse_variables", "get_log_config"]
+
+REDUCTION_TASKS = ["r0_to_dl1", "dl1ab", "dl1_datacheck", "dl1_to_dl2"]
+
+
+def parse_variables(class_instance):
+    """Parse variables needed in model"""
+
+    # calibration_pipeline.py
+    # -c sequencer.cfg
+    # -d 2020_02_18
+    # --drs4-pedestal-run 01804
+    # --pedcal-run 01805
+    # LST1
+
+    # datasequence.py
+    # -c sequencer.cfg
+    # -d 2020_02_18
+    # --prod-id v0.4
+    # --pedcal-file .../20200218/v0.4.3/calibration_filters_52.Run02006.0000.h5
+    # --drs4-pedestal-file .../20200218/v0.4.3/drs4_pedestal.Run02005.0000.h5
+    # --time-calib-file .../20191124/pro/time_calibration.Run01625.0000.h5
+    # --systematic_correction_file .../20200725/pro/ffactor_systematics_20200725.h5
+    # --drive-file .../DrivePositioning/DrivePosition_20200218.txt
+    # --run-summary .../monitoring/RunSummary/RunSummary_20200101.ecsv
+    # 02006.0000
+    # LST1
+
+    flat_date = date_to_dir(options.date)
+    configfile_dl1b = cfg.get("lstchain", "dl1b_config")
+    configfile_dl2 = cfg.get("lstchain", "dl2_config")
+    raw_dir = Path(cfg.get("LST1", "R0_DIR"))
+    rf_models_directory = Path(cfg.get("lstchain", "RF_MODELS"))
+    dl1_dir = Path(cfg.get("LST1", "DL1_DIR"))
+    dl2_dir = Path(cfg.get("LST1", "DL2_DIR"))
+    calib_dir = Path(cfg.get("LST1", "CALIB_DIR"))
+    pedestal_dir = Path(cfg.get("LST1", "PEDESTAL_DIR"))
+
+    class_instance.SoftwareVersion = get_lstchain_version()
+    class_instance.ProcessingConfigFile = str(options.configfile)
+    class_instance.ObservationDate = flat_date
+    if class_instance.__name__ in REDUCTION_TASKS:
+        muon_dir = dl1_dir / flat_date / options.prod_id / "muons"
+        outdir_dl1 = dl1_dir / flat_date / options.prod_id / options.dl1_prod_id
+        outdir_dl2 = dl2_dir / flat_date / options.prod_id / options.dl2_prod_id
+
+    if class_instance.__name__ in ["drs4_pedestal", "calibrate_charge"]:
+        # drs4_pedestal_run_id  [0] 1804
+        # pedcal_run_id         [1] 1805
+        # history_file          [2] .../20210913/v0.7.5/sequence_LST1_01805.0000.history
+        class_instance.PedestalRun = f"{class_instance.args[0]:05d}"
+        class_instance.CalibrationRun = f"{class_instance.args[1]:05d}"
+
+        version = get_lstchain_version()
+
+        # according to code in onsite scripts in lstchain
+        class_instance.RawObservationFilePedestal = str((
+            raw_dir / flat_date / f"LST-1.1.Run{class_instance.args[0]:05d}.fits.fz"
+        ).resolve())
+        class_instance.RawObservationFileCalibration = str((
+            raw_dir / flat_date / f"LST-1.1.Run{class_instance.args[1]:05d}.fits.fz"
+        ).resolve())
+        class_instance.PedestalCheckPlot = str((
+            pedestal_dir
+            / flat_date
+            / version
+            / f"log/drs4_pedestal.Run{class_instance.args[0]:05d}.0000.pdf"
+        ).resolve())
+        class_instance.CalibrationCheckPlot = str((
+            calib_dir
+            / flat_date
+            / version
+            / f"log/calibration_filters_52.Run{class_instance.args[1]:05d}.0000.pdf"
+        ).resolve())
+
+        # according to code in sequence_calibration_files from paths.py
+        class_instance.PedestalFile = str((
+            pedestal_dir
+            / flat_date
+            / version
+            / f"drs4_pedestal.Run{class_instance.args[0]:05d}.0000.h5"
+        ).resolve())
+        class_instance.CoefficientsCalibrationFile = str((
+            calib_dir
+            / flat_date
+            / version
+            / f"calibration_filters_52.Run{class_instance.args[1]:05d}.0000.h5"
+        ).resolve())
+
+    if class_instance.__name__ == "r0_to_dl1":
+        # calibration_file   [0] .../20200218/v0.4.3/calibration_filters_52.Run02006.0000.h5
+        # drs4_pedestal_file [1] .../20200218/v0.4.3/drs4_pedestal.Run02005.0000.h5
+        # time_calib_file    [2] .../20191124/v0.4.3/time_calibration.Run01625.0000.h5
+        # systematic_corr    [3] .../20200101/v0.4.3/no_sys_corrected_calib_20210514.0000.h5
+        # drive_file         [4] .../DrivePositioning/DrivePosition_20200218.txt
+        # run_summary_file   [5] .../RunSummary/RunSummary_20200101.ecsv
+        # pedestal_ids_file  [6] .../path/to/interleaved/pedestal/events.h5
+        # run_str            [7] 02006.0000
+
+        run_subrun = class_instance.args[7]
+        run = run_subrun.split(".")[0]
+        class_instance.ObservationRun = run
+
+        calibration_file = Path(class_instance.args[0]).resolve()
+        pedestal_file = Path(class_instance.args[1]).resolve()
+        timecalibration_file = Path(class_instance.args[2]).resolve()
+        systematic_correction_file = Path(class_instance.args[3]).resolve()
+        class_instance.R0SubrunDataset = str((
+            raw_dir / flat_date / f"LST-1.1.Run{run_subrun}.fits.fz"
+        ).resolve())
+        class_instance.CoefficientsCalibrationFile = str(calibration_file)
+        class_instance.PedestalFile = str(pedestal_file)
+        class_instance.TimeCalibrationFile = str(timecalibration_file)
+        class_instance.SystematicCorrectionFile = str(systematic_correction_file)
+        class_instance.PointingFile = str(Path(class_instance.args[4]).resolve())
+        class_instance.RunSummaryFile = str(Path(class_instance.args[5]).resolve())
+        class_instance.DL1SubrunDataset = str(
+            (outdir_dl1 / f"dl1_LST-1.Run{run_subrun}.h5").resolve()
+        )
+        class_instance.MuonsSubrunDataset = str((
+            muon_dir / f"muons_LST-1.Run{run_subrun}.fits"
+        ).resolve())
+        class_instance.InterleavedPedestalEventsFile = None
+        if class_instance.args[6] is not None:
+            class_instance.InterleavedPedestalEventsFile = str(Path(class_instance.args[6]))
+
+    if class_instance.__name__ == "dl1ab":
+        # run_str       [0] 02006.0000
+
+        class_instance.Analysisconfigfile_dl1 = str(Path(configfile_dl1b))
+        class_instance.ObservationRun = class_instance.args[0].split(".")[0]
+        class_instance.StoreImage = cfg.getboolean("lstchain", "store_image_dl1ab")
+        class_instance.DL1SubrunDataset = str((
+            outdir_dl1 / f"dl1_LST-1.Run{class_instance.args[0]}.h5"
+        ).resolve())
+
+    if class_instance.__name__ == "dl1_datacheck":
+        # run_str       [0] 02006.0000
+        run_subrun = class_instance.args[0]
+        run = run_subrun.split(".")[0]
+
+        class_instance.ObservationRun = run
+        class_instance.DL1SubrunDataset = str(
+            (outdir_dl1 / f"dl1_LST-1.Run{run_subrun}.h5").resolve()
+        )
+        class_instance.MuonsSubrunDataset = str((
+            muon_dir / f"muons_LST-1.Run{run_subrun}.fits"
+        ).resolve())
+        class_instance.DL1CheckSubrunDataset = str((
+            outdir_dl1 / f"datacheck_dl1_LST-1.Run{run_subrun}.h5"
+        ).resolve())
+        class_instance.DL1CheckHDF5File = str((
+            outdir_dl1 / f"datacheck_dl1_LST-1.Run{run}.h5"
+        ).resolve())
+        class_instance.DL1CheckPDFFile = str((
+            outdir_dl1 / f"datacheck_dl1_LST-1.Run{run}.pdf"
+        ).resolve())
+
+    if class_instance.__name__ == "dl1_to_dl2":
+        # run_str       [0] 02006.0000
+        run_subrun = class_instance.args[0]
+        run = run_subrun.split(".")[0]
+
+        class_instance.Analysisconfigfile_dl2 = configfile_dl2
+        class_instance.ObservationRun = run
+        class_instance.RFModelEnergyFile = str((rf_models_directory / "reg_energy.sav").resolve())
+        class_instance.RFModelDispNormFile = str(
+            (rf_models_directory / "reg_disp_norm.sav").resolve()
+        )
+        class_instance.RFModelDispSignFile = str(
+            (rf_models_directory / "reg_disp_sign.sav").resolve()
+        )
+        class_instance.RFModelGammanessFile = str((rf_models_directory / "cls_gh.sav").resolve())
+        class_instance.DL1SubrunDataset = str(
+            (outdir_dl1 / f"dl1_LST-1.Run{run_subrun}.h5").resolve()
+        )
+        class_instance.DL2SubrunDataset = str(
+            (outdir_dl2 / f"dl2_LST-1.Run{run_subrun}.h5").resolve()
+        )
+        class_instance.DL2MergedFile = str((outdir_dl2 / f"dl2_LST-1.Run{run}.h5").resolve())
+
+    if class_instance.__name__ in REDUCTION_TASKS:
+        class_instance.session_name = class_instance.ObservationRun
+
+    return class_instance
+
+
+def get_log_config():
+    """Get logging configuration from provenance logger config file."""
+    std_logger_file = files("osa.provenance") / "config/logger.yaml"
+    return std_logger_file.read_text()

{lstosa-0.10.4 → lstosa-0.10.6}/src/osa/scripts/calibration_pipeline.py

@@ -37,7 +37,8 @@ def is_calibration_produced(drs4_pedestal_run_id: int, pedcal_run_id: int) -> bo
     Check if both daily calibration (DRS4 baseline and
     charge calibration) files are already produced.
     """
-    return drs4_pedestal_exists(drs4_pedestal_run_id) and calibration_file_exists(pedcal_run_id)
+    return drs4_pedestal_exists(drs4_pedestal_run_id, options.prod_id) \
+        and calibration_file_exists(pedcal_run_id, options.prod_id)
 
 
 def drs4_pedestal_command(drs4_pedestal_run_id: int) -> list:
@@ -122,7 +123,7 @@ def drs4_pedestal(
     rc : int
         Return code
     """
-    if options.simulate or drs4_pedestal_exists(drs4_pedestal_run_id):
+    if options.simulate or drs4_pedestal_exists(drs4_pedestal_run_id, options.prod_id):
         return 0
 
     cmd = drs4_pedestal_command(drs4_pedestal_run_id)
@@ -154,7 +155,7 @@ def calibrate_charge(
     rc: int
         Return code
     """
-    if options.simulate or calibration_file_exists(pedcal_run_id):
+    if options.simulate or calibration_file_exists(pedcal_run_id, options.prod_id):
         return 0
 
     cmd = calibration_file_command(