corgidrp 0.1__tar.gz

corgidrp-0.1/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2024, corgidrp Developers
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+may be used to endorse or promote products derived from this software without
+specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

corgidrp-0.1/MANIFEST.in

@@ -0,0 +1,7 @@
+include requirements.txt
+include corgidrp/recipe_templates/*.json
+include tests/test_data/example_L1_input.fits
+include tests/test_data/metadata.yaml
+include tests/test_data/metadata_eng.yaml
+include tests/test_data/nonlin_sample.csv
+include tests/test_data/nonlin_sample.fits

corgidrp-0.1/PKG-INFO

@@ -0,0 +1,17 @@
+Metadata-Version: 2.1
+Name: corgidrp
+Version: 0.1
+Summary: (Roman Space Telescope) CORonaGraph Instrument Data Reduction Pipeline
+Home-page: https://github.com/roman-corgi/corgidrp
+Author: Roman Coronagraph Instrument CPP
+License: BSD
+Keywords: Roman Space Telescope Exoplanets Astronomy
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Astronomy
+Classifier: Programming Language :: Python :: 3
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: astropy
+Requires-Dist: pandas
+Requires-Dist: pytest
+Requires-Dist: scipy

corgidrp-0.1/README.md

@@ -0,0 +1,158 @@
+# corgidrp: CORonaGraph Instrument Data Reduction Pipeline
+This is the data reduction pipeline for the Nancy Grace Roman Space Telescope Coronagraph Instrument
+
+![Testing Badge](https://github.com/roman-corgi/corgidrp/actions/workflows/python-app.yml/badge.svg)
+
+## Install
+As the code is very much still in development, clone this repository, enter the top-level folder, and run the following command:
+```
+pip install -e .
+```
+Then you can import `corgidrp` like any other python package!
+
+The installation will create a configuration folder in your home directory called `.corgidrp`. 
+That configuration directory will be used to locate things on your computer such as the location of the calibration database and the pipeline configuration file. The configuration files stores setting such as whether to track each individual error term added to the noise. 
+
+## How to Contribute
+
+We encourage you to chat with Jason, Max, and Marie (e.g., on Slack) to discuss what to do before you get started. Brainstorming
+about how to implement something is a very good use of time and makes sure you aren't going down the wrong path. 
+Contact Jason is you have any questions on how to get started on programming details (e.g., git). 
+
+Below is a quick tutorial  that outlines the general contribution process. 
+
+### The basics of getting setup
+#### Find a task to work on
+Check out the [Github issues page](https://github.com/roman-corgi/corgidrp/issues) for tasks that need attention. Alternatively, contact Jason (@semaphoreP). _Make sure to tag yourself on the issue and mention in the comments if you start working on it._ 
+
+#### Clone the git repository and install
+
+See install instructions above. Contact Jason (@semaphoreP) if you need write access to push changes as you make edits.  If you do not have write access to the repository, you can still contribute by creating a fork of the repository under your own GitHub user.  See here for details: https://docs.github.com/en/get-started/quickstart/fork-a-repo.  You can then use the same commands given below, but just replace `roman-corgi` with your own GitHub username. If you fork the repository, you will need to make sure that your fork is up to date with the main repository (https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork).
+
+To quickly get up an running with the repository, execute the following commands in a terminal (or command prompt - you will need to have the git executable installed on your system):
+```
+> git clone https://github.com/roman-corgi/corgidrp.git
+> cd corgidrp
+> pip install -e .
+```
+
+##### Make a new git branch to work on
+
+You will create a "feature branch" so you can develop your feature without impacting other people's code. Let's say I'm working on dark subtraction. I could create a feature branch and switch to it like this
+
+```
+> git branch dark-sub
+> git checkout dark-sub
+```
+
+### Write your pipeline step
+
+In `corgidrp`, each pipeline step is a function, that is contained in one of the lX_to_lY.py files (where X and Y are various data levels). 
+Think about how your feature can be implemented as a function that takes in some data and outputs processed data. Please see below for some 
+`corgidrp` design principles. 
+
+All functions should follow this example:
+
+```
+def example_step(dataset, calib_data, tuneable_arg=1, another_arg="test"):
+    """
+    Function docstrings are required and should follow Google style docstrings. 
+    We will not demonstrate it here for brevity.
+    """
+    # unless you don't alter the input dataset at all, plan to make a copy of the data
+    # this is to ensure functions are reproducible
+    processed_dataset = input_dataset.copy()
+
+    ### Your code here that does the real work
+    # here is a convience field to grab all the data in a dataset
+    all_data = processed_dataset.all_data
+    ### End of your code that does the real work
+
+    # update the header of the new dataset with your processing step
+    history_msg = "I did an example step"
+    # update the output dataset with the new data and update the history
+    processed_dataset.update_after_processing_step(history_msg, new_all_data=all_data)
+
+    # return the processed data
+    return processed_dataset
+```
+
+Inside the function can be nearly anything you want, but the function signature and start/end of the function should follow a few rules.
+
+  * Each function should include a docstring that descibes what the function is doing, what the inputs (including units if appropriate) are and what the outputs (also with units). The dosctrings should be [goggle style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html). 
+  * The input dataset should always be the first input
+  * Additional arguments and keywords exist only if you need them--many relevant parameters might already by in Dataset headers. A pipeline step can only have a single argument (the input dataset) if needed.
+  * All additional function arguments/keywords should only consist of the following types: int, float, str, or a class defined in corgidrp.Data. 
+    * (Long explaination for the curious: The reason for this is that pipeline steps can be written out as text files. Int/float/str are easily represented succinctly by textfiles. All classes in corgidrp.Data can be created simply by passing in a filepath. Therefore, all pipeline steps have easily recordable arguments for easy reproducibility.)
+  * The first line of the function generally should be creating a copy of the dataset (which will be the output dataset). This way, the output dataset is not the same instance as the input dataset. This will make it easier to ensure reproducibility. 
+  * The function should always end with updating the header and (typically) the data of the output dataset. The history of running this pipeline step should be recorded in the header. 
+
+You can check out `corgidrp.l2a_to_l2b.dark_subtraction` function as an example of a basic pipeline step.
+
+### Write a unit test to debug your pipeline step
+
+We are required to write tests to verify the functionality of the code. Instead of seeing this as an extra chore, I encourage you to write unit tests to be your debug script to get your code working (this is called "test-driven development"). 
+
+All tests are stored in the `tests` folder and each test is a function that starts with `test_`. See `tests/test_dark_sub.py` as an example. Within each test, you will likely need to simulate some mock data, run it through your function you wrote, and verify it ran correctly using assert statements. Your tests should cover the primary use cases of your code, and check that the function outputs what you expect. You do not need high fidelity data for your test: focus on making sure the data is in the correct format as real data, and less on making sure the data values are simulated to high fidelity (see the examples in the `mocks.py` module). 
+
+Importantly, these tests will allow code reviewers to test and understand your code. We will also run these tests in an automated test suite (continuous integration) with the pipeline to verify the functions continue to work (e.g., as dependencies change).
+
+#### How to run your tests locally
+
+You can either run tests individually yourself (to debug individual tests) or run the entire test suite to make sure you didn't break anything.
+
+To run an individual test, call the test function you want to test at the bottom of its `test_*.py` script. Then, you just need to run the `test_*.py` script. See `tests/test_dark_sub.py` for an example.
+
+To run all the tests in the test suite, go to the base corgidrp folder in a terminal and run the `pytest` command. 
+
+### Linting
+
+In addition to unit tests, your code will need to pass a static analysis before being merged.  `corgidrp` currently runs a subset of flake8 tests, which you can replicate on your local system by running:
+
+```
+flake8 . --count --select=E9,F63,F7,F82,DCO020,DCO021,DCO022,DCO023,DCO024,DCO030,DCO031,DCO032,DCO060,DCO061,DCO062,DCO063,DCO064,DCO065 --show-source --statistics
+```
+from the top-level directory of the repository.  In order to run these tests you will need to have `flake8` and `flake8-docstrings-complete` installed (both are pip-installable).  Note that the test subset may be updated in the future.  To see the current set of tests being applied, look in the continuous integration GitHub action, located in the repository in file `.github/workflows/python-app.yml`.
+
+### Create a pull request to merge your changes
+Before creating a pull request, review the design Principles below. Use the Github pull request feature to request that your changes get merged into the `main` branch. Assign Jason/Max to be your reviewers. Your changes will be reviewed, and possibly some edits will be requested. You can simply make additional pushes to your branch to update the pull request with those changes (you don't need to delete the PR and make a new one). When the branch is satisfactory, we will pull your changes in. When preparing your pull request, you may find it helpful to follow this checklist:
+
+- [ ] If working with a fork, sync your fork to the upstream repository
+- [ ] Ensure that your pull can be merged automatically by merging main onto the branch you wish to pull from
+- [ ] Ensure that all of your new additions have properly formatted docstrings (this will happen automatically if you run the `flake8` command given above
+- [ ] Ensure that all of the commits going in to your pull have informative messages
+- [ ] Ensure that all unit tests pass locally, and that you have provided new unit tests for all new functionality in your pull
+- [ ] Create a new pull request, fully describing all changes/additions
+
+
+## Overarching Design Principles
+* Minimize the use of external packages, unless it saves us a lot of time. If you need to use something external, default to well-established and maintained packages, such as `numpy`, `scipy` or `Astropy`. If you think you need to use something else, please check with Jason and Max. 
+* Minimize the use of new classes, with the exception of new classes that extend the existing data framework.
+* The python module files (i.e. the *.py files) should typically hold on the order of 5-10 different functions. You can create new ones if need be, but the new files should be general enough in topic to encapsulate other future functions as well.
+* All the image data in the Dataset and Image objects should be saved as standard numpy arrays. Other types of arrays (such as masked arrays) can be used as intermediate products within a function.
+* Keep things simple
+* Use _descriptive_ variable names **always**.
+* Comments should be used to describe a section of the code where it's not immediately obvious what the code is doing. Using descriptive variable names will minimize the amount of comments required.
+
+## FAQ
+
+  * Does my pipeline function need to save files?
+    * Files will be saved by a higher level pipeline code. As long as you output an object that's an instance of a `corgidrp.Data` class, it will have a `save()` function that will be used.
+  * Can I create new data classes?
+    * Yes, you can feel free to make new data classes. Generally, they should be a subclass of the `Image` class, and you can look at the `Dark` class as an example. Each calibration type should have its own `Image` subclass defined. Talk with Jason and Max to discuss how your class should be implemented!
+
+* What python version should I develop in?
+  * Python 3.12
+    
+* How should I treat hard-coded variables
+  * If a variable value is extremely unlikely to change AND is only required by one module, it can be hard coded inside that module.
+  * If it is unlikely to change but will need to be referenced by multiple modules, it should be added to the central constants file (name TBD). 
+  * If it is a setting about pipeline behavior that may change, it should be implemented by a config file (examples are location of the calibration database and whether to save individual error terms in the output FITS files).
+ 
+* Where should I store computed variables so they can be referenced later in the pipeline?
+  * If possible, in the header of the dataset being processed or in a hew HDU extension
+  * If not possible, then let's chat!
+ 
+* Where do I save FITS files or other data files I need to use for my tests?
+  * Auxiliary data to run tests should be stored in the tests/test_data folder
+  * If they are larger than 1 MB, they should be stored using `git lfs`. Ask Jason about setting up git lfs (as of writing, we have not set up git lfs yet). 

corgidrp-0.1/corgidrp/__init__.py

@@ -0,0 +1,63 @@
+import configparser
+import os
+import pathlib
+import configparser
+
+__version__ = "0.1"
+version = __version__ # temporary backwards compatability 
+
+#### Create a configuration file for the corgidrp if it doesn't exist. 
+def create_config_dir():
+    """
+    Checks if the default .corgidrp directory exists, and if not, it sets it up
+    """
+    homedir = pathlib.Path.home()
+    config_folder = os.path.join(homedir, ".corgidrp")
+    # replace legacy file with folder if needed
+    if os.path.isfile(config_folder):
+        oldconfig = configparser.ConfigParser()
+        oldconfig.read(config_folder)
+        os.remove(config_folder)
+    else:
+        oldconfig = None
+
+    # make folder if doesn't exist
+    if not os.path.isdir(config_folder):
+        os.mkdir(config_folder)
+
+        # make default calibrations folder
+        default_cal_dir = os.path.join(config_folder, "default_calibs")
+        if not os.path.exists(default_cal_dir):
+            os.mkdir(default_cal_dir)
+
+        # write config 
+        config_filepath = os.path.join(config_folder, "corgidrp.cfg")
+        if not os.path.exists(config_filepath):
+            config = configparser.ConfigParser()
+            config["PATH"] = {}
+            config["PATH"]["caldb"] = os.path.join(config_folder, "corgidrp_caldb.csv") # location to store caldb
+            config["PATH"]["default_calibs"] = default_cal_dir
+            config["DATA"] = {}
+            config["DATA"]["track_individual_errors"] = "False"
+            # overwrite with old settings if needed
+            if oldconfig is not None:
+                config["PATH"]["caldb"] = oldconfig["PATH"]["caldb"]
+
+            with open(config_filepath, 'w') as f:
+                config.write(f)
+
+            print("corgidrp: Configuration file written to {0}. Please edit if you want things stored in different locations.".format(config_filepath))
+create_config_dir()
+    
+_bool_map = {"true" : True, "false" : False}
+
+# borrowed from the kpicdrp caldb
+# load in default caldbs based on configuration file
+config_filepath = os.path.join(pathlib.Path.home(), ".corgidrp", "corgidrp.cfg")
+config = configparser.ConfigParser()
+config.read(config_filepath)
+
+## pipeline settings
+caldb_filepath = config.get("PATH", "caldb", fallback=None)
+default_cal_dir = config.get("PATH", "default_calibs", fallback=None)
+track_individual_errors = _bool_map[config.get("DATA", "track_individual_errors").lower()]

corgidrp-0.1/corgidrp/caldb.py

@@ -0,0 +1,304 @@
+"""
+Calibration tracking system. Modified from kpicdrp caldb implmentation (Copyright (c) 2024, KPIC Team)
+"""
+import os
+import numpy as np
+import pandas as pd
+import corgidrp
+import corgidrp.data as data
+import astropy.time as time
+
+
+column_names = [
+    "Filepath",
+    "Type",
+    "MJD",
+    "EXPTIME",
+    "Files Used",
+    "Date Created",
+    "Hash",
+    "DRPVERSN",
+    "OBSID",
+    "NAXIS1",
+    "NAXIS2",
+    "OPMODE",
+    "CMDGAIN",
+    "EXCAMT",
+]
+
+labels = {data.Dark: "Dark", 
+          data.NonLinearityCalibration: "NonLinearityCalibration", 
+          data.BadPixelMap: "BadPixelMap", 
+          data.KGain : "KGain", 
+          data.DetectorParams : "DetectorParams"}
+
+class CalDB:
+    """
+    Database for tracking calibration files saved to disk. Modified from the kpicdrp version
+
+    Note that database is not parallelism-safe, but should be ok in most cases.
+    (Jason: look at using posix_ipc to guarantee thread safety if we really need it)
+
+    Args:
+        filepath (str): [optional] filepath to a CSV file with an existing database
+
+    Fields:
+        columns (list): column names of dataframe
+        filepath(str): full filepath to data
+    """
+
+    def __init__(self, filepath=""):
+        """
+        Args:
+            filepath (str): [optional] filepath to a CSV file with an existing database
+        """
+
+        # If filepath is not passed in, use the default (majority of case)
+        if len(filepath) == 0:
+            self.filepath = corgidrp.caldb_filepath
+        else:
+            # possibly edge case where we want to use specialized caldb
+            self.filepath = filepath
+
+        # if database does't exist, create a blank one
+        if not os.path.exists(self.filepath):
+            # new database
+            self.columns = column_names
+            self._db = pd.DataFrame(columns=self.columns)
+            self.save()
+        else:
+            # already a database exists
+            self.load()
+            self.columns = list(self._db.columns.values)
+
+    def load(self):
+        """
+        Load/update db from filepath
+        """
+        self._db = pd.read_csv(self.filepath)
+
+    def save(self):
+        """
+        Save file without numbered index to disk with user specified filepath as a CSV file
+        """
+        self._db.to_csv(self.filepath, index=False)
+
+    def _get_values_from_entry(self, entry, is_calib=True):
+        """
+        Extract the properties from this data entry to ingest them into the database
+
+        Args:
+            entry (corgidrp.data.Image subclass): calibration frame to add to the database
+            is_calib (bool): is a calibration frame. if Not, it won't look up filetype.
+                             Only used in get_calib() to grab metadata for science frames
+
+        Returns:
+            tuple:
+                row (list):
+                    List of data entry properties
+                row_dict (dict):
+                    Dictionary of data entry properties keyed by column names
+
+        """
+        filepath = os.path.abspath(entry.filepath)
+        if is_calib:
+            datatype = labels[entry.__class__]  # get the database str representation
+        else:
+            datatype = "Sci"
+        mjd = time.Time(entry.ext_hdr["SCTSRT"]).mjd
+        exptime = entry.ext_hdr["EXPTIME"]
+
+        # check if this exists. will be a keyword written by corgidrp
+        if "DRPNFILE" in entry.ext_hdr:
+            files_used = entry.ext_hdr["DRPNFILE"]
+        else:
+            files_used = 0
+
+        if "DRPCTIME" in entry.ext_hdr:
+            date_created = time.Time(entry.ext_hdr["DRPCTIME"]).mjd
+        else:
+            date_created = -1
+
+        if "DRPVERSN" in entry.ext_hdr:
+            drp_version = entry.ext_hdr["DRPVERSN"]
+        else:
+            drp_version = ""
+
+        obsid = entry.pri_hdr["OBSID"]
+
+        hash_val = entry.get_hash()
+
+        # this only works for 2D images. may need to adapt for non-2D calibration frames
+        naxis1 = entry.data.shape[-1]
+        naxis2 = entry.data.shape[-2]
+
+        row = [
+            filepath,
+            datatype,
+            mjd,
+            exptime,
+            files_used,
+            date_created,
+            hash_val,
+            drp_version,
+            obsid,
+            naxis1,
+            naxis2,
+        ]
+
+        # rest are ext_hdr keys we can copy
+        start_index = len(row)
+        for i in range(start_index, len(self.columns)):
+            row.append(entry.ext_hdr[self.columns[i]])  # add value staright from header
+
+        row_dict = {}
+        for key, val in zip(self.columns, row):
+            row_dict[key] = val
+
+        return row, row_dict
+
+    def create_entry(self, entry, to_disk=True):
+        """
+        Add a new entry to or update an existing one in the database. Note that function by default will load and save db to disk
+
+        Args:
+            entry (corgidrp.data.Image subclass): calibration frame to add to the database
+            to_disk (bool): True by default, will update DB from disk before adding entry and saving it back to disk
+        """
+        new_row, row_dict = self._get_values_from_entry(entry)
+
+        # update database from disk in case anything changed
+        if to_disk:
+            self.load()
+
+        # use filepath as key to see if it's already in database
+        if row_dict["Filepath"] in self._db.values:
+            row_index = self._db[
+                self._db["Filepath"] == row_dict["Filepath"]
+            ].index.values
+            self._db.loc[row_index, self.columns] = new_row
+        # otherwise create new entry
+        else:
+            new_entry = pd.DataFrame([new_row], columns=self.columns)
+            self._db = pd.concat([self._db, new_entry], ignore_index=True)
+
+        # save to disk to update changes
+        if to_disk:
+            self.save()
+
+    def remove_entry(self, entry, to_disk=True):
+        """
+        Remove an entry from the database. Removes the entire row
+
+        Args:
+            entry (corgidrp.data.Image subclass): calibration frame to add to the database
+            to_disk (bool): True by default, will update DB from disk before adding entry and saving it back to disk
+        """
+        new_row, row_dict = self._get_values_from_entry(entry)
+
+        # update database from disk in case anything changed
+        if to_disk:
+            self.load()
+
+        if row_dict["Filepath"] in self._db.values:
+            entry_index = self._db[
+                self._db["Filepath"] == row_dict["Filepath"]
+            ].index.values
+            self._db = self._db.drop(self._db.index[entry_index])
+            self._db = self._db.reset_index(drop=True)
+        else:
+            raise ValueError("No filepath found so could not remove.")
+
+        # save to disk to update changes
+        if to_disk:
+            self.save()
+
+    def get_calib(self, frame, dtype, to_disk=True):
+        """
+        Outputs the best calibration file of the given type for the input sciene frame.
+
+        Args:
+            frame (corgidrp.data.Image): an image frame to request a calibratio for
+            dtype (corgidrp.data Class): for example: corgidrp.data.Dark (TODO: document the entire list of options)
+            to_disk (bool): True by default, will update DB from disk before matching
+
+        Returns:
+            corgidrp.data.*: an instance of the appropriate calibration type (Exact type depends on calibration type)
+        """
+        if dtype not in labels:
+            raise ValueError(
+                "Requested calibration dtype of {0} not a valid option".format(dtype)
+            )
+        dtype_label = labels[dtype]
+
+        # get values for this science frame
+        _, frame_dict = self._get_values_from_entry(frame, is_calib=False)
+
+        # update database from disk in case anything changed
+        if to_disk:
+            self.load()
+
+        # downselect to only calibs of this type
+        calibdf = self._db[self._db["Type"] == dtype_label]
+
+        if dtype_label in ["Dark"]:
+            # general selection criteria for 2D image frames. Can use different selection criteria for different dtypes
+            options = calibdf.loc[
+                (
+                    (calibdf["EXPTIME"] == frame_dict["EXPTIME"])
+                    & (calibdf["NAXIS1"] == frame_dict["NAXIS1"])
+                    & (calibdf["NAXIS2"] == frame_dict["NAXIS2"])
+                )
+            ]
+        else:
+            options = calibdf
+
+        # select the one closest in time
+        result_index = np.abs(options["MJD"] - frame_dict["MJD"]).argmin()
+        calib_filepath = options.iloc[result_index, 0]
+
+        # load the object from disk and return it
+        return dtype(calib_filepath)
+
+    def scan_dir_for_new_entries(self, filedir, look_in_subfolders=True, to_disk=True):
+        """
+        Scan a folder and subfolder for calibration files and add them all to the caldb
+
+        Args:
+            filedir (str): path to folder to scan (includes all subfolders by default)
+            look_in_subfolders (bool): whether to look in subfolders for files. True by default
+            to_disk (bool): True by default, will update DB from disk before adding entry and saving it back to disk
+        """
+        calib_frames = []
+        # walk the directory to find all the calibration files
+        for dirpath, subfolders, filenames in os.walk(filedir):
+            for filename in filenames:
+                # hard coded check only for files that end in .fits
+                if filename[-5:] != ".fits":
+                    continue
+
+                filepath = os.path.join(dirpath, filename)
+                frame = data.autoload(filepath)
+
+                # check what class it has been loaded as. only save frames that fall into calibration classes
+                if frame.__class__ in labels:
+                    calib_frames.append(frame)
+
+            # the first iteration looks in the basedir
+            # if we don't wnat to look in subdirs now, we should break
+            if not look_in_subfolders:
+                break
+
+        # load all these files into the caldb
+        for calib_frame in calib_frames:
+            self.create_entry(calib_frame, to_disk=to_disk)
+
+### Create set of default calibrations
+# Add default detector_params calibration file if it doesn't exist
+if not os.path.exists(os.path.join(corgidrp.default_cal_dir, "DetectorParams_2023-11-01T00:00:00.000.fits")):
+    default_detparams = data.DetectorParams({}, date_valid=time.Time("2023-11-01 00:00:00", scale='utc'))
+    default_detparams.save(filedir=corgidrp.default_cal_dir)
+
+    # add default caldb entries
+    default_caldb = CalDB()
+    default_caldb.scan_dir_for_new_entries(corgidrp.default_cal_dir)