roman-snpit-snappl 0.21.0__tar.gz → 0.23.0__tar.gz

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0}/.github/workflows/changelog.yml

@@ -23,6 +23,9 @@ jobs:
       - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
         with:
           fetch-depth: 0
+      - run: sudo apt-get -y update && sudo apt-get install -y libfftw3-double3
+      - run: sudo ln -s /usr/lib/x86_64-linux-gnu/libfftw3.so.3 /usr/lib/libfftw3.so
+      - run: ls -l /usr/lib/libff*
       - run: pip install .
       - run: pip install towncrier
       - run: towncrier check
@@ -34,4 +37,4 @@ jobs:
         with:
           fetch-depth: 0
       - name: prevent direct changes to `CHANGES.rst`
-        run: git diff HEAD ${{ github.event.pull_request.base.sha }} --no-patch --exit-code CHANGES.rst
+        run: git diff HEAD ${{ github.event.pull_request.base.sha }} --no-patch --exit-code CHANGES.rst

{roman_snpit_snappl-0.21.0/roman_snpit_snappl.egg-info → roman_snpit_snappl-0.23.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: roman_snpit_snappl
-Version: 0.21.0
+Version: 0.23.0
 Summary: General, database, and photometry utilities for the Roman SNPIT
 Author: Roman Supernova Project Infrastructure Team
 Maintainer-email: Roman SN PIT <raknop@lbl.gov>

roman_snpit_snappl-0.23.0/changes/103.snappl.rst

@@ -0,0 +1 @@
+Small changes for Nov 2025 run

roman_snpit_snappl-0.23.0/changes/105.docs.rst

@@ -0,0 +1 @@
+Write recipes in Usage docs for the nov2025 run

roman_snpit_snappl-0.23.0/changes/106.docs.rst

@@ -0,0 +1 @@
+Config file in docs for nov 2025

roman_snpit_snappl-0.23.0/changes/107.feature.rst

@@ -0,0 +1 @@
+Add position_angle to segmap and l2image database tables

roman_snpit_snappl-0.23.0/changes/109.bugfix.rst

@@ -0,0 +1 @@
+Fix issue #90 (spurious error messages when saving a provenance to database)

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0}/docs/usage.rst

@@ -27,16 +27,287 @@ November 2025 SNPIT Pipeline Test Primer
 The database connection is under heavy development, and more things are showing up every day.  Right now, the following is available:
 
 * Find image L2 images in the database
+* Find segmentation maps in the database
 * Saving newly discovered DiaObjects to the database
 * Finding DiaObjects
 * Saving updated positions for DiaObjects
-
-I *hope* that we will have saving lightcurves and finding lightcurves also available by the November test.  How much spectrum saving, and how much characterization saving, is implemented will depend on my time and what we get from those working groups.
-
+* Saving lightcurves to the database
+* Finding and reading lightcurves from the database
+  
 This section describes what you need to do in order to connect to the database.
 
 **WARNING**: because everything is under heavy development, it's possible that interfaces will change.  We will try to avoid doing this, because it's a pain when everybody has to adapt, but we're still building this, so it may be inevitable.
 
+Recipes
+=======
+
+Read everything to understand what's going on, but these are intended as a quick start.
+
+These recipes assume you have a working environment (see :ref:`nov2025-working-env`), which hopefully is as simple as ``pip install roman-snpit-snappl``, but see that section for all the details and where you eventually need to be heading.
+
+They also assume you have set up a config file.  If you're on NERSC, *not* running in a container, then save `this config file <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/nov2025_nersc_native_config.yaml>`_.   (this is the file ``nov2025_nersc_native_config.yaml`` from the top level of the ``environment`` roman snpit github archive).  If you are running in a podman container, then look at `this config file <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/nov2025_container_config.yaml>`_ (this is the file ``nov2025_nersc_container_config.yaml`` from the top level of the ``environment`` roman snpit github archive); you will also need to download `interactive-podman-nov2025.sh <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/interactive-podman-nov2025.sh>`_.   If you are elsewhere, you will need to edit the config file to have the right paths to find things on your system.
+
+You need to set the environment variable ``SNPIT_CONFIG`` to point to where this configuration file lives.
+
+Finally, once at the top of your code you need to do::
+  
+  from snappl.dbclient import SNPITDBClient
+
+  dbclient = SNPITDBClient()
+
+Most of the recipes below use the ``dbclient`` variable.
+
+  
+.. _recipe-command-line-args:
+
+Command-line Arguments
+----------------------
+
+Below, you will be told you need to know a number of object ids and/or provenance-related values.  These will generally be provided by orchestration.  You should make them things that can be passed on the command line.  I recommend using the following command-line arguments — choose the ones that you need (they are all string values)::
+
+  --diaobject-id
+  --diaobject-provenance-tag
+  --diaobject-process
+  --diaobject-position-provenance-tag
+  --diaobject-position-process
+  --image-id
+  --image-provenance-tag
+  --image-process
+  --segmap-provenance-tag
+  --segmap-process
+  --ltcv-provenance-tag
+  --ltcv-process
+  --spec1d-provenance-tag
+  --spec1d-process
+
+
+.. _recipe-find-diaobject:
+
+Finding a DiaObject
+-------------------
+
+You need to know *either* the ``diaobjectd_id`` of the object (which you will generally be given), or you need to know the ``diaobject_provenance_tag`` and ``diaobject_process``, and you must have enough search criteria to find the object.  If you're doing the latter, read the docstring on ``snappl.diaobject.DiaObject.find_objects``.  For the former::
+
+  from snappl.diaobject import DiaObject
+
+  diaobject = DiaObject.get_object( diaobject_id=diaobject_id, dbclient=dbclient )
+
+The returned ``DiaObject`` object has, among other things, properties ``.ra`` and ``.dec``.
+
+
+.. _recipe-diaobject-position:
+
+Getting an updated diaobject position
+-------------------------------------
+
+You need the ``diaobject_position_provenance_tag`` and ``diaobject_position_process``.
+
+Do::
+
+  diaobj_pos = diaobject.get_position( provenance_tag=diaobject_position_provenance_tag,
+                                       process=diaobject_position_process,
+                                       dbclient=dbclient )
+
+You get back a dictionary that has a number of keys including ``ra`` and ``dec``.
+
+   
+
+
+Getting a Specific Image
+------------------------
+
+ROB IMPLEMENT THIS BETTER
+
+Orchestration has given you a ``image_id`` that you are supposed to do something with.  E.g.,, you are running sidecar, and you're supposed to subtract and search this image.  Right now you also need to know the images's ``image_provenance_tag`` and ``image_process``, but that requirement will go away when Rob fixes it::
+
+  from snappl.imagecollection import ImageCollection
+  from snappl.image import Image
+
+  collection = ImageCollection.get_collection( provenance_tag=image_provenance_tag,
+                                               process=image_process, dbclient=dbclient )
+  image = collection.get_image( image_id=image_id, dbclient=dbclient )
+
+You will get back an ``Image`` object.  It has a number of properties.  Most important are ``.data``, ``.noise``, and ``.flags``, which hold 2d numpy arrays.  There is also a ``.get_fits_header()`` method that currently works, **but be careful using this as this method will not work in the future when we're using ASDF files**.  See the docstrings in ``snappl.image.Image`` for more details.  Some of the stuff you might want is available directly as properties of and ``Image`` object.
+                                               
+
+Finding Images
+--------------
+
+You need to know the ``image_provenance_tag`` and ``image_process``.
+
+See the docstring on ``snappl.imagecollection.ImageCollection.find_images`` if you want to do more than what's below.
+
+
+Finding all images that include a ra and dec
+********************************************
+
+Do::
+
+  from snappl.imagecollection import ImageCollection
+
+  collection = ImageCollection.get_collection( provenance_tag=image_provenance_tag,
+                                               process=image_process, dbclient=dbclient )
+  images = collection.find_images( ra=ra, dec=dec, band=band, dbclient=dbclient )
+
+where ``band=band`` is optional but often useful.  You will get back a list of ``Image`` objects, which have a number of properties.  Most important are ``.data``, ``.noise``, and ``.flags``, which hold 2d numpy arrays.  There is also a ``.get_fits_header()`` method that currently works, **but be careful using this as this method will not work in the future when we're using ASDF files**.  See the docstrings in ``snappl.image.Image`` for more details.
+
+Finding Segmentation Maps
+-------------------------
+
+You need to know the ``segmap_provenance_tag`` and the ``segmap_process``.
+
+See the dockstring on ``snappl.segmap.SegmentationMap.find_segmaps`` for more information on searches you can do beyond what's below.
+
+Finding all segmaps that include a ra and dec
+*********************************************
+
+Do::
+
+  from snappl.segmap import SegmentationMap
+
+  segmaps = SegmentationMap.find_segmaps( provenance_tag=segmap_provenance_tag,
+                                          process=segmap_process,
+                                          ra=ra, dec=dec,
+                                          dbclient=dbclient )
+
+You get back a list of ``SegmentationMap`` objects.  These have a number of properties, most import of which is ``image``, which holds an ``Image`` object.  You can get the image data for the segmentation map for the first element of the list with ``segmaps[0].image.data`` (a 2d numpy array).
+
+
+Saving a new DiaObject
+----------------------
+
+You are running sidecar and you've found a new diaobject you want to save.  You need a ``process`` (we shall assume ``process='sidecar'`` here), the ``major`` and ``minor`` version of your code, and the ``params`` that define how the code runs.  The latter is just a dictionary; you can build it yourself, but see :ref:`nov2025-making-prov` below.  Finally, assume that ``images`` is an list that has the ``snappl.image.Image`` objects of the images that you've used; replace ``images[0]`` below with wherever you have your ``Image`` object::
+
+  from snappl.provenance import Provenance
+  from snappl.diaobject import DiaObject
+
+  imageprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  prov = Provenance( process='sidecar', major=major, minor=minor, params=params,
+                     upstreams=[ imageprov ] )
+  # You only have to do this next line once for a given provenance;
+  #   once the provenance is in the databse, you never need to save it again.
+  prov.save_to_db( tag=diaobject_provenance_tag, dbclient=dbclient )   # See note below
+
+  diaobj = DiaObject( provenance_id=prov.id, ra=ra, dec=dec, name=optional, mjd_discovery=mjd )
+  diaobj.save_object( dbclient=dbclient )
+  
+
+*Note*: right now, you'll get exceptions on the ``prov.save_to_db`` line, but it probaby did work.  There is a snappl issue out for Rob to fix this.
+
+This will save the object to the database.  You can then look at ``diaobj.id`` to see what UUID it was assigned.  You do not need to give it a ``name``, but you can if you want to.  (The database uses the ``id`` as the unique identifier.)  ``mjd_discovery`` should be the MJD of the science image that the object was found on.
+
+Finding and reading lightcurves
+-------------------------------
+
+You need to know the ``ltcv_provenance_tag`` and ``ltcv_process``, and the ``diaobject_id`` of the object for which you want to get lightcurves::
+
+  from snappl.lightcurve import Lightcurve
+
+  ltcvs = Lightcurve.find_lightcurves( provenance_tag=ltcv_provenance_tag,
+                                       process=ltcv_process,
+                                       diaobject=diaobject_id,
+                                       band=band,         # optional
+                                       dbclient=dbclient )
+
+You will get back a list of ``Lightcurve`` objects.  You can find the actual lightcurve data of the first lightcurve from the list with ``ltcvs[0].lightcurve``.  This is an astropy QTable.  You can read the metadata from ``ltcvs[0].lightcurve.meta``.
+
+**Coming soon**: a way to read a combined lightcurve that has all of the bands mixed together.  (Not implemented yet.)
+
+
+Saving lightcurves
+------------------
+
+You need to make sure you've created a dictionary with `all the necessary metadata <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/lightcurve>`_.  Also make sure you've created a data table with the necessary columns; this can be an astropy Table, a pandas DataFrame, or a dict of lists.  We shall call these two things ``meta`` and ``data``.
+
+Assume that you've made the lightcurve for object ``diaobject`` (a ``DiaObject`` object), and that you have a list of your images in ``images``.  Adjust below for the variables where you really have things.  Finally, if you used an updated :ref:`DiaObject position <recipe-diaobject-position>`, make sure you have set the ``ra`` and ``dec`` in ``meta`` from that.
+
+Finally, you will need to know the ``ltcv_provenance_tag`` we're using.
+
+Below, ``process`` is probably either ``campari`` or ``phrosty``.  ``major`` and ``minor`` are the major and minor parts of the version, which you should parse from ``campari.__version__`` or ``phrosty.__version__``.  ``params`` are the parameters as described below in :ref:`nov2025-making-prov`.
+
+Do::
+
+  from snappl.provenance import Provenance
+  from snappl.lightcurve import Lightcurve
+
+  imgprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  objprov = Provenance.get_by_id( diaobject.provenance_id, dbclient=dbclient )
+  objposprov = Provenance.get_by_id( diaobj_pos['provenance_id'] )
+
+  ltcvprov = Provenance( process=process, major=major, minor=minor, params=params,
+                         upstreams=[imgprov, objprov, objposprov], dbclient=dbclient )
+  # The next line only needs to be run once.  Once you've saved it to the database,
+  #   you never need to do this again.
+  ltcvprov.save_to_db( tag=ltcv_provenance_tag )
+
+  meta['provenance_id'] = ltcvprov.id
+  meta['diaobject_id'] = diaobject.id
+  meta['diaobject_position_id'] = diaobj_pos['id']
+  for att in [ 'ra', 'dec', 'ra_err', 'dec_err', 'ra_dec_covar' ]:
+      meta[att] = diaobj_pos[att]
+
+  ltcv = Lightcurve( data=data, meta=meta )
+  ltcv.write()
+  ltcv.save_to_db( dbclient=dbclient )
+
+You can look at ``ltcv.id`` to see the ``UUID`` of the lightcurve you saved, in case you are curious.
+  
+If you used the ``ra`` and ``dec`` that was in ``DiaObject``, then ``meta['diaobject_position_id']`` should be ``None``.  Skip everything else above that refers to ``diaobj_pos``.
+  
+
+
+Finding and reading 1d Spectra
+------------------------------
+
+(Not implemented yet.)
+
+Saving 1d Spectra
+-----------------
+
+**Warning: this is not implemented yet.  When it is, the process will look *something* like hte following.**
+
+You need to have the ``diaobject`` (a ``DiaObject`` object) for which you made the spectrum, potentially a ``diaobj_pos``, an improved position for the object, and ``images``, a list of ``Image`` object that held the dispersed images from which you are making the spectrum.  You need to know the ``spec1d_provenance_tag``.
+
+You need to know the ``process`` (which is probably just the name of your code), and the ``major`` and ``minor`` versions of your code.  Finally, you need to know the ``params`` that define how your code runs.   The latter is just a dictionary; you can build it yourself, but see :ref:`nov2025-making-prov` below.
+
+You build a data structure that is described on `the wiki <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/spectrum_1d>`_; call that ``spec_struct``.  Some of lines below make sure that some of this metadata is right.
+
+Do::
+
+  from snappl.provenance import Provenance
+  from snappl.spec1d import Spectrum1d
+
+  diaobj_prov = Provenance.get_by_id( diaobject.provenance_id, dbclient=dbclient )
+  imageprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  diaobj_pos_prov = Provenance.get_by_id( diaobj_pos['id'], dbclient=dbclient )
+
+  spec1d_prov = Provenance( process=process, major=major, minor=minor, params=params,
+                            upstreams=[ diaobj_prov, imageprov, diaobj_pos_prov ] )
+  # The next line only needs to be run once.  Once
+  #   you have saved a Provenance to the databse you
+  #   never need to save it again
+  spec1d_prov.save_to_db( tag=spec1d_provenance_tag, dbclient-dbclient )
+
+  spec_struct['meta']['provenance_id'] = spec1d_prov.id
+  spec_struct['meta']['diaobject_id'] = diaobject.id
+  spec_struct['meta']['diaobject_position_id'] = diaobj_pos['id']
+  spec_struct['meta']['image_ids'] = [ i.id for i in images ]
+
+  # Make sure that all (but see below) of the other mandatory metadata is there
+
+  spec1d = Spectrum1d( spec_struct )
+  spec1d.write()
+  spec1d.save_to_db( dbclient=dbclient )
+  
+  
+You do *not* need to set ``meta['id']`` or ``meta['filepath']`` yourself; those will be set automatically when you save the sepctrum.
+
+Note that when you create a ``Spectrum1d``, it will keep a *copy* of the ``spec_struct`` object you passed in its ``spec_struct`` property.  It will also modify this object, in particular, setting the ``id`` when the ``Specrtrum1d`` object is constructed, and setting the ``filepath`` when it is saved.
+
+
+
+
+.. _nov2025-working-env:
 
 Choose a working environment
 ============================

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0/roman_snpit_snappl.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: roman_snpit_snappl
-Version: 0.21.0
+Version: 0.23.0
 Summary: General, database, and photometry utilities for the Roman SNPIT
 Author: Roman Supernova Project Infrastructure Team
 Maintainer-email: Roman SN PIT <raknop@lbl.gov>

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0}/roman_snpit_snappl.egg-info/SOURCES.txt

@@ -25,6 +25,11 @@ tox.ini
 .github/workflows/sub_package_update.yml
 changes/.gitkeep
 changes/10.snappl.rst
+changes/103.snappl.rst
+changes/105.docs.rst
+changes/106.docs.rst
+changes/107.feature.rst
+changes/109.bugfix.rst
 changes/13.bugfix.rst
 changes/14.snappl.rst
 changes/15.feature.rst
@@ -122,7 +127,9 @@ snappl/utils.py
 snappl/wcs.py
 snappl/_dev/__init__.py
 snappl/_dev/scm_version.py
+snappl/admin/load_nov2025_truth_diaobjects.py
 snappl/admin/load_ou2024_l2images.py
+snappl/admin/load_ou2024_nov2025_segmaps.py
 snappl/admin/load_snana_ou2024_diaobject.py
 snappl/data/README.rst
 snappl/db/baseview.py
@@ -133,6 +140,7 @@ snappl/db/migrations/20251017_objdetcount.sql
 snappl/db/migrations/20251024_l2image_mjd.sql
 snappl/db/migrations/20251025_lightcurve.sql
 snappl/db/migrations/20251028_segmap.sql
+snappl/db/migrations/20251031_posang.sql
 snappl/db/migrations/apply_migrations.py
 snappl/db/migrations/schema_to_rst.py
 snappl/db/migrations/scorched_earth.py

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0}/snappl/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.21.0'
-__version_tuple__ = version_tuple = (0, 21, 0)
+__version__ = version = '0.23.0'
+__version_tuple__ = version_tuple = (0, 23, 0)
 
-__commit_id__ = commit_id = 'g7a9941597'
+__commit_id__ = commit_id = 'g19a7a9205'

roman_snpit_snappl-0.23.0/snappl/admin/load_nov2025_truth_diaobjects.py

@@ -0,0 +1,190 @@
+import re
+import uuid
+import simplejson
+import pathlib
+import functools
+import argparse
+import multiprocessing
+
+import pandas
+import shapely
+
+from snappl.logger import SNLogger
+from snappl.utils import SNPITJsonEncoder
+import snappl.db.db
+
+
+def load_snana_ou2024_diaobject( provid, polys, pqf, logevery=100, dbcon=None ):
+    match = re.search( r'^snana_([0-9]+)\.parquet$', pqf.name )
+    if match is None:
+        raise ValueError( f"Failed to parse filename {match}" )
+    healpix = int( match.group(1) )
+    df = pandas.read_parquet( pqf )
+
+    thingstoinsert = []
+    nloaded = 0
+    for n, row in enumerate( df.itertuples() ):
+        if n % logevery == 0:
+            SNLogger.info( f"File {pqf.name}, {n} of {len(df)} done, {nloaded} prepped for load" )
+
+        params = { k: v for k, v in zip( row.model_param_names, row.model_param_values ) }
+
+        included = False
+        for polydex, poly in enumerate( polys ):
+            # if polydex % 500 == 0:
+            #     SNLogger.info( f"...evaluating polygon {polydex} of {len(polys)}" )
+            if poly.contains( shapely.Point( row.ra, row.dec ) ):
+                # import random
+                # import remote_pdb; remote_pdb.RemotePdb('127.0.0.1', random.randint(4000,60000) ).set_trace()
+                included = True
+                break
+
+        if not included:
+            continue
+
+        subdict = { 'id':  uuid.uuid4(),
+                    'provenance_id': provid,
+                    'name': str(row.id),
+                    'iauname': None,
+                    'ra': float(row.ra),
+                    'dec': float(row.dec),
+                    'mjd_discovery': float(row.start_mjd),
+                    'mjd_peak': float(row.peak_mjd),
+                    'mjd_start': float(row.start_mjd),
+                    'mjd_end': float(row.end_mjd),
+                    'ndetected': 2,
+                    'properties': simplejson.dumps(
+                        { 'healpix': healpix,
+                          'host_id': int(row.host_id),
+                          'gentype': int(row.gentype),
+                          'model_name': row.model_name,
+                          'z_cmb': float(row.z_CMB),
+                          'mw_ebv': float(row.mw_EBV),
+                          'mw_extinction_applied': float(row.mw_extinction_applied),
+                          'av': float(row.AV),
+                          'rv': float(row.RV),
+                          'v_pec': float(row.v_pec),
+                          'host_ra': float(row.host_ra),
+                          'host_dec': float(row.host_dec),
+                          'host_mag_g': float(row.host_mag_g),
+                          'host_mag_i': float(row.host_mag_i),
+                          'host_mag_f': float(row.host_mag_F),
+                          'host_sn_sep': float(row.host_sn_sep),
+                          'peak_mjd': float(row.peak_mjd),
+                          'peak_mag_g': float(row.peak_mag_g),
+                          'peak_mag_i': float(row.peak_mag_i),
+                          'peak_mag_f': float(row.peak_mag_F),
+                          'lens_dmu': float(row.lens_dmu),
+                          'lens_dmu_applied': bool(row.lens_dmu_applied),
+                          'model_params': params },
+                        sort_keys=True,
+                        cls=SNPITJsonEncoder )
+                   }
+        thingstoinsert.append( subdict )
+        nloaded += 1
+
+    snappl.db.db.DiaObject.bulk_insert_or_upsert( thingstoinsert, dbcon=dbcon )
+
+    return len( thingstoinsert )
+
+
+def get_rects( dbcon=None ):
+    with snappl.db.db.DBCon( dictcursor=True ) as dbcon:
+        rows = dbcon.execute( "SELECT ra_corner_00, ra_corner_01, ra_corner_10, ra_corner_11,"
+                              "  dec_corner_00, dec_corner_01, dec_corner_10, dec_corner_11 "
+                              "FROM l2image" )
+    polys = []
+    for row in rows:
+        polys.append( shapely.Polygon( ( (row['ra_corner_00'], row['dec_corner_00']),
+                                         (row['ra_corner_01'], row['dec_corner_01']),
+                                         (row['ra_corner_11'], row['dec_corner_11']),
+                                         (row['ra_corner_10'], row['dec_corner_10']),
+                                         (row['ra_corner_00'], row['dec_corner_00'])
+                                        )
+                                      ) )
+    return polys
+
+
+
+def main():
+    parser = argparse.ArgumentParser( 'load_snana_ou2024_diaobject',
+                                      description='Load all parquet files below a directory.' )
+    parser.add_argument( '-p', '--provid', required=True, help="Provenance id" )
+    parser.add_argument( '-n', '--nprocs', type=int, default=12,
+                         help="Number of processes to run at once [default: 12]" )
+    parser.add_argument( 'basedir', help='Base directory.' )
+    args = parser.parse_args()
+
+    with snappl.db.db.DBCon( dictcursor=True ) as dbcon:
+        rows = dbcon.execute( "SELECT * FROM provenance WHERE id=%(id)s", { 'id': args.provid } )
+        if len(rows) == 0:
+            raise ValueError( "Invalid provenance {args.provid}" )
+        SNLogger.info( f"Loading provenance for process {rows[0]['process']} {rows[0]['major']}.{rows[0]['minor']}" )
+
+    SNLogger.info( "Creating shapely polygons of image footprints...." )
+    polys = get_rects()
+    SNLogger.info( f"...have {len(polys)} polygons." )
+
+    # First, collect all the parquet files
+
+    def find_pqfiles( direc ):
+        pqfiles = []
+        subdirs = []
+        for f in direc.iterdir():
+            f = f.resolve()
+            if ( f.name[:5] == 'snana') and ( f.name[-8:] == '.parquet' ):
+                pqfiles.append( f )
+            elif f.is_dir():
+                subdirs.append( f )
+
+        for subdir in subdirs:
+            pqfiles.extend( find_pqfiles( subdir ) )
+
+        return pqfiles
+
+    basedir = pathlib.Path( args.basedir )
+    pqfiles = find_pqfiles( basedir )
+    SNLogger.info( f"{len(pqfiles)} parquet files to load" )
+    SNLogger.debug( f"All pqfiles we're going to do: {pqfiles}" )
+
+    # Launch parallel processes to load these files
+
+    did = []
+    tot = 0
+
+    def add_to_did( n ):
+        nonlocal tot, did
+        tot += n
+        did.append( n )
+        SNLogger.info( f"Completed {len(did)} of {len(pqfiles)} parquet files ({tot} objects)" )
+
+    errors = []
+
+    def omg( e ):
+        nonlocal errors
+        SNLogger.error( f"Subprocess returned error: {e}" )
+        errors.append( e )
+
+    do_load = functools.partial( load_snana_ou2024_diaobject, args.provid, polys )
+
+    if args.nprocs > 1:
+        with multiprocessing.Pool( args.nprocs ) as pool:
+            for pqf in pqfiles:
+                pool.apply_async( do_load, [ pqf ], callback=add_to_did, error_callback=omg )
+            pool.close()
+            pool.join()
+    else:
+        for pqf in pqfiles:
+            add_to_did( do_load( pqf ) )
+
+    if len(errors) > 0:
+        nl = '\n'
+        SNLogger.error( f"There were errors:\n{nl.join(str(e) for e in errors)}" )
+        raise RuntimeError( "There were errors!" )
+
+    SNLogger.info( f"Loaded {tot} objects in {len(did)} parquet files." )
+
+
+# ======================================================================
+if __name__ == "__main__":
+    main()

{roman_snpit_snappl-0.21.0 → roman_snpit_snappl-0.23.0}/snappl/admin/load_ou2024_l2images.py

@@ -54,6 +54,7 @@ def _parse_fits_file( relpath, base_path=None, provid=None ):
                'height': height,
                'format': 1,
                'mjd': image.mjd,
+               'position_angle': image.position_angle,
                'exptime': exptime,
                'properties': psycopg.types.json.Jsonb( {} )
               }