roman-snpit-snappl 0.20.0__tar.gz → 0.22.0__tar.gz

{roman_snpit_snappl-0.20.0/roman_snpit_snappl.egg-info → roman_snpit_snappl-0.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: roman_snpit_snappl
-Version: 0.20.0
+Version: 0.22.0
 Summary: General, database, and photometry utilities for the Roman SNPIT
 Author: Roman Supernova Project Infrastructure Team
 Maintainer-email: Roman SN PIT <raknop@lbl.gov>

roman_snpit_snappl-0.22.0/changes/103.snappl.rst

@@ -0,0 +1 @@
+Small changes for Nov 2025 run

roman_snpit_snappl-0.22.0/changes/105.docs.rst

@@ -0,0 +1 @@
+Write recipes in Usage docs for the nov2025 run

roman_snpit_snappl-0.22.0/changes/106.docs.rst

@@ -0,0 +1 @@
+Config file in docs for nov 2025

roman_snpit_snappl-0.22.0/changes/107.feature.rst

@@ -0,0 +1 @@
+Add position_angle to segmap and l2image database tables

roman_snpit_snappl-0.22.0/changes/98.docs.rst

@@ -0,0 +1 @@
+Add lightcurve reading and writing to the nov 2025 primer in the usage docs

roman_snpit_snappl-0.22.0/changes/99.feature.rst

@@ -0,0 +1 @@
+Add storage and database tracking of segmentation maps

{roman_snpit_snappl-0.20.0 → roman_snpit_snappl-0.22.0}/docs/usage.rst

@@ -27,16 +27,287 @@ November 2025 SNPIT Pipeline Test Primer
 The database connection is under heavy development, and more things are showing up every day.  Right now, the following is available:
 
 * Find image L2 images in the database
+* Find segmentation maps in the database
 * Saving newly discovered DiaObjects to the database
 * Finding DiaObjects
 * Saving updated positions for DiaObjects
-
-I *hope* that we will have saving lightcurves and finding lightcurves also available by the November test.  How much spectrum saving, and how much characterization saving, is implemented will depend on my time and what we get from those working groups.
-
+* Saving lightcurves to the database
+* Finding and reading lightcurves from the database
+  
 This section describes what you need to do in order to connect to the database.
 
 **WARNING**: because everything is under heavy development, it's possible that interfaces will change.  We will try to avoid doing this, because it's a pain when everybody has to adapt, but we're still building this, so it may be inevitable.
 
+Recipes
+=======
+
+Read everything to understand what's going on, but these are intended as a quick start.
+
+These recipes assume you have a working environment (see :ref:`nov2025-working-env`), which hopefully is as simple as ``pip install roman-snpit-snappl``, but see that section for all the details and where you eventually need to be heading.
+
+They also assume you have set up a config file.  If you're on NERSC, *not* running in a container, then save `this config file <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/nov2025_nersc_native_config.yaml>`_.   (this is the file ``nov2025_nersc_native_config.yaml`` from the top level of the ``environment`` roman snpit github archive).  If you are running in a podman container, then look at `this config file <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/nov2025_container_config.yaml>`_ (this is the file ``nov2025_nersc_container_config.yaml`` from the top level of the ``environment`` roman snpit github archive); you will also need to download `interactive-podman-nov2025.sh <https://raw.githubusercontent.com/Roman-Supernova-PIT/environment/refs/heads/main/interactive-podman-nov2025.sh>`_.   If you are elsewhere, you will need to edit the config file to have the right paths to find things on your system.
+
+You need to set the environment variable ``SNPIT_CONFIG`` to point to where this configuration file lives.
+
+Finally, once at the top of your code you need to do::
+  
+  from snappl.dbclient import SNPITDBClient
+
+  dbclient = SNPITDBClient()
+
+Most of the recipes below use the ``dbclient`` variable.
+
+  
+.. _recipe-command-line-args:
+
+Command-line Arguments
+----------------------
+
+Below, you will be told you need to know a number of object ids and/or provenance-related values.  These will generally be provided by orchestration.  You should make them things that can be passed on the command line.  I recommend using the following command-line arguments — choose the ones that you need (they are all string values)::
+
+  --diaobject-id
+  --diaobject-provenance-tag
+  --diaobject-process
+  --diaobject-position-provenance-tag
+  --diaobject-position-process
+  --image-id
+  --image-provenance-tag
+  --image-process
+  --segmap-provenance-tag
+  --segmap-process
+  --ltcv-provenance-tag
+  --ltcv-process
+  --spec1d-provenance-tag
+  --spec1d-process
+
+
+.. _recipe-find-diaobject:
+
+Finding a DiaObject
+-------------------
+
+You need to know *either* the ``diaobjectd_id`` of the object (which you will generally be given), or you need to know the ``diaobject_provenance_tag`` and ``diaobject_process``, and you must have enough search criteria to find the object.  If you're doing the latter, read the docstring on ``snappl.diaobject.DiaObject.find_objects``.  For the former::
+
+  from snappl.diaobject import DiaObject
+
+  diaobject = DiaObject.get_object( diaobject_id=diaobject_id, dbclient=dbclient )
+
+The returned ``DiaObject`` object has, among other things, properties ``.ra`` and ``.dec``.
+
+
+.. _recipe-diaobject-position:
+
+Getting an updated diaobject position
+-------------------------------------
+
+You need the ``diaobject_position_provenance_tag`` and ``diaobject_position_process``.
+
+Do::
+
+  diaobj_pos = diaobject.get_position( provenance_tag=diaobject_position_provenance_tag,
+                                       process=diaobject_position_process,
+                                       dbclient=dbclient )
+
+You get back a dictionary that has a number of keys including ``ra`` and ``dec``.
+
+   
+
+
+Getting a Specific Image
+------------------------
+
+ROB IMPLEMENT THIS BETTER
+
+Orchestration has given you a ``image_id`` that you are supposed to do something with.  E.g.,, you are running sidecar, and you're supposed to subtract and search this image.  Right now you also need to know the images's ``image_provenance_tag`` and ``image_process``, but that requirement will go away when Rob fixes it::
+
+  from snappl.imagecollection import ImageCollection
+  from snappl.image import Image
+
+  collection = ImageCollection.get_collection( provenance_tag=image_provenance_tag,
+                                               process=image_process, dbclient=dbclient )
+  image = collection.get_image( image_id=image_id, dbclient=dbclient )
+
+You will get back an ``Image`` object.  It has a number of properties.  Most important are ``.data``, ``.noise``, and ``.flags``, which hold 2d numpy arrays.  There is also a ``.get_fits_header()`` method that currently works, **but be careful using this as this method will not work in the future when we're using ASDF files**.  See the docstrings in ``snappl.image.Image`` for more details.  Some of the stuff you might want is available directly as properties of and ``Image`` object.
+                                               
+
+Finding Images
+--------------
+
+You need to know the ``image_provenance_tag`` and ``image_process``.
+
+See the docstring on ``snappl.imagecollection.ImageCollection.find_images`` if you want to do more than what's below.
+
+
+Finding all images that include a ra and dec
+********************************************
+
+Do::
+
+  from snappl.imagecollection import ImageCollection
+
+  collection = ImageCollection.get_collection( provenance_tag=image_provenance_tag,
+                                               process=image_process, dbclient=dbclient )
+  images = collection.find_images( ra=ra, dec=dec, band=band, dbclient=dbclient )
+
+where ``band=band`` is optional but often useful.  You will get back a list of ``Image`` objects, which have a number of properties.  Most important are ``.data``, ``.noise``, and ``.flags``, which hold 2d numpy arrays.  There is also a ``.get_fits_header()`` method that currently works, **but be careful using this as this method will not work in the future when we're using ASDF files**.  See the docstrings in ``snappl.image.Image`` for more details.
+
+Finding Segmentation Maps
+-------------------------
+
+You need to know the ``segmap_provenance_tag`` and the ``segmap_process``.
+
+See the dockstring on ``snappl.segmap.SegmentationMap.find_segmaps`` for more information on searches you can do beyond what's below.
+
+Finding all segmaps that include a ra and dec
+*********************************************
+
+Do::
+
+  from snappl.segmap import SegmentationMap
+
+  segmaps = SegmentationMap.find_segmaps( provenance_tag=segmap_provenance_tag,
+                                          process=segmap_process,
+                                          ra=ra, dec=dec,
+                                          dbclient=dbclient )
+
+You get back a list of ``SegmentationMap`` objects.  These have a number of properties, most import of which is ``image``, which holds an ``Image`` object.  You can get the image data for the segmentation map for the first element of the list with ``segmaps[0].image.data`` (a 2d numpy array).
+
+
+Saving a new DiaObject
+----------------------
+
+You are running sidecar and you've found a new diaobject you want to save.  You need a ``process`` (we shall assume ``process='sidecar'`` here), the ``major`` and ``minor`` version of your code, and the ``params`` that define how the code runs.  The latter is just a dictionary; you can build it yourself, but see :ref:`nov2025-making-prov` below.  Finally, assume that ``images`` is an list that has the ``snappl.image.Image`` objects of the images that you've used; replace ``images[0]`` below with wherever you have your ``Image`` object::
+
+  from snappl.provenance import Provenance
+  from snappl.diaobject import DiaObject
+
+  imageprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  prov = Provenance( process='sidecar', major=major, minor=minor, params=params,
+                     upstreams=[ imageprov ] )
+  # You only have to do this next line once for a given provenance;
+  #   once the provenance is in the databse, you never need to save it again.
+  prov.save_to_db( tag=diaobject_provenance_tag, dbclient=dbclient )   # See note below
+
+  diaobj = DiaObject( provenance_id=prov.id, ra=ra, dec=dec, name=optional, mjd_discovery=mjd )
+  diaobj.save_object( dbclient=dbclient )
+  
+
+*Note*: right now, you'll get exceptions on the ``prov.save_to_db`` line, but it probaby did work.  There is a snappl issue out for Rob to fix this.
+
+This will save the object to the database.  You can then look at ``diaobj.id`` to see what UUID it was assigned.  You do not need to give it a ``name``, but you can if you want to.  (The database uses the ``id`` as the unique identifier.)  ``mjd_discovery`` should be the MJD of the science image that the object was found on.
+
+Finding and reading lightcurves
+-------------------------------
+
+You need to know the ``ltcv_provenance_tag`` and ``ltcv_process``, and the ``diaobject_id`` of the object for which you want to get lightcurves::
+
+  from snappl.lightcurve import Lightcurve
+
+  ltcvs = Lightcurve.find_lightcurves( provenance_tag=ltcv_provenance_tag,
+                                       process=ltcv_process,
+                                       diaobject=diaobject_id,
+                                       band=band,         # optional
+                                       dbclient=dbclient )
+
+You will get back a list of ``Lightcurve`` objects.  You can find the actual lightcurve data of the first lightcurve from the list with ``ltcvs[0].lightcurve``.  This is an astropy QTable.  You can read the metadata from ``ltcvs[0].lightcurve.meta``.
+
+**Coming soon**: a way to read a combined lightcurve that has all of the bands mixed together.  (Not implemented yet.)
+
+
+Saving lightcurves
+------------------
+
+You need to make sure you've created a dictionary with `all the necessary metadata <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/lightcurve>`_.  Also make sure you've created a data table with the necessary columns; this can be an astropy Table, a pandas DataFrame, or a dict of lists.  We shall call these two things ``meta`` and ``data``.
+
+Assume that you've made the lightcurve for object ``diaobject`` (a ``DiaObject`` object), and that you have a list of your images in ``images``.  Adjust below for the variables where you really have things.  Finally, if you used an updated :ref:`DiaObject position <recipe-diaobject-position>`, make sure you have set the ``ra`` and ``dec`` in ``meta`` from that.
+
+Finally, you will need to know the ``ltcv_provenance_tag`` we're using.
+
+Below, ``process`` is probably either ``campari`` or ``phrosty``.  ``major`` and ``minor`` are the major and minor parts of the version, which you should parse from ``campari.__version__`` or ``phrosty.__version__``.  ``params`` are the parameters as described below in :ref:`nov2025-making-prov`.
+
+Do::
+
+  from snappl.provenance import Provenance
+  from snappl.lightcurve import Lightcurve
+
+  imgprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  objprov = Provenance.get_by_id( diaobject.provenance_id, dbclient=dbclient )
+  objposprov = Provenance.get_by_id( diaobj_pos['provenance_id'] )
+
+  ltcvprov = Provenance( process=process, major=major, minor=minor, params=params,
+                         upstreams=[imgprov, objprov, objposprov], dbclient=dbclient )
+  # The next line only needs to be run once.  Once you've saved it to the database,
+  #   you never need to do this again.
+  ltcvprov.save_to_db( tag=ltcv_provenance_tag )
+
+  meta['provenance_id'] = ltcvprov.id
+  meta['diaobject_id'] = diaobject.id
+  meta['diaobject_position_id'] = diaobj_pos['id']
+  for att in [ 'ra', 'dec', 'ra_err', 'dec_err', 'ra_dec_covar' ]:
+      meta[att] = diaobj_pos[att]
+
+  ltcv = Lightcurve( data=data, meta=meta )
+  ltcv.write()
+  ltcv.save_to_db( dbclient=dbclient )
+
+You can look at ``ltcv.id`` to see the ``UUID`` of the lightcurve you saved, in case you are curious.
+  
+If you used the ``ra`` and ``dec`` that was in ``DiaObject``, then ``meta['diaobject_position_id']`` should be ``None``.  Skip everything else above that refers to ``diaobj_pos``.
+  
+
+
+Finding and reading 1d Spectra
+------------------------------
+
+(Not implemented yet.)
+
+Saving 1d Spectra
+-----------------
+
+**Warning: this is not implemented yet.  When it is, the process will look *something* like hte following.**
+
+You need to have the ``diaobject`` (a ``DiaObject`` object) for which you made the spectrum, potentially a ``diaobj_pos``, an improved position for the object, and ``images``, a list of ``Image`` object that held the dispersed images from which you are making the spectrum.  You need to know the ``spec1d_provenance_tag``.
+
+You need to know the ``process`` (which is probably just the name of your code), and the ``major`` and ``minor`` versions of your code.  Finally, you need to know the ``params`` that define how your code runs.   The latter is just a dictionary; you can build it yourself, but see :ref:`nov2025-making-prov` below.
+
+You build a data structure that is described on `the wiki <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/spectrum_1d>`_; call that ``spec_struct``.  Some of lines below make sure that some of this metadata is right.
+
+Do::
+
+  from snappl.provenance import Provenance
+  from snappl.spec1d import Spectrum1d
+
+  diaobj_prov = Provenance.get_by_id( diaobject.provenance_id, dbclient=dbclient )
+  imageprov = Provenance.get_by_id( images[0].provenance_id, dbclient=dbclient )
+  diaobj_pos_prov = Provenance.get_by_id( diaobj_pos['id'], dbclient=dbclient )
+
+  spec1d_prov = Provenance( process=process, major=major, minor=minor, params=params,
+                            upstreams=[ diaobj_prov, imageprov, diaobj_pos_prov ] )
+  # The next line only needs to be run once.  Once
+  #   you have saved a Provenance to the databse you
+  #   never need to save it again
+  spec1d_prov.save_to_db( tag=spec1d_provenance_tag, dbclient-dbclient )
+
+  spec_struct['meta']['provenance_id'] = spec1d_prov.id
+  spec_struct['meta']['diaobject_id'] = diaobject.id
+  spec_struct['meta']['diaobject_position_id'] = diaobj_pos['id']
+  spec_struct['meta']['image_ids'] = [ i.id for i in images ]
+
+  # Make sure that all (but see below) of the other mandatory metadata is there
+
+  spec1d = Spectrum1d( spec_struct )
+  spec1d.write()
+  spec1d.save_to_db( dbclient=dbclient )
+  
+  
+You do *not* need to set ``meta['id']`` or ``meta['filepath']`` yourself; those will be set automatically when you save the sepctrum.
+
+Note that when you create a ``Spectrum1d``, it will keep a *copy* of the ``spec_struct`` object you passed in its ``spec_struct`` property.  It will also modify this object, in particular, setting the ``id`` when the ``Specrtrum1d`` object is constructed, and setting the ``filepath`` when it is saved.
+
+
+
+
+.. _nov2025-working-env:
 
 Choose a working environment
 ============================
@@ -145,6 +416,14 @@ However, we recommend against that.  While 4500 is perhaps not an overwhelming n
 Finding Objects
 ===============
 
+You may just be given an diaobject id.  In that case, all you have to do is::
+
+  from snappl.diaobject import DiaObject
+
+  obj = DiaObject.get_object( diaobject_id=<id> )
+
+where ``<id>`` is the diaobject id you were given.  ``obj`` will be a ``DiaObject`` object.
+
 There is also an interface that lets you find objects.  For instance, if you want to find all objects within 100 arcseconds of a given location, you could run::
 
   from snappl.diaobject import DiaObject
@@ -158,7 +437,7 @@ Here, you can use ``ou2024`` for ``TAG`` and ``load_ou2024_diaobject`` for ``PRO
 Getting Better Object Positions
 ===============================
 
-``DiaObject.find_objects`` will return a list of ``DiaObject`` objects, and these include properties ``ra`` and ``dec``.  **However, the positions in the DiaObject object should be viewed as approximate.**  They will be the position it had when the object was first discovered.  For objects loaded from truth tables, they will be perfect, but of course we won't have truth tables for the real survey.  Often, the first discovery will be a relatively low S/N point, and much better positions can be determined; doing so will be one of the jobs of ``phrosty``.
+The ``DiaObject`` you got from ``DiaObject.get_object`` or ``DiaObject.find_object`` include properties ``ra`` and ``dec``.  **However, the positions in the DiaObject object should be viewed as approximate.**  They will be the position it had when the object was first discovered.  For objects loaded from truth tables, they will be perfect, but of course we won't have truth tables for the real survey.  Often, the first discovery will be a relatively low S/N point, and much better positions can be determined; doing so will be one of the jobs of ``phrosty``.
 
 To get an improved position for an object, assume you have the object in the variable ``diaobj``.  You can then call::
 
@@ -181,10 +460,36 @@ See :ref:`nov2025-provtags` below to figure out what ``TAG`` and ``PROCESS`` sho
 **Important**: if you use an updated DiaObject position, then the provenance of that position should be one of your upstream provenances; see :ref:`nov2025-making-prov`.
 
 
-Finding Lightcurves
+Reading Lightcurves
 ===================
 
-TODO
+To read a lightcurve, you need three or four things:
+
+  * The ``diaobject_id`` of the object whose lightcurve you want
+  * The ``provenance_tag`` of the provenance you want to get lightcurves from
+  * The ``process`` to go with the provenance tag
+  * (Optional) The ``band`` of the lightcurve
+
+The ``provenance_tag`` and ``process`` will be given to you through our fake orchestration process, and may possibly be found in :ref:`nov2025-provtags`.  The ``diaobject_id`` will be given to you through the fake orchestration process.
+
+To get the lightcurves::
+
+  from snappl.lightcurve import Lightcurve
+
+  ltcvs = Lightcurve.find_lightcurves( <diaobject_id>, provenance_tag=<tag>, process=<proc>, dbclient=dbclient )
+
+That will return a list of lightcurves.  If you also specify ``band=<band>``, that will be a 1-element list with the lightcurve just for that band.  (Or a 0-element list if it's not found.)
+
+Each element of the list will be a ``Lightcurve`` object.  You can find the actual lightcurve data in the ``.lightcurve`` property as an astropy QTable.  You can find the metadata dictionary either in the ``.lightcurve.meta`` or in the ``.meta`` property (though the latter will intially be ``None`` until you access the ``.lightcurve`` property).  Guaranteed metadata can be found in the `lightcurve schema specification on the PIT wiki <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/lightcurve>`_.  You should probably ignore the ``filepath`` in the metadata, because ``snappl`` has already read the file for you (and put it in the ``.lightcurve`` property).
+
+
+Finding Segmentation Maps
+=========================
+
+You will need to ``from snappl.segmap import SegmentationMap`` and then call ``SegmentationMap.find_segmaps``.  You need to pass a provenance tag and a process to ``find_segmaps``; you will be given these (see :ref:`nov2025-provtags`).  Beyond that, look at the docstring for that function to see what you can search on.
+
+You will get back a list of ``SegmentationMap`` objects.  A ``SegmentationMap`` object has several attributes, including ``band``, and eight attributes ``ra_corner_00``, ``dec_corner_00``, etc., where ``00`` is the lower-left pixel, ``01`` upper-left pixel, ``10`` is the lower-right pixel, and ``11`` is the upper-right pixel.  The most important field is ``.image``.  This is a ``snappl.image.Image`` object.  You can get the 2d numpy array of the image data of the segmentaton map from the ``.data`` property of the ``Image`` object.
+
 
 .. _nov2025-making-prov:
 
@@ -250,9 +555,24 @@ This will (I believe) return a dictionary that's the same as what you'd get back
 Saving Lightcurves
 ==================
 
-TODO
+Lightcurves saved to the database are for only a single band.  If you have a multiband lightcurve, from the point of view of the database that's several different lightcurves.
+
+To write a Lightcurve, first create a ``Lightcurve`` object::
+
+  from snappl.lightcurve import Lightcurve
+
+  ltcv = Lightcurve( data=<data>, meta=<meta> )
+
+where ``<meta>`` is a dictionary with metadata, and ``<data>`` is one of an astropy Table, a pandas DataFrame, or a dictionary of lists.  On the `lightcurve schema specification on the PIT wiki <https://github.com/Roman-Supernova-PIT/Roman-Supernova-PIT/wiki/lightcurve>`_ you can find the mandatory fields in the metadata dictionary; it's allowed to have additional ones as well.  Likewise, there you can find the mandatory columns (and the order of those columns) in the data array.  You may also have additional columns in that data array.
+
+In order to create a lightcurve, you will need to make a provenance for it; see :ref:`nov2025-making-prov`.  Make sure to include the ``diaobject position`` provenance as an upstream provenance if you used a position other than the cheesy approximate one that comes with the ``DiaObject``.
+
+Once you have your lightcurve object, do two things::
 
+  ltcv.write()
+  ltcv.save_to_db( dbclient=dbclient )
 
+The first one writes the actual file; it will write it in the standard location, and will populate the ``.filepath`` property with the location of the file *relative to the configured base directory for lightcurves* (which is in config option ``system.paths.lightcurves``).  The second call saves a record to the database with information about your lightcurve.
 
 .. _nov2025-provtags:
 

{roman_snpit_snappl-0.20.0 → roman_snpit_snappl-0.22.0/roman_snpit_snappl.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: roman_snpit_snappl
-Version: 0.20.0
+Version: 0.22.0
 Summary: General, database, and photometry utilities for the Roman SNPIT
 Author: Roman Supernova Project Infrastructure Team
 Maintainer-email: Roman SN PIT <raknop@lbl.gov>

{roman_snpit_snappl-0.20.0 → roman_snpit_snappl-0.22.0}/roman_snpit_snappl.egg-info/SOURCES.txt

@@ -25,6 +25,10 @@ tox.ini
 .github/workflows/sub_package_update.yml
 changes/.gitkeep
 changes/10.snappl.rst
+changes/103.snappl.rst
+changes/105.docs.rst
+changes/106.docs.rst
+changes/107.feature.rst
 changes/13.bugfix.rst
 changes/14.snappl.rst
 changes/15.feature.rst
@@ -70,6 +74,8 @@ changes/9.snappl.rst
 changes/91.feature.rst
 changes/92.docs.rst
 changes/94.snappl.rst
+changes/98.docs.rst
+changes/99.feature.rst
 docker/postgres/Dockerfile
 docker/postgres/postgresql.conf
 docker/postgres/run_postgres.sh
@@ -115,11 +121,14 @@ snappl/logger.py
 snappl/provenance.py
 snappl/psf.py
 snappl/sed.py
+snappl/segmap.py
 snappl/utils.py
 snappl/wcs.py
 snappl/_dev/__init__.py
 snappl/_dev/scm_version.py
+snappl/admin/load_nov2025_truth_diaobjects.py
 snappl/admin/load_ou2024_l2images.py
+snappl/admin/load_ou2024_nov2025_segmaps.py
 snappl/admin/load_snana_ou2024_diaobject.py
 snappl/data/README.rst
 snappl/db/baseview.py
@@ -129,6 +138,8 @@ snappl/db/migrations/20251008_init.sql
 snappl/db/migrations/20251017_objdetcount.sql
 snappl/db/migrations/20251024_l2image_mjd.sql
 snappl/db/migrations/20251025_lightcurve.sql
+snappl/db/migrations/20251028_segmap.sql
+snappl/db/migrations/20251031_posang.sql
 snappl/db/migrations/apply_migrations.py
 snappl/db/migrations/schema_to_rst.py
 snappl/db/migrations/scorched_earth.py

{roman_snpit_snappl-0.20.0 → roman_snpit_snappl-0.22.0}/snappl/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.20.0'
-__version_tuple__ = version_tuple = (0, 20, 0)
+__version__ = version = '0.22.0'
+__version_tuple__ = version_tuple = (0, 22, 0)
 
-__commit_id__ = commit_id = 'g3f6286e64'
+__commit_id__ = commit_id = 'gb84fe5a43'

roman_snpit_snappl-0.22.0/snappl/admin/load_nov2025_truth_diaobjects.py

@@ -0,0 +1,190 @@
+import re
+import uuid
+import simplejson
+import pathlib
+import functools
+import argparse
+import multiprocessing
+
+import pandas
+import shapely
+
+from snappl.logger import SNLogger
+from snappl.utils import SNPITJsonEncoder
+import snappl.db.db
+
+
+def load_snana_ou2024_diaobject( provid, polys, pqf, logevery=100, dbcon=None ):
+    match = re.search( r'^snana_([0-9]+)\.parquet$', pqf.name )
+    if match is None:
+        raise ValueError( f"Failed to parse filename {match}" )
+    healpix = int( match.group(1) )
+    df = pandas.read_parquet( pqf )
+
+    thingstoinsert = []
+    nloaded = 0
+    for n, row in enumerate( df.itertuples() ):
+        if n % logevery == 0:
+            SNLogger.info( f"File {pqf.name}, {n} of {len(df)} done, {nloaded} prepped for load" )
+
+        params = { k: v for k, v in zip( row.model_param_names, row.model_param_values ) }
+
+        included = False
+        for polydex, poly in enumerate( polys ):
+            # if polydex % 500 == 0:
+            #     SNLogger.info( f"...evaluating polygon {polydex} of {len(polys)}" )
+            if poly.contains( shapely.Point( row.ra, row.dec ) ):
+                # import random
+                # import remote_pdb; remote_pdb.RemotePdb('127.0.0.1', random.randint(4000,60000) ).set_trace()
+                included = True
+                break
+
+        if not included:
+            continue
+
+        subdict = { 'id':  uuid.uuid4(),
+                    'provenance_id': provid,
+                    'name': str(row.id),
+                    'iauname': None,
+                    'ra': float(row.ra),
+                    'dec': float(row.dec),
+                    'mjd_discovery': float(row.start_mjd),
+                    'mjd_peak': float(row.peak_mjd),
+                    'mjd_start': float(row.start_mjd),
+                    'mjd_end': float(row.end_mjd),
+                    'ndetected': 2,
+                    'properties': simplejson.dumps(
+                        { 'healpix': healpix,
+                          'host_id': int(row.host_id),
+                          'gentype': int(row.gentype),
+                          'model_name': row.model_name,
+                          'z_cmb': float(row.z_CMB),
+                          'mw_ebv': float(row.mw_EBV),
+                          'mw_extinction_applied': float(row.mw_extinction_applied),
+                          'av': float(row.AV),
+                          'rv': float(row.RV),
+                          'v_pec': float(row.v_pec),
+                          'host_ra': float(row.host_ra),
+                          'host_dec': float(row.host_dec),
+                          'host_mag_g': float(row.host_mag_g),
+                          'host_mag_i': float(row.host_mag_i),
+                          'host_mag_f': float(row.host_mag_F),
+                          'host_sn_sep': float(row.host_sn_sep),
+                          'peak_mjd': float(row.peak_mjd),
+                          'peak_mag_g': float(row.peak_mag_g),
+                          'peak_mag_i': float(row.peak_mag_i),
+                          'peak_mag_f': float(row.peak_mag_F),
+                          'lens_dmu': float(row.lens_dmu),
+                          'lens_dmu_applied': bool(row.lens_dmu_applied),
+                          'model_params': params },
+                        sort_keys=True,
+                        cls=SNPITJsonEncoder )
+                   }
+        thingstoinsert.append( subdict )
+        nloaded += 1
+
+    snappl.db.db.DiaObject.bulk_insert_or_upsert( thingstoinsert, dbcon=dbcon )
+
+    return len( thingstoinsert )
+
+
+def get_rects( dbcon=None ):
+    with snappl.db.db.DBCon( dictcursor=True ) as dbcon:
+        rows = dbcon.execute( "SELECT ra_corner_00, ra_corner_01, ra_corner_10, ra_corner_11,"
+                              "  dec_corner_00, dec_corner_01, dec_corner_10, dec_corner_11 "
+                              "FROM l2image" )
+    polys = []
+    for row in rows:
+        polys.append( shapely.Polygon( ( (row['ra_corner_00'], row['dec_corner_00']),
+                                         (row['ra_corner_01'], row['dec_corner_01']),
+                                         (row['ra_corner_11'], row['dec_corner_11']),
+                                         (row['ra_corner_10'], row['dec_corner_10']),
+                                         (row['ra_corner_00'], row['dec_corner_00'])
+                                        )
+                                      ) )
+    return polys
+
+
+
+def main():
+    parser = argparse.ArgumentParser( 'load_snana_ou2024_diaobject',
+                                      description='Load all parquet files below a directory.' )
+    parser.add_argument( '-p', '--provid', required=True, help="Provenance id" )
+    parser.add_argument( '-n', '--nprocs', type=int, default=12,
+                         help="Number of processes to run at once [default: 12]" )
+    parser.add_argument( 'basedir', help='Base directory.' )
+    args = parser.parse_args()
+
+    with snappl.db.db.DBCon( dictcursor=True ) as dbcon:
+        rows = dbcon.execute( "SELECT * FROM provenance WHERE id=%(id)s", { 'id': args.provid } )
+        if len(rows) == 0:
+            raise ValueError( "Invalid provenance {args.provid}" )
+        SNLogger.info( f"Loading provenance for process {rows[0]['process']} {rows[0]['major']}.{rows[0]['minor']}" )
+
+    SNLogger.info( "Creating shapely polygons of image footprints...." )
+    polys = get_rects()
+    SNLogger.info( f"...have {len(polys)} polygons." )
+
+    # First, collect all the parquet files
+
+    def find_pqfiles( direc ):
+        pqfiles = []
+        subdirs = []
+        for f in direc.iterdir():
+            f = f.resolve()
+            if ( f.name[:5] == 'snana') and ( f.name[-8:] == '.parquet' ):
+                pqfiles.append( f )
+            elif f.is_dir():
+                subdirs.append( f )
+
+        for subdir in subdirs:
+            pqfiles.extend( find_pqfiles( subdir ) )
+
+        return pqfiles
+
+    basedir = pathlib.Path( args.basedir )
+    pqfiles = find_pqfiles( basedir )
+    SNLogger.info( f"{len(pqfiles)} parquet files to load" )
+    SNLogger.debug( f"All pqfiles we're going to do: {pqfiles}" )
+
+    # Launch parallel processes to load these files
+
+    did = []
+    tot = 0
+
+    def add_to_did( n ):
+        nonlocal tot, did
+        tot += n
+        did.append( n )
+        SNLogger.info( f"Completed {len(did)} of {len(pqfiles)} parquet files ({tot} objects)" )
+
+    errors = []
+
+    def omg( e ):
+        nonlocal errors
+        SNLogger.error( f"Subprocess returned error: {e}" )
+        errors.append( e )
+
+    do_load = functools.partial( load_snana_ou2024_diaobject, args.provid, polys )
+
+    if args.nprocs > 1:
+        with multiprocessing.Pool( args.nprocs ) as pool:
+            for pqf in pqfiles:
+                pool.apply_async( do_load, [ pqf ], callback=add_to_did, error_callback=omg )
+            pool.close()
+            pool.join()
+    else:
+        for pqf in pqfiles:
+            add_to_did( do_load( pqf ) )
+
+    if len(errors) > 0:
+        nl = '\n'
+        SNLogger.error( f"There were errors:\n{nl.join(str(e) for e in errors)}" )
+        raise RuntimeError( "There were errors!" )
+
+    SNLogger.info( f"Loaded {tot} objects in {len(did)} parquet files." )
+
+
+# ======================================================================
+if __name__ == "__main__":
+    main()