gmprocess 2.1.2__tar.gz → 2.3.0__tar.gz

{gmprocess-2.1.2 → gmprocess-2.3.0}/.gitignore

@@ -9,6 +9,7 @@ build
 .coverage.*
 coverage.xml
 htmlcov
+.pymon
 **/gmprocess.egg-info/
 .pytest_cache
 *.so

{gmprocess-2.1.2 → gmprocess-2.3.0}/.gitlab-ci.yml

@@ -23,6 +23,12 @@ default:
   variables:
     ENVIRONMENT: staging
 
+.parallel:
+  parallel:
+    matrix:
+      - PYVER: [3.10-build, 3.11-build, 3.12-build]
+        DIST: [linux]
+
 stages:
   - test
   - build
@@ -47,6 +53,8 @@ run tests and coverage:
     - pytest -x --cov=src/gmprocess/ --cov-report xml:coverage.xml --cov-report term --junitxml junit.xml
     - coverage xml
   stage: test
+#  parallel: !reference [.parallel,parallel]
+
 
 run upload pypi:
   rules:
@@ -67,6 +75,11 @@ run upload pypi:
   stage: build
 
 pages:
+  image: code.usgs.gov:5001/devops/images/usgs/python:3.12-build
+  before_script:
+    - python --version
+    - export PATH="/home/usgs-user/.local/bin:$PATH"
+    - apt-get install tree
   artifacts:
     paths:
       # The folder that contains the files to be exposed at the Page URL

{gmprocess-2.1.2 → gmprocess-2.3.0}/CHANGELOG.md

@@ -1,6 +1,59 @@
 ## main
 
 
+## 2.3.0 / 2025-01-23
+
+- Enhancements
+  - Substantial changes to the default processing steps.
+  - Add support for psudo-response spectra (PSA and PSV) and change default metrics to use PSA rather than SA.
+  - Allow for user to set a different query window duration for different tectonic envirnments.
+  - Increase default query window duraiton.
+  - Add a new "source_path" method to compute the signal end that uses 1/fc as the source duration, which is then added to a path duration that is parameterized with coefficients for an intercept and distance term.
+  - Exclude network 7D, which is for ocean bottom stations.
+  - Apply clipping check to strong motion instruments. In the past, we thought it was safe to assume strong motion instruments did not clip but we have found exceptions to this. 
+  - Update summary plots so that the time axis is the same across raw, accel, vel, and disp plots.
+  - Update report to plot stations in order of epicentral distance.
+  - Update setup_env.sh to use miniforge rather than miniconda and renamed to install.sh to be consistent with other repos.
+  - Add taper before differentiation for velocity instruments.
+  - Expand instrument response QA to include consistency of units with instrument type. 
+  - Set unit registry to treat "counts" as having a unique dimension, rather than being dimensionless. 
+  - Simplify SNR failure message.
+  - Allow water_level to be specified as "None" for the instrument response in config file.
+  - Add QA check for inconsistency in sample rate reported in data and metadata.
+  - Increase `sec_before_split` default value for trimming the record.
+  - Add config options to allow for using a Butterworth filter for filtering prior to instrument response removal, rather than the default options that ObsPy uses with their `remove_response` method. 
+  - Add `bandpass` processing step.
+- Documentation
+  - Fix the description of the --textfile option for gmrecords.
+- Bugfix
+  - Do not exit when event id not found in comcat while looking for a rupture file.
+  - Ensure fill value matches the data type of the trace.
+  - Add provenance entry for when the detrend method is "pre".
+  - Fix errors in some filtering provenance entries.
+  - Bugfix in normalizing pre-event noise spectra.
+
+## 2.2.0 / 2024-12-23
+- Enhancements
+  - Add the PRISM/Jones et al. (doi: 10.1785/0220160200) adaptive baseline correction step.
+  - Add support for SV and SD metrics (for cosmos writer V3)
+  - Add support for V2 (processed acc, vel, disp) and V3 COSMOS writer
+  - Add number of threads configuration of FDSN clients to more reliably retrieve data from FDSN servers that may restrict the number of simultaneous downloads.
+  - Add QA check comparing the combined stage gains to the overall instrument sensitivity.
+  - Add QA step to check that the data type of miniseed raw data are integers.
+  - Add options for downloading origin/strec/rupture/waveform data individually to the download subcommand.
+  - StationTrace methods:
+    - Override the Trace taper method so that the provenance is set when this activity is performed.
+    - Ditto for detrend.
+    - Ditto for trim.
+    - Add a zero_pad method as a helper for zero padding.
+  - Add zero pad processing step.
+  - Add progress bar while creating diagnostic plots.
+- Bug fixes
+  - Ensure that the FFT is taken for the length record as is used for normalizing the SNR.
+- Other
+  - When trying to read the config file, allow for yaml read error to be raised rather than returning None. 
+  - Increase tolerance for misaligned traces to be resampled.
+
 ## 2.1.2 / 2024-11-14
 - Enhancements
   - Station distance (epicentral distance) added to the reports

{gmprocess-2.1.2 → gmprocess-2.3.0}/CONTRIBUTING.md

@@ -82,12 +82,12 @@ Common reasons to rebase include:
    ```
 3. Update `CHANGELOG.md` to include the changes for this version. The goal is for the changelog to be kept up to date with each merge request, so this step should largely consist of creating a new section for this release and moving content into it from "main". 
 4. Commit changes to the `v1.2.1.rc0` branch.
-5. Merge the release candidate back into main. 
+5. Push the branch to your origin.
    ```
    git push origin v1.2.1.rc0
    ```
-   Then create a merge request in to upstream main.
-6. Create an annotated tag **from main** and push the tag upstream
+6. Create a merge request in to upstream main, merge it, and rebase locally.
+7. Create an annotated tag **from main** and push the tag upstream
    ```
    git tag -a v1.2.1 -m "Version 1.2.1"
    git push upstream v1.2.1
@@ -96,8 +96,8 @@ Common reasons to rebase include:
    - *Note that we cannot have a hyphen in the tag name.* 
    - Also, if we end the tag name with "dev" then it will be build and uploaded to pypi as a pre-release version, which means that it will never be pip installed unless the user specifies the exact version of the pre-release.
    - Maintainence note: the PYPI tokens are maintained in gitlab for the ESI group, under Settings then CI/CD, then Variables. These tokens are associated with the "pager_cicd" pypi service account.  
-7. Create release from tag in gitlab. Give it a release title like `v1.2.1`.
-8. Copy/paste the relevant part of the changelog into the "describe this release" section.
+8. Create release from tag in gitlab. Give it a release title like `v1.2.1`.
+9. Copy/paste the relevant part of the changelog into the "describe this release" section.
 
 
 ## Build Documentation

gmprocess-2.3.0/DISCLAIMER.md

@@ -0,0 +1,9 @@
+This software is preliminary or provisional and is subject to revision.
+It is being provided to meet the need for timely best science.
+The software has not received final approval by the U.S. Geological Survey (USGS).
+No warranty, expressed or implied, is made by the USGS or the U.S. Government
+as to the functionality of the software and related material
+nor shall the fact of release constitute any such warranty.
+The software is provided on the condition that neither the USGS nor the U.S. Government
+shall be held liable for any damages resulting from the authorized or unauthorized
+use of the software.

{gmprocess-2.1.2 → gmprocess-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: gmprocess
-Version: 2.1.2
+Version: 2.3.0
 Summary: USGS Automated Ground Motion Processing Software
 Author-email: Eric Thompson <emthompsone@usgs.gov>, Mike Hearne <mhearne@usgs.gov>, Brad Aagaard <baagaard@usgs.gov>, Bruce Worden <cbworden@contractor.usgs.gov>, John Rekoske <jrekoske@ucsd.edu>, Heather Hunsinger <hhunsinger@usgs.gov>, Gabe Ferragut <gferragut@usgs.gov>
 License: License
@@ -68,6 +68,7 @@ Requires-Dist: obspy>=1.4.1
 Requires-Dist: openpyxl>=3.0.8
 Requires-Dist: openquake.engine>=3.18
 Requires-Dist: pandas>=1.0
+Requires-Dist: Pint>0.24.3
 Requires-Dist: ps2ff>=1.5.2
 Requires-Dist: pyasdf>=0.7
 Requires-Dist: pydantic>=2.0
@@ -77,6 +78,7 @@ Requires-Dist: schema>=0.7
 Requires-Dist: scipy>=1.7
 Requires-Dist: setuptools-scm>=6.3.2
 Requires-Dist: statsmodels>=0.12.2
+Requires-Dist: tqdm>4.67
 Requires-Dist: usgs-strec>=2.3.2
 Requires-Dist: xlrd>=2.0
 Provides-Extra: dev

{gmprocess-2.1.2 → gmprocess-2.3.0}/code.json

@@ -1,4 +1,86 @@
 [
+    {
+        "name": "groundmotion-processing",
+        "description": "Read and process ground motion waveform data",
+        "version": "v2.3.0",
+        "status": "Production",
+        "organization": "U.S. Geological Survey",
+        "vcs": "git",
+        "repositoryURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing.git",
+        "homepageURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing",
+        "downloadURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/archive/v2.3.0/groundmotion-processing-v2.3.0.zip",
+        "disclaimerURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/raw/v2.3.0/DISCLAIMER.md",
+        "permissions": {
+            "licenses": [
+                {
+                    "name": "Public Domain, CC0-1.0",
+                    "URL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/raw/v2.3.0/LICENSE.md"
+                }
+            ],
+            "usageType": "openSource",
+            "exemptionText": null
+        },
+        "laborHours": 1,
+        "languages": [
+            "python"
+        ],
+        "tags": [
+            "earthquake",
+            "impact",
+            "strong motion",
+            "amplitudes",
+            "ground motion",
+            "usg-artificial-intelligence"
+        ],
+        "contact": {
+            "name": "Eric Thompson",
+            "email": "emthompson@usgs.gov"
+        },
+        "date": {
+            "metadataLastUpdated": "2025-01-23"
+        }
+    },
+    {
+        "name": "groundmotion-processing",
+        "description": "Read and process ground motion waveform data",
+        "version": "v2.2.0",
+        "status": "Production",
+        "organization": "U.S. Geological Survey",
+        "vcs": "git",
+        "repositoryURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing.git",
+        "homepageURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing",
+        "downloadURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/archive/v2.2.0/groundmotion-processing-v2.2.0.zip",
+        "disclaimerURL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/raw/v2.2.0/DISCLAIMER.md",
+        "permissions": {
+            "licenses": [
+                {
+                    "name": "Public Domain, CC0-1.0",
+                    "URL": "https://code.usgs.gov/ghsc/esi/groundmotion-processing/-/raw/v2.2.0/LICENSE.md"
+                }
+            ],
+            "usageType": "openSource",
+            "exemptionText": null
+        },
+        "laborHours": 1,
+        "languages": [
+            "python"
+        ],
+        "tags": [
+            "earthquake",
+            "impact",
+            "strong motion",
+            "amplitudes",
+            "ground motion",
+            "usg-artificial-intelligence"
+        ],
+        "contact": {
+            "name": "Eric Thompson",
+            "email": "emthompson@usgs.gov"
+        },
+        "date": {
+            "metadataLastUpdated": "2024-12-23"
+        }
+    },
     {
         "name": "groundmotion-processing",
         "description": "Read and process ground motion waveform data",

{gmprocess-2.1.2 → gmprocess-2.3.0}/pyproject.toml

@@ -34,6 +34,7 @@ dependencies = [
     "openpyxl>=3.0.8",
     "openquake.engine>=3.18",
     "pandas>=1.0",
+    "Pint>0.24.3",
     "ps2ff>=1.5.2",
     "pyasdf>=0.7",
     "pydantic>=2.0",
@@ -43,6 +44,7 @@ dependencies = [
     "scipy>=1.7",
     "setuptools-scm>=6.3.2",
     "statsmodels>=0.12.2",
+    "tqdm>4.67",
     "usgs-strec>=2.3.2",
     "xlrd>=2.0",
 ]
@@ -88,7 +90,6 @@ gminfo = "gmprocess.bin.gminfo:cli"
 gmrecords = "gmprocess.bin.gmrecords:main"
 gmworkspace = "gmprocess.bin.gmworkspace:main"
 gmprocess_config = "gmprocess.bin.gmprocess_config:main"
-list_metrics = "gmprocess.bin.list_metrics:main"
 cwa_gather = "gmprocess.bin.cwa_gather:cli"
 fix_inventory = "gmprocess.bin.fix_inventory:main"
 fetch_orfeus = "gmprocess.bin.fetch_orfeus:cli"

{gmprocess-2.1.2 → gmprocess-2.3.0}/setup_env.sh

@@ -8,11 +8,9 @@ echo "Installing ${VENV}...${cwd}"
 unamestr=`uname`
 if [ "$unamestr" == 'Linux' ]; then
     prof=~/.bashrc
-    mini_conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
     matplotlibdir=~/.config/matplotlib
 elif [ "$unamestr" == 'FreeBSD' ] || [ "$unamestr" == 'Darwin' ]; then
     prof=~/.bash_profile
-    mini_conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
     matplotlibdir=~/.matplotlib
 else
     echo "Unsupported environment. Exiting."
@@ -39,9 +37,7 @@ elif [ ! grep -Fxq "backend" $matplotlibrc ]; then
     echo "NOTE: A non-interactive matplotlib backend (Agg) has been set for this user."
 else
     sed -i '' 's/backend.*/backend : Agg/' $matplotlibrc
-    echo "###############"
     echo "NOTE: $matplotlibrc has been changed to set 'backend : Agg'"
-    echo "###############"
 fi
 
 
@@ -50,21 +46,45 @@ conda --version
 if [ $? -ne 0 ]; then
     echo "No conda detected, installing miniconda..."
 
-    curl -L $mini_conda_url -o miniconda.sh;
+    command -v curl >/dev/null 2>&1 || { echo >&2 "Script requires curl but it's not installed. Aborting."; exit 1; }
+
+    miniforge_url="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
+    curl -L $miniforge_url -o miniforge.sh &>/dev/null
+
+    # if miniforge.sh fails, bow out gracefully
+    if [ $? -ne 0 ];then
+        echo "Failed to run miniconda installer shell script. Exiting."
+        exit 1
+    fi
+
     echo "Install directory: $HOME/miniconda"
 
-    bash miniconda.sh -f -b -p $HOME/miniconda
+    bash miniforge.sh -f -b -p $HOME/miniconda
 
     # Need this to get conda into path
     . $HOME/miniconda/etc/profile.d/conda.sh
+
+    rm miniforge.sh
 else
     echo "conda detected, installing $VENV environment..."
 fi
 
-echo "PATH:"
-echo $PATH
-echo ""
-
+# Update the conda tool
+CVNUM=`conda -V | cut -f2 -d' '`
+LATEST=`conda search conda | tail -1 | tr -s ' ' | cut -f2 -d" "`
+echo "${CVNUM}"
+echo "${LATEST}"
+if [ ${LATEST} != ${CVNUM} ]; then
+    echo "##################Updating conda tool..."
+    CVERSION=`conda -V`
+    echo "Current conda version: ${CVERSION}"
+    conda update -n base -c defaults conda -y
+    CVERSION=`conda -V`
+    echo "New conda version: ${CVERSION}"
+    echo "##################Done updating conda tool..."
+else
+    echo "conda ${CVNUM} already matches latest version ${LATEST}. No update required."
+fi
 
 # only add this line if it does not already exist
 grep "/etc/profile.d/conda.sh" $prof
@@ -72,6 +92,9 @@ if [ $? -ne 0 ]; then
     echo ". $_CONDA_ROOT/etc/profile.d/conda.sh" >> $prof
 fi
 
+# Set libmamba as solver
+conda config --set solver libmamba &>/dev/null
+
 # Start in conda base environment
 echo "Activate base virtual environment"
 conda activate base
@@ -87,14 +110,13 @@ if [ $? -ne 0 ]; then
     exit
 fi
 
-
 # Activate the new environment
 echo "Activating the $VENV virtual environment"
 conda activate $VENV
 
 # Do mac-specific conda installs
 if [ "$unamestr" == 'FreeBSD' ] || [ "$unamestr" == 'Darwin' ]; then
-    # This is motivated by the mysterios pyproj/rasterio error and incorrect results
+    # This is motivated by the mysterious pyproj/rasterio error and incorrect results
     # that only happen on ARM macs. 
     # https://github.com/conda-forge/pyproj-feedstock/issues/156
     conda install -c conda-forge -y libgdal-netcdf

{gmprocess-2.1.2 → gmprocess-2.3.0}/src/gmprocess/apps/gmrecords.py

@@ -358,7 +358,8 @@ class GMrecordsApp(object):
                 "(2) six columns in which those columns are: "
                 "event_id (string, no spaces), time (any ISO standard for date/time), "
                 "latitutde (float, decimal degrees), longitude (float, decimal "
-                "degrees), depth (float, km), magnitude (float)."
+                "degrees), depth (float, km), magnitude (float), and magnitude type "
+                "(string)."
             ),
         )
 

{gmprocess-2.1.2 → gmprocess-2.3.0}/src/gmprocess/bin/bundle_datarelease.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python3
 """Utility script for bundling workspace files for archiving a data release as a special project at CESMD."""
 
-GMPROCESS_ASDF_URL = "https://gmprocess.readthedocs.io/en/latest/contents/manual/workspace.html"
 
 import pathlib
 import shutil
@@ -11,7 +10,12 @@ import io
 import h5py
 import obspy
 
-class HtmlIndex():
+GMPROCESS_ASDF_URL = (
+    "https://gmprocess.readthedocs.io/en/latest/contents/manual/workspace.html"
+)
+
+
+class HtmlIndex:
 
     def __init__(self, project_name):
         self.project_name = project_name
@@ -66,11 +70,11 @@ class HtmlIndex():
         lines = (
             """<p>Earthquake are ordered by origin time.</p>""",
             """<table style = "width: 30%">""",
-        	"""\t<tr>""",
-		    """\t\t<th style = "width: 20%">Filename</th>""",
-		    """\t\t<th style = "width: 12.5%">Event ID</th>""",
-		    """\t\t<th style = "width: 12.5%">Filesize</th>""",
-	        """\t</tr>""",
+            """\t<tr>""",
+            """\t\t<th style = "width: 20%">Filename</th>""",
+            """\t\t<th style = "width: 12.5%">Event ID</th>""",
+            """\t\t<th style = "width: 12.5%">Filesize</th>""",
+            """\t</tr>""",
         )
         for eq_id, file_size in event_info:
             lines += (
@@ -84,8 +88,8 @@ class HtmlIndex():
             self.fout.write(line + "\n")
 
 
-class DataReleaseApp():
-    
+class DataReleaseApp:
+
     def main(self, project_name: str, project_title: str, data_dir: pathlib.Path):
         self.project_name = project_name
         self.project_title = project_title
@@ -136,9 +140,9 @@ class DataReleaseApp():
         filename = self.data_dir / event_id / "workspace.h5"
         nbytes = os.path.getsize(filename)
         if nbytes >= 2**30:
-            size = f"{nbytes / 2**30:.2f} GB" 
+            size = f"{nbytes / 2**30:.2f} GB"
         elif nbytes >= 2**20:
-            size = f"{nbytes / 2**20:.2f} MB" 
+            size = f"{nbytes / 2**20:.2f} MB"
         else:
             size = f"{nbytes // 2**10:.2f} KB"
         return size
@@ -150,10 +154,26 @@ def cli():
 
     import argparse
 
-    parser = argparse.ArgumentParser(description=DESCRIPTION, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument("--project", action="store", dest="project", required=True, help="CESMD project identifier")
-    parser.add_argument("--title", action="store", dest="title", required=True, help="Title of project")
-    parser.add_argument("--data-dir", action="store", dest="data_dir", required=True, help="Data directory containing events.")
+    parser = argparse.ArgumentParser(
+        description=DESCRIPTION, formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument(
+        "--project",
+        action="store",
+        dest="project",
+        required=True,
+        help="CESMD project identifier",
+    )
+    parser.add_argument(
+        "--title", action="store", dest="title", required=True, help="Title of project"
+    )
+    parser.add_argument(
+        "--data-dir",
+        action="store",
+        dest="data_dir",
+        required=True,
+        help="Data directory containing events.",
+    )
     args = parser.parse_args()
 
     app = DataReleaseApp()
@@ -162,8 +182,8 @@ def cli():
         "project_title": args.title,
         "data_dir": pathlib.Path(args.data_dir).expanduser(),
     }
-    import pdb; pdb.set_trace()
     app.main(**kwds)
 
+
 if __name__ == "__main__":
     cli()

{gmprocess-2.1.2 → gmprocess-2.3.0}/src/gmprocess/core/scalar_event.py

@@ -241,6 +241,7 @@ class ScalarEvent(Event):
             "magnitude": self.magnitude,
             "magnitude_type": self.magnitude_type,
         }
+        logging.info(f"Creating event file: {filename}")
         with open(filename, "w", encoding="utf-8") as f:
             json.dump(data, f)
 
@@ -275,7 +276,7 @@ def get_events_from_file(filename=None):
                 fields = line.split(",")
                 if len(fields) != 7:
                     raise IOError(
-                        f"Expected 6 columns in events file '{filename}'. Error parsing line '{line}'."
+                        f"Expected 7 columns in events file '{filename}'. Error parsing line '{line}'."
                     )
                 event_values = {
                     "id": fields[0].strip(),
@@ -348,5 +349,6 @@ def write_geojson(data, event_dir):
             Event directory.
     """
     filename = event_dir / constants.EVENT_FILE
+    logging.info(f"Creating event file: {filename}")
     with open(filename, "w", encoding="utf-8") as fout:
         json.dump(data, fout)

{gmprocess-2.1.2 → gmprocess-2.3.0}/src/gmprocess/core/stationstream.py

@@ -39,7 +39,7 @@ UNUSED_STANDARD_PARAMS = [
     "source_format",
 ]
 
-TIME_TOL = 1e-6
+TIME_TOL = 1e-5
 
 
 class StationStream(Stream):
@@ -184,8 +184,6 @@ class StationStream(Stream):
 
     def validate(self):
         """Validation checks for Traces within a StationStream."""
-        logging.debug(self)
-
         # Check that channel codes are unique_npts
         self.__check_channels()
 
@@ -520,7 +518,6 @@ def _channel_from_stats(stats):
         response = Response(instrument_sensitivity=sensitivity)
 
     comments = Comment(stats.standard.comments)
-    logging.debug("channel: %s", stats.channel)
     channel = Channel(
         stats.channel,
         stats.location,