requake 0.8.2__tar.gz → 0.8.3__tar.gz

{requake-0.8.2 → requake-0.8.3}/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 Copyright (c) 2021-2026 Claudio Satriano <satriano@ipgp.fr>
 
+## [0.8.3] - 2026-06-19
+
+Note: you should run `requake update_config` to update your config file
+to the latest version.
+
+The waveform disk cache is now a single SQLite database
+(`OUTDIR/waveform_cache.sqlite`) with a persistent negative cache that
+remembers failed downloads across runs, eliminating redundant HTTP requests.
+
+- New `wfcache` CLI: `prefetch`, `print`/`inspect`, `extract`,
+  `reset-failures`.
+- New config: `catalog_waveform_cache_failure_max_retries`,
+  `catalog_waveform_cache_failure_backoff_s`,
+  `catalog_waveform_require_prefetch`.
+- Several optimization for parallel runs in `scan_catalog` (especially on
+  Slurm clusters) and for startup time of `scan_catalog -c` (continue an
+  interrupted scan).
+- Extended documentation for `read_catalog`, `scan_catalog` and `wfcache`.
+
 ## [0.8.2] - 2026-06-03
 
 Note: you might want to run `requake update_config` to update your config file
@@ -187,6 +206,7 @@ Mostly a bugfix release with a slightly improved trace plotting.
 
 - Initial release, not yet feature complete
 
+[0.8.3]: https://github.com/SeismicSource/requake/compare/v0.8.2...v0.8.3
 [0.8.2]: https://github.com/SeismicSource/requake/compare/v0.8.1...v0.8.2
 [0.8.1]: https://github.com/SeismicSource/requake/compare/v0.8...v0.8.1
 [0.8]: https://github.com/SeismicSource/requake/compare/v0.7.3...v0.8

{requake-0.8.2 → requake-0.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: requake
-Version: 0.8.2
+Version: 0.8.3
 Summary: Repeating earthquakes search and analysis
 Author-email: Claudio Satriano <satriano@ipgp.fr>
 License-Expression: GPL-3.0-or-later
@@ -37,7 +37,7 @@ Dynamic: description
 Dynamic: description-content-type
 Dynamic: license-file
 
-<img src="https://cdn.jsdelivr.net/gh/SeismicSource/requake@9071fd7343802f3ef642a16f6bb02ff5ce1d88f5/imgs/Requake_logo.svg" width="400">
+<img src="https://cdn.jsdelivr.net/gh/SeismicSource/requake@407e6c96ec3574f6e3743741763ca5b7c4daf914/imgs/Requake_logo.svg" width="400">
 
 # Requake
 
@@ -181,6 +181,8 @@ Different commands are available:
                         families
     scan_templates      scan a continuous waveform stream using one or more
                         templates
+    wfcache             manage persistent waveform cache (prefetch, print,
+                        inspect, extract, reset_failures)
 
 Certain commands (e.g., `plot_pair`) require further arguments (use, e.g.,
 `requake plot_pair -h` to get help).
@@ -216,6 +218,14 @@ Now, build the catalog of event pairs with:
 
     requake scan_catalog
 
+When relying on FDSN web services for waveform data, it is strongly
+recommended to prefetch all waveform windows before running the scan.
+This downloads every required waveform once and stores it in a local
+SQLite cache, avoiding repeated downloads and dramatically reducing
+overall runtime for large catalogs:
+
+    requake wfcache prefetch
+
 Once done ([it will take time!](#performances)), you are ready to build
 repeating earthquake families:
 
@@ -232,6 +242,12 @@ repeating earthquake families:
   performance is typically dominated by waveform download latency rather than
   computation.
 
+- For large FDSN-based runs, use `requake wfcache prefetch` before
+  `requake scan_catalog` to download all waveforms upfront into a local
+  SQLite cache.  This eliminates repeated downloads, lets the scan read
+  exclusively from disk, and is the single most effective way to speed up
+  catalog scanning.
+
 - `requake build_families` is fast™.
 
 ## How to Cite
@@ -244,7 +260,7 @@ If you used Requake for a scientific paper, please cite it as:
 Please replace `X.Y` with the Requake version number you used.
 
 You can also cite the following abstract presented at the
-2016 AGU Fall Meeting:
+2021 AGU Fall Meeting:
 
 > Satriano, C., Doucet, A. & Bouin, M.-P. (2021).
 > Probing the creep rate along the Lesser Antilles Subduction Zone through repeating earthquakes.

{requake-0.8.2 → requake-0.8.3}/README.md

@@ -142,6 +142,8 @@ Different commands are available:
                         families
     scan_templates      scan a continuous waveform stream using one or more
                         templates
+    wfcache             manage persistent waveform cache (prefetch, print,
+                        inspect, extract, reset_failures)
 
 Certain commands (e.g., `plot_pair`) require further arguments (use, e.g.,
 `requake plot_pair -h` to get help).
@@ -177,6 +179,14 @@ Now, build the catalog of event pairs with:
 
     requake scan_catalog
 
+When relying on FDSN web services for waveform data, it is strongly
+recommended to prefetch all waveform windows before running the scan.
+This downloads every required waveform once and stores it in a local
+SQLite cache, avoiding repeated downloads and dramatically reducing
+overall runtime for large catalogs:
+
+    requake wfcache prefetch
+
 Once done ([it will take time!](#performances)), you are ready to build
 repeating earthquake families:
 
@@ -193,6 +203,12 @@ repeating earthquake families:
   performance is typically dominated by waveform download latency rather than
   computation.
 
+- For large FDSN-based runs, use `requake wfcache prefetch` before
+  `requake scan_catalog` to download all waveforms upfront into a local
+  SQLite cache.  This eliminates repeated downloads, lets the scan read
+  exclusively from disk, and is the single most effective way to speed up
+  catalog scanning.
+
 - `requake build_families` is fast™.
 
 ## How to Cite
@@ -205,7 +221,7 @@ If you used Requake for a scientific paper, please cite it as:
 Please replace `X.Y` with the Requake version number you used.
 
 You can also cite the following abstract presented at the
-2016 AGU Fall Meeting:
+2021 AGU Fall Meeting:
 
 > Satriano, C., Doucet, A. & Bouin, M.-P. (2021).
 > Probing the creep rate along the Lesser Antilles Subduction Zone through repeating earthquakes.

{requake-0.8.2 → requake-0.8.3}/requake/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2026-06-03T14:39:27+0200",
+ "date": "2026-06-19T10:48:17+0200",
  "dirty": false,
  "error": null,
- "full-revisionid": "9071fd7343802f3ef642a16f6bb02ff5ce1d88f5",
- "version": "0.8.2"
+ "full-revisionid": "407e6c96ec3574f6e3743741763ca5b7c4daf914",
+ "version": "0.8.3"
 }
 '''  # END VERSION_JSON
 

{requake-0.8.2 → requake-0.8.3}/requake/config/configspec.conf

@@ -76,9 +76,19 @@ catalog_waveform_cache_size = integer(min=2, default=5000)
 ## Set to 0 to derive it automatically from catalog_waveform_cache_size.
 catalog_waveform_cache_size_parallel = integer(min=0, default=0)
 ## Keep a persistent copy of fetched event waveforms on disk to speed up
-## reruns. This on-disk cache lives in OUTDIR/waveform_cache and is not
+## reruns. This on-disk cache lives in OUTDIR/waveform_cache.sqlite and is
 ## limited by catalog_waveform_cache_size.
 catalog_waveform_disk_cache_enabled = boolean(default=true)
+## Maximum number of retries for a failed waveform download before
+## suppressing new attempts for the same cache key.
+catalog_waveform_cache_failure_max_retries = integer(min=0, default=3)
+## Base backoff in seconds for persistent waveform download failures.
+## The retry delay grows exponentially with each failure.
+catalog_waveform_cache_failure_backoff_s = float(min=0, default=600)
+## When True, skip live downloads for events absent from both positive
+## and negative cache, trusting a prior prefetch run to have covered
+## all fetchable events. Requires disk cache to be enabled.
+catalog_waveform_require_prefetch = boolean(default=false)
 ## Number of worker processes for catalog scan.
 ## 0 means automatic selection; 1 disables parallelism.
 catalog_scan_nprocs = integer(min=0, default=0)

{requake-0.8.2 → requake-0.8.3}/requake/config/parse_arguments.py

@@ -498,6 +498,179 @@ def parse_arguments(progname='requake'):
              '"A" header field'
     )
     # ---
+    # --- wfcache
+    wfcache = subparser.add_parser(
+        'wfcache',
+        help='manage persistent waveform cache'
+    )
+    wfcache_sub = wfcache.add_subparsers(dest='wfcache_action')
+    wfcache_sub.required = True
+
+    wfcache_row_filters = argparse.ArgumentParser(add_help=False)
+    wfcache_row_filters.add_argument(
+        '--event-id',
+        action='append',
+        default=[],
+        help='event ID filter (repeatable)'
+    )
+    wfcache_row_filters.add_argument(
+        '--event-id-file',
+        type=str,
+        default=None,
+        help='path to text file with event IDs'
+    )
+    wfcache_row_filters.add_argument(
+        '--trace-id',
+        action='append',
+        default=[],
+        help='trace-ID filter (repeatable)'
+    )
+    wfcache_row_filters.add_argument(
+        '--start-time',
+        type=str,
+        default=None,
+        help='minimum start time (UTCDateTime-compatible string)'
+    )
+    wfcache_row_filters.add_argument(
+        '--end-time',
+        type=str,
+        default=None,
+        help='maximum end time (UTCDateTime-compatible string)'
+    )
+    wfcache_row_filters.add_argument(
+        '--limit',
+        type=_nonnegative_int,
+        default=None,
+        help='limit number of returned rows'
+    )
+
+    wfcache_prefetch = wfcache_sub.add_parser(
+        'prefetch',
+        help='prefetch waveform windows into the persistent cache'
+    )
+    wfcache_prefetch.add_argument(
+        '--event-id',
+        action='append',
+        default=[],
+        help='event ID filter (repeatable)'
+    )
+    wfcache_prefetch.add_argument(
+        '--event-id-file',
+        type=str,
+        default=None,
+        help='path to text file with event IDs'
+    )
+    wfcache_prefetch.add_argument(
+        '--trace-id',
+        action='append',
+        default=[],
+        help='trace-ID filter (repeatable)'
+    )
+    wfcache_prefetch.add_argument(
+        '--max-events',
+        type=_nonnegative_int,
+        default=None,
+        help='maximum number of catalog events to prefetch'
+    )
+    wfcache_prefetch.add_argument(
+        '--batch-size',
+        type=_nonnegative_int,
+        default=500,
+        help='progress logging batch size (default: %(default)s)'
+    )
+    wfcache_prefetch.add_argument(
+        '--group-window',
+        type=str,
+        default='1h',
+        metavar='DURATION',
+        help='max grouped download span (e.g., 30m, 1h; default: '
+             '%(default)s)'
+    )
+    wfcache_prefetch.set_defaults(action='wfcache_prefetch')
+
+    wfcache_print = wfcache_sub.add_parser(
+        'print',
+        parents=[wfcache_row_filters],
+        help='print cached waveform rows'
+    )
+    wfcache_print.add_argument(
+        '--json',
+        action='store_true',
+        help='print rows as JSON'
+    )
+    wfcache_print.set_defaults(action='wfcache_print')
+
+    wfcache_inspect = wfcache_sub.add_parser(
+        'inspect',
+        help='print persistent waveform-cache summary'
+    )
+    wfcache_inspect.add_argument(
+        '--integrity',
+        action='store_true',
+        help='run PRAGMA integrity_check'
+    )
+    wfcache_inspect.add_argument(
+        '--json',
+        action='store_true',
+        help='print summary as JSON'
+    )
+    wfcache_inspect.set_defaults(action='wfcache_inspect')
+
+    wfcache_extract = wfcache_sub.add_parser(
+        'extract',
+        parents=[wfcache_row_filters],
+        help='extract cached waveforms to external files'
+    )
+    wfcache_extract.add_argument(
+        '--format',
+        type=str,
+        choices=['mseed', 'sac'],
+        default='mseed',
+        help='output format (default: %(default)s)'
+    )
+    wfcache_extract.add_argument(
+        '--output-dir',
+        type=str,
+        default='waveform_cache',
+        help='output directory (default: %(default)s)'
+    )
+    wfcache_extract.set_defaults(action='wfcache_extract')
+
+    wfcache_reset = wfcache_sub.add_parser(
+        'reset-failures',
+        help='reset persistent waveform failure-cache rows'
+    )
+    wfcache_reset.add_argument(
+        '--event-id',
+        action='append',
+        default=[],
+        help='event ID to reset (repeatable)'
+    )
+    wfcache_reset.add_argument(
+        '--event-id-file',
+        type=str,
+        default=None,
+        help='path to text file with event IDs to reset'
+    )
+    wfcache_reset.add_argument(
+        '--all',
+        action='store_true',
+        help='reset all failure rows'
+    )
+    wfcache_reset.add_argument(
+        '--older-than',
+        type=str,
+        default=None,
+        metavar='DURATION',
+        help='reset rows older than DURATION (e.g., 6h, 7d)'
+    )
+    wfcache_reset.add_argument(
+        '--dry-run',
+        action='store_true',
+        help='show affected rows without deleting'
+    )
+    wfcache_reset.set_defaults(action='wfcache_reset_failures')
+    # ---
     argcomplete.autocomplete(parser)
     args = parser.parse_args()
     if args.action is None:

{requake-0.8.2 → requake-0.8.3}/requake/config/rq_setup.py

@@ -28,6 +28,12 @@ from .utils import (
 # Note: modules are lazily imported to speed up the startup time.
 # pylint: disable=relative-beyond-top-level,import-outside-toplevel
 
+
+def _memory_log_filter(record):
+    """Suppress [MEM] messages from TTY console handlers."""
+    return '[MEM]' not in record.getMessage()
+
+
 logger = None  # pylint: disable=invalid-name
 PYTHON_VERSION_STR = None
 NUMPY_VERSION_STR = None
@@ -112,6 +118,7 @@ def _setup_tqdm_logging(logger_root):
                 self.handleError(record)
     console = TqdmLoggingHandler()
     console.setLevel(logging.INFO)
+    console.addFilter(_memory_log_filter)
     # Add logger color coding on all platforms but win32
     if sys.platform != 'win32' and sys.stdout.isatty():
         console.emit = _color_handler_emit(console.emit)
@@ -172,23 +179,13 @@ def _connect_station_dataselect():
     Connect to station and dataselect services.
 
     Those can be either FDSN web services or local files.
+    Both FDSN clients are created lazily on first use to avoid
+    unnecessary network I/O when data comes from local sources.
     """
-    from obspy.clients.fdsn import Client as FDSNClient
-    if config.station_metadata_path is None:
-        config.station_client = FDSNClient(config.fdsn_station_url)
-        logger.info(
-            f'Connected to FDSN station server: {config.fdsn_station_url}'
-        )
+    config.station_client = None
     config.dataselect_client = None
     if config.sds_data_path is not None:
         _connect_sds()
-    if config.event_data_path is not None:
-        return
-    config.dataselect_client = FDSNClient(config.fdsn_dataselect_url)
-    logger.info(
-        'Connected to FDSN dataselect server: '
-        f'{config.fdsn_dataselect_url}'
-    )
 
 
 def _connect_sds():
@@ -267,6 +264,7 @@ def configure(args):
     # update config with the contents of config_obj
     config.update(config_obj)
     config.args = args
+    config.args.outdir = os.path.abspath(config.args.outdir)
     config.scan_catalog_pairs_file = os.path.join(
         config.args.outdir, 'requake.event_pairs.csv'
     )
@@ -293,14 +291,14 @@ def configure(args):
     # save config to output dir (only for actions that write to outdir)
     actions_writing_to_outdir = (
         'read_catalog', 'scan_catalog', 'build_families',
-        'build_templates', 'scan_templates'
+        'build_templates', 'scan_templates', 'wfcache_prefetch'
     )
     if args.action in actions_writing_to_outdir:
         shutil.copy(args.configfile, args.outdir)
     _parse_catalog_options()
     actions_needing_fdsn_station_dataselect = (
         'scan_catalog', 'plot_pair', 'plot_families', 'build_templates',
-        'scan_templates'
+        'scan_templates', 'wfcache_prefetch'
     )
     try:
         if args.action in actions_needing_fdsn_station_dataselect:

{requake-0.8.2 → requake-0.8.3}/requake/config/utils.py

@@ -13,12 +13,43 @@ import os
 import sys
 import locale
 import shutil
+import hashlib
 from datetime import datetime
 from .configobj import ConfigObj, ParseError
 from .configobj.validate import Validator
 locale.setlocale(locale.LC_ALL, '')
 
 
+_seen_msgs = set()
+
+
+def log_once(logger, level, msg, *args, **kwargs):
+    """
+    Log a message only once.
+
+    :param logger: Logger object.
+    :type logger: logging.Logger
+    :param level: Log level.
+    :type level: str
+    :param msg: Message to log.
+    :type msg: str
+    :param args: Arguments for the message.
+    :param kwargs: Keyword arguments for the message.
+    """
+    text = str(msg)
+    digest = hashlib.blake2b(
+        text.encode('utf-8'),
+        digest_size=8,
+    ).digest()
+    if digest in _seen_msgs:
+        return
+    _seen_msgs.add(digest)
+    log_fn = getattr(logger, level, None)
+    if log_fn is None:
+        raise ValueError(f'Invalid log level: {level}')
+    log_fn(text, *args, **kwargs)
+
+
 def err_exit(msg):
     """
     Exit with error message.

{requake-0.8.2 → requake-0.8.3}/requake/database/families.py

@@ -95,8 +95,14 @@ def write_families(families):
         conn.close()
 
 
-def read_families():
-    """Read catalog-scan families from SQLite."""
+def read_families(family_numbers=None):
+    """Read catalog-scan families from SQLite.
+
+    :param family_numbers: If given, only read families with these numbers.
+    :type family_numbers: iterable of int or None
+    :return: List of families.
+    :rtype: list of Family
+    """
     from ..catalog import RequakeEvent
     from ..families.families import Family
 
@@ -104,12 +110,24 @@ def read_families():
     try:
         cursor = conn.cursor()
         try:
-            rows = cursor.execute(
-                f'''
-                SELECT * FROM {FAMILIES_TABLE}
-                ORDER BY family_number, orig_time, evid, trace_id
-                '''
-            ).fetchall()
+            if family_numbers is not None:
+                family_numbers = sorted(set(family_numbers))
+                placeholders = ','.join('?' * len(family_numbers))
+                rows = cursor.execute(
+                    f'''
+                    SELECT * FROM {FAMILIES_TABLE}
+                    WHERE family_number IN ({placeholders})
+                    ORDER BY family_number, orig_time, evid, trace_id
+                    ''',
+                    family_numbers,
+                ).fetchall()
+            else:
+                rows = cursor.execute(
+                    f'''
+                    SELECT * FROM {FAMILIES_TABLE}
+                    ORDER BY family_number, orig_time, evid, trace_id
+                    '''
+                ).fetchall()
         except sqlite3.OperationalError as err:
             if MISSING_FAMILIES_TABLE in str(err):
                 raise FileNotFoundError(

{requake-0.8.2 → requake-0.8.3}/requake/database/pairs.py

@@ -315,13 +315,19 @@ def write_pair_records(pairs, append=True):
 
 
 def count_pairs():
-    """Return the number of stored event pairs."""
+    """Return the number of stored event pairs.
+
+    Uses ``SELECT 1 ... LIMIT 1`` instead of ``COUNT(*)`` because
+    the latter performs a full table scan, which is prohibitively
+    slow (20+ minutes for 300M rows) on network filesystems such
+    as GPFS.
+    """
     conn = get_db_connection(initdb=False)
     try:
         cursor = conn.cursor()
         try:
             row = cursor.execute(
-                f'SELECT COUNT(*) AS npairs FROM {EVENT_PAIRS_TABLE}'
+                f'SELECT 1 FROM {EVENT_PAIRS_TABLE} LIMIT 1'
             ).fetchone()
         except sqlite3.OperationalError as err:
             if _is_missing_pairs_table_error(err):
@@ -329,7 +335,7 @@ def count_pairs():
             raise
     finally:
         conn.close()
-    return int(row['npairs'])
+    return 1 if row is not None else 0
 
 
 def read_event_key_rows():
@@ -343,7 +349,7 @@ def read_event_key_rows():
                 SELECT event_id, evid
                 FROM {EVENT_KEYS_TABLE}
                 '''
-            ).fetchall()
+            )
         except sqlite3.OperationalError as err:
             if _is_missing_pairs_table_error(err):
                 return []
@@ -358,18 +364,25 @@ def read_event_key_rows():
     return [(int(row['event_id']), row['evid']) for row in rows]
 
 
-def read_pair_key_ids():
-    """Read stored event-pair foreign keys as integer ID tuples."""
+def read_packed_pair_ids(event_id_to_idx, nevents):
+    """Stream pair keys from DB and return packed ``uint64`` IDs.
+
+    Reads ``(event1_id, event2_id)`` rows from the database and
+    packs them into a set of canonical ``uint64`` values in a
+    single pass, avoiding the intermediate tuple set.
+
+    :param event_id_to_idx: mapping from ``event_id`` to catalog index
+    :param nevents: number of events in the catalog
+    :return: set of packed IDs, each computed as
+        ``min(idx1, idx2) * nevents + max(idx1, idx2)``
+    """
     conn = get_db_connection(initdb=False)
     try:
         cursor = conn.cursor()
         try:
             rows = cursor.execute(
-                f'''
-                SELECT DISTINCT event1_id, event2_id
-                FROM {EVENT_PAIRS_TABLE}
-                '''
-            ).fetchall()
+                f'SELECT event1_id, event2_id FROM {EVENT_PAIRS_TABLE}'
+            )
         except sqlite3.OperationalError as err:
             if _is_missing_pairs_table_error(err):
                 return set()
@@ -379,12 +392,21 @@ def read_pair_key_ids():
                     f'Requake version in db file {get_db_path()}'
                 ) from err
             raise
+        get_idx = event_id_to_idx.get
+        result = set()
+        add_id = result.add
+        for event1_id, event2_id in rows:
+            idx1 = get_idx(int(event1_id))
+            idx2 = get_idx(int(event2_id))
+            if idx1 is None or idx2 is None:
+                continue
+            if idx1 < idx2:
+                add_id(idx1 * nevents + idx2)
+            else:
+                add_id(idx2 * nevents + idx1)
+        return result
     finally:
         conn.close()
-    return {
-        (int(row['event1_id']), int(row['event2_id']))
-        for row in rows
-    }
 
 
 def read_pairs(cc_min=None, cc_max=None):

{requake-0.8.2 → requake-0.8.3}/requake/families/families.py

@@ -151,15 +151,17 @@ class Family(list):
         return distance / 1e3
 
 
-def _read_families_from_catalog_scan():
+def _read_families_from_catalog_scan(family_numbers=None):
     """
     Read a list of families from the catalog scan output.
 
+    :param family_numbers: If given, only read families with these numbers.
+    :type family_numbers: iterable of int or None
     :return: List of families.
     :rtype: list of Family
     """
     from ..database.families import read_families as read_families_from_db
-    return read_families_from_db()
+    return read_families_from_db(family_numbers=family_numbers)
 
 
 def _read_families_from_template_scan():
@@ -175,16 +177,18 @@ def _read_families_from_template_scan():
     return read_template_families_from_db()
 
 
-def read_families():
+def read_families(family_numbers=None):
     """
     Read families from the catalog scan output or template scan output.
 
+    :param family_numbers: If given, only read families with these numbers.
+    :type family_numbers: iterable of int or None
     :return: List of families.
     :rtype: list of Family
     """
     if getattr(config.args, 'template', False):
         return _read_families_from_template_scan()
-    return _read_families_from_catalog_scan()
+    return _read_families_from_catalog_scan(family_numbers=family_numbers)
 
 
 def read_selected_families():
@@ -197,7 +201,7 @@ def read_selected_families():
     :raises FamilyNotFoundError: if no family is found
     """
     family_numbers = _build_family_number_list()
-    families = read_families()
+    families = read_families(family_numbers=family_numbers)
     families_selected = []
     for family in families:
         if family.number not in family_numbers: