timewise 1.0.0a2__tar.gz → 1.0.0a5__tar.gz

{timewise-1.0.0a2 → timewise-1.0.0a5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: timewise
-Version: 1.0.0a2
+Version: 1.0.0a5
 Summary: Download WISE infrared data for many objects and process them with AMPEL
 License: MIT
 License-File: LICENSE
@@ -10,12 +10,13 @@ Requires-Python: >=3.11,<3.12
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
+Provides-Extra: ampel
 Provides-Extra: dev
 Provides-Extra: docs
-Requires-Dist: ampel-alerts (==0.10.3a5)
-Requires-Dist: ampel-core (>=0.10.4.post0,<0.11.0)
-Requires-Dist: ampel-photometry (>=0.10.1,<0.11.0)
-Requires-Dist: ampel-plot (>=0.9.1,<0.10.0)
+Requires-Dist: ampel-alerts (==0.10.3a5) ; extra == "ampel"
+Requires-Dist: ampel-core (>=0.10.4.post0,<0.11.0) ; extra == "ampel"
+Requires-Dist: ampel-photometry (>=0.10.1,<0.11.0) ; extra == "ampel"
+Requires-Dist: ampel-plot (>=0.9.1,<0.10.0) ; extra == "ampel"
 Requires-Dist: astropy (>=5.1,<6.0.0)
 Requires-Dist: autodoc_pydantic[erdantic] (>=2.2.0,<3.0.0) ; extra == "docs"
 Requires-Dist: backoff (>=2.1.2,<3.0.0)
@@ -36,7 +37,6 @@ Requires-Dist: ruff (>=0.13.0,<0.14.0) ; extra == "dev"
 Requires-Dist: scikit-image (>=0.19.3,<0.22.0)
 Requires-Dist: scikit-learn (>=1.3.0,<2.0.0)
 Requires-Dist: scipy-stubs (>=1.16.2.0,<2.0.0.0) ; extra == "dev"
-Requires-Dist: seaborn (>=0.11.2,<0.14.0)
 Requires-Dist: sphinx-rtd-theme (>=1.3.0,<2.0.0) ; extra == "docs"
 Requires-Dist: tqdm (>=4.64.0,<5.0.0)
 Requires-Dist: typer (>=0.19.2,<0.20.0)
@@ -60,23 +60,33 @@ Description-Content-Type: text/markdown
 This package downloads WISE data for positions on the sky and stacks single-exposure photometry per visit
 
 ## Prerequisites
+Python version 3.11.
 
-`timewise` makes use of [AMPEL](https://ampelproject.github.io/ampelastro/) and needs a running [MongoDB](https://www.mongodb.com/).
+If you want to not only download individual exposure photometry but also stack detections per visit (see below),
+you must have access to a running [MongoDB](https://www.mongodb.com/)*. 
+
+<sub>* On MacOS have alook at the custom `brew` tap 
+[here](https://stackoverflow.com/questions/57856809/installing-mongodb-with-homebrew)
+to get the MongoDB community edition. </sub>
 
 ## Installation
-The package can be installed via `pip`:
+
+### If you use timewise only for downloading
+The package can be installed via `pip` (but make sure to install the v1 pre-release):
 ```bash
-pip install timewise
+pip install --pre timewise==1.0.0a5
+```
+### If you use timewise also for stacking individual exposures
+You must install with the `ampel` extra:
+```bash
+pip install --pre timewise[ampel]==1.0.0a5
 ```
-
 To tell AMPEL which modules, aka units, to use, build the corresponding configuration file:
 ```bash
 ampel config build -distributions ampel timewise -stop-on-errors 0 -out <path-to-ampel-config-file>
 ```
 
-## Usage
-
-### Command line interface
+## Command line interface
 
 ```
  Usage: timewise [OPTIONS] COMMAND [ARGS]...                                                                
@@ -161,40 +171,50 @@ ampel:
 This configuration file will be the input to all subcommands. Downloading and stacking can be run together or separate.
 
 
-#### All-in-one:
-Run download, stacking, and export:
+### To only download the data:
 ```bash
-timewise run-chain <path-to-config-file> <path-to-ampel-config-file> <output-directory>
+timewise download <path-to-config-file>
 ```
+The photometry can be found in FITS files in the working directory specified in the configuration file\
+along with metadata JSON files. These tell `timewise` which quries have already completed (per chunk) so the
+download process can be interrupted and re-started at a later time.
+
+### Stack individual exposure by visits
+As mentioned above, this needs installation with the ampel extra.
+
 
-#### Separate download and processing:
-To only download the data:
+To **execute the stacking** after the download:
 ```bash
-timewise download <path-to-config-file>
+timewise process <path-to-config-file> <path-to-ampel-config-file>
 ```
 
-To execute the stacking:
+Make some **diagnostic plots** to check the datapoint selection and binning:
 ```bash
-timewise process <path-to-config-file> <path-to-ampel-config-file>
+timewise plot <path-to-config-file> <indices-to-plot> <output-directory>
 ```
 
-#### Run AMPEL manually
-Prepare an AMPEL job file for stacking the single-exposure data:
+As a shortcut, you can also run **download, stacking, and export in one command**:
 ```bash
-timewise prepare-ampel <path-to-config-file>
+timewise run-chain <path-to-config-file> <path-to-ampel-config-file> <output-directory>
 ```
-The result will contain the path to the prepared AMPEL job file that can be run with
+
+For more configuration options of the stacking, you can **run AMPEL manually**.
+
+1. Prepare an AMPEL job file for stacking the single-exposure data:
 ```bash
-ampel job -config <path-to-ampel-config-file> -schema <path-to-ampel-job-file>
+timewise prepare-ampel <path-to-config-file>
 ```
+The result will contain the path to the prepared AMPEL job file.
 
-#### Make some diagnostic plots
-To check the datapoint selection and binning, take a quick look at the data:
+2. Run the AMPEL job
 ```bash
-timewise plot <path-to-config-file> <indices-to-plot> <output-directory>
+ampel job -config <path-to-ampel-config-file> -schema <path-to-ampel-job-file>
 ```
 
 
+
+
+
 ## Citation
 If you use `timewise` please make sure to cite [Necker et al. A&A 695, A228 (2025)](https://www.aanda.org/articles/aa/abs/2025/03/aa51340-24/aa51340-24.html).
 Additionally, you might want to include a reference to the specific version you are using: [![DOI](https://zenodo.org/badge/449677569.svg)](https://zenodo.org/badge/latestdoi/449677569)

{timewise-1.0.0a2 → timewise-1.0.0a5}/README.md

@@ -10,23 +10,33 @@
 This package downloads WISE data for positions on the sky and stacks single-exposure photometry per visit
 
 ## Prerequisites
+Python version 3.11.
 
-`timewise` makes use of [AMPEL](https://ampelproject.github.io/ampelastro/) and needs a running [MongoDB](https://www.mongodb.com/).
+If you want to not only download individual exposure photometry but also stack detections per visit (see below),
+you must have access to a running [MongoDB](https://www.mongodb.com/)*. 
+
+<sub>* On MacOS have alook at the custom `brew` tap 
+[here](https://stackoverflow.com/questions/57856809/installing-mongodb-with-homebrew)
+to get the MongoDB community edition. </sub>
 
 ## Installation
-The package can be installed via `pip`:
+
+### If you use timewise only for downloading
+The package can be installed via `pip` (but make sure to install the v1 pre-release):
 ```bash
-pip install timewise
+pip install --pre timewise==1.0.0a5
+```
+### If you use timewise also for stacking individual exposures
+You must install with the `ampel` extra:
+```bash
+pip install --pre timewise[ampel]==1.0.0a5
 ```
-
 To tell AMPEL which modules, aka units, to use, build the corresponding configuration file:
 ```bash
 ampel config build -distributions ampel timewise -stop-on-errors 0 -out <path-to-ampel-config-file>
 ```
 
-## Usage
-
-### Command line interface
+## Command line interface
 
 ```
  Usage: timewise [OPTIONS] COMMAND [ARGS]...                                                                
@@ -111,40 +121,50 @@ ampel:
 This configuration file will be the input to all subcommands. Downloading and stacking can be run together or separate.
 
 
-#### All-in-one:
-Run download, stacking, and export:
+### To only download the data:
 ```bash
-timewise run-chain <path-to-config-file> <path-to-ampel-config-file> <output-directory>
+timewise download <path-to-config-file>
 ```
+The photometry can be found in FITS files in the working directory specified in the configuration file\
+along with metadata JSON files. These tell `timewise` which quries have already completed (per chunk) so the
+download process can be interrupted and re-started at a later time.
+
+### Stack individual exposure by visits
+As mentioned above, this needs installation with the ampel extra.
+
 
-#### Separate download and processing:
-To only download the data:
+To **execute the stacking** after the download:
 ```bash
-timewise download <path-to-config-file>
+timewise process <path-to-config-file> <path-to-ampel-config-file>
 ```
 
-To execute the stacking:
+Make some **diagnostic plots** to check the datapoint selection and binning:
 ```bash
-timewise process <path-to-config-file> <path-to-ampel-config-file>
+timewise plot <path-to-config-file> <indices-to-plot> <output-directory>
 ```
 
-#### Run AMPEL manually
-Prepare an AMPEL job file for stacking the single-exposure data:
+As a shortcut, you can also run **download, stacking, and export in one command**:
 ```bash
-timewise prepare-ampel <path-to-config-file>
+timewise run-chain <path-to-config-file> <path-to-ampel-config-file> <output-directory>
 ```
-The result will contain the path to the prepared AMPEL job file that can be run with
+
+For more configuration options of the stacking, you can **run AMPEL manually**.
+
+1. Prepare an AMPEL job file for stacking the single-exposure data:
 ```bash
-ampel job -config <path-to-ampel-config-file> -schema <path-to-ampel-job-file>
+timewise prepare-ampel <path-to-config-file>
 ```
+The result will contain the path to the prepared AMPEL job file.
 
-#### Make some diagnostic plots
-To check the datapoint selection and binning, take a quick look at the data:
+2. Run the AMPEL job
 ```bash
-timewise plot <path-to-config-file> <indices-to-plot> <output-directory>
+ampel job -config <path-to-ampel-config-file> -schema <path-to-ampel-job-file>
 ```
 
 
+
+
+
 ## Citation
 If you use `timewise` please make sure to cite [Necker et al. A&A 695, A228 (2025)](https://www.aanda.org/articles/aa/abs/2025/03/aa51340-24/aa51340-24.html).
 Additionally, you might want to include a reference to the specific version you are using: [![DOI](https://zenodo.org/badge/449677569.svg)](https://zenodo.org/badge/latestdoi/449677569)

timewise-1.0.0a5/ampel/timewise/alert/TimewiseAlertSupplier.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# File:                timewise/ampel/timewise/alert/TimewiseAlertSupplier.py
+# License:             BSD-3-Clause
+# Author:              Jannis Necker <jannis.necker@gmail.com>
+# Date:                16.09.2025
+# Last Modified Date:  16.09.2025
+# Last Modified By:    Jannis Necker <jannis.necker@gmail.com>
+
+import sys
+from hashlib import blake2b
+from typing import Literal, List
+
+import pandas as pd
+
+from bson import encode
+
+from ampel.alert.AmpelAlert import AmpelAlert
+from ampel.alert.BaseAlertSupplier import BaseAlertSupplier
+from ampel.view.ReadOnlyDict import ReadOnlyDict
+
+
+class TimewiseAlertSupplier(BaseAlertSupplier):
+    """
+    Iterable class that, for each transient name provided by the underlying alert_loader
+    returns a PhotoAlert instance.
+    """
+
+    stat_pps: int = 0
+    stat_uls: int = 0
+
+    dpid: Literal["hash", "inc"] = "hash"
+    #    external_directory: Optional[ str ]
+    #    deserialize: None | Literal["avro", "json"]
+
+    bands: List[str] = ["w1", "w2"]
+
+    def __init__(self, **kwargs) -> None:
+        super().__init__(**kwargs)
+        self.counter = 0 if self.dpid == "hash" else 1
+
+    def __next__(self) -> AmpelAlert:
+        """
+        :returns: a dict with a structure that AlertProcessor understands
+        :raises StopIteration: when alert_loader dries out.
+        :raises AttributeError: if alert_loader was not set properly before this method is called
+        """
+        table: pd.DataFrame = self._deserialize(next(self.alert_loader))  # type: ignore
+
+        stock_ids = table["stock_id"].unique()
+        assert len(stock_ids) == 1
+        stock_id = stock_ids[0]
+
+        # make the tables into a list of dictionaries for ampel to understand
+        all_ids = b""
+        pps = []
+
+        # remove the _ep at the end of AllWISE MEP data
+        columns_to_rename = [c for c in table.columns if c.endswith("_ep")]
+        if len(columns_to_rename):
+            rename = {
+                c: c.replace("_ep", "")
+                for c in columns_to_rename
+                if c.replace("_ep", "") not in table.columns
+            }
+            if rename:
+                # in this case only the allwise column eith the _ep extension exists
+                # and we can simply rename the columns
+                table.rename(columns=rename, inplace=True)
+
+            move = {
+                c: c.replace("_ep", "")
+                for c in columns_to_rename
+                if c.replace("_ep", "") in table.columns
+            }
+            if move:
+                # In this case, the columns already exists because the neowise data is present
+                # we have to insert the values form the columns with the _ep extension into the
+                # respective neowise columns
+                for c, nc in move.items():
+                    na_mask = table[nc].isna()
+                    table.loc[na_mask, nc] = table[c][na_mask]
+                pd.options.mode.chained_assignment = None
+                table.drop(columns=[c for c in move], inplace=True)
+                pd.options.mode.chained_assignment = "warn"
+
+        for i, row in table.iterrows():
+            # convert table row to dict, convert data types from numpy to native python
+            # Respect masked fields and convert to None
+            pp = {k: None if pd.isna(v) else v for k, v in row.to_dict().items()}
+            pp_hash = blake2b(encode(pp), digest_size=7).digest()
+            if self.counter:
+                pp["candid"] = self.counter
+                self.counter += 1
+            else:
+                pp["candid"] = int.from_bytes(pp_hash, byteorder=sys.byteorder)
+
+            all_ids += pp_hash
+            pps.append(ReadOnlyDict(pp))
+
+        if not pps:
+            return self.__next__()
+
+        # Update stats
+        self.stat_pps += len(pps)
+
+        return AmpelAlert(
+            id=int.from_bytes(  # alert id
+                blake2b(all_ids, digest_size=7).digest(), byteorder=sys.byteorder
+            ),
+            stock=int(stock_id),  # internal ampel id
+            datapoints=tuple(pps),
+        )

timewise-1.0.0a5/ampel/timewise/alert/load/TimewiseFileLoader.py

@@ -0,0 +1,118 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# File:                timewise/ampel/timewise/alert/load/TimewiseFileLoader.py
+# License:             BSD-3-Clause
+# Author:              Jannis Necker <jannis.necker@gmail.com>
+# Date:                16.09.2025
+# Last Modified Date:  16.09.2025
+# Last Modified By:    Jannis Necker <jannis.necker@gmail.com>
+from typing import Dict, get_args
+
+import numpy as np
+import pandas as pd
+from astropy.table import Table, vstack
+from ampel.abstract.AbsAlertLoader import AbsAlertLoader
+from timewise.tables import TableType
+from timewise.config import TimewiseConfig
+from timewise.types import TaskID
+
+
+class TimewiseFileLoader(AbsAlertLoader[Dict]):
+    """
+    Load alerts from one of more files.
+    """
+
+    # path to timewise download config file
+    timewise_config_file: str
+
+    # column name of id
+    stock_id_column_name: str
+
+    chunks: list[int] | None = None
+
+    def __init__(self, **kwargs) -> None:
+        super().__init__(**kwargs)
+
+        self.logger.info(f"loading timewise config file {self.timewise_config_file}")
+        timewise_config = TimewiseConfig.from_yaml(self.timewise_config_file)
+        dl = timewise_config.download.build_downloader()
+        self._timewise_backend = dl.backend
+
+        # selecting tasks to run
+        _tasks = [tasks for tasks in dl.iter_tasks_per_chunk()]
+        if self.chunks is not None:
+            self._tasks = [_tasks[i] for i in self.chunks]
+        else:
+            self._tasks = _tasks
+        if self.logger:
+            self.logger.info(
+                f"Registering {len(self._tasks)} chunk(s) to load: {self._tasks}"
+            )
+
+        self._table_types = get_args(TableType.__origin__)  # type: ignore
+        self._gen = self.iter_stocks()
+
+    @staticmethod
+    def encode_result(res: pd.DataFrame) -> pd.DataFrame:
+        if isinstance(res, pd.Series):
+            return pd.DataFrame([res])
+        return res
+
+    def find_table_from_task(self, task: TaskID) -> TableType:
+        tables = [
+            t for t in self._table_types if t.model_fields["name"].default in str(task)
+        ]
+        assert len(tables) > 0, f"No matching table found for {task}!"
+        assert len(tables) < 2, f"More than one matching table found for {task}!"
+        self.logger.debug(
+            f"{task} is from table {tables[0].model_fields['name'].default}"
+        )
+        return tables[0]
+
+    def iter_stocks(self):
+        # emit all datapoints per stock id
+        # This way ampel runs not per datapoint but per object
+        backend = self._timewise_backend
+        for tasks in self._tasks:
+            data = []
+            for task in tasks:
+                self.logger.debug(f"reading {task}")
+                idata = backend.load_data(task)
+
+                # add table name
+                idata["table_name"] = (
+                    self.find_table_from_task(task).model_fields["name"].default
+                )
+
+                data.append(idata)
+
+            data = vstack(data).to_pandas()
+
+            # rename stock id column
+            data.rename(columns={self.stock_id_column_name: "stock_id"}, inplace=True)
+
+            # Find the indices for each stock id. This is much faster than making a mask
+            # each loop and accessing the table then. Shown below is a comparison.
+            # The top example is the access provided by pandas which would be
+            # again a factor 3 faster.
+            #
+            # In [45]: %timeit test_df()
+            # 5.62 μs ± 47.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
+            #
+            # In [46]: %timeit test_index()
+            # 14.6 μs ± 45 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
+            #
+            # In [47]: %timeit test_mask()
+            # 2.61 ms ± 18 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
+            data.set_index(data.stock_id, inplace=True)
+
+            # iterate over all stock ids
+            for stock_id in np.unique(data["stock_id"]):
+                selection = data.loc[stock_id]
+                yield self.encode_result(selection)
+
+    def __iter__(self):
+        return self
+
+    def __next__(self) -> pd.DataFrame:  # type: ignore
+        return next(self._gen)

timewise-1.0.0a5/ampel/timewise/ingest/TiCompilerOptions.py

@@ -0,0 +1,20 @@
+#!/usr/bin/env python
+# File:                Ampel-ZTF/ampel/ztf/ingest/ZiCompilerOptions.py
+# License:             BSD-3-Clause
+# Author:              valery brinnel <firstname.lastname@gmail.com>
+# Date:                14.05.2021
+# Last Modified Date:  14.05.2021
+# Last Modified By:    valery brinnel <firstname.lastname@gmail.com>
+
+from typing import Any
+
+from ampel.model.ingest.CompilerOptions import CompilerOptions
+
+
+class TiCompilerOptions(CompilerOptions):
+    stock: dict[str, Any] = {"tag": "TIMEWISE"}
+    t0: dict[str, Any] = {"tag": "TIMEWISE"}
+    t1: dict[str, Any] = {"tag": "TIMEWISE"}
+    state_t2: dict[str, Any] = {"tag": "TIMEWISE"}
+    point_t2: dict[str, Any] = {"tag": "TIMEWISE"}
+    stock_t2: dict[str, Any] = {"tag": "TIMEWISE"}

timewise-1.0.0a5/ampel/timewise/ingest/TiDataPointShaper.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python
+# File:                timewise/ampel/timewise/ingest/TiDataPointShaper.py
+# License:             BSD-3-Clause
+# Author:              valery brinnel <firstname.lastname@gmail.com>
+# Date:                14.12.2017
+# Last Modified Date:  19.09.2025
+# Last Modified By:    Jannis Necker <jannis.necker@gmail.com>
+
+from collections.abc import Iterable, Sequence
+from typing import Any
+
+from bson import encode
+
+from ampel.abstract.AbsT0Unit import AbsT0Unit
+from ampel.base.AmpelUnit import AmpelUnit
+from ampel.content.DataPoint import DataPoint
+from ampel.types import StockId, Tag
+from ampel.util.hash import hash_payload
+
+from ampel.timewise.ingest.tags import tags
+
+
+class TiDataPointShaperBase(AmpelUnit):
+    """
+    This class 'shapes' datapoints in a format suitable
+    to be saved into the ampel database
+    """
+
+    # JD2017 is used to define upper limits primary IDs
+    JD2017: float = 2457754.5
+    #: Byte width of datapoint ids
+    digest_size: int = 8
+
+    # Mandatory implementation
+    def process(self, arg: Iterable[dict[str, Any]], stock: StockId) -> list[DataPoint]:
+        """
+        :param arg: sequence of unshaped pps
+        IMPORTANT:
+        1) This method *modifies* the input dicts (it removes 'candid' and programpi),
+        even if the unshaped pps are ReadOnlyDict instances
+        2) 'stock' is not set here on purpose since it will conflict with the $addToSet operation
+        """
+
+        ret_list: list[DataPoint] = []
+        popitem = dict.pop
+
+        for photo_dict in arg:
+            # Photopoint
+            assert photo_dict.get("candid"), "photometry points does not have 'candid'!"
+            ret_list.append(
+                {  # type: ignore[typeddict-item]
+                    "id": photo_dict["candid"],
+                    "stock": stock,
+                    "tag": tags[photo_dict["table_name"]],
+                    "body": photo_dict,
+                }
+            )
+
+            popitem(photo_dict, "candid", None)
+
+        return ret_list
+
+    def _create_datapoint(
+        self, stock: StockId, tag: Sequence[Tag], body: dict[str, Any]
+    ) -> DataPoint:
+        """
+        Create a Datapoint from stock, body, and tags, using the hash of the body as id
+        """
+        # ensure that keys are ordered
+        sorted_body = dict(sorted(body.items()))
+        # The following is a comment from the original ampel.ztf.ingest.ZiDataPointShaperBase:
+        # This is not a complete DataPoint as (channel,meta) is missing, set later.
+        # Should these be optional? or added default?
+        return {  # type: ignore
+            "id": hash_payload(encode(sorted_body), size=-self.digest_size * 8),
+            "stock": stock,
+            "tag": [*tags[body["table_name"]], *tag],
+            "body": sorted_body,
+        }
+
+    def ul_identity(self, uld: dict[str, Any]) -> int:
+        """
+        This should not happen
+        """
+        raise NotImplementedError
+
+
+class TiDataPointShaper(TiDataPointShaperBase, AbsT0Unit):
+    def process(self, arg: Any, stock: None | StockId = None) -> list[DataPoint]:
+        assert stock is not None
+        return super().process(arg, stock)