atomscale 0.10.2__tar.gz → 0.12.0__tar.gz

{atomscale-0.10.2 → atomscale-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atomscale
-Version: 0.10.2
+Version: 0.12.0
 Summary: Python SDK for Atomscale.
 Author-email: Atomscale <info@atomscale.ai>
 License: GPL-3.0-only

atomscale-0.12.0/examples/changepoints.ipynb

@@ -0,0 +1,191 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Changepoint Detection\n",
+    "This short notebook shows how to pull changepoint detection records from the Atomscale API using `client.get_changepoints()`. Changepoints are time-bounded regions where an automated detector flagged a meaningful shift in one of the analyzed properties of a data item."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Install package"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install atomscale"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### API Client Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from atomscale import Client\n",
+    "from atomscale.results import ChangepointResult\n",
+    "\n",
+    "api_key = \"YOUR_API_KEY_HERE\"\n",
+    "client = Client(api_key=api_key)\n",
+    "\n",
+    "# Use a data ID from your catalogue. Search or grab one from the web interface.\n",
+    "data_id = \"YOUR_DATA_ID_HERE\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Fetching Changepoints\n",
+    "Call `get_changepoints` with one or more `data_ids` to get back a pandas `DataFrame` of detected changepoints. With default arguments it returns the most recent intensity-profile detection run, filtered to the critical severity tier — a good starting point for most users."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "changepoints = client.get_changepoints(data_ids=data_id)\n",
+    "changepoints"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Each row is a single changepoint. The columns are:\n",
+    "```\n",
+    "['id', 'data_id', 'data_modality', 'property_name', 'severity', 'score',\n",
+    " 'window_start_elapsed', 'window_end_elapsed', 'detection_method', 'detail', 'label']\n",
+    "```\n",
+    "- `score` is a normalized magnitude in `[0, 1]` — higher means stronger signal.\n",
+    "- `window_start_elapsed` / `window_end_elapsed` are seconds from the start of the source time series, so you can line changepoints up against timeseries data pulled via `client.get()`.\n",
+    "- `detail` holds method-specific metadata (e.g. RMS error, comparison window size).\n",
+    "- `label` is the applied category label if one has been set, otherwise `None`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Filtering Options\n",
+    "The defaults keep the view focused, but all three filter arguments can be adjusted:\n",
+    "\n",
+    "- `detection_method` — `\"forecasting\"`, `\"clustering\"`, `\"intensity_profile\"`, or `None` for all methods. Defaults to `\"intensity_profile\"`.\n",
+    "- `severity` — `\"info\"`, `\"warning\"`, `\"critical\"`, or `None` for all levels. Defaults to `\"critical\"`.\n",
+    "- `latest_only` — when `True` (default), only anomalies from the most recently completed run per `(data_id, detection_method)` are returned. Set to `False` to include every historical run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Everything the detector flagged, no filters applied.\n",
+    "all_changepoints = client.get_changepoints(\n",
+    "    data_ids=data_id,\n",
+    "    detection_method=None,\n",
+    "    severity=None,\n",
+    ")\n",
+    "all_changepoints[[\"detection_method\", \"severity\", \"score\", \"property_name\"]]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# All historical runs for this data_id, not just the latest.\n",
+    "history = client.get_changepoints(\n",
+    "    data_ids=data_id,\n",
+    "    detection_method=None,\n",
+    "    severity=None,\n",
+    "    latest_only=False,\n",
+    ")\n",
+    "print(f\"latest run only: {len(all_changepoints)} rows\")\n",
+    "print(f\"all historical runs: {len(history)} rows\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Result Objects\n",
+    "Pass `as_dataframe=False` to get a list of `ChangepointResult` objects instead of a DataFrame. Useful when you want to iterate and work with the records programmatically."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = client.get_changepoints(data_ids=data_id, as_dataframe=False)\n",
+    "\n",
+    "for cp in results[:5]:\n",
+    "    window = f\"[{cp.window_start_elapsed:.1f}s \\u2192 {cp.window_end_elapsed:.1f}s]\"\n",
+    "    print(f\"{cp.severity:>8}  score={cp.score:.3f}  {window}  {cp.property_name}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch Over Multiple Data IDs\n",
+    "`data_ids` accepts a list; requests are chunked under the hood so you can pass an arbitrary number of IDs in a single call."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Use a search to pull a batch of data IDs, then fetch changepoints across all of them.\n",
+    "search_results = client.search(data_type=\"rheed_stationary\", status=\"success\")\n",
+    "data_ids = search_results[\"Data ID\"].to_list()\n",
+    "\n",
+    "batch = client.get_changepoints(data_ids=data_ids)\n",
+    "batch.groupby(\"data_id\").size().sort_values(ascending=False).head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For more information on other data from the API or other example use see notebooks in the code repository (https://github.com/atomic-data-sciences/api-client) and the documentation (https://atomic-data-sciences.github.io/api-client/)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

{atomscale-0.10.2 → atomscale-0.12.0}/pyproject.toml

@@ -63,6 +63,7 @@ package-dir = { "" = "src" }
 [tool.setuptools.packages.find]
 where = ["src"]
 include = ["atomscale*", "atomicds"]
+exclude = ["atomscale.streaming.target*"]
 
 [tool.setuptools-rust]
 debug = false

{atomscale-0.10.2 → atomscale-0.12.0}/src/atomscale/client.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import os
+import re
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from datetime import datetime
 from pathlib import Path
@@ -14,12 +15,14 @@ from pandas import DataFrame
 from atomscale.core import BaseClient, ClientError, _FileSlice
 from atomscale.core.utils import _make_progress, normalize_path
 from atomscale.results import (
+    ChangepointResult,
     PhotoluminescenceResult,
     RamanResult,
     RHEEDImageResult,
     RHEEDVideoResult,
     UnknownResult,
     XPSResult,
+    XRDResult,
     _get_rheed_image_result,
 )
 from atomscale.results.group import PhysicalSampleResult, ProjectResult
@@ -71,6 +74,7 @@ class Client(BaseClient):
             "rheed_stationary",
             "rheed_rotating",
             "xps",
+            "xrd",
             "photoluminescence",
             "pl",
             "raman",
@@ -102,7 +106,7 @@ class Client(BaseClient):
             data_ids (str | list[str] | None): Data ID or list of data IDs. Defaults to None.
             physical_sample_ids (str | list[str] | None): Physical sample ID or list of IDs. Defaults to None.
             project_ids (str | list[str] | None): Project ID or list of IDs. Defaults to None.
-            data_type (Literal["rheed_image", "rheed_stationary", "rheed_rotating", "xps", "photoluminescence", "raman", "all"]): Type of data. Defaults to "all".
+            data_type (Literal["rheed_image", "rheed_stationary", "rheed_rotating", "xps", "xrd", "photoluminescence", "raman", "all"]): Type of data. Defaults to "all".
             status (Literal["success", "pending", "error", "running", "all"]): Analyzed status of the data. Defaults to "all".
             growth_length (tuple[int | None, int | None]): Minimum and maximum values of the growth length in seconds.
                 Defaults to (None, None) which will include all non-video data.
@@ -172,10 +176,10 @@ class Client(BaseClient):
 
         if "projects" in catalogue.columns:
             catalogue["project_ids"] = catalogue["projects"].apply(
-                lambda projects: (projects[0].get("id") if projects else None)
+                lambda projects: projects[0].get("id") if projects else None
             )
             catalogue["project_names"] = catalogue["projects"].apply(
-                lambda projects: (projects[0].get("name") if projects else None)
+                lambda projects: projects[0].get("name") if projects else None
             )
 
         if len(catalogue):
@@ -224,6 +228,7 @@ class Client(BaseClient):
         RHEEDVideoResult
         | RHEEDImageResult
         | XPSResult
+        | XRDResult
         | PhotoluminescenceResult
         | RamanResult
         | UnknownResult
@@ -234,7 +239,7 @@ class Client(BaseClient):
             data_ids (str | list[str]): Data ID or list of data IDs from the data catalogue to obtain analyzed results for.
 
         Returns:
-            list[atomscale.results.RHEEDVideoResult | atomscale.results.RHEEDImageResult | atomscale.results.XPSResult]:
+            list[atomscale.results.RHEEDVideoResult | atomscale.results.RHEEDImageResult | atomscale.results.XPSResult | atomscale.results.XRDResult]:
                 List of result objects
 
         """
@@ -285,6 +290,80 @@ class Client(BaseClient):
                 progress_description="Obtaining data results",
             )
 
+    def get_changepoints(
+        self,
+        data_ids: str | list[str],
+        latest_only: bool = True,
+        detection_method: (
+            Literal["forecasting", "clustering", "intensity_profile"] | None
+        ) = "intensity_profile",
+        severity: Literal["info", "warning", "critical"] | None = "critical",
+        as_dataframe: bool = True,
+    ) -> DataFrame | list[ChangepointResult]:
+        """Get changepoint detection records for one or more data IDs.
+
+        Args:
+            data_ids (str | list[str]): Data ID or list of data IDs from the data catalogue.
+            latest_only (bool): If True (default), only return changepoints from the most
+                recently completed detection run for each (data_id, detection_method) pair.
+                If False, return all changepoints from every historical run.
+            detection_method (str | None): Filter to a single detection method. One of
+                "forecasting", "clustering", "intensity_profile". Defaults to "intensity_profile".
+                Pass None to include all detection methods.
+            severity (str | None): Filter to a single severity level. One of "info",
+                "warning", "critical". Defaults to "critical". Pass None to include all
+                severities.
+            as_dataframe (bool): If True (default) return a pandas DataFrame. If False
+                return a list of ChangepointResult objects.
+
+        Returns:
+            DataFrame | list[ChangepointResult]: Changepoint records matching the filters.
+        """
+        if isinstance(data_ids, str):
+            data_ids = [data_ids]
+
+        records: list[dict] = []
+        chunk_size = 100
+        chunks = [
+            data_ids[i : i + chunk_size] for i in range(0, len(data_ids), chunk_size)
+        ]
+
+        for chunk in chunks:
+            payload: dict | None = self._get(  # type: ignore[assignment]
+                sub_url="changepoints/",
+                params={"data_ids": chunk, "latest_only": latest_only},
+            )
+            if payload:
+                records.extend(payload.get("changepoints", []))
+
+        if detection_method is not None:
+            records = [
+                r for r in records if r.get("detection_method") == detection_method
+            ]
+        if severity is not None:
+            records = [r for r in records if r.get("severity") == severity]
+
+        if as_dataframe:
+            # Keep only the label itself; drop label provenance/metadata fields.
+            _drop = {
+                "label_category",
+                "label_notes",
+                "label_source",
+                "label_confidence",
+                "labeled_at",
+                "labeled_by_user_id",
+                "similar_neighbor_ids",
+            }
+            rows = [
+                {
+                    **{k: v for k, v in r.items() if k not in _drop},
+                    "label": r.get("label_category"),
+                }
+                for r in records
+            ]
+            return DataFrame(rows)
+        return [ChangepointResult.from_api(r) for r in records]
+
     def list_physical_samples(self) -> DataFrame:
         """List physical samples available to the user."""
         data = self._get(sub_url="physical_samples/")
@@ -294,10 +373,10 @@ class Client(BaseClient):
 
         if "projects" in samples.columns:
             samples["project_id"] = samples["projects"].apply(
-                lambda projects: (projects[0].get("id") if projects else None)
+                lambda projects: projects[0].get("id") if projects else None
             )
             samples["project_name"] = samples["projects"].apply(
-                lambda projects: (projects[0].get("name") if projects else None)
+                lambda projects: projects[0].get("name") if projects else None
             )
 
         if "detail_notes" in samples.columns:
@@ -305,9 +384,9 @@ class Client(BaseClient):
                 lambda note: note.get("content") if isinstance(note, dict) else None
             )
             samples["detail_note_last_updated"] = samples["detail_notes"].apply(
-                lambda note: note.get("last_updated")
-                if isinstance(note, dict)
-                else None
+                lambda note: (
+                    note.get("last_updated") if isinstance(note, dict) else None
+                )
             )
             samples["detail_note_last_updated"] = samples[
                 "detail_note_last_updated"
@@ -315,13 +394,15 @@ class Client(BaseClient):
 
         if "target_material" in samples.columns:
             samples["target_material"] = samples["target_material"].apply(
-                lambda tm: {
-                    k: tm.get(k)
-                    for k in ("substrate", "sample_name")
-                    if isinstance(tm, dict) and k in tm
-                }
-                if isinstance(tm, dict)
-                else tm
+                lambda tm: (
+                    {
+                        k: tm.get(k)
+                        for k in ("substrate", "sample_name")
+                        if isinstance(tm, dict) and k in tm
+                    }
+                    if isinstance(tm, dict)
+                    else tm
+                )
             )
 
         columns_to_drop = [
@@ -390,9 +471,9 @@ class Client(BaseClient):
                 lambda note: note.get("content") if isinstance(note, dict) else None
             )
             projects["detail_note_last_updated"] = projects["detail_note"].apply(
-                lambda note: note.get("last_updated")
-                if isinstance(note, dict)
-                else None
+                lambda note: (
+                    note.get("last_updated") if isinstance(note, dict) else None
+                )
             )
             projects["detail_note_last_updated"] = projects[
                 "detail_note_last_updated"
@@ -552,6 +633,7 @@ class Client(BaseClient):
         data_id: str,
         data_type: Literal[
             "xps",
+            "xrd",
             "photoluminescence",
             "pl",
             "raman",
@@ -570,20 +652,41 @@ class Client(BaseClient):
         | XPSResult
         | PhotoluminescenceResult
         | RamanResult
+        | XRDResult
         | UnknownResult
         | None
     ):
+        collected_dt = (
+            catalogue_entry.get("collected_datetime") if catalogue_entry else None
+        )
+
         if data_type == "xps":
-            result: dict = self._get(sub_url=f"xps/{data_id}")  # type: ignore  # noqa: PGH003
+            result: dict = self._get(sub_url=f"xps/{data_id}") or {}  # type: ignore  # noqa: PGH003
 
             return XPSResult(
                 data_id=data_id,
-                xps_id=result["xps_id"],
-                binding_energies=result["binding_energies"],
-                intensities=result["intensities"],
-                predicted_composition=result["predicted_composition"],
-                detected_peaks=result["detected_peaks"],
-                elements_manually_set=bool(result["set_elements"]),
+                xps_id=result.get("xps_id"),
+                binding_energies=result.get("binding_energies", []),
+                intensities=result.get("intensities", []),
+                predicted_composition=result.get("predicted_composition", {}),
+                detected_peaks=result.get("detected_peaks", {}),
+                elements_manually_set=bool(result.get("set_elements", False)),
+                collected_datetime=collected_dt,
+            )
+
+        if data_type == "xrd":
+            result = self._get(sub_url=f"xrd/{data_id}") or {}  # type: ignore  # noqa: PGH003
+            return XRDResult(
+                data_id=data_id,
+                xrd_id=result.get("id"),
+                two_theta=result.get("two_theta", []),
+                intensities=result.get("intensities", []),
+                detected_peaks=result.get("detected_peaks", []),
+                wavelength_angstrom=result.get("wavelength_angstrom", 1.5406),
+                two_theta_unit=result.get("two_theta_unit", "degrees"),
+                spectral_metadata=result.get("spectral_metadata", {}),
+                last_updated=result.get("last_updated"),
+                collected_datetime=collected_dt,
             )
 
         if data_type in ("photoluminescence", "pl"):
@@ -602,6 +705,7 @@ class Client(BaseClient):
                 intensities=result.get("intensities", []),
                 detected_peaks=result.get("detected_peaks", {}),
                 last_updated=result.get("last_updated"),
+                collected_datetime=collected_dt,
             )
 
         if data_type == "raman":
@@ -613,10 +717,14 @@ class Client(BaseClient):
                 intensities=result.get("intensities", []),
                 detected_peaks=result.get("detected_peaks", {}),
                 last_updated=result.get("last_updated"),
+                collected_datetime=collected_dt,
             )
 
         if data_type == "rheed_image":
-            return _get_rheed_image_result(self, data_id)
+            result_obj = _get_rheed_image_result(self, data_id)
+            if result_obj is not None:
+                result_obj.collected_datetime = collected_dt
+            return result_obj
 
         if data_type in [
             "rheed_stationary",
@@ -646,6 +754,7 @@ class Client(BaseClient):
                 upload_dt = catalogue_entry.get("upload_datetime")
                 if upload_dt:
                     result_obj.upload_datetime = upload_dt
+            result_obj.collected_datetime = collected_dt
             return result_obj
 
         # Fallback for unknown/unsupported data types
@@ -653,16 +762,80 @@ class Client(BaseClient):
             data_id=data_id,
             data_type=data_type,
             catalogue_entry=catalogue_entry,
+            collected_datetime=collected_dt,
+        )
+
+    _UUID_RE = re.compile(
+        r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.IGNORECASE
+    )
+
+    def _resolve_physical_sample(self, physical_sample: str) -> tuple[str, str]:
+        """Resolve a physical sample name or UUID to (id, name).
+
+        If UUID: look up existing sample, error if not found.
+        If name: case-insensitive match, auto-create if not found.
+
+        Returns:
+            Tuple of (physical_sample_id, physical_sample_name).
+        """
+        physical_sample = physical_sample.strip()
+        if not physical_sample:
+            raise ClientError("physical_sample cannot be empty")
+
+        samples_df = self.list_physical_samples()
+
+        if self._UUID_RE.match(physical_sample):
+            match = samples_df[samples_df["Physical Sample ID"] == physical_sample]
+            if match.empty:
+                raise ClientError(
+                    f"Physical sample with id '{physical_sample}' not found"
+                )
+            return physical_sample, match.iloc[0]["Physical Sample Name"]
+
+        # Name lookup: case-insensitive exact match
+        names_lower = samples_df["Physical Sample Name"].str.strip().str.lower()
+        mask = names_lower == physical_sample.lower()
+        match = samples_df[mask]
+
+        if not match.empty:
+            return match.iloc[0]["Physical Sample ID"], match.iloc[0][
+                "Physical Sample Name"
+            ]
+
+        # Not found — create a new physical sample
+        resp: dict = self._post_or_put(  # type: ignore  # noqa: PGH003
+            method="POST",
+            sub_url="physical_samples/",
+            body={"name": physical_sample},
         )
+        return resp["id"], physical_sample
 
-    def upload(self, files: list[str | BinaryIO]):
-        """Upload and process files
+    def upload(
+        self,
+        files: list[str | BinaryIO],
+        physical_sample: str | None = None,
+    ) -> list[str]:
+        """Upload and process files.
 
         Args:
-            files (list[str | BinaryIO]): List containing string paths to files, or BinaryIO objects from `open`.
+            files (list[str | BinaryIO]): List containing string paths to files, or BinaryIO objects from ``open``.
+            physical_sample (str | None): Physical sample name or UUID to link uploads to.
+                If a name is given and no matching sample exists, one is created automatically.
+
+        Returns:
+            list[str]: Data IDs assigned to the uploaded files.
         """
         chunk_size = 40 * 1024 * 1024  # 40 MiB
 
+        # Resolve physical sample before uploading so we fail fast on bad input
+        metadata_body: dict[str, str] | None = None
+        if physical_sample is not None:
+            ps_id, ps_name = self._resolve_physical_sample(physical_sample)
+            metadata_body = {
+                "physical_sample_id": ps_id,
+                "physical_sample_name": ps_name,
+            }
+
         # Check to make sure list is valid and get pre-signed URL nums
         file_data = []
         for file in files:
@@ -697,7 +870,7 @@ class Client(BaseClient):
             file_info: dict[
                 Literal["num_urls", "file_name", "file_size", "file_path"], int | str
             ],
-        ):
+        ) -> str:
             url_data: list[dict[str, str | int]] = self._post_or_put(
                 method="POST",
                 sub_url="data_entries/raw_data/staged/upload_urls/",
@@ -706,6 +879,7 @@ class Client(BaseClient):
                     "num_parts": file_info["num_urls"],
                     "staging_type": "core",
                 },
+                body=metadata_body,
             )  # type: ignore  # noqa: PGH003
 
             # Iterate through data structure above and upload file using multi-part S3 urls. Multithread appropriately.
@@ -783,6 +957,9 @@ class Client(BaseClient):
                     },
                 )
 
+            return str(first_part["data_id"])
+
+        data_ids: list[str] = []
         main_task = None
         file_count = len(file_data)
         with _make_progress(self.mute_bars, False) as progress:
@@ -803,10 +980,12 @@ class Client(BaseClient):
                     for file_info in file_data
                 }
                 for future in as_completed(futures):
-                    future.result()  # raise early if anything went wrong
+                    data_ids.append(future.result())
                     if main_task is not None:
                         progress.update(main_task, advance=1, refresh=True)
 
+        return data_ids
+
     def download_videos(
         self,
         data_ids: str | list[str],

{atomscale-0.10.2 → atomscale-0.12.0}/src/atomscale/results/__init__.py

@@ -1,3 +1,4 @@
+from .changepoint import ChangepointResult
 from .group import PhysicalSampleResult, ProjectResult
 from .metrology import MetrologyResult
 from .optical import OpticalResult
@@ -8,8 +9,10 @@ from .rheed_video import RHEEDVideoResult
 from .similarity_trajectory import SimilarityTrajectoryResult
 from .unknown import UnknownResult
 from .xps import XPSResult
+from .xrd import XRDResult
 
 __all__ = [
+    "ChangepointResult",
     "MetrologyResult",
     "OpticalResult",
     "PhotoluminescenceResult",
@@ -22,5 +25,6 @@ __all__ = [
     "SimilarityTrajectoryResult",
     "UnknownResult",
     "XPSResult",
+    "XRDResult",
     "_get_rheed_image_result",
 ]