fmu-sumo 2.4.10__tar.gz → 2.6.0__tar.gz

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fmu-sumo
-Version: 2.4.10
+Version: 2.6.0
 Summary: Python package for interacting with Sumo in an FMU setting
 Author: Equinor
 License:                                  Apache License

fmu_sumo-2.6.0/examples/explorer.ipynb

@@ -0,0 +1,460 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "These are basic examples of how the `fmu-sumo` package can be used for consuming FMU results via the Python API.\n",
+    "\n",
+    "To install fmu-sumo, use `pip install fmu-sumo`\n",
+    "\n",
+    "We are always grateful for feedback and any question you might have. See the <a href=\"https://doc-sumo-doc-prod.radix.equinor.com/\">Sumo Documentation</a> for how to get in touch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from fmu.sumo.explorer import Explorer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%html\n",
+    "<style>table {margin-left: 0 !important;}</style>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Terminology\n",
+    "\n",
+    "Hiearchy of FMU results:\n",
+    "| term | description | disk equivalent | examples of data in this context |\n",
+    "| -- | -- | -- | -- |\n",
+    "| model | The workflow that, when realized, created these results | /project/.../model/**revision**/ | |\n",
+    "| | | | |\n",
+    "| case | A collection of _ensembles_ | /scratch/field/user/**case** | Observations |\n",
+    "| ensemble | A collection of _realizations_ | /scratch/field/user/case/realization-*/**ensemble**/ | Aggregated results |\n",
+    "| realization | A collection of data objects produced by the same realization | /scratch/field/user/case/**realization**/ensemble/ | Exported results |\n",
+    "| | | | |\n",
+    "| entity | The array of objects that are representations of the same result | iter-\\*/realization-\\*/**share/results/type/myfile.xyz | |\n",
+    "\n",
+    "For more details, refer to https://doc-sumo-doc-prod.radix.equinor.com/fmu_results\n",
+    "\n",
+    "### Principles\n",
+    "The basic entry point for consuming FMU results is the _case_. On disk, we are used to the case concept as a folder with a name. A name, however, is not unique. Therefore, cases in terms of FMU results are assigned unique IDs allowing us to refer explicitly to a specific case regardless of its name.\n",
+    "\n",
+    "When working with FMU results, we mostly operate within the bounds of a specific _ensemble_. Note, however, that FMU results exists both in the case context, the ensemble context and in the realization context."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Initialize Explorer\n",
+    "Initializing the explorer will establish a connection to Sumo. If you don't have a valid access token stored, you will be taken through the authentication process."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sumo = Explorer()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get Drogon cases\n",
+    "myassetname = \"Drogon\"  # Must be a valid asset on Sumo\n",
+    "cases = sumo.cases.filter(asset=myassetname)\n",
+    "\n",
+    "# Filter on user\n",
+    "cases = cases.filter(user=\"peesv\")\n",
+    "\n",
+    "# Iterate over results\n",
+    "print(f\"\\nFound {len(cases)} cases:\")\n",
+    "\n",
+    "for case in cases:\n",
+    "    print(f\"Case: {case.name} | {case.uuid}\")\n",
+    "    for ensemble in case.ensembles:\n",
+    "        print(f\"    Ensemble: {ensemble.name} (n={len(ensemble.realizations)})\")\n",
+    "\n",
+    "\n",
+    "# Option 1 (recommended): Get case by uuid\n",
+    "mycaseuuid = cases[0].uuid  # for sake of example\n",
+    "unique_cases = sumo.cases.filter(uuid=mycaseuuid)[0]\n",
+    "\n",
+    "# Option 2: Get case by name (name is not guaranteed to be unique)\n",
+    "mycasename = cases[0].name  # for sake of example\n",
+    "named_cases = sumo.cases.filter(name=mycasename)\n",
+    "\n",
+    "if len(named_cases) > 1:\n",
+    "    raise ValueError(f\"More than one case exist with name {mycasename}, please use UUID.\")\n",
+    "elif len(named_cases) == 0:\n",
+    "    raise ValueError(f\"No case with name {mycasename} exist.\")\n",
+    "else:\n",
+    "    mycase = named_cases[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Select case\n",
+    "mycase = cases[-1] # for sake of the example we pick the last - you might want to select your case differently\n",
+    "print(\"Selected case: \")\n",
+    "print(f\"  -> {mycase.name} ({mycase.uuid}) [{mycase.status}]\")\n",
+    "\n",
+    "# Select ensemble\n",
+    "myensemble = mycase.ensembles[-1] # for sake of the example - you might want to be more explitic on which ensemble you select\n",
+    "print(\"Selected ensemble:\")\n",
+    "print(f\"  -> {myensemble.name}\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "At this point, we have identified our case and the ensemble we want to work with. Now we will consume data:\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "As opposed to on the disk, FMU results stored in Sumo are stored in a flat structure and we use metadata to identify individual data objects and arrays of data across an ensemble. The Explorer is fairly versitale, and backed by a powerful search engine. This means that there are usually multiple ways of getting to the same data, depending on your use case. In these examples, we show a few of these patterns.\n",
+    "</div>\n",
+    "\n",
+    "### Search context\n",
+    "An important concept in the Explorer is the _SearchContext_. This is a collection of data, filtered according to your needs. Here are some examples of SearchContexts:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# all _surfaces_ in our ensemble\n",
+    "myensemble.surfaces\n",
+    "\n",
+    "# all _tables_ in a case\n",
+    "myensemble.tables"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Filtering\n",
+    "The Search Context allows for further filtering:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# list all names available for these surfaces\n",
+    "myensemble.surfaces.names\n",
+    "\n",
+    "# Filter on a specific name\n",
+    "myensemble.surfaces.filter(name=\"Therys Fm. Top\")\n",
+    "\n",
+    "# list all tagnames available for surfaces with this name\n",
+    "myensemble.surfaces.filter(name=\"Therys Fm. Top\").tagnames\n",
+    "\n",
+    "# Filter on a specific name and tagname\n",
+    "myensemble.surfaces.filter(name=\"Therys Fm. Top\", tagname=\"DS_extract_geogrid\")\n",
+    "\n",
+    "# Filter on a specific name, tagname and realization\n",
+    "myensemble.surfaces.filter(name=\"Therys Fm. Top\", tagname=\"DS_extract_geogrid\", realization=0)\n",
+    "\n",
+    "# In the Drogon example, this has now filtered down to exactly 1 surface object.\n",
+    "# This may not be the situation for your data, and more filters might be required."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we go from \"finding data\" to \"using data\". For the Surface example, we recommend using XTgeo:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mysurfs = myensemble.surfaces.filter(name=\"Therys Fm. Top\", tagname=\"DS_extract_geogrid\", realization=0)\n",
+    "\n",
+    "if len(mysurfs) != 1:\n",
+    "    print(f\"Warning! The collection has {len(mysurfs)} surfaces, which is not exactly 1 surface object.\")\n",
+    "\n",
+    "mysurf = mysurfs[0].to_regular_surface() # `mysurf` is now a RegularSurface object\n",
+    "\n",
+    "mysurf.quickplot(title=\"A surface!\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Statistical aggregations\n",
+    "A key feature of FMU is it's ability to represent uncertainty, by realizing the same result many times with different parameters. As opposed to classical model workflows that create a single, atomic instance of each result - FMU workflows produce results in the form of distributions.\n",
+    "\n",
+    "To analyze such distributions, it is frequently useful to create statistical aggregations:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Perform statistical aggregation on SurfaceCollection\n",
+    "\n",
+    "surfs = myensemble.surfaces.filter(name=\"Therys Fm. Top\", tagname=\"DS_extract_geogrid\", realization=True)\n",
+    "\n",
+    "print(f\"There are {len(myensemble.realizations)} realizations, and the Search Context has {len(surfs)} individual surface objects.\")\n",
+    "\n",
+    "mean = surfs.aggregation(operation=\"mean\")  # operations: max, mean, std, p10, p90, p50\n",
+    "mean.to_regular_surface().quickplot(title=\"Mean surface!\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Through the <a href=\"https://fmu-sumo.app.radix.equinor.com/\">Sumo web interface</a>, you can also call bulk aggregation on all surfaces in your ensemble. When aggregated surfaces are made, you can also access them directly with a filter, as shown below:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mymean = myensemble.surfaces.filter(name=\"Therys Fm. Top\", tagname=\"DS_extract_geogrid\", aggregation=\"mean\")\n",
+    "mymean[0].to_regular_surface().quickplot(title=\"Still the mean\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Time filtration\n",
+    "The `TimeFilter` class can be used to construct a time filter which can be passed to the `SurfaceCollection.filter` method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from fmu.sumo.explorer import TimeFilter, TimeType\n",
+    "\n",
+    "# get surfaces with timestamps\n",
+    "time = TimeFilter(time_type=TimeType.TIMESTAMP)\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "print(\"Timestamp:\", len(surfs))\n",
+    "\n",
+    "# get surfaces with time intervals\n",
+    "time = TimeFilter(time_type=TimeType.INTERVAL)\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "print(\"Interval:\", len(surfs))\n",
+    "\n",
+    "\n",
+    "# get surfaces with time data (timestamp or interval)\n",
+    "time = TimeFilter(time_type=TimeType.ALL)\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "print(\"Time data:\", len(surfs))\n",
+    "\n",
+    "\n",
+    "# get surfaces without time data\n",
+    "time = TimeFilter(time_type=TimeType.NONE)\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "print(\"No time data:\", len(surfs))\n",
+    "\n",
+    "\n",
+    "# get avaiable timestamps\n",
+    "timestamps = myensemble.surfaces.timestamps\n",
+    "print(\"\\nTimestamps:\", timestamps)\n",
+    "\n",
+    "# get available intervals\n",
+    "intervals = myensemble.surfaces.intervals\n",
+    "print(\"Intervals:\", intervals)\n",
+    "\n",
+    "\n",
+    "# get surfaces with timestamp in range\n",
+    "time = TimeFilter(\n",
+    "    time_type=TimeType.TIMESTAMP, start=\"2018-01-01\", end=\"2022-01-01\"\n",
+    ")\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "\n",
+    "# get surfaces with time intervals in range\n",
+    "time = TimeFilter(\n",
+    "    time_type=TimeType.INTERVAL, start=\"2018-01-01\", end=\"2022-01-01\"\n",
+    ")\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "\n",
+    "# get surfaces where intervals overlap with range\n",
+    "time = TimeFilter(\n",
+    "    time_type=TimeType.INTERVAL,\n",
+    "    start=\"2018-01-01\",\n",
+    "    end=\"2022-01-01\",\n",
+    "    overlap=True,\n",
+    ")\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "\n",
+    "# get surfaces with exact timestamp matching (t0 == start)\n",
+    "time = TimeFilter(time_type=TimeType.TIMESTAMP, start=\"2018-01-01\", exact=True)\n",
+    "surfs = myensemble.surfaces.filter(time=time)\n",
+    "\n",
+    "# get surfaces with exact interval matching (t0 == start AND t1 == end)\n",
+    "time = TimeFilter(\n",
+    "    time_type=TimeType.INTERVAL,\n",
+    "    start=\"2018-01-01\",\n",
+    "    end=\"2022-01-01\",\n",
+    "    exact=True,\n",
+    ")\n",
+    "surfs = myensemble.surfaces.filter(time=time)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tables\n",
+    "FMU produces results across almost all types of data, including a significant amount of tables. While most tables are relatively small, some tables - such as the `UNSMRY` - can be very large. To deal with these large tables, Sumo is doing some data transformation. In short, we flip the data from being realization-oriented, to being vector-oriented.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    Refer to the <a href=\"https://doc-sumo-doc-prod.radix.equinor.com/\">Sumo documentation</a> for more information about data transformation of tables.</div>\n",
+    "\n",
+    "In this example, we will use the same ensemble as before. We will first find a summary file for a specific realization, and cast that to a Pandas dataframe.\n",
+    "\n",
+    "### Getting a single realization of a table\n",
+    "Example: Inplace volumes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mysinglevolumes = myensemble.tables.filter(tagname=\"vol\", name=\"geogrid\", realization=0)\n",
+    "if len(mysinglevolumes) != 1:\n",
+    "    raise ValueError(f\"Got {len(mysinglevolumes)} which is not exactly 1.\")\n",
+    "print(\"Inplace volumes (geogrid) for realization 0:\")\n",
+    "df = mysinglevolumes[0].to_pandas()\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Example: UNSMRY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mysinglesummary = myensemble.tables.filter(tagname=\"summary\", realization=0)\n",
+    "if len(mysinglesummary) != 1:\n",
+    "    raise ValueError(f\"Got {len(mysinglesummary)} which is not exactly 1.\")\n",
+    "print(\"Summary for realization 0:\")\n",
+    "df = mysinglesummary[0].to_pandas()\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Commonly, however, we don't want the Summary for a single realization but rather a specific set of columns for the ensemble. It might be tempting to start looping through all realizations, but this is not recommended! Rather, you can call for some data transformation (we call this \"aggregation\" as well) to provide easy access to single columns across an ensemble of large tables:\n",
+    "\n",
+    "### Getting a column across many realizations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = myensemble.tables.filter(tagname=\"summary\").aggregation(operation=\"collection\", column=\"FOPT\").to_pandas()\n",
+    "print(\"FOPT across all realizations\")\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that in addition to the \"FOPT\" column, you also get the **DATE** and **REAL** columns as categorical columns.\n",
+    "\n",
+    "Data transformations of large tables can be tedious. However, when data has been transformed, they are also stored in Sumo and you can access them quicker:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "myensemble.tables.filter(tagname=\"summary\", aggregation=\"collection\", column=\"FOPT\")[0].to_pandas()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "886a3bf2c68989186a10fcef5692724f98e2557aca959586571e913a0aec80e7"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/pyproject.toml

@@ -55,7 +55,7 @@ platforms = ["any"]
 where = ["src"]
 
 [tool.ruff]
-exclude = [".env", ".git", ".github", ".venv", "venv"]
+exclude = [".env", ".git", ".github", ".venv", "venv", "examples"]
 
 line-length = 79
 

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/src/fmu/sumo/explorer/_version.py

@@ -1,7 +1,14 @@
 # file generated by setuptools-scm
 # don't change, don't track in version control
 
-__all__ = ["__version__", "__version_tuple__", "version", "version_tuple"]
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
 
 TYPE_CHECKING = False
 if TYPE_CHECKING:
@@ -9,13 +16,19 @@ if TYPE_CHECKING:
     from typing import Union
 
     VERSION_TUPLE = Tuple[Union[int, str], ...]
+    COMMIT_ID = Union[str, None]
 else:
     VERSION_TUPLE = object
+    COMMIT_ID = object
 
 version: str
 __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
+commit_id: COMMIT_ID
+__commit_id__: COMMIT_ID
 
-__version__ = version = '2.4.10'
-__version_tuple__ = version_tuple = (2, 4, 10)
+__version__ = version = '2.6.0'
+__version_tuple__ = version_tuple = (2, 6, 0)
+
+__commit_id__ = commit_id = 'g7c93d8bdc'

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/src/fmu/sumo/explorer/objects/_child.py

@@ -21,7 +21,7 @@ class Child(Document):
         self._sumo = sumo
         self._blob = blob
 
-    def __repr__(self):
+    def __str__(self):
         if self.stage == "case" and self.__class__.__name__ != "Case":
             return (
                 f"<{self.__class__.__name__}: {self.name} {self.uuid}(uuid) "
@@ -46,7 +46,10 @@ class Child(Document):
                     f"in asset {self.asset}>"
                 )
             else:
-                return super().__repr__()
+                return super().__str__()
+
+    def __repr__(self):
+        return self.__str__()
 
     @property
     def blob(self) -> BytesIO:

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/src/fmu/sumo/explorer/objects/_document.py

@@ -1,6 +1,5 @@
 """Contains class for one document"""
 
-import json
 import re
 from typing import Any, Dict, List, Union
 
@@ -20,14 +19,14 @@ class Document:
         self._metadata = metadata["_source"]
 
     def __str__(self):
-        return f"{json.dumps(self.metadata, indent=4)}"
-
-    def __repr__(self):
         return (
             f"<{self.__class__.__name__}: {self.name} {self.uuid}(uuid) "
             f"in asset {self.asset}>"
         )
 
+    def __repr__(self):
+        return self.__str__()
+
     @property
     def uuid(self):
         """Return uuid

{fmu_sumo-2.4.10 → fmu_sumo-2.6.0}/src/fmu/sumo/explorer/objects/_metrics.py

@@ -272,3 +272,56 @@ class Metrics:
                 "percentiles", field=field, percents=percents
             )
         )["values"]
+
+    def _fnv1a_script(self, field):
+        return {
+            "init_script": """
+                state.h = state.count = state.total = 0L;
+            """,
+            "map_script": f"""
+                state.total++;
+                if (doc['{field}'].size() == 0) return;
+                def s = doc.get('{field}').value;
+                long h = -3750763034362895579L;
+                for (int i = 0; i < s.length(); i++) {{
+                    h ^= (long) s.charAt(i);
+                    h *= 1099511628211L;
+                }}
+                state.h ^= h;
+                state.count++;
+            """,
+            "combine_script": """
+                return state;
+            """,
+            "reduce_script": """
+                long h = 0, c = 0, t = 0;
+                for (st in states) {
+                    h ^= st.h; c += st.count; t += st.total
+                }
+                return ['checksum': Long.toHexString(h), 'docs_in_checksum': c, 'docs_total': t];
+            """,
+        }
+
+    def fnv1a(self, field):
+        """Compute the 64-bit FNV-1a checksum for field over the current set of objects.
+
+        Arguments:
+            - field (str): the name of a property in the metadata.
+
+        Returns:
+            - a dict with the keys "docs_all", "docs_seen" and "xor_fnv64_hex".
+        """
+        return self._aggregate("scripted_metric", **self._fnv1a_script(field))
+
+    async def fnv1a_async(self, field):
+        """Compute the 64-bit FNV-1a checksum for field over the current set of objects.
+
+        Arguments:
+            - field (str): the name of a property in the metadata.
+
+        Returns:
+            - a dict with the keys "docs_all", "docs_seen" and "xor_fnv64_hex".
+        """
+        return await self._aggregate_async(
+            "scripted_metric", **self._fnv1a_script(field)
+        )