sdf-xarray 0.8.0__tar.gz → 0.10.0__tar.gz

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/.github/workflows/build_publish.yml

@@ -20,7 +20,7 @@ jobs:
         submodules: "recursive"
 
     - name: Cache Zenodo datasets
-      uses: actions/cache@v4
+      uses: actions/cache@v6
       with:
         enableCrossOsArchive: true
         path: ~/.cache/sdf_datasets

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/.github/workflows/tests.yml

@@ -31,7 +31,7 @@ jobs:
         uv sync --python ${{ matrix.python-version }} --frozen
 
     - name: Cache Zenodo datasets
-      uses: actions/cache@v4
+      uses: actions/cache@v6
       with:
         path: ~/.cache/sdf_datasets
         key: sdf-datasets-17991042

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sdf-xarray
-Version: 0.8.0
+Version: 0.10.0
 Summary: Provides a backend for xarray to read SDF files as created by the EPOCH plasma PIC code.
 Author-Email: Peter Hill <peter.hill@york.ac.uk>, Joel Adams <joel.adams@york.ac.uk>, Shaun Doherty <shaun.doherty@york.ac.uk>, Chris Herdman <chris.herdman@york.ac.uk>, Liam Pattinson <liam.pattinson@york.ac.uk>, Sviatoslav Shekhanov <sviatoslav.shekhanov@york.ac.uk>
 License-Expression: BSD-3-Clause

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/docs/animation.md

@@ -12,7 +12,6 @@ mimic <inv:#xarray.DataArray.plot>.
 
 ```{code-cell} ipython3
 import sdf_xarray as sdfxr
-import xarray as xr
 import matplotlib.pyplot as plt
 from matplotlib.animation import FuncAnimation
 from IPython.display import HTML
@@ -35,9 +34,10 @@ resolved data has 2 dimensions.
 
 | Dimensions |      Plotting function        |        Notes          |
 | ---------- | ----------------------------- | --------------------- |
-|    `2`     | <inv:#xarray.plot.line>       |                       |
-|    `3`     | <inv:#xarray.plot.pcolormesh> |                       |
-|    `>3`    | <inv:#xarray.plot.hist>       | Not fully implemented |
+|     2      | <inv:#xarray.plot.line>       |                       |
+|     3      | <inv:#xarray.plot.pcolormesh> |                       |
+|     4      | <inv:#mpl_toolkits.mplot3d.axes3d.Axes3D.voxels>| WIP |
+|     >4     | <inv:#xarray.plot.hist>       | Not fully implemented |
 
 
 ### 1D simulation
@@ -103,6 +103,10 @@ anim.show()
 
 ### 3D simulation
 
+```{warning}
+[Voxel](./plotting.md#voxel-plots) animations are not currently implemented. 
+```
+
 Opening a 3D simulation as a multi-file dataset and plotting it will
 return a <inv:#xarray.plot.hist>. However, this may not be
 desirable. We can plot a 3D simulation along a certain plane in the
@@ -134,31 +138,23 @@ EPOCH allows for simulations that have a moving simulation window
 `move_window` and change the x-axis limits accordingly.
 
 ```{warning}
-<project:#sdf_xarray.open_mfdataset> does not currently function with moving window data.
-You must use <inv:#xarray.open_mfdataset> and specify arguments in the following way.
+Importing some datasets with moving windows can cause vertical banding
+in the <inv:#xarray.Dataset>, which will affect the animation. Each output stores
+its own x grid, and the same position can differ by floating point rounding between
+outputs. Both values are then kept on the combined x axis, so each time step has
+nan values in some columns. Setting `join = "override"` in
+<inv:#xarray.open_mfdataset> does not fix this for moving window data, as it
+relabels every time step with the first output's x grid.
 ```
 
 ```{code-cell} ipython3
-ds = xr.open_mfdataset(
-   "tutorial_dataset_2d_moving_window/*.sdf",
-   preprocess = sdfxr.SDFPreprocess(),
-   combine = "nested",
-   join = "outer",
-   compat="no_conflicts",
-   concat_dim="time",
-   )
+ds = sdfxr.open_mfdataset("tutorial_dataset_2d_moving_window/*.sdf")
 
 da = ds["Derived_Number_Density_Beam_Electrons"]
 anim = da.epoch.animate(move_window = True, fps = 5)
 anim.show()
 ```
 
-```{warning}
-Importing some datasets with moving windows can cause vertical banding
-in the <inv:#xarray.Dataset>, which will affect the animation. The cause for
-this is unknown but can be circumvented by setting `join = "override"`.
-```
-
 ## Customisation
 
 The animation can be customised in much the same way as <inv:#xarray.DataArray.plot>,

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/docs/index.md

@@ -21,6 +21,7 @@ why_sdf_xarray.md
 loading_data.md
 understanding_datasets.md
 unit_conversion.md
+plotting.md
 animation.md
 ```
 

sdf_xarray-0.10.0/docs/known_issues.md

@@ -0,0 +1,8 @@
+# Known Issues
+
+These are the known 'quirks' in `sdf_xarray`:
+
+- [Issue #112](https://github.com/epochpic/sdf-xarray/issues/112) Due to the 
+way [cylindrical EPOCH](https://github.com/epochpic/cylindrical_epoch) sets up 
+grids we can't currently load it's SDF files. If there is enough interest in this
+we will attempt to find a solution otherwise it shall remain low priority.

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/docs/loading_data.md

@@ -85,7 +85,11 @@ is by using the <project:#sdf_xarray.open_mfdataset>.
 ```{tip}
 If your simulation includes multiple `output` blocks that specify different variables
 for output at various time steps, variables not present at a specific step will default
-to a nan value. To remove these nan values we suggest using the <inv:#xarray.DataArray.dropna>
+to a nan value. This includes distribution functions and particle data that are only
+output at some time steps. Grids that change between outputs (such as a moving window,
+or a distribution function with an automatic range) are combined onto the union of
+every output's grid points, with nan values outside the part of the grid each output
+covers. To remove these nan values we suggest using the <inv:#xarray.DataArray.dropna>
 function or following our implmentation in [](#loading-sparse-data).
 ```
 
@@ -98,13 +102,20 @@ sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf")
 Alternatively, files can be loaded using <inv:#xarray.open_mfdataset>. However, when loading in
 all the files we have do some processing of the data so that we can correctly align it along
 the time dimension; This is done via the `preprocess` parameter utilising the
-<project:#sdf_xarray.SDFPreprocess> function.
+<project:#sdf_xarray.SDFPreprocess> function. Passing `combine="nested"` and
+`concat_dim="time"` makes sure the files are only joined along time. Otherwise
+<inv:#xarray> treats any coordinate that differs between files (such as a moving
+window's x grid) as another dimension to concatenate along, which can fail or request
+far more memory than needed. Unlike <project:#sdf_xarray.open_mfdataset>, this does
+not handle particle positions that are missing from some files.
 
 ```{code-cell} ipython3
 import xarray as xr
 
 xr.open_mfdataset(
    "tutorial_dataset_1d/*.sdf",
+   combine="nested",
+   concat_dim="time",
    join="outer",
    compat="no_conflicts",
    preprocess=sdfxr.SDFPreprocess()
@@ -123,6 +134,9 @@ sdfxr.open_mfdatatree("tutorial_dataset_1d/*.sdf")
 
 ## Loading sparse data
 
+By default, <project:#sdf_xarray.open_mfdataset> puts every variable on a single
+time dimension, and a variable missing from an output (including distribution
+functions and particle data) is filled with nan values at that time step.
 When dealing with sparse data (where different variables are saved at different,
 non-overlapping time steps) you can optimize memory usage by loading the data with
 <project:#sdf_xarray.open_mfdataset> using the parameter `separate_times=True`. This
@@ -140,12 +154,17 @@ sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf", separate_times=True)
 ## Loading particle data
 
 ```{warning}
-It is **not recommended** to use <inv:#xarray.open_mfdataset> or
-<project:#sdf_xarray.open_mfdataset> to load particle data from multiple
-SDF outputs. The number of particles often varies between outputs,
-which can lead to inconsistent array shapes that these functions
-cannot handle. Instead, consider loading each file individually and
-then concatenating them manually.
+<project:#sdf_xarray.open_mfdataset> can load particle data from multiple
+SDF outputs, and outputs that don't include particles are filled with nan
+values. However, every output that does include a species must contain the
+same number of particles of that species. The number of particles often
+varies between outputs (for example with particle injection, losses or probes),
+in which case loading fails with a conflicting dimension sizes error, and you
+should load each file individually instead.
+
+The `ID_<species>` dimension is the position in each output's particle array,
+not a particle identity, so the same index at two time steps is not
+necessarily the same particle.
 ```
 
 ```{note}
@@ -157,7 +176,7 @@ Failing to do so will result in dimension name conflicts.
 
 By default, particle data isn't kept as it takes up a lot of space.
 Pass `keep_particles=True` as a keyword argument to
-<inv:#xarray.open_dataset> (for single files) or <inv:#xarray.open_mfdataset> (for
+<project:#sdf_xarray.open_dataset> (for single files) or <project:#sdf_xarray.open_mfdataset> (for
 multiple files).
 
 ```{code-cell} ipython3
@@ -168,14 +187,10 @@ sdfxr.open_dataset("tutorial_dataset_1d/0010.sdf", keep_particles=True)
 
 ## Loading specific variables
 
-When loading datasets containing several (`>10`) coordinates/dimensions
-using <project:#sdf_xarray.open_mfdataset>, <inv:#xarray> may struggle to locate
-the necessary RAM to concatenate all of the data (as seen in
-[Issue #57](https://github.com/epochpic/sdf-xarray/issues/57)).
-In this instance, you can optimize memory usage by loading only the data
-you need using the keyword argument `data_vars` and passing one or more
-variables. This creates a dataset consisting only of the given variable(s)
-and the relevant coordinates/dimensions, significantly reducing memory
+If you only need some of the variables, you can load just those using the
+keyword argument `data_vars` and passing one or more variables to
+<project:#sdf_xarray.open_mfdataset>. This creates a dataset consisting only of
+the given variable(s) and the relevant coordinates/dimensions, reducing memory
 consumption.
 
 ```{code-cell} ipython3

sdf_xarray-0.10.0/docs/plotting.md

@@ -0,0 +1,95 @@
+---
+file_format: mystnb
+kernelspec:
+  name: python3
+---
+
+# Plotting
+
+You can plot datasets using
+[`xarray.DataArray.epoch.plot`](project:#sdf_xarray.dataarray_accessor.EpochAccessor.plot).
+This is a custom <project:#sdf_xarray> plotting routine that builds on top of
+<inv:#xarray.DataArray.plot>, so you keep the familiar <inv:#xarray> plotting
+behaviour while using <project:#sdf_xarray> conveniences (see 
+[here](project:#sdf_xarray.dataarray_accessor.EpochAccessor.plot) for details).
+Under the hood, plotting is still handled by <inv:#matplotlib>, which means you
+can use the full <inv:#matplotlib> API to customise your figure.
+
+| Dimensions | Base plotting function        |        Notes          |
+| ---------- | ----------------------------- | --------------------- |
+|     1      | <inv:#xarray.plot.line>       |                       |
+|     2      | <inv:#xarray.plot.pcolormesh> |                       |
+|     3      | <inv:#mpl_toolkits.mplot3d.axes3d.Axes3D.voxels>| Custom functionality built for <project:#sdf_xarray> |
+|     >3     | <inv:#xarray.plot.hist>       | Not fully implemented |
+
+## Line plots
+
+One dimensional data will be plotted as a line. Multiple lines can be easily
+plotted at the same time.
+
+```{code-cell} ipython3
+import sdf_xarray as sdfxr
+import matplotlib.pyplot as plt
+ds = sdfxr.open_dataset("tutorial_dataset_1d/0010.sdf")
+da_1 = ds["Derived_Number_Density_Electron"]
+da_2 = ds["Derived_Number_Density_Ion"]
+da_1.epoch.plot(label = "Electron")
+da_2.epoch.plot(label = "Ion")
+plt.legend()
+plt.show()
+```
+
+## Mesh plots
+
+Two dimensional data will be plotted as a mesh.
+
+```{code-cell} ipython3
+ds = sdfxr.open_dataset("tutorial_dataset_2d/0010.sdf")
+da = ds["Derived_Number_Density_Electron"]
+da.epoch.plot()
+plt.show()
+```
+
+## Voxel plots
+
+Three dimensional data will be plotted using voxels. This behaviour is not native
+to <inv:#xarray>, it has been custom built specifically for this package.
+
+```python
+ds = sdfxr.open_dataset("tutorial_dataset_3d/0000.sdf")
+da = ds["Derived_Number_Density"]
+da.epoch.plot(vmin = 1e27)
+plt.show()
+```
+
+![voxel_plot_high_res](./figures/voxel_plot_high_res.png)
+
+```{warning}
+Voxel plots can be extremely computationally expensive and may take longer than
+expected to plot.
+```
+
+Because voxel plots are very expensive, even with relatively small data arrays,
+plotting can be sped up by using [`xarray.DataArray.epoch.resize`](project:#sdf_xarray.dataarray_accessor.EpochAccessor.resize),
+which uses interpolation to reduce the resolution.
+
+```{code-cell} ipython3
+ds = sdfxr.open_dataset("tutorial_dataset_3d/0005.sdf")
+da = ds["Derived_Number_Density"]
+da_resized = da.epoch.resize((20, 20, 20))
+da_resized.epoch.plot(vmin = 1e27)
+plt.show()
+```
+
+## Histograms
+
+When the data array has four or more dimensions (or is specified by the user), it will
+be plotted as a histogram.
+
+```{code-cell} ipython3
+ds = sdfxr.open_dataset("tutorial_dataset_3d/0005.sdf")
+da = ds["Derived_Number_Density"]
+
+da.epoch.plot(hist = True)
+plt.show()
+```

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/docs/understanding_datasets.md

@@ -29,23 +29,10 @@ ds = sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf")
 ds["Electric_Field_Ex"]
 ```
 
-## Plotting
+## Visualisation
 
-You can plot datasets using
-[`xarray.DataArray.epoch.plot`](project:#sdf_xarray.dataarray_accessor.EpochAccessor.plot).
-This is a custom <project:#sdf_xarray> plotting routine that builds on top of
-<inv:#xarray.DataArray.plot>, so you keep the familiar <inv:#xarray> plotting
-behaviour while using <project:#sdf_xarray> conveniences (see 
-[here](project:#sdf_xarray.dataarray_accessor.EpochAccessor.plot) for details).
-Under the hood, plotting is still handled by <inv:#matplotlib>, which means you
-can use the full <inv:#matplotlib> API to customise your figure.
-
-```{code-cell} ipython3
-# This is discretized in both space and time
-ds["Electric_Field_Ex"].epoch.plot()
-plt.title("Electric field along the x-axis")
-plt.show()
-```
+`sdf-xarray` has built-in plotting and animation functionality, more details can
+be found on [](./plotting.md) and [](./animation.md).
 
 ## Dimension slicing
 
@@ -56,6 +43,7 @@ for easy indexing. To quickly determine the number of time steps available,
 you can check the size of the time dimension.
 
 ```{code-cell} ipython3
+ds = sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf")
 # This corresponds to the number of individual SDF files loaded
 print(f"There are a total of {ds['time'].size} time steps")
 
@@ -66,7 +54,7 @@ print(f"The time at the 20th simulation step is {sim_time:.2e} s")
 
 You can select and extract a single simulation snapshot using the integer
 index of the time step with the <inv:#xarray.Dataset.isel> function. This can be
-done by passsing the index to the `time` parameter (e.g., `time=0` for
+done by passing the index to the `time` parameter (e.g., `time=0` for
 the first snapshot).
 
 ```{code-cell} ipython3
@@ -74,7 +62,7 @@ ds["Electric_Field_Ex"].isel(time=20)
 ```
 
 We can also use the <inv:#xarray.Dataset.sel> function if you wish to pass a
-value intead of an index.
+value instead of an index.
 
 ```{tip}
 If you know roughly what time you wish to select but not the exact value
@@ -122,6 +110,51 @@ print(f"Total particle energy absorbed: {ds["Total_Particle_Energy_in_Simulation
 print(f"The laser absorption fraction: {ds["Laser_Absorption_Fraction_in_Simulation"][-1].values:.1f} %")
 ```
 
+## Limit
+
+The `.epoch` accessor has the ability to "limit" data arrays. This is a helper
+function which applies <inv:#xarray.DataArray.where> to the upper and lower bounds
+of each dimension. You can use `None` to preserve the existing limit.
+
+```{code-cell} ipython3
+ds = sdfxr.open_dataset("tutorial_dataset_2d/0020.sdf")
+da = ds["Derived_Number_Density_Electron"]
+da.epoch.plot()
+print(f"Orignal shape: {da.shape}")
+plt.show()
+```
+
+```{code-cell} ipython3
+da_limit = da.epoch.limit(((-2e-6, 2e-6), (None, 0)))
+da_limit.epoch.plot()
+print(f"New shape: {da_limit.shape}")
+plt.show()
+```
+
+## Resize
+
+The data arrays can be interpolated with <project:#sdf_xarray.dataarray_accessor.EpochAccessor.resize>
+to either increase or decrease the resolution while mantaining the general structure.
+Decreasing the size may be useful to download large data from HPCs to local machines
+or ploting with [](./plotting.md#voxel-plots). Be aware that the original
+data will not be preserved.
+
+```{code-cell} ipython3
+ds = sdfxr.open_dataset("tutorial_dataset_2d/0020.sdf")
+da = ds["Derived_Number_Density_Electron"]
+
+da.epoch.plot()
+print(f"Orignal shape: {da.shape}")
+plt.show()
+```
+
+```{code-cell} ipython3
+da_resized = da.epoch.resize((50, 50))
+da_resized.epoch.plot()
+print(f"New shape: {da_resized.shape}")
+plt.show()
+```
+
 ## Visualisation on HPC Machines
 
 In many cases you will be running EPOCH simulations via a HPC cluster and your

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/docs/why_sdf_xarray.md

@@ -7,7 +7,7 @@ There are several benefits to using this package over the [`sdf_helper`](https:/
 - [data to numpy and pandas](https://docs.xarray.dev/en/stable/user-guide/pandas.html) - Data can be easily converted to [numpy](https://www.numpy.org/) or [pandas](https://pandas.pydata.org/) if you prefer to work with raw data.
 - [data labelling](project:understanding_datasets.md) - All SDF files come with a bunch of information about them such as the `dimensions`, `units` and `name` of the variable. In <inv:#xarray> these are added to each variable making it much easier to read.
 - [lazy/partial loading](https://docs.xarray.dev/en/stable/internals/internal-design.html#lazy-loading) - This reduces the RAM requirements when loading large SDF files as we only load the actual array values when they are needed.
-- [plotting](project:understanding_datasets.md#plotting) - You can easily plot variables across several dimensions including time using built in <inv:#matplotlib> support.
+- [plotting](project:plotting.md) - You can easily plot variables across several dimensions including time using built in <inv:#matplotlib> support.
 - [animating](project:animation.md) - Visualise your data across time or slice through another dimension and make beautiful GIFs for your presentations.
 - [deck loading](loading-input-deck) - <project:#sdf_xarray> automatically loads in the associated `input.deck` utilised when creating the simulation so that you can access setup variables.
 - [easier unit conversion](project:unit_conversion.md) - Convert dimensions and variable units to other types with the help of [`pint`](https://pint.readthedocs.io/en/stable).

{sdf_xarray-0.8.0 → sdf_xarray-0.10.0}/pyproject.toml

@@ -69,6 +69,7 @@ docs = [
   "sphinx-design>=0.7.0",
   "pooch>=1.8.2",
   "tqdm",
+  "scipy",
 ]
 build = ["cibuildwheel[uv]"]
 lint = ["ruff"]
@@ -80,6 +81,7 @@ test = [
   "tqdm",
   "pint",
   "pint-xarray",
+  "scipy",
 ]
 
 [project.entry-points."xarray.backends"]
@@ -133,6 +135,10 @@ ignore = [
   "PLR2004", # magic-comparison
   "B9",      # flake8-bugbear opinionated warnings
   "PLR0913", # remove maximum number of arguments in a function
+  "PLR0917", # remove maximum number of positional arguments in a function
+  "BLE001",  # allow capturing of base exception
 ]
-exclude = ["**/_version.py"]
-# Auto-generated file
+exclude = ["**/_version.py"] # Auto-generated file
+
+[tool.pytest.ini_options]
+filterwarnings = ['ignore::UserWarning']