xobjects 0.4.4__tar.gz → 0.4.6__tar.gz

xobjects-0.4.6/.github/workflows/black.yml

@@ -0,0 +1,10 @@
+name: Lint
+
+on: [push, pull_request]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: psf/black@stable

xobjects-0.4.6/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.pyc
+*.pyo
+*.swp
+*.egg-info/
+docs/_build
+cov_html
+.vscode
+.pytest_cache
+.coverage
+*.c

xobjects-0.4.6/.pre-commit-config.yaml

@@ -0,0 +1,12 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: trailing-whitespace
+  - repo: https://github.com/psf/black
+    rev: 23.10.1
+    hooks:
+      - id: black
+        language_version: python3 # Should be a command that runs python3.6+

xobjects-0.4.6/.readthedocs.yaml

@@ -0,0 +1,20 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF
+formats:
+   - pdf
+
+# Optionally set the version of Python and requirements required to build your docs
+python:
+   version: 3.7
+#   install:
+#   - requirements: docs/requirements.txt

xobjects-0.4.6/Architecture.md

@@ -0,0 +1,202 @@
+# Architecture
+
+## Introduction
+
+The library is based on:
+- Buffers: manage chunck of bytes on CPU and GPU (Cuda and OpenCL) memory using different backends
+- Contexts: keep track of buffers, launch functions operating on buffers
+- Basic types: a set of basic types that allow to describe structured data. Basic types are scalars, strings, structs, arrays and unions. User defined classes, composed with basic types, allow to interact with data on buffers and generate C functions to operate with data.
+- C-API generation: The type information allows to write C functions that can be interact with data. The C functions are specialized for CPU, OpenCL and Cuda. Also loops described by annotation in comments are automatically generated for CPU, OpenCL and Cuda.
+
+When a user defined class is created, Python methods and additional metadeta is attached to the class.
+When an instance of a user defined class is instantiated, memory is written to a buffer and a Python object is created to interact with the data.
+The size of an instance may not be known at class defintion time, but it cannot be changed after initialization.
+
+## Contexts and Buffers
+
+A context contain:
+
+- `new_buffer(capacity)`: create new buffer of initial capacity
+- `buffers`: list of weakref to buffers
+-  kernels : Object containing compiled code from sources/
+
+Buffer:
+- `buffer`: low level buffer
+- `allocate(self,size)`: return the offset of the first free memory chunk of given size and grows the buffer if necessary
+- `free(self,offset,size)`: free a chuck of memory of a given size
+- `grow(self,capacity)`: increase the size of a buffer
+- `_new_buffer(capacity)`: creae lowlevel buffer
+- `update_from_native(...)`: Copy data from native buffer into self.buffer starting from offset
+- `update_from_buffer(...)`: Copy data from python buffer such as bytearray, bytes, memoryview, numpy array.data
+- `update_from_nplike(...)`: Copy data from nplike matching dest_dtype
+- `update_from_xbuffer(...)`: Copy data from any xbuffer, don't pass through cpu if possible
+- `copy_to_native(...)`: Copy data from self.buffer into dest
+- `to_native(...)`: Return native data with content at from offset and nbytes
+- `to_nplike(...)`: Return a view in of an nplike
+- `to_bytearray(...)`: Return a byte array: used in update_from_xbuffer
+- `to_pointer_arg(...)`: Return data that can be used as argument in kernel
+
+## Types
+
+Types can be composed of:
+- scalar: numbers, String
+- compound: Struct, Array, Ref, UnionRef 
+
+### Scalars
+- examples: Float64, Int64, ...
+- create: Float64(3.14)
+- memory layout
+    - data 
+
+### String:
+- create: String(string_or_int)
+- memory layout
+    - size
+    - data 
+
+
+### Struct
+- example: struct MyStruct field1 field2 ...
+- create: Struct(dict), Struct(**args)
+- memory layout:
+  - [ instance size ]
+  - static-field1
+  -  ..
+  - static-fieldn
+  - [ offset field 2 ]
+  - [ offset ...     ]
+  - [ offset field n ]
+  - [ dynamic-field1 ]
+  - [ ... ]
+  - [ dynamic-fieldn ]
+
+### Array:
+- example: 'array' 'f64' d1 d2 d3 'C' ; 'array' 'i64' d1 ':' d3 'F' ;
+- where:
+   - 'C'/'F' is C-contiguos or F-contiguous
+   - d1, d2, .. are the dimensions for each axis and ':' means not specified
+- create: Array(d1,d2,...) or Array([...]) or Array(np-array)
+- memory layout:
+  - [size] if not _is_static_shape or not _is_static_type
+  - [dims ... ] len(_dynamic_shape)
+  - [strides...] if nd>1 and dynamic shapes
+  - [offsets] if not _is_static_type
+  - data
+
+### Union:
+- create: Union(obj), Union( (typename,args) )
+- memory layout
+  - typeid
+  - data
+
+### UnionRef:
+- create: UnionRef(obj), UnionRef( (typename,args) )
+- memory layout
+  - typeid
+  - offset
+
+
+### Ref:
+- memory layout
+  - offset
+
+## Implementation details
+
+Assumptions:
+- Python user API cannot change the internal structure of the object but only the content
+- Python object caches the structure information to avoid round trips to access data
+- C user API does not manipulate structures
+
+
+Operation:
+
+- _inspect_args(*args, **args) -> info: Info
+  - check value and calculate size, offsets and dimensions from arguments, recursively
+  - returns at least `size` for `_get_a_buffer` and `value` for `_to_buffer`
+
+- _get_a_buffer(size, _context, _buffer, _offset):
+  - make sure a valid buffer and offset is created
+
+- _to_buffer(buffer, offset, value, info, size?):
+  - set data on buffer with offset from pyhon value, using info and implicitely respecting size
+  - if value same class -> binary copy (passing from context if needed)
+
+- _from_buffer(buffer, offset) -> value:
+  - create python object from data on buffer. Can be a native type (scalar or string) or another XObject (struct, array)
+
+
+- __init__(self,*args,_context=None, _buffer=None, _offset=None,*nargs)
+  - initialize object data on buffer from args and create python object:
+    - use _pre_init(*args,**nargs) -> return args, nargs standard initialization value
+    - check values and calculate size using `_inspect_args`, return info object containing size, offsets and values to be later used by _to_buffer
+    - allocate using `_get_a_buffer`
+    - write data using `_to_buffer`
+    - use `_post_init` to complete object
+`
+- _update(self, value, size=None):
+  - Optional update object content using value, in case respecting size for string
+
+- __get__(field,instance) or __getitem__(self,index...)
+  - if return instance._cache[field.index] else  #should implement item caching for struct and array
+  - else return _from_buffer
+
+- __set__(..., value) or __setitem__(...,value)
+  - if hasattr(self.ftype._update) get object and update in place
+    else call _to_buffer
+
+- json enconding
+  nested object can be initialized using a json enconding
+  d:dict -> type(**d)
+  t:tuple -> type(*t)
+  a:other -> type(a)
+
+
+
+Class content:
+
+- _inspect_args(cls, *args, **nargs) -> size, offsets
+  compute size and sizes and offsets from arguments
+
+- _to_buffer(cls, buffer, offset, value, offsets=None):
+  initialize object on buffer
+
+- _from_buffer(cls, buffer,offset):
+  create python object from initlialized object on buffer,offset
+
+- _size: object size
+  None if dynamic
+
+Instance content:
+
+- _get_size(self) -> size
+
+## C-API
+
+Access
+- `<Class>_get<_attrs>(obj, <indexes>)` -> scalar or object pointer
+- `<Class>_set<_attrs>(obj, <indexes>, scalar)`
+
+e.g. Multipole_get_field_normal(3)
+
+Kernel convention:
+
+```
+<fun>(void **refs){
+          Type0 arg0 = (Type0) refs[0];
+          ....
+          Type1 argN = (TypeN) refs[N];
+        ...
+        }
+```
+
+``` <fun>(buffer1, offset, ..., offset3){
+         Type0 arg0 = (Type0) refs[offset0];
+         ,,,
+         TypeN argN = (TypeN) refs[offsetN];
+}
+```
+
+## TODO
+
+- add free methods to delete data from buffer
+- implement SOA types

{xobjects-0.4.4/xobjects.egg-info → xobjects-0.4.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: xobjects
-Version: 0.4.4
+Version: 0.4.6
 Summary: In-memory serialization and code generator for CPU and GPU
 Home-page: https://xsuite.readthedocs.io/
 Download-URL: https://pypi.python.org/pypi/xobjects

xobjects-0.4.6/Readme.md

@@ -0,0 +1,10 @@
+# XObjects
+
+Library to create and manipulate serialized object in CPU and GPU memory efficiently.
+
+The library defines:
+-  Contexts: create buffers and manage computations e.g. BytearrayContext, CLContext, ...
+-  Buffers: reference to internal memory buffer with an allocator e.g.
+   BytearrayContext, CLBuffer, ...
+-  Data types: define users define types to create and manipulate objects on
+   buffers and generate C api, e.g. Float64, Int64, String, Struct, Array, Union

xobjects-0.4.6/contributors_and_credits.txt

@@ -0,0 +1,7 @@
+The following people contributed to the development of this package:
+
+CERN contributors:
+ - R. De Maria
+ - G. Iadarola
+ - P. Kiksiny
+ - S. Lopaciuk

xobjects-0.4.6/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

xobjects-0.4.6/docs/architecture/contexts.rst

@@ -0,0 +1,164 @@
+Contexts
+========
+
+Devices
+-------
+
+
+Buffers
+-------
+
+Buffers are able to store data in the host or device from several object types and return data in device or host.
+
+
+Instance methods:
+
+*  write(self, source, offset, nbytes): write nbytes from source that can be a np-like array, bytearray, bytes or another buffer
+*  to_nplike(self,dtype,shape,offset): return a view of the data in a np-like arrays
+*  to_bytearray(self,offset,count): return
+
+.. code::
+
+   def update_from_xbuffer(self,offset, source, source_offset, nbytes):
+       "update from any xbuffer, don't pass through gpu if possible"
+
+   def update_from_buffer(self, offset, buffer):
+       """source is python buffer such as bytearray, numpy array .data"""
+
+   def copy_native(self,offset,nbytes):
+        """return native data with content at from offset and nbytes
+        return native_data
+
+   def update_from_native(self, offset, source, source_offset, nbytes):
+       """update native source into buffer usef in update_from_xbuffer"""
+
+   def to_nplike(self,offset,shape,dtype)
+       """view in nplike"""
+
+   def to_bytearray(self,offset,nbytes)
+       """copy in byte array: used in update_from_xbuffer"""
+
+   def to_pointer_arg(self,offset,nbytes):
+       return self.data[offset:offset+nbytes] #offset can lead to alignement error in opencl if not multiple of 4 bytes
+
+
+
+Arguments for kernels:
+
+- CPU:
+  - numpy array ->  ``ffi.cast("double *", ffi.from_buffer(arr))``
+  - numpy scalar -> no change
+  - s string -> ``ss.decode("utf8")``
+
+- PyOpenCL:
+  - pyopencl.Buffer: [offset:offset+nbytes] or copy back and fort if needs to be aligned
+  - pyopencl.array:  arr.base_data[arr.offset:arr.offset+arr.nbytes] or copy back and fort
+
+- cuda:
+    -  pycuda.array: arr or arr.data
+
+
+nparray.data:    python buffer pointing to start of the array arrays
+clarray.data:  Optional Opencl buffer pointing to the start of the array
+clarray.base_data,clarray.base_data:  Optional Opencl buffer  and offset from where the pointer start
+cupyaray.
+
+
+clarray.base_data[]:
+
+Code generation
+---------------
+
+Each context is able to compile source code and make available python callable functions.
+
+The callable functions can take as arguments Python numbers, strings,
+numpy-like arrays (depending on the context) and xobject allocated in the same
+contexts (if not copies will be done).
+
+Source code is optionally pre-processed to adapt it to the context.
+
+The callable functions are defined by:
+*  the name of the function in the source code
+*  the type of parameters
+*  how to pass parallalization instruction (threads for openmp, grid for GPU)
+
+The callable functions are parametrized by name and argument types.
+
+Each xobject type can provide the API source code  the ``_get_c_api()`` class method.
+
+
+
+C Api
+^^^^^^^^^^^^^^^^^^^
+
+General:
+
+- ``<pointer type> <typename>_getp_<field_names>(<typename> obj, <indexes>)``
+- ``<scalar type>  <typename>_get_<field_names>(<typename> obj, <indexes>)``
+- ``<typename>_set_<field_names>(<typename> obj, <indexes>, <scalar_type> value)``
+- ``<int64> <typename>_size_<field_names>(<typename> obj,<indexes>)``
+
+Array:
+- ``<int64> <typename>_len...(<typename> obj) : number of itemns``
+- ``<int64> <typename>_dim...(<typename> obj, int index): size of dimension``
+- ``<int64> <typename>_ndim...(<typename> obj): number of dimensions``
+- ``void <typename>_strides(int* result)``
+
+UnionRef:
+- ``<int64> <typename>_typeid...(<typename> obj) : get member
+- ``void* <typename>_member...(int* result)``
+
+
+Kernels and Functions
+^^^^^^^^^^^^^^^^^^^^^
+
+Context can execute:
+
+- kernels: no return function and executed on a grid
+- functions: only for cpu kernels
+
+
+.. code:: python
+  Kernel(
+    cname,
+    args=Arg(dtype, name: String, const: Boolean)
+    grid=integer, string or list of string or functions of arguments
+    ret=None)
+
+
+Arg:
+
+* atype: argument type:
+  - provide validation function: cls._validate(value) or atype(value)
+  - provide C-type name: atype._get_ctype()  or dtype -> converted to c types,
+  - generate raw argument from value .atype._get_carg() for the interface
+* name: optional, String needed for the grid
+* const: boolean (used in case of copy back)
+
+Code:
+
+* define sources, list of string or files
+* define kernels to expose, list of kernels
+* define preprocessors
+* attach callable kernels to context
+* to be used by CMethod or CProperty
+
+.. code::
+    double x -> Arg(dtype=xo.Float64,name='x'): takes python scalar and np scalar
+    double *restrict x -> Arg(dtype=xo.Float64,pointer=True,const=False,name='x'): takes np.array, xo.Float64 arrays
+    const double *restrict x -> Arg(dtype=xo.Float64,pointer=True,const=True,name='x'): takes np.array, xo.Float64 arrays
+    const Float64_N -> Arg(dtype=xo.Float64[:],const=True,name='x'): takes xo.Float64[:]
+    const Float64_6by6 -> Arg(dtype=xo.Float64[6,6],const=True,name='x'): takes xo.Float64[6,6]
+
+
+
+
+
+*  function: general function
+*  method: implies first attribute is the instance of the class where it is defined
+*  kernel: implies some iteration and no return type
+
+The function can be defined using:
+*   header
+*   source
+*   body

xobjects-0.4.6/docs/architecture/types.rst

@@ -0,0 +1,179 @@
+Types
+=====
+
+Introduction
+-------------
+
+
+Data layout and API can be defined by a composition of:
+*   Scalars: Int8, Int16, Int32, Int64, Float32, Float64
+*   Compounds: Struct, Array, Ref
+
+Types can have a fixed size (*static types*) or a variable size (*dynamic size*).
+The size of one instance cannot change after creation, but their content can.
+
+
+Data Layout
+-------------
+
+Data structure is organized in 64bit slots.
+
+Table indicates slot size and the description.
+
+Scalar
+
+=====  ===========
+Slots  Description
+=====  ===========
+8      Data
+=====  ===========
+
+String of n-bytes
+
+=============     ===============================
+Slots             Description
+=============     ===============================
+8                 Size in bytes
+ceil((n+1)/8)     Data in UTF-8 null terminated
+=============     ===============================
+
+
+Struct with static and *n* dynamical fields
+
+=======  ===============================
+Slots    Description
+=======  ===============================
+8        Optional: size if n>0
+...      Fixed-size field data
+8*(n-1)  Optional: Offsets of the 2nd to n-th last variable sized
+...      Optional: Variable sized data
+=======  ===============================
+
+Array with n dimensions, whose l are dynamic of *m* items.
+
+=======  ===============================
+Slots    Description
+=======  ===============================
+8        Optional: size
+8*l      Optional: size of l-variable dimensions
+8*n      Optional: strides for n-dimensions larger than 2
+m*s      Data or offsets for m-items (s is the item-size or 8)
+...      Optional: Items data if the item type is dynamic
+=======  ===============================
+
+Union
+
+=======  ===============================
+Slots    Description
+=======  ===============================
+8        Type id
+...      Item data
+=======  ===============================
+
+
+Ref
+
+=======  ======================================
+Slots    Description
+=======  ======================================
+8        Offset from start of the buffer
+8        Optional: typeid
+=======  ======================================
+
+
+Python type interface
+----------------------
+
+The following describe the generic interface
+
+Class variables:
+
+==============  ===============================
+name            Description
+==============  ===============================
+_size           Size in bytes, None if not static size
+==============  ===============================
+
+Class methods:
+
+============= ============== ============ ============================================
+name          args           return                  Description
+============= ============== ============ ============================================
+_inspect_args *args,         Info         Return at least info.size and optionally
+              *kwargs                     other metadata to build objects in memory
+                                          from python objects. Args are propocessed
+                                          by _pre_init
+_pre_init     *args,         args,        Preprocess arguments and generate standard
+              *kwargs        kwargs       values
+_to_buffer    buffer         None         Serialize python in buffer from offset.
+              offset                      Assume sufficient size is available
+              value
+_from_buffer  buffer         instance     Return Python instance from buffer and
+              offset                      and offset
+_to_schema                   string       TODO String representation of the type to
+                                          deserialize objects
+============= ============== ============ ============================================
+
+
+Instance variables:
+
+=============  ========================================
+name           Description
+=============  ========================================
+_size          Optional: size in bytes for dynamic-size
+=============  ========================================
+
+
+Class methods:
+
+============= ============== ============ ============================================
+name          args           return                  Description
+============= ============== ============ ============================================
+_post_init                                Run after object creation in __init__
+copy          buffer or      None         TODO Return a copy into a buffer
+              context                      Assume sufficient size is available
+_update       value          instance     Update values of an exsiting objec
+_to_json                     json         TODO json
+============= ============== ============ ============================================
+
+
+
+**__init__**
+
+init is used to  create a new xobject by allocating memory on the buffer.
+Ref and Scalars cannot be initialized alone (maybe temporarily)
+
+Init can take by default:
+
+*  a json object:
+    *  dict for Struct
+    *  list for Array,
+    *  number of Scalars
+    *  String for String
+    *  (Typename, {}) for Ref of multiple types
+*  dimensions (for string or arrays)
+*  an xobject of the correct type
+*  None for Ref
+
+The data is dispatched to _pre_init using the following convetion:
+* a dict is passed with **
+* a tuple is passed with *
+* any other object is passed verbatim
+
+Init uses the following steps:
+
+*  pre_init to pre_process input
+*  inspect_args to calculate sizes and collect metadata
+*  get_a_buffer to allocate space
+*  _to_buffer to write to memory
+*  _from_buffer to build a python object
+*  post_init
+
+**_from_buffer**
+
+When accessing data inside compound object the _from_buffer method is used.
+For scalar and string, python data is returned. For coumpund object,
+XObjects are returned
+
+**setters**
+Setters uses to_buffer or _update to update existing data.

xobjects-0.4.6/docs/architecture.rst

@@ -0,0 +1,16 @@
+Architecture
+==============
+
+
+
+The architucture is based on
+
+* Context managers implementing buffer, compilation, running
+* Type system to specify data layout and user API
+
+
+.. toctree::
+    :maxdepth: 2
+
+    architecture/contexts
+    architecture/types