b10-msgpack-numpy 0.4.8__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- b10_msgpack_numpy-0.4.8/AUTHORS.md +18 -0
- b10_msgpack_numpy-0.4.8/CHANGES.md +156 -0
- b10_msgpack_numpy-0.4.8/LICENSE.md +35 -0
- b10_msgpack_numpy-0.4.8/PKG-INFO +163 -0
- b10_msgpack_numpy-0.4.8/README.md +131 -0
- b10_msgpack_numpy-0.4.8/b10_msgpack_numpy.egg-info/PKG-INFO +163 -0
- b10_msgpack_numpy-0.4.8/b10_msgpack_numpy.egg-info/SOURCES.txt +14 -0
- b10_msgpack_numpy-0.4.8/b10_msgpack_numpy.egg-info/dependency_links.txt +1 -0
- b10_msgpack_numpy-0.4.8/b10_msgpack_numpy.egg-info/requires.txt +2 -0
- b10_msgpack_numpy-0.4.8/b10_msgpack_numpy.egg-info/top_level.txt +1 -0
- b10_msgpack_numpy-0.4.8/msgpack_numpy.py +343 -0
- b10_msgpack_numpy-0.4.8/pyproject.toml +3 -0
- b10_msgpack_numpy-0.4.8/setup.cfg +7 -0
- b10_msgpack_numpy-0.4.8/setup.py +55 -0
- b10_msgpack_numpy-0.4.8/tests.py +336 -0
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
<!---
|
|
2
|
+
-*- mode:markdown -*-
|
|
3
|
+
vi:ft=markdown
|
|
4
|
+
-->
|
|
5
|
+
|
|
6
|
+
Authors
|
|
7
|
+
=======
|
|
8
|
+
This software was written and packaged by [Lev E. Givon](https://github.com/lebedov)
|
|
9
|
+
|
|
10
|
+
Special thanks are due to the following parties for their contributions:
|
|
11
|
+
|
|
12
|
+
- [Alex Ford](https://github.com/asford) - bug fix.
|
|
13
|
+
- [Colin Jermain](https://github.com/cjermain) - Python 3 support.
|
|
14
|
+
- [Etienne Wodey](https://github.com/airwoodix) - support for ndarrays with dtype='O'
|
|
15
|
+
- [John Tyree](https://github.com/johntyree) - support for numpy scalar booleans.
|
|
16
|
+
- [Mehdi Sadeghi](https://github.com/mehdisadeghi) - bug reports.
|
|
17
|
+
- [Sujoy Roy](https://github.com/tvkpz) - bug reports.
|
|
18
|
+
- [Jose Coutinho](https://github.com/tiagocoutinho) - make msgpack-numpy params consistent with msgpack-python.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
-*- mode:markdown -*-
|
|
3
|
+
vi:ft=markdown
|
|
4
|
+
-->
|
|
5
|
+
Change Log
|
|
6
|
+
==========
|
|
7
|
+
|
|
8
|
+
Release 0.4.8 (Baseten fork, November 2026)
|
|
9
|
+
-------------------------------------------
|
|
10
|
+
* SECURITY (CWE-502): object-dtype ndarrays are no longer unpickled by
|
|
11
|
+
default. ``decode``, ``unpackb``/``unpack`` and ``Unpacker`` now take an
|
|
12
|
+
``allow_pickle`` argument (default ``False``) and raise ``PickleNotAllowed``
|
|
13
|
+
instead of calling ``pickle.loads`` on attacker-controlled bytes. Enable it
|
|
14
|
+
only for payloads from a trusted encoder.
|
|
15
|
+
|
|
16
|
+
Release 0.4.8 (April 28, 2022)
|
|
17
|
+
------------------------------
|
|
18
|
+
* Add support for ndarrays with dtype=object (#46).
|
|
19
|
+
|
|
20
|
+
Release 0.4.7.1 (September 30, 2020)
|
|
21
|
+
------------------------------------
|
|
22
|
+
* Fix Python 2.7 regression (#45).
|
|
23
|
+
|
|
24
|
+
Release 0.4.7 (September 16, 2020)
|
|
25
|
+
----------------------------------
|
|
26
|
+
* Fix bug unpacking nested, structured dtypes (#42).
|
|
27
|
+
|
|
28
|
+
Release 0.4.6.1 (July 28, 2020)
|
|
29
|
+
-------------------------------
|
|
30
|
+
* Deprecate support for msgpack < 1.0.0 (#41).
|
|
31
|
+
|
|
32
|
+
Release 0.4.6 (May 25, 2020)
|
|
33
|
+
----------------------------
|
|
34
|
+
* Set default use_bin_type and raw params of Packer and Unpacker classes the same as in msgpack (#39).
|
|
35
|
+
* Convert docs to Markdown.
|
|
36
|
+
|
|
37
|
+
Release 0.4.5 (April 12, 2020)
|
|
38
|
+
------------------------------
|
|
39
|
+
* Set defaults appropriate to msgpack 1.0.0 (#38).
|
|
40
|
+
|
|
41
|
+
Release 0.4.4.3 (May 16, 2019)
|
|
42
|
+
------------------------------
|
|
43
|
+
* Configure contiguous integration with Travis.
|
|
44
|
+
* Move unit tests out of msgpack_numpy module.
|
|
45
|
+
* Add workaround for issue handling n-dim arrays on MacOS (#35).
|
|
46
|
+
|
|
47
|
+
Release 0.4.4.2 (November 8, 2018)
|
|
48
|
+
----------------------------------
|
|
49
|
+
* Fix regression handling noncontiguous arrays (#34).
|
|
50
|
+
|
|
51
|
+
Release 0.4.4.1 (September 26, 2018)
|
|
52
|
+
------------------------------------
|
|
53
|
+
* Check Py version before defining encode and tostr funcs.
|
|
54
|
+
* Eliminate deprecation warnings, raise minimum required msgpack version (#31).
|
|
55
|
+
* Fix Python 2 support (#32).
|
|
56
|
+
|
|
57
|
+
Release 0.4.4 (September 25, 2018)
|
|
58
|
+
----------------------------------
|
|
59
|
+
* Access ndarray memory view directly to slightly speed up encoding (#30).
|
|
60
|
+
|
|
61
|
+
Release 0.4.3.2 - (September 17, 2018)
|
|
62
|
+
--------------------------------------
|
|
63
|
+
* Update classifiers to list Py3 support (#29).
|
|
64
|
+
|
|
65
|
+
Release 0.4.3.1 - (July 11, 2018)
|
|
66
|
+
---------------------------------
|
|
67
|
+
* Switch to numpy.frombuffer to avoid deprecation warning (#27).
|
|
68
|
+
|
|
69
|
+
Release 0.4.3 - (February 25, 2018)
|
|
70
|
+
-----------------------------------
|
|
71
|
+
* Change dependency name due to package name change (#25).
|
|
72
|
+
|
|
73
|
+
Release 0.4.2 - (December 2, 2017)
|
|
74
|
+
----------------------------------
|
|
75
|
+
* Make decoding/encoding defaults identical to those of msgpack-python 0.4.* (#19).
|
|
76
|
+
* Fix handling of nested arrays in Py3 (#21).
|
|
77
|
+
* Improve decoding of complex values in Py3.
|
|
78
|
+
* Use numpy.testing functions to make unit test errors more informative.
|
|
79
|
+
* Make object_hook params consistent with those of msgack-python (#23).
|
|
80
|
+
|
|
81
|
+
Release 0.4.1 - (July 6, 2017)
|
|
82
|
+
------------------------------
|
|
83
|
+
* Improve ability to decode data serialized with versions before 0.3.9 (#20).
|
|
84
|
+
|
|
85
|
+
Release 0.4.0 - (May 24, 2017)
|
|
86
|
+
------------------------------
|
|
87
|
+
* Remove deprecated ez_setup.
|
|
88
|
+
* Add numpy >= 1.9.0 dependency (#18).
|
|
89
|
+
|
|
90
|
+
Release 0.3.9 - (February 7, 2017)
|
|
91
|
+
----------------------------------
|
|
92
|
+
* Complex type handling fixes.
|
|
93
|
+
* Handle structured dtypes (#15).
|
|
94
|
+
* Note: data serialized with earlier versions cannot be deserialized with 0.3.9
|
|
95
|
+
(#17).
|
|
96
|
+
|
|
97
|
+
Release 0.3.8 - (Februrary 4, 2017)
|
|
98
|
+
-----------------------------------
|
|
99
|
+
* Fix deserialization with Python 3.5 (#13).
|
|
100
|
+
|
|
101
|
+
Release 0.3.7 - (March 12, 2016)
|
|
102
|
+
--------------------------------
|
|
103
|
+
* Add Python 3 support (#11).
|
|
104
|
+
|
|
105
|
+
Release 0.3.6 - (July 8, 2015)
|
|
106
|
+
------------------------------
|
|
107
|
+
* Fall back to pure Python msgpack if compiled extension is unavailable (#6).
|
|
108
|
+
|
|
109
|
+
Release 0.3.5 - (February 12, 2015)
|
|
110
|
+
-----------------------------------
|
|
111
|
+
* Add support for numpy scalar booleans (#8).
|
|
112
|
+
|
|
113
|
+
Release 0.3.4 - (December 12, 2014)
|
|
114
|
+
-----------------------------------
|
|
115
|
+
* Add badges, update README.
|
|
116
|
+
|
|
117
|
+
Release 0.3.3 - (October 21, 2014)
|
|
118
|
+
----------------------------------
|
|
119
|
+
* Add missing ez_setup.py file.
|
|
120
|
+
|
|
121
|
+
Release 0.3.2 - (May 6, 2014)
|
|
122
|
+
-----------------------------
|
|
123
|
+
* Simplify encoding/decoding of numpy scalars.
|
|
124
|
+
|
|
125
|
+
Release 0.3.1.1 - (March 2, 2014)
|
|
126
|
+
---------------------------------
|
|
127
|
+
* Make package a simple module to fix dependency installation issues.
|
|
128
|
+
|
|
129
|
+
Release 0.3.1 - (March 2, 2014)
|
|
130
|
+
-------------------------------
|
|
131
|
+
* Switch to PEP 440 version numbering.
|
|
132
|
+
* Update to setuptools 2.2 ez_setup.py
|
|
133
|
+
|
|
134
|
+
Release 0.03 - (October 30, 2013)
|
|
135
|
+
---------------------------------
|
|
136
|
+
* Add support for msgpack 0.4.0.
|
|
137
|
+
* Rename to msgpack-numpy.
|
|
138
|
+
|
|
139
|
+
Release 0.022 - (September 10, 2013)
|
|
140
|
+
------------------------------------
|
|
141
|
+
* Fix decoding of string arrays (#4).
|
|
142
|
+
* Fix decoding of dicts containing ndarrays (#5).
|
|
143
|
+
|
|
144
|
+
Release 0.021 - (May 29, 2013)
|
|
145
|
+
------------------------------
|
|
146
|
+
* Improve encoding/decoding performance for arrays.
|
|
147
|
+
* Fix numpy type support on different platforms (#3).
|
|
148
|
+
|
|
149
|
+
Release 0.02 - (February 21, 2013)
|
|
150
|
+
----------------------------------
|
|
151
|
+
* Add support for msgpack 0.3.0 (contributed by crispamares).
|
|
152
|
+
|
|
153
|
+
Release 0.01 - (February 07, 2013)
|
|
154
|
+
----------------------------------
|
|
155
|
+
* First public release.
|
|
156
|
+
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
<!---
|
|
2
|
+
-*- mode:markdown -*-
|
|
3
|
+
vi:ft=markdown
|
|
4
|
+
-->
|
|
5
|
+
License
|
|
6
|
+
=======
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2013-2022, Lev E. Givon.
|
|
9
|
+
All rights reserved.
|
|
10
|
+
|
|
11
|
+
Redistribution and use in source and binary forms, with or without
|
|
12
|
+
modification, are permitted provided that the following conditions are
|
|
13
|
+
met:
|
|
14
|
+
|
|
15
|
+
* Redistributions of source code must retain the above copyright
|
|
16
|
+
notice, this list of conditions and the following disclaimer.
|
|
17
|
+
* Redistributions in binary form must reproduce the above
|
|
18
|
+
copyright notice, this list of conditions and the following
|
|
19
|
+
disclaimer in the documentation and/or other materials provided
|
|
20
|
+
with the distribution.
|
|
21
|
+
* Neither the name of Lev E. Givon nor the names of any
|
|
22
|
+
contributors may be used to endorse or promote products derived
|
|
23
|
+
from this software without specific prior written permission.
|
|
24
|
+
|
|
25
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
26
|
+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
27
|
+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
28
|
+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
29
|
+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
30
|
+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
31
|
+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
32
|
+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
33
|
+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
34
|
+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
35
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: b10-msgpack-numpy
|
|
3
|
+
Version: 0.4.8
|
|
4
|
+
Summary: Numpy data serialization using msgpack (Baseten fork with CWE-502 fix)
|
|
5
|
+
Home-page: https://github.com/basetenlabs/b10-msgpack-numpy
|
|
6
|
+
Author: Lev E. Givon
|
|
7
|
+
Author-email: lev@columbia.edu
|
|
8
|
+
License: BSD
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: License :: OSI Approved :: BSD License
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python
|
|
15
|
+
Classifier: Programming Language :: Python :: 2.7
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.4
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.5
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.6
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
22
|
+
Classifier: Topic :: Scientific/Engineering
|
|
23
|
+
Classifier: Topic :: Software Development
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE.md
|
|
26
|
+
License-File: AUTHORS.md
|
|
27
|
+
Requires-Dist: numpy>=1.9.0
|
|
28
|
+
Requires-Dist: msgpack>=0.5.2
|
|
29
|
+
Dynamic: author
|
|
30
|
+
Dynamic: author-email
|
|
31
|
+
Dynamic: classifier
|
|
32
|
+
Dynamic: description
|
|
33
|
+
Dynamic: description-content-type
|
|
34
|
+
Dynamic: home-page
|
|
35
|
+
Dynamic: license
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
Dynamic: requires-dist
|
|
38
|
+
Dynamic: summary
|
|
39
|
+
|
|
40
|
+
Package Description
|
|
41
|
+
-------------------
|
|
42
|
+
This package provides encoding and decoding routines that enable the
|
|
43
|
+
serialization and deserialization of numerical and array data types provided by
|
|
44
|
+
[numpy](http://www.numpy.org/) using the highly efficient
|
|
45
|
+
[msgpack](http://msgpack.org/) format. Serialization of Python's
|
|
46
|
+
native complex data types is also supported.
|
|
47
|
+
|
|
48
|
+
[](https://pypi.python.org/pypi/msgpack-numpy)
|
|
49
|
+
[](https://travis-ci.org/lebedov/msgpack-numpy)
|
|
50
|
+
|
|
51
|
+
Installation
|
|
52
|
+
------------
|
|
53
|
+
msgpack-numpy requires msgpack-python and numpy. If you
|
|
54
|
+
have [pip](http://www.pip-installer.org/) installed on your
|
|
55
|
+
system, run
|
|
56
|
+
|
|
57
|
+
pip install msgpack-numpy
|
|
58
|
+
|
|
59
|
+
to install the package and all dependencies. You can also download
|
|
60
|
+
the source tarball, unpack it, and run
|
|
61
|
+
|
|
62
|
+
python setup.py install
|
|
63
|
+
|
|
64
|
+
from within the source directory.
|
|
65
|
+
|
|
66
|
+
Usage
|
|
67
|
+
-----
|
|
68
|
+
The easiest way to use msgpack-numpy is to call its monkey patching
|
|
69
|
+
function after importing the Python msgpack package:
|
|
70
|
+
|
|
71
|
+
import msgpack
|
|
72
|
+
import msgpack_numpy as m
|
|
73
|
+
m.patch()
|
|
74
|
+
|
|
75
|
+
This will automatically force all msgpack serialization and deserialization
|
|
76
|
+
routines (and other packages that use them) to become numpy-aware.
|
|
77
|
+
Of course, one can also manually pass the encoder and
|
|
78
|
+
decoder provided by msgpack-numpy to the msgpack routines:
|
|
79
|
+
|
|
80
|
+
import msgpack
|
|
81
|
+
import msgpack_numpy as m
|
|
82
|
+
import numpy as np
|
|
83
|
+
|
|
84
|
+
x = np.random.rand(5)
|
|
85
|
+
x_enc = msgpack.packb(x, default=m.encode)
|
|
86
|
+
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)
|
|
87
|
+
|
|
88
|
+
msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
|
|
89
|
+
If msgpack was not compiled with Cython (or if the ``MSGPACK_PUREPYTHON``
|
|
90
|
+
variable is set), it will fall back to using the slower pure Python msgpack
|
|
91
|
+
implementation.
|
|
92
|
+
|
|
93
|
+
Notes
|
|
94
|
+
-----
|
|
95
|
+
The primary design goal of msgpack-numpy is ensuring preservation of numerical
|
|
96
|
+
data types during msgpack serialization and deserialization. Inclusion of type
|
|
97
|
+
information in the serialized data necessarily incurs some storage overhead; if
|
|
98
|
+
preservation of type information is not needed, one may be able to avoid some
|
|
99
|
+
of this overhead by writing a custom encoder/decoder pair that produces more
|
|
100
|
+
efficient serializations for those specific use cases.
|
|
101
|
+
|
|
102
|
+
Numpy arrays with a dtype of 'O' are serialized using pickle as a fallback
|
|
103
|
+
solution. Deserializing them requires unpickling, which executes arbitrary
|
|
104
|
+
code from the payload, so decoding is **disabled by default** (CWE-502):
|
|
105
|
+
``decode``, ``unpackb``/``unpack`` and ``Unpacker`` raise
|
|
106
|
+
``msgpack_numpy.PickleNotAllowed`` when they encounter an object-dtype array.
|
|
107
|
+
If you control both ends, opt in explicitly:
|
|
108
|
+
|
|
109
|
+
x_rec = msgpack.unpackb(x_enc, allow_pickle=True)
|
|
110
|
+
|
|
111
|
+
Never enable ``allow_pickle`` for untrusted input. As the overhead of pickle
|
|
112
|
+
serialization negates one of the reasons to use msgpack, consider writing a
|
|
113
|
+
custom encoder/decoder for your specific use case instead.
|
|
114
|
+
|
|
115
|
+
Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied
|
|
116
|
+
if they are to be modified.
|
|
117
|
+
|
|
118
|
+
Development
|
|
119
|
+
-----------
|
|
120
|
+
The latest source code can be obtained from
|
|
121
|
+
[GitHub](https://github.com/basetenlabs/b10-msgpack-numpy/). This is the Baseten
|
|
122
|
+
fork of [lebedov/msgpack-numpy](https://github.com/lebedov/msgpack-numpy); the
|
|
123
|
+
distribution is named ``b10-msgpack-numpy`` and still provides the
|
|
124
|
+
``msgpack_numpy`` module.
|
|
125
|
+
|
|
126
|
+
msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.
|
|
127
|
+
|
|
128
|
+
Install [`tox`](https://tox.readthedocs.io/en/latest/) to support testing
|
|
129
|
+
across multiple python versions in your development environment. If you
|
|
130
|
+
use [`conda`](https://docs.conda.io/en/latest/) to install `python` use
|
|
131
|
+
[`tox-conda`](https://github.com/tox-dev/tox-conda) to automatically manage
|
|
132
|
+
testing across all supported python versions.
|
|
133
|
+
|
|
134
|
+
# Using a system python
|
|
135
|
+
pip install tox
|
|
136
|
+
|
|
137
|
+
# Additionally, using a conda-provided python
|
|
138
|
+
pip install tox tox-conda
|
|
139
|
+
|
|
140
|
+
Execute tests across supported python versions:
|
|
141
|
+
|
|
142
|
+
tox
|
|
143
|
+
|
|
144
|
+
Authors
|
|
145
|
+
-------
|
|
146
|
+
See the included [AUTHORS.md](https://github.com/lebedov/msgpack-numpy/blob/master/AUTHORS.md) file for
|
|
147
|
+
more information.
|
|
148
|
+
|
|
149
|
+
License
|
|
150
|
+
-------
|
|
151
|
+
This software is licensed under the [BSD License](http://www.opensource.org/licenses/bsd-license).
|
|
152
|
+
See the included [LICENSE.md](https://github.com/lebedov/msgpack-numpy/blob/master/LICENSE.md) file for
|
|
153
|
+
more information.
|
|
154
|
+
|
|
155
|
+
See Also
|
|
156
|
+
--------
|
|
157
|
+
[msgpack-numpy-js](https://github.com/emerald-geomodelling/msgpack-numpy-js) by
|
|
158
|
+
[Egil Möller](https://github.com/redhog) is a JavaScript library that can
|
|
159
|
+
serialize and deserialize JS typed arrays with msgpack.
|
|
160
|
+
|
|
161
|
+
[msgpack-numpy-rs](https://github.com/clearviewai/msgpack-numpy-rs) by [
|
|
162
|
+
Terrence Liu](https://github.com/terencezl) is a Rust crate (mostly) compatible with msgpack-numpy
|
|
163
|
+
that can serialize and deserialize numpy arrays with msgpack.
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
<!---
|
|
2
|
+
-*- mode:markdown -*-
|
|
3
|
+
vi:ft=markdown
|
|
4
|
+
-->
|
|
5
|
+
Numpy Data Type Serialization Using Msgpack
|
|
6
|
+
===========================================
|
|
7
|
+
|
|
8
|
+
Package Description
|
|
9
|
+
-------------------
|
|
10
|
+
This package provides encoding and decoding routines that enable the
|
|
11
|
+
serialization and deserialization of numerical and array data types provided by
|
|
12
|
+
[numpy](http://www.numpy.org/) using the highly efficient
|
|
13
|
+
[msgpack](http://msgpack.org/) format. Serialization of Python's
|
|
14
|
+
native complex data types is also supported.
|
|
15
|
+
|
|
16
|
+
[](https://pypi.python.org/pypi/msgpack-numpy)
|
|
17
|
+
[](https://travis-ci.org/lebedov/msgpack-numpy)
|
|
18
|
+
|
|
19
|
+
Installation
|
|
20
|
+
------------
|
|
21
|
+
msgpack-numpy requires msgpack-python and numpy. If you
|
|
22
|
+
have [pip](http://www.pip-installer.org/) installed on your
|
|
23
|
+
system, run
|
|
24
|
+
|
|
25
|
+
pip install msgpack-numpy
|
|
26
|
+
|
|
27
|
+
to install the package and all dependencies. You can also download
|
|
28
|
+
the source tarball, unpack it, and run
|
|
29
|
+
|
|
30
|
+
python setup.py install
|
|
31
|
+
|
|
32
|
+
from within the source directory.
|
|
33
|
+
|
|
34
|
+
Usage
|
|
35
|
+
-----
|
|
36
|
+
The easiest way to use msgpack-numpy is to call its monkey patching
|
|
37
|
+
function after importing the Python msgpack package:
|
|
38
|
+
|
|
39
|
+
import msgpack
|
|
40
|
+
import msgpack_numpy as m
|
|
41
|
+
m.patch()
|
|
42
|
+
|
|
43
|
+
This will automatically force all msgpack serialization and deserialization
|
|
44
|
+
routines (and other packages that use them) to become numpy-aware.
|
|
45
|
+
Of course, one can also manually pass the encoder and
|
|
46
|
+
decoder provided by msgpack-numpy to the msgpack routines:
|
|
47
|
+
|
|
48
|
+
import msgpack
|
|
49
|
+
import msgpack_numpy as m
|
|
50
|
+
import numpy as np
|
|
51
|
+
|
|
52
|
+
x = np.random.rand(5)
|
|
53
|
+
x_enc = msgpack.packb(x, default=m.encode)
|
|
54
|
+
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)
|
|
55
|
+
|
|
56
|
+
msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
|
|
57
|
+
If msgpack was not compiled with Cython (or if the ``MSGPACK_PUREPYTHON``
|
|
58
|
+
variable is set), it will fall back to using the slower pure Python msgpack
|
|
59
|
+
implementation.
|
|
60
|
+
|
|
61
|
+
Notes
|
|
62
|
+
-----
|
|
63
|
+
The primary design goal of msgpack-numpy is ensuring preservation of numerical
|
|
64
|
+
data types during msgpack serialization and deserialization. Inclusion of type
|
|
65
|
+
information in the serialized data necessarily incurs some storage overhead; if
|
|
66
|
+
preservation of type information is not needed, one may be able to avoid some
|
|
67
|
+
of this overhead by writing a custom encoder/decoder pair that produces more
|
|
68
|
+
efficient serializations for those specific use cases.
|
|
69
|
+
|
|
70
|
+
Numpy arrays with a dtype of 'O' are serialized using pickle as a fallback
|
|
71
|
+
solution. Deserializing them requires unpickling, which executes arbitrary
|
|
72
|
+
code from the payload, so decoding is **disabled by default** (CWE-502):
|
|
73
|
+
``decode``, ``unpackb``/``unpack`` and ``Unpacker`` raise
|
|
74
|
+
``msgpack_numpy.PickleNotAllowed`` when they encounter an object-dtype array.
|
|
75
|
+
If you control both ends, opt in explicitly:
|
|
76
|
+
|
|
77
|
+
x_rec = msgpack.unpackb(x_enc, allow_pickle=True)
|
|
78
|
+
|
|
79
|
+
Never enable ``allow_pickle`` for untrusted input. As the overhead of pickle
|
|
80
|
+
serialization negates one of the reasons to use msgpack, consider writing a
|
|
81
|
+
custom encoder/decoder for your specific use case instead.
|
|
82
|
+
|
|
83
|
+
Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied
|
|
84
|
+
if they are to be modified.
|
|
85
|
+
|
|
86
|
+
Development
|
|
87
|
+
-----------
|
|
88
|
+
The latest source code can be obtained from
|
|
89
|
+
[GitHub](https://github.com/basetenlabs/b10-msgpack-numpy/). This is the Baseten
|
|
90
|
+
fork of [lebedov/msgpack-numpy](https://github.com/lebedov/msgpack-numpy); the
|
|
91
|
+
distribution is named ``b10-msgpack-numpy`` and still provides the
|
|
92
|
+
``msgpack_numpy`` module.
|
|
93
|
+
|
|
94
|
+
msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.
|
|
95
|
+
|
|
96
|
+
Install [`tox`](https://tox.readthedocs.io/en/latest/) to support testing
|
|
97
|
+
across multiple python versions in your development environment. If you
|
|
98
|
+
use [`conda`](https://docs.conda.io/en/latest/) to install `python` use
|
|
99
|
+
[`tox-conda`](https://github.com/tox-dev/tox-conda) to automatically manage
|
|
100
|
+
testing across all supported python versions.
|
|
101
|
+
|
|
102
|
+
# Using a system python
|
|
103
|
+
pip install tox
|
|
104
|
+
|
|
105
|
+
# Additionally, using a conda-provided python
|
|
106
|
+
pip install tox tox-conda
|
|
107
|
+
|
|
108
|
+
Execute tests across supported python versions:
|
|
109
|
+
|
|
110
|
+
tox
|
|
111
|
+
|
|
112
|
+
Authors
|
|
113
|
+
-------
|
|
114
|
+
See the included [AUTHORS.md](https://github.com/lebedov/msgpack-numpy/blob/master/AUTHORS.md) file for
|
|
115
|
+
more information.
|
|
116
|
+
|
|
117
|
+
License
|
|
118
|
+
-------
|
|
119
|
+
This software is licensed under the [BSD License](http://www.opensource.org/licenses/bsd-license).
|
|
120
|
+
See the included [LICENSE.md](https://github.com/lebedov/msgpack-numpy/blob/master/LICENSE.md) file for
|
|
121
|
+
more information.
|
|
122
|
+
|
|
123
|
+
See Also
|
|
124
|
+
--------
|
|
125
|
+
[msgpack-numpy-js](https://github.com/emerald-geomodelling/msgpack-numpy-js) by
|
|
126
|
+
[Egil Möller](https://github.com/redhog) is a JavaScript library that can
|
|
127
|
+
serialize and deserialize JS typed arrays with msgpack.
|
|
128
|
+
|
|
129
|
+
[msgpack-numpy-rs](https://github.com/clearviewai/msgpack-numpy-rs) by [
|
|
130
|
+
Terrence Liu](https://github.com/terencezl) is a Rust crate (mostly) compatible with msgpack-numpy
|
|
131
|
+
that can serialize and deserialize numpy arrays with msgpack.
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: b10-msgpack-numpy
|
|
3
|
+
Version: 0.4.8
|
|
4
|
+
Summary: Numpy data serialization using msgpack (Baseten fork with CWE-502 fix)
|
|
5
|
+
Home-page: https://github.com/basetenlabs/b10-msgpack-numpy
|
|
6
|
+
Author: Lev E. Givon
|
|
7
|
+
Author-email: lev@columbia.edu
|
|
8
|
+
License: BSD
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: License :: OSI Approved :: BSD License
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python
|
|
15
|
+
Classifier: Programming Language :: Python :: 2.7
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.4
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.5
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.6
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
22
|
+
Classifier: Topic :: Scientific/Engineering
|
|
23
|
+
Classifier: Topic :: Software Development
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE.md
|
|
26
|
+
License-File: AUTHORS.md
|
|
27
|
+
Requires-Dist: numpy>=1.9.0
|
|
28
|
+
Requires-Dist: msgpack>=0.5.2
|
|
29
|
+
Dynamic: author
|
|
30
|
+
Dynamic: author-email
|
|
31
|
+
Dynamic: classifier
|
|
32
|
+
Dynamic: description
|
|
33
|
+
Dynamic: description-content-type
|
|
34
|
+
Dynamic: home-page
|
|
35
|
+
Dynamic: license
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
Dynamic: requires-dist
|
|
38
|
+
Dynamic: summary
|
|
39
|
+
|
|
40
|
+
Package Description
|
|
41
|
+
-------------------
|
|
42
|
+
This package provides encoding and decoding routines that enable the
|
|
43
|
+
serialization and deserialization of numerical and array data types provided by
|
|
44
|
+
[numpy](http://www.numpy.org/) using the highly efficient
|
|
45
|
+
[msgpack](http://msgpack.org/) format. Serialization of Python's
|
|
46
|
+
native complex data types is also supported.
|
|
47
|
+
|
|
48
|
+
[](https://pypi.python.org/pypi/msgpack-numpy)
|
|
49
|
+
[](https://travis-ci.org/lebedov/msgpack-numpy)
|
|
50
|
+
|
|
51
|
+
Installation
|
|
52
|
+
------------
|
|
53
|
+
msgpack-numpy requires msgpack-python and numpy. If you
|
|
54
|
+
have [pip](http://www.pip-installer.org/) installed on your
|
|
55
|
+
system, run
|
|
56
|
+
|
|
57
|
+
pip install msgpack-numpy
|
|
58
|
+
|
|
59
|
+
to install the package and all dependencies. You can also download
|
|
60
|
+
the source tarball, unpack it, and run
|
|
61
|
+
|
|
62
|
+
python setup.py install
|
|
63
|
+
|
|
64
|
+
from within the source directory.
|
|
65
|
+
|
|
66
|
+
Usage
|
|
67
|
+
-----
|
|
68
|
+
The easiest way to use msgpack-numpy is to call its monkey patching
|
|
69
|
+
function after importing the Python msgpack package:
|
|
70
|
+
|
|
71
|
+
import msgpack
|
|
72
|
+
import msgpack_numpy as m
|
|
73
|
+
m.patch()
|
|
74
|
+
|
|
75
|
+
This will automatically force all msgpack serialization and deserialization
|
|
76
|
+
routines (and other packages that use them) to become numpy-aware.
|
|
77
|
+
Of course, one can also manually pass the encoder and
|
|
78
|
+
decoder provided by msgpack-numpy to the msgpack routines:
|
|
79
|
+
|
|
80
|
+
import msgpack
|
|
81
|
+
import msgpack_numpy as m
|
|
82
|
+
import numpy as np
|
|
83
|
+
|
|
84
|
+
x = np.random.rand(5)
|
|
85
|
+
x_enc = msgpack.packb(x, default=m.encode)
|
|
86
|
+
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)
|
|
87
|
+
|
|
88
|
+
msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
|
|
89
|
+
If msgpack was not compiled with Cython (or if the ``MSGPACK_PUREPYTHON``
|
|
90
|
+
variable is set), it will fall back to using the slower pure Python msgpack
|
|
91
|
+
implementation.
|
|
92
|
+
|
|
93
|
+
Notes
|
|
94
|
+
-----
|
|
95
|
+
The primary design goal of msgpack-numpy is ensuring preservation of numerical
|
|
96
|
+
data types during msgpack serialization and deserialization. Inclusion of type
|
|
97
|
+
information in the serialized data necessarily incurs some storage overhead; if
|
|
98
|
+
preservation of type information is not needed, one may be able to avoid some
|
|
99
|
+
of this overhead by writing a custom encoder/decoder pair that produces more
|
|
100
|
+
efficient serializations for those specific use cases.
|
|
101
|
+
|
|
102
|
+
Numpy arrays with a dtype of 'O' are serialized using pickle as a fallback
|
|
103
|
+
solution. Deserializing them requires unpickling, which executes arbitrary
|
|
104
|
+
code from the payload, so decoding is **disabled by default** (CWE-502):
|
|
105
|
+
``decode``, ``unpackb``/``unpack`` and ``Unpacker`` raise
|
|
106
|
+
``msgpack_numpy.PickleNotAllowed`` when they encounter an object-dtype array.
|
|
107
|
+
If you control both ends, opt in explicitly:
|
|
108
|
+
|
|
109
|
+
x_rec = msgpack.unpackb(x_enc, allow_pickle=True)
|
|
110
|
+
|
|
111
|
+
Never enable ``allow_pickle`` for untrusted input. As the overhead of pickle
|
|
112
|
+
serialization negates one of the reasons to use msgpack, consider writing a
|
|
113
|
+
custom encoder/decoder for your specific use case instead.
|
|
114
|
+
|
|
115
|
+
Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied
|
|
116
|
+
if they are to be modified.
|
|
117
|
+
|
|
118
|
+
Development
|
|
119
|
+
-----------
|
|
120
|
+
The latest source code can be obtained from
|
|
121
|
+
[GitHub](https://github.com/basetenlabs/b10-msgpack-numpy/). This is the Baseten
|
|
122
|
+
fork of [lebedov/msgpack-numpy](https://github.com/lebedov/msgpack-numpy); the
|
|
123
|
+
distribution is named ``b10-msgpack-numpy`` and still provides the
|
|
124
|
+
``msgpack_numpy`` module.
|
|
125
|
+
|
|
126
|
+
msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.
|
|
127
|
+
|
|
128
|
+
Install [`tox`](https://tox.readthedocs.io/en/latest/) to support testing
|
|
129
|
+
across multiple python versions in your development environment. If you
|
|
130
|
+
use [`conda`](https://docs.conda.io/en/latest/) to install `python` use
|
|
131
|
+
[`tox-conda`](https://github.com/tox-dev/tox-conda) to automatically manage
|
|
132
|
+
testing across all supported python versions.
|
|
133
|
+
|
|
134
|
+
# Using a system python
|
|
135
|
+
pip install tox
|
|
136
|
+
|
|
137
|
+
# Additionally, using a conda-provided python
|
|
138
|
+
pip install tox tox-conda
|
|
139
|
+
|
|
140
|
+
Execute tests across supported python versions:
|
|
141
|
+
|
|
142
|
+
tox
|
|
143
|
+
|
|
144
|
+
Authors
|
|
145
|
+
-------
|
|
146
|
+
See the included [AUTHORS.md](https://github.com/lebedov/msgpack-numpy/blob/master/AUTHORS.md) file for
|
|
147
|
+
more information.
|
|
148
|
+
|
|
149
|
+
License
|
|
150
|
+
-------
|
|
151
|
+
This software is licensed under the [BSD License](http://www.opensource.org/licenses/bsd-license).
|
|
152
|
+
See the included [LICENSE.md](https://github.com/lebedov/msgpack-numpy/blob/master/LICENSE.md) file for
|
|
153
|
+
more information.
|
|
154
|
+
|
|
155
|
+
See Also
|
|
156
|
+
--------
|
|
157
|
+
[msgpack-numpy-js](https://github.com/emerald-geomodelling/msgpack-numpy-js) by
|
|
158
|
+
[Egil Möller](https://github.com/redhog) is a JavaScript library that can
|
|
159
|
+
serialize and deserialize JS typed arrays with msgpack.
|
|
160
|
+
|
|
161
|
+
[msgpack-numpy-rs](https://github.com/clearviewai/msgpack-numpy-rs) by [
|
|
162
|
+
Terrence Liu](https://github.com/terencezl) is a Rust crate (mostly) compatible with msgpack-numpy
|
|
163
|
+
that can serialize and deserialize numpy arrays with msgpack.
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
AUTHORS.md
|
|
2
|
+
CHANGES.md
|
|
3
|
+
LICENSE.md
|
|
4
|
+
README.md
|
|
5
|
+
msgpack_numpy.py
|
|
6
|
+
pyproject.toml
|
|
7
|
+
setup.cfg
|
|
8
|
+
setup.py
|
|
9
|
+
tests.py
|
|
10
|
+
b10_msgpack_numpy.egg-info/PKG-INFO
|
|
11
|
+
b10_msgpack_numpy.egg-info/SOURCES.txt
|
|
12
|
+
b10_msgpack_numpy.egg-info/dependency_links.txt
|
|
13
|
+
b10_msgpack_numpy.egg-info/requires.txt
|
|
14
|
+
b10_msgpack_numpy.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|