numcodecs-random-projection 0.1.0a1__tar.gz

numcodecs_random_projection-0.1.0a1/.gitignore

@@ -0,0 +1,14 @@
+/.venv
+/.mypy_cache
+/.pytest_cache
+/.ruff_cache
+/uv.lock
+/test_implementation.py
+RPAnalysis.ipynb
+.ipynb_checkpoints/
+
+/build
+/site
+
+**/__pycache__
+**/*.egg-info

numcodecs_random_projection-0.1.0a1/LICENSE

@@ -0,0 +1,375 @@
+Copyright (c) 2025-2026, Sami Sidoroff and Juniper Tyree
+
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.

numcodecs_random_projection-0.1.0a1/PKG-INFO

@@ -0,0 +1,64 @@
+Metadata-Version: 2.4
+Name: numcodecs-random-projection
+Version: 0.1.0a1
+Summary: Random projection codec for the `numcodecs` buffer compression API
+Project-URL: Documentation, https://sidorow.github.io/numcodecs-random-projection/
+Project-URL: Repository, https://github.com/Sidorow/numcodecs-random-projection.git
+Project-URL: Issues, https://github.com/Sidorow/numcodecs-random-projection/issues
+Author-email: Sami Sidoroff <sami.sidoroff@helsinki.fi>, Juniper Tyree <juniper.tyree@helsinki.fi>
+Maintainer-email: Sami Sidoroff <sami.sidoroff@helsinki.fi>, Juniper Tyree <juniper.tyree@helsinki.fi>
+License-Expression: MPL-2.0
+License-File: LICENSE
+Keywords: compression,lossy-compression,numcodecs,random-projection
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Archiving :: Compression
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: leb128~=1.0.8
+Requires-Dist: numcodecs<0.17,>=0.13.0
+Requires-Dist: numpy~=2.0
+Requires-Dist: tqdm~=4.60
+Requires-Dist: types-tqdm~=4.60
+Requires-Dist: typing-extensions~=4.6
+Provides-Extra: dev
+Requires-Dist: mypy~=1.14; extra == 'dev'
+Requires-Dist: pytest~=8.3; extra == 'dev'
+Provides-Extra: test
+Requires-Dist: scipy>=1.15.3; extra == 'test'
+Requires-Dist: xarray>=2025.6.1; extra == 'test'
+Description-Content-Type: text/markdown
+
+[![image](https://img.shields.io/github/actions/workflow/status/Sidorow/numcodecs-random-projection/ci.yml?branch=main)](https://github.com/Sidorow/numcodecs-random-projection/actions/workflows/ci.yml?query=branch%3Amain)
+[![image](https://img.shields.io/pypi/v/numcodecs-random-projection.svg)](https://pypi.python.org/pypi/numcodecs-random-projection)
+[![image](https://img.shields.io/pypi/l/numcodecs-random-projection.svg)](https://github.com/Sidorow/numcodecs-random-projection/blob/main/LICENSE)
+[![image](https://img.shields.io/pypi/pyversions/numcodecs-random-projection.svg)](https://pypi.python.org/pypi/numcodecs-random-projection)
+[![image](https://img.shields.io/github/actions/workflow/status/Sidorow/numcodecs-random-projection/docs.yml?branch=main&label=docs)](https://sidorow.github.io/numcodecs-random-projection/)
+
+# numcodecs-random-projection
+
+`RPCodec` for the [`numcodecs`] buffer compression API.
+
+[`numcodecs`]: https://numcodecs.readthedocs.io/en/stable/
+
+## License
+
+Licensed under the Mozilla Public License, Version 2.0 ([LICENSE](LICENSE) or https://www.mozilla.org/en-US/MPL/2.0/).
+
+
+## Funding
+
+The `numcodecs-random-projection` package has been developed as part of [ESiWACE3](https://www.esiwace.eu), the third phase of the Centre of Excellence in Simulation of Weather and Climate in Europe.
+
+Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093054.

numcodecs_random_projection-0.1.0a1/README.md

@@ -0,0 +1,22 @@
+[![image](https://img.shields.io/github/actions/workflow/status/Sidorow/numcodecs-random-projection/ci.yml?branch=main)](https://github.com/Sidorow/numcodecs-random-projection/actions/workflows/ci.yml?query=branch%3Amain)
+[![image](https://img.shields.io/pypi/v/numcodecs-random-projection.svg)](https://pypi.python.org/pypi/numcodecs-random-projection)
+[![image](https://img.shields.io/pypi/l/numcodecs-random-projection.svg)](https://github.com/Sidorow/numcodecs-random-projection/blob/main/LICENSE)
+[![image](https://img.shields.io/pypi/pyversions/numcodecs-random-projection.svg)](https://pypi.python.org/pypi/numcodecs-random-projection)
+[![image](https://img.shields.io/github/actions/workflow/status/Sidorow/numcodecs-random-projection/docs.yml?branch=main&label=docs)](https://sidorow.github.io/numcodecs-random-projection/)
+
+# numcodecs-random-projection
+
+`RPCodec` for the [`numcodecs`] buffer compression API.
+
+[`numcodecs`]: https://numcodecs.readthedocs.io/en/stable/
+
+## License
+
+Licensed under the Mozilla Public License, Version 2.0 ([LICENSE](LICENSE) or https://www.mozilla.org/en-US/MPL/2.0/).
+
+
+## Funding
+
+The `numcodecs-random-projection` package has been developed as part of [ESiWACE3](https://www.esiwace.eu), the third phase of the Centre of Excellence in Simulation of Weather and Climate in Europe.
+
+Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093054.

numcodecs_random_projection-0.1.0a1/pyproject.toml

@@ -0,0 +1,78 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "numcodecs-random-projection"
+version = "0.1.0a1"
+dependencies = [
+    "leb128~=1.0.8",
+    "numcodecs>=0.13.0,<0.17",
+    "numpy~=2.0",
+    "typing-extensions~=4.6",
+    "tqdm~=4.60",
+    "types-tqdm~=4.60",
+]
+requires-python = ">=3.10"
+authors = [{ name = "Sami Sidoroff", email = "sami.sidoroff@helsinki.fi" }, { name = "Juniper Tyree", email = "juniper.tyree@helsinki.fi" }]
+maintainers = [{ name = "Sami Sidoroff", email = "sami.sidoroff@helsinki.fi" }, { name = "Juniper Tyree", email = "juniper.tyree@helsinki.fi" }]
+description = "Random projection codec for the `numcodecs` buffer compression API"
+readme = "README.md"
+license = "MPL-2.0"
+keywords = ["random-projection", "compression", "lossy-compression", "numcodecs"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: End Users/Desktop",
+    "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
+    "Natural Language :: English",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering",
+    "Topic :: System :: Archiving :: Compression",
+    "Typing :: Typed",
+]
+
+[project.urls]
+Documentation = "https://sidorow.github.io/numcodecs-random-projection/"
+Repository = "https://github.com/Sidorow/numcodecs-random-projection.git"
+Issues = "https://github.com/Sidorow/numcodecs-random-projection/issues"
+
+[project.optional-dependencies]
+dev = [
+    "mypy~=1.14",
+    "pytest~=8.3",
+]
+
+test = [
+    "scipy>=1.15.3",
+    "xarray>=2025.6.1",
+]
+
+[project.entry-points."numcodecs.codecs"]
+"rp" = "numcodecs_random_projection:RPCodec"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"numcodecs_random_projection" = ["py.typed"]
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "I"]
+
+[tool.pytest.ini_options]
+addopts = ["--import-mode=importlib"]
+xfail_strict = true
+
+[[tool.mypy.overrides]]
+module = ["leb128.*", "numcodecs.*"]
+follow_untyped_imports = true
+
+[tool.hatch.build.targets.sdist]
+include = ["/src", "/LICENSE", "/pyproject.toml", "/README.md"]

numcodecs_random_projection-0.1.0a1/src/numcodecs_random_projection/__init__.py

@@ -0,0 +1,629 @@
+"""
+[`RPCodec`][numcodecs_random_projection.RPCodec] for the [`numcodecs`][numcodecs] buffer compression API.
+"""
+
+__all__ = ["RPCodec", "RPMethod"]
+
+import logging
+import time
+from contextlib import contextmanager
+from enum import Enum
+from io import BytesIO
+from math import ceil
+from sys import byteorder
+
+import leb128
+import numcodecs.compat
+import numcodecs.registry
+import numpy as np
+import tqdm
+from numcodecs.abc import Codec
+from typing_extensions import (
+    Buffer,  # MSPV 3.12
+    assert_never,  # MSPV 3.11
+)
+
+from .mt_rng import MultithreadedRNG
+
+LOG = logging.getLogger(__name__)
+
+_BLOCK_THRESHOLD = 1000
+_BLOCK_SIZE = 512
+
+
+class RPMethod(Enum):
+    """Random projection method."""
+
+    dct = "dct"
+    """
+    Discrete Cosine Transform (DCT).
+
+    Generate DxK projection matrix R using Type II Discrete Cosine Transform (DCT) basis ¹.
+    """
+    gaussian = "gaussian"
+    """
+    Gaussian random projection.
+
+    Generate random DxK matrix R with entries drawn from N(0, 1/√K) distribution,
+    which preserves expected distances according to Johnson-Lindenstrauss lemma.
+
+
+    [^1]:   José J. Amador,
+            Random projection and orthonormality for lossy image compression, Image and Vision Computing, Volume 25, Issue 5, 2007, Pages 754-766, ISSN 0262-8856, Available from:
+            [https://doi.org/10.1016/j.imavis.2006.05.018](https://doi.org/10.1016/j.imavis.2006.05.018)
+    """
+
+
+class RPCodec(Codec):
+    """
+    Random projection codec for lossy compression of numerical data.
+
+    Compresses 2D finite floating point data by projecting it onto a lower-dimensional subspace using a specified method.
+    Discrete Cosine Transform (DCT) is used by default.
+
+    A two-dimensional array of shape N x D is encoded as an array of
+    shape N x K, where `k` is either set explicitly or chosen with compression ratio `cr`.
+    Alternatively, `k` can be estimated from the data during encoding by giving a specified Mean Absolute Error (MAE) during initialization.
+
+    """
+
+    __slots__ = ("_mae", "_cr", "_k", "_method", "_seed", "_debug")
+    _mae: None | float
+    _cr: None | float
+    _k: None | int
+    _method: RPMethod
+    _seed: int
+    _debug: bool
+
+    codec_id: str = "rp"  # type: ignore
+
+    def __init__(
+        self,
+        mae: None | float = None,
+        cr: None | float = None,
+        k: None | int = None,
+        method: str | RPMethod = RPMethod.dct,
+        seed: int | None = None,
+        debug: bool = False,
+    ) -> None:
+        """
+        Initialize Random Projection codec.
+
+        Parameters
+        ----------
+        mae : float
+            Target mean absolute error. If specified, `k` will be estimated from
+            data during encoding. Note that the bound is *not* guaranteed to be
+            met.
+        cr : float
+            Target compression ratio. If specified, `k` will be calculated as D/`cr`
+            where D is the number of features in the input data.
+        k : int
+            Number of dimensions in the projected space. Will be used over `cr` if
+            both are specified. Estimated if `mae` is specified.
+        method : str | RPMethod
+            Method for generating the projection matrix. Please refer to the
+            [`RPMethod`][numcodecs_random_projection.RPMethod] enumeration for
+            all supported methods.
+        seed : int
+            Random seed for reproducible results. If None, results will be
+            non-deterministic when using the Gaussian method.
+        debug : bool
+            Whether debug information should be printed during encoding and
+            decoding.
+
+        Raises
+        ------
+        ValueError
+            If not exactly one of `mae`, `cr`, or `k` is set.
+        """
+
+        if sum([(mae is not None), (cr is not None), (k is not None)]) != 1:
+            raise ValueError("exactly one of `mae`, `cr` or `k` must be set")
+
+        self._mae = mae
+        self._cr = cr
+        self._k = k
+
+        try:
+            self._method = method if isinstance(method, RPMethod) else RPMethod[method]
+        except KeyError:
+            hy = "'"
+            raise ValueError(
+                f"unknown method '{method}', expected one of {', '.join(f'{hy}{m.name}{hy}' for m in RPMethod)}."
+            )
+
+        if seed is None:
+            self._seed = np.random.randint(0, 2**31 - 1)
+        else:
+            self._seed = seed
+
+        self._debug = debug
+
+    def encode(self, buf: Buffer) -> Buffer:
+        """
+        Encode data using random projection.
+
+        During encode, the input data is standardized (mean=0, std=1) before projection.
+
+        If `mae` is specified, the number of projected dimensions `k` is estimated based on the standardized data.
+
+        Parameters
+        ----------
+        buf : Buffer
+            Input data buffer. Must be a 2D array with shape (n_samples, d_features).
+
+        Returns
+        -------
+        enc : bytes
+            Serialized encoded data containing:
+            - Standardized data statistics (mean, std)
+            - Original data shape and dtype
+            - Projected data
+            - Compression parameters
+        """
+        data = numcodecs.compat.ensure_ndarray(buf)
+
+        validations = [
+            not np.issubdtype(data.dtype, np.floating),
+            data.ndim != 2,
+        ]
+
+        if any(validations):
+            raise ValueError(
+                f"RPCodec requires 2D floating-point data, got {data.dtype} and {data.ndim}D data"
+            )
+
+        np.nan_to_num(data, copy=False, nan=0.0, posinf=0.0, neginf=0.0)
+
+        data_mean = np.mean(data)
+        data_std = np.std(data)
+        if data_std == 0:
+            data_std = 1
+
+        standardized_data = (data - data_mean) / data_std
+
+        original_shape = data.shape
+        original_dtype = data.dtype
+
+        k: int
+        if self._mae is not None:
+            k = self._estimate_k_for_target_mae(standardized_data)
+        elif self._cr is not None:
+            assert self._cr is not None
+            k = int(ceil(data.shape[1] / self._cr))
+        else:
+            assert self._k is not None
+            k = self._k
+
+        if self._debug:
+            LOG.debug(f"encode with k={k}")
+
+        if k > _BLOCK_THRESHOLD:
+            block_size = _BLOCK_SIZE
+            projected = self._project_blocks(
+                standardized_data, data.shape[1], k, original_dtype, block_size
+            )
+        else:
+            R = self._gen_R(data.shape[1], k, original_dtype, self._seed)
+            projected = np.matmul(standardized_data, R)
+
+        bio = BytesIO()
+
+        bio.write(leb128.u.encode(len(original_shape)))
+        for dim in original_shape:
+            bio.write(leb128.u.encode(dim))
+
+        dtype_str = original_dtype.str.encode("ascii")
+        bio.write(leb128.u.encode(len(dtype_str)))
+        bio.write(dtype_str)
+
+        bio.write(leb128.u.encode(k))
+        bio.write(leb128.u.encode(self._seed))
+
+        mean_bytes = np.array(data_mean, dtype=original_dtype).tobytes()
+        bio.write(leb128.u.encode(len(mean_bytes)))
+        bio.write(mean_bytes)
+
+        std_bytes = np.array(data_std, dtype=original_dtype).tobytes()
+        bio.write(leb128.u.encode(len(std_bytes)))
+        bio.write(std_bytes)
+
+        projected_byteorder = projected.dtype.byteorder
+
+        projected_byteorder = (
+            projected_byteorder
+            if projected_byteorder in ("<", ">")
+            else ("<" if (byteorder == "little") else ">")
+        )
+
+        if projected_byteorder != "<":
+            projected = projected.byteswap()
+
+        proj_bytes = projected.tobytes()
+        bio.write(leb128.u.encode(len(proj_bytes)))
+        bio.write(proj_bytes)
+
+        return bio.getvalue()
+
+    def decode(self, buf: Buffer, out: None | Buffer = None) -> Buffer:
+        """
+        Decode random projection encoded data.
+
+        During decode, the standardized data is reconstructed and denormalized.
+
+        Parameters
+        ----------
+        buf : Buffer
+            Encoded data from RPCodec.
+        out : Buffer, optional
+            Writeable buffer to store decoded data.
+
+        Returns
+        -------
+        dec : Buffer
+            Reconstructed data with original shape and dtype.
+        """
+
+        data = numcodecs.compat.ensure_bytes(buf)
+
+        bio = BytesIO(data)
+
+        ndim, _ = leb128.u.decode_reader(bio)
+        original_shape = tuple(leb128.u.decode_reader(bio)[0] for _ in range(ndim))
+
+        dtype_len, _ = leb128.u.decode_reader(bio)
+        dtype_str = bio.read(dtype_len).decode("ascii")
+        original_dtype = np.dtype(dtype_str)
+
+        k, _ = leb128.u.decode_reader(bio)
+        seed, _ = leb128.u.decode_reader(bio)
+
+        mean_len, _ = leb128.u.decode_reader(bio)
+        mean_bytes = bio.read(mean_len)
+        data_mean = np.frombuffer(mean_bytes, dtype=original_dtype)
+
+        std_len, _ = leb128.u.decode_reader(bio)
+        std_bytes = bio.read(std_len)
+        data_std = np.frombuffer(std_bytes, dtype=original_dtype)
+
+        proj_len, _ = leb128.u.decode_reader(bio)
+        proj_bytes = bio.read(proj_len)
+
+        projected = np.frombuffer(
+            proj_bytes, dtype=original_dtype.newbyteorder("<")
+        ).reshape((original_shape[0], k))
+
+        projected_byteorder = projected.dtype.byteorder
+
+        projected_byteorder = (
+            projected_byteorder
+            if projected_byteorder in ("<", ">")
+            else ("<" if (byteorder == "little") else ">")
+        )
+
+        if byteorder == "big":
+            projected = projected.byteswap()
+
+        if k > _BLOCK_THRESHOLD:
+            block_size = _BLOCK_SIZE
+            reconstructed = self._reconstruct_blocks(
+                projected, original_shape[1], k, original_dtype, block_size, seed
+            )
+        else:
+            R = self._gen_R(original_shape[1], k, original_dtype, seed)
+            reconstructed = np.matmul(projected, R.T)
+
+        reconstructed = reconstructed * data_std + data_mean
+        reconstructed = reconstructed.reshape(original_shape)
+        return numcodecs.compat.ndarray_copy(reconstructed, out)  # type: ignore
+
+    def get_config(self) -> dict:
+        """
+        Get codec configuration.
+
+        Returns:
+            dict: Codec configuration.
+        """
+        config: dict[str, str | int | float] = dict(id=type(self).codec_id)
+
+        if self._mae is not None:
+            config["mae"] = self._mae
+        if self._cr is not None:
+            config["cr"] = self._cr
+        if self._k is not None:
+            config["k"] = self._k
+
+        config["method"] = self._method.name
+        config["seed"] = self._seed
+
+        return config
+
+    def _estimate_k_for_target_mae(
+        self,
+        data: np.ndarray,
+    ) -> int:
+        """
+        Estimate the number of dimensions 'k' for the projected space based on the standardized
+        input data and targeted MAE.
+
+        This method assumes standardized input data prior to calling via encode method.
+
+        Parameters
+        ----------
+        data : np.ndarray
+            Standardized input data (mean=0, std=1).
+
+        Returns
+        -------
+        int
+            Estimated k (number of projected dimensions)
+        """
+
+        D = data.shape[1]
+
+        assert self._mae is not None
+        target_mae = self._mae
+        # normalized_mae = self._mae / data_std if data_std > 0 else self._mae
+
+        match self._method:
+            case RPMethod.gaussian:
+                ratio = 1 - target_mae
+            case RPMethod.dct:
+                ratio = 1 - np.sqrt(target_mae * 4)
+            case _:
+                assert_never(self._method)
+
+        estimated_k = int(D * ratio)
+        K = max(1, min(estimated_k, D))
+
+        return K
+
+    def _project_blocks(
+        self, data: np.ndarray, D: int, K: int, dtype: np.dtype, block_size: int
+    ) -> np.ndarray:
+        """
+        Project input data to a lower-dimensional subspace using block-wise matrix generation.
+        Processes projection matrix R in blocks of shape (D, block_size) instead of generating the full DxK matrix to reduce memory usage when K is large.
+
+        Note that due to how blocks are processed and projection matrix is generated, the output is not the same as if the full matrix was used (see _gen_R_block notes).
+
+        Parameters
+        ----------
+        data : np.ndarray
+            Input data array with shape (N, D), where N is the number of samples
+            and D is the number of input features.
+        D : int
+            Number of input features (columns in data).
+        K : int
+            Number of dimensions in the projected space.
+        dtype : np.dtype
+            Data type for the projection matrix and output. Should match the
+            original data dtype for consistency.
+        block_size : int
+            Number of features to process in each block. Determines the
+            size of each R_block as (D, block_size).
+
+        Returns
+        -------
+        np.ndarray
+            Projected data with shape (N, K)
+        """
+
+        rng = MultithreadedRNG(seed=self._seed)
+
+        out = np.empty((data.shape[0], K), dtype=dtype)
+        R_block = None
+        out_block = None
+
+        if self._debug:
+            progress = tqdm.tqdm(total=K)
+
+        for k_start in range(0, K, block_size):
+            k_end = min(k_start + block_size, K)
+            actual_block_size = k_end - k_start
+
+            if R_block is None or R_block.shape != (D, actual_block_size):
+                R_block = np.empty((D, actual_block_size), dtype=dtype)
+
+            if out_block is None or out_block.shape != (
+                data.shape[0],
+                actual_block_size,
+            ):
+                out_block = np.empty((data.shape[0], actual_block_size), dtype=dtype)
+
+            block_timing = [0.0]
+            with self._debug_timing(block_timing):
+                self._gen_R_block(K, k_start, rng, out=R_block)
+
+            matmul_timing = [0.0]
+            with self._debug_timing(matmul_timing):
+                np.matmul(data, R_block, out=out_block)
+                out[:, k_start:k_end] = out_block
+
+            if self._debug:
+                progress.set_postfix_str(
+                    f"encode N={data.shape[0]} D={D} Kb={actual_block_size} Rgen={np.round(block_timing[0], 2)}s matmul={np.round(matmul_timing[0], 2)}s"
+                )
+                progress.update(actual_block_size)
+
+        return out
+
+    def _reconstruct_blocks(
+        self,
+        projected: np.ndarray,
+        D: int,
+        K: int,
+        dtype: np.dtype,
+        block_size: int,
+        seed: int,
+    ) -> np.ndarray:
+        """
+        Reconstruct data using block-wise matrix generation.
+
+        Performs the inverse operation of _project_blocks by computing projected @ R.T in blocks to reduce memory usage. Accumulate each processed block and return the full reconstructed matrix of shape (N, D).
+
+        Parameters
+        ----------
+        projected : np.ndarray
+            Projected data array with shape (N, K), where N is the number of samples
+            and K is the number of projected features.
+        D : int
+            Number of input features (columns in data).
+        K : int
+            Number of dimensions in the projected space.
+        dtype : np.dtype
+            Data type for the reconstructed matrix. Should match the
+            original data dtype for consistency.
+        block_size : int
+            Number of features to process in each block.
+
+        Returns
+        -------
+        np.ndarray
+            Reconstructed data with shape (N, D)
+        """
+
+        rng = MultithreadedRNG(seed=seed)
+
+        out = np.zeros((projected.shape[0], D), dtype=dtype)
+        R_block = None
+        rec_block = np.empty((projected.shape[0], D), dtype=dtype)
+
+        if self._debug:
+            progress = tqdm.tqdm(total=K)
+
+        for k_start in range(0, K, block_size):
+            k_end = min(k_start + block_size, K)
+            actual_block_size = k_end - k_start
+
+            if R_block is None or R_block.shape != (D, actual_block_size):
+                R_block = np.empty((D, actual_block_size), dtype=dtype)
+
+            block_timing = [0.0]
+            with self._debug_timing(block_timing):
+                self._gen_R_block(K, k_start, rng, out=R_block)
+
+            matmul_timing = [0.0]
+            with self._debug_timing(matmul_timing):
+                projected_block = projected[:, k_start:k_end]
+                np.matmul(projected_block, R_block.T, out=rec_block)
+
+            acc_timing = [0.0]
+            with self._debug_timing(acc_timing):
+                out += rec_block
+
+            if self._debug:
+                progress.set_postfix_str(
+                    f"decode N={out.shape[0]} D={D} Kb={actual_block_size} Rgen={np.round(block_timing[0], 2)}s matmul={np.round(matmul_timing[0], 2)}s acc={np.round(acc_timing[0], 2)}s"
+                )
+                progress.update(actual_block_size)
+
+        return out
+
+    def _gen_R(
+        self, D: int, K: int, dtype: np.dtype, seed: int | None = None
+    ) -> np.ndarray:
+        """
+        Generate a projection matrix using specified method.
+
+        DCT method:
+            Generates a DxK projection matrix R using Type II Discrete Cosine Transform (DCT) basis.
+
+        Gaussian method:
+            Creates a random DxK matrix R with entries drawn from N(0, 1/√K) distribution,
+            which preserves expected distances according to Johnson-Lindenstrauss lemma.
+
+        Parameters
+        ----------
+        D : int
+            Input dimensionality (number of features).
+        K : int
+            Output dimensionality (number of projected features).
+        seed : int, optional
+            Random seed of reproducible matrix generation.
+
+        Returns
+        -------
+        np.ndarray
+            Projection matrix of shape (D, K)
+        """
+
+        match self._method:
+            case RPMethod.dct:
+                i = np.arange(D, dtype=dtype).reshape(-1, 1)
+                m = np.arange(K, dtype=dtype).reshape(1, -1)
+                alpha_m = np.where(m == 0, np.sqrt(1 / D), np.sqrt(2 / D))
+                R = alpha_m * np.cos((np.pi * (2 * i + 1) * m) / (2 * D))
+            case RPMethod.gaussian:
+                scale = np.sqrt(1 / K)
+                rng = MultithreadedRNG(seed=seed)
+                rng.fill_arr(shape=(D, K))
+                R = rng.values * scale
+            case _:
+                assert_never(self._method)
+
+        return R.astype(dtype)
+
+    def _gen_R_block(
+        self,
+        K: int,
+        k_start: int,
+        rng: MultithreadedRNG,
+        out: np.ndarray,
+    ) -> None:
+        """
+        Generate a block of projection matrix R using a specified method.
+
+        Parameters
+        ----------
+        K : int
+            Number of projected features.
+        k_start : int
+            Starting index for the projected space.
+        rng : MultithreadedRNG
+            Random number generator based on https://numpy.org/doc/stable/reference/random/multithreading.html.
+        out : np.ndarray
+            Block of matrix R with shape (D, block_size) that will be filled
+            by this method.
+
+        Notes
+        -----
+        - Generating Gaussian R matrix block by block does not produce the same numbers as the Gen_R method.
+            This is because the random numbers are generated in chunks based on the block size,
+            so the sequence of random numbers used for each block is different than if the full matrix was generated at once.
+        """
+
+        D, block_size = out.shape
+        dtype = out.dtype
+
+        match self._method:
+            case RPMethod.dct:
+                i = np.arange(D, dtype=dtype).reshape(-1, 1)
+                m = np.arange(k_start, k_start + block_size, dtype=dtype).reshape(1, -1)
+                alpha_m = np.where(m == 0, np.sqrt(1 / D), np.sqrt(2 / D))
+                out[:] = 2 * i + 1
+                out[:] *= m * np.pi
+                out[:] /= 2 * D
+                np.cos(out, out=out)
+                out *= alpha_m
+            case RPMethod.gaussian:
+                scale = np.sqrt(1 / K)
+                if hasattr(rng, "fill_arr"):
+                    rng.fill_arr(shape=(D, block_size))
+                    np.multiply(rng.values, scale, out=out)
+            case _:
+                assert_never(self._method)
+
+    @contextmanager
+    def _debug_timing(self, out: list[float]):
+        if self._debug:
+            start = time.perf_counter()
+            yield
+            end = time.perf_counter()
+            out[0] = end - start
+        else:
+            yield
+
+
+numcodecs.registry.register_codec(RPCodec)

numcodecs_random_projection-0.1.0a1/src/numcodecs_random_projection/mt_rng.py

@@ -0,0 +1,88 @@
+import concurrent.futures
+import multiprocessing
+
+import numpy as np
+from numpy.random import SeedSequence, default_rng
+
+
+class MultithreadedRNG:
+    """
+    Multithreaded random number generator using numpy's default_rng
+    Based on https://numpy.org/doc/stable/reference/random/multithreading.html.
+    """
+
+    def __init__(self, seed, threads=None):
+        """
+        Initialize the multithreaded RNG.
+
+        Parameters
+        ----------
+        seed : int
+            The seed for the random number generator.
+        threads : int, optional
+            The number of threads to use. If None, uses the number of CPU cores.
+        """
+        if threads is None:
+            threads = multiprocessing.cpu_count()
+        self.threads = threads
+
+        self.seed_seq = SeedSequence(seed)
+        self.shape = None
+        self.values = None
+        self.step = None
+
+        self.executor = concurrent.futures.ThreadPoolExecutor(self.threads)
+
+    def fill_arr(self, shape: tuple[int, int]) -> np.ndarray:
+        """
+        Fill an array of given shape with random numbers in parallel using threads.
+
+        The number of RNG chunks is determined by the shape,
+        so results are reproducible regardless of the number of threads used.
+
+        Parameters
+        ----------
+        shape : tuple[int]
+            The shape of the array to fill.
+
+        Returns
+        -------
+        np.ndarray
+            The filled array with random floating-point numbers.
+        """
+        if isinstance(shape, int):
+            shape = (shape,)
+        self.shape = tuple(shape)
+        self.values = np.empty(self.shape)
+        n_rows = self.values.shape[0]
+
+        rows_per_chunk = shape[1]
+        n_chunks = max(1, int(np.ceil(n_rows / rows_per_chunk)))
+        chunk_step = int(np.ceil(n_rows / n_chunks))
+
+        child_seeds = self.seed_seq.spawn(n_chunks)
+
+        chunks = []
+        for i in range(n_chunks):
+            first = i * chunk_step
+            last = min((i + 1) * chunk_step, n_rows)
+            if first >= last:
+                break
+            chunks.append((i, first, last))
+
+        def _fill_chunk(chunk_idx, out, first, last):
+            rng = default_rng(child_seeds[chunk_idx])
+            view = out[first:last]
+            view[...] = rng.standard_normal(view.shape)
+
+        futures = [
+            self.executor.submit(_fill_chunk, idx, self.values, first, last)
+            for idx, first, last in chunks
+        ]
+
+        concurrent.futures.wait(futures)
+
+        return self.values
+
+    def __del__(self):
+        self.executor.shutdown(False)