swapcollection 1.0.0__tar.gz

swapcollection-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 renren5977
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

swapcollection-1.0.0/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: swapcollection
+Version: 1.0.0
+Summary: A library that automatically offloads large pickle-able objects stored in a dict/list to SQLite, reducing memory usage. It can be used with the same syntax as a regular dict/list.
+Author: llinghyami
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/llinghyami/swapcollection
+Project-URL: Repository, https://github.com/llinghyami/swapcollection.git
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqlite-utils
+Dynamic: license-file
+
+> **English** · [日本語 (Japanese)](README.md)
+
+# swapcollection
+
+A library that automatically offloads large objects stored in a
+`dict`/`list` to SQLite, reducing memory usage. It can be used with
+the same syntax as a regular `dict`/`list`.
+
+## Quick Start
+
+```bash
+pip install swapcollection
+```
+
+```python
+from swapcollection import SwapDict, SwapList
+
+# Spill values >= 1 MB (pickle size) to SQLite
+data = SwapDict(size_threshold=1024)
+
+data["small"] = b"hello"
+data["large"] = b"x" * 100_000
+
+# Access transparently — same as a normal dict
+print(data["small"])
+print(len(data["large"]))
+```
+
+## How It Works
+
+`SwapDict` / `SwapList` automatically offloads values whose pickled size
+meets or exceeds `size_threshold` to a SQLite database. Any pickle-able
+object (not just `bytes`) is eligible.
+
+| Value | Storage |
+|---|---|
+| Pickle size < `size_threshold` | In-memory |
+| Pickle size >= `size_threshold` | SQLite (transparent) |
+
+Spilled values are retrieved, updated, and deleted using the same `dict`/`list`
+interface — no special API calls needed.
+
+## Caveats
+
+- `size_threshold` is the pickle-size threshold in bytes; values >= this size
+  are spilled.
+- `size_threshold` cannot be changed after instantiation.
+- Any pickle-able object type (not only `bytes`) can be spilled.
+- Deleting the SQLite database will cause errors on previously spilled items.
+
+## License
+
+MIT

swapcollection-1.0.0/README.en.md

@@ -0,0 +1,53 @@
+> **English** · [日本語 (Japanese)](README.md)
+
+# swapcollection
+
+A library that automatically offloads large objects stored in a
+`dict`/`list` to SQLite, reducing memory usage. It can be used with
+the same syntax as a regular `dict`/`list`.
+
+## Quick Start
+
+```bash
+pip install swapcollection
+```
+
+```python
+from swapcollection import SwapDict, SwapList
+
+# Spill values >= 1 MB (pickle size) to SQLite
+data = SwapDict(size_threshold=1024)
+
+data["small"] = b"hello"
+data["large"] = b"x" * 100_000
+
+# Access transparently — same as a normal dict
+print(data["small"])
+print(len(data["large"]))
+```
+
+## How It Works
+
+`SwapDict` / `SwapList` automatically offloads values whose pickled size
+meets or exceeds `size_threshold` to a SQLite database. Any pickle-able
+object (not just `bytes`) is eligible.
+
+| Value | Storage |
+|---|---|
+| Pickle size < `size_threshold` | In-memory |
+| Pickle size >= `size_threshold` | SQLite (transparent) |
+
+Spilled values are retrieved, updated, and deleted using the same `dict`/`list`
+interface — no special API calls needed.
+
+## Caveats
+
+- `size_threshold` is the pickle-size threshold in bytes; values >= this size
+  are spilled.
+- `size_threshold` cannot be changed after instantiation.
+- Any pickle-able object type (not only `bytes`) can be spilled.
+- Deleting the SQLite database will cause errors on previously spilled items.
+
+## License
+
+MIT

swapcollection-1.0.0/README.md

@@ -0,0 +1,48 @@
+> [English (English)](README.en.md) · **日本語**
+
+# swapcollection
+
+辞書型やリストに格納された大容量のオブジェクトを自動的にSQLiteへ退避し、メモリ使用量を削減するライブラリです。通常の `dict` / `list` と同じ書き方で使用できます。
+
+## クイックスタート
+
+```bash
+pip install swapcollection
+```
+
+```python
+from swapcollection import SwapDict, SwapList
+
+# 1 MB以上の値をSQLiteへ退避
+data = SwapDict(size_threshold=1024)
+
+data["small"] = b"hello"
+data["large"] = b"x" * 100_000
+
+# 通常のdictと同じように取得
+print(data["small"])
+print(len(data["large"]))
+```
+
+## 仕組み
+
+`SwapDict` / `SwapList` は、格納された値を pickle したサイズが `size_threshold` 以上の場合に、値をSQLiteへ自動的に退避します。`bytes` に限らず、任意のpickle可能なオブジェクトが対象です。
+
+| 値 | 保存先 |
+|---|---|
+| pickleサイズ < `size_threshold` | メモリ |
+| pickleサイズ >= `size_threshold` | SQLite |
+
+SQLiteへ退避された値も、通常の `dict` / `list` と同じ方法で取得、更新、削除できます。
+
+## 注意事項
+
+- `size_threshold` は、SQLiteへ退避する pickle サイズの閾値をバイト単位で指定します。
+- サイズが `size_threshold` と等しい値もSQLiteへ退避されます。
+- `size_threshold` はインスタンス作成後に変更できません。
+- 値の種類（`bytes` 以外も含む）は pickle サイズでのみ判断されます。
+- SQLiteデータベースを削除すると、退避済みの値を取得できなくなり、エラーが発生します。
+
+## ライセンス
+
+MIT

swapcollection-1.0.0/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "swapcollection"
+version = "1.0.0"
+description = "A library that automatically offloads large pickle-able objects stored in a dict/list to SQLite, reducing memory usage. It can be used with the same syntax as a regular dict/list."
+readme = "README.en.md"
+license = "MIT"
+authors = [
+    {name = "llinghyami"},
+]
+requires-python = ">=3.10"
+dependencies = [
+    "sqlite-utils",
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+]
+
+[project.urls]
+Homepage = "https://github.com/llinghyami/swapcollection"
+Repository = "https://github.com/llinghyami/swapcollection.git"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["swapcollection*"]

swapcollection-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

swapcollection-1.0.0/swapcollection/__init__.py

@@ -0,0 +1,3 @@
+from .swapcollection import SwapDict, SwapList
+
+__all__ = ["SwapDict", "SwapList"]

swapcollection-1.0.0/swapcollection/swapcollection.py

@@ -0,0 +1,234 @@
+from sqlite_utils import Database
+from sqlite_utils.db import NotFoundError
+import uuid
+from collections import UserDict, UserList
+import pickle
+
+class SwapDict(UserDict):
+    PREFIX = "swapdict_"
+    PATH = "swapcollections_cache.db"
+        
+    def __init__(self, data={}, size_threshold: int = 1048576):
+        self.db = Database(self.PATH)
+        self.table = self.db["DICT"]
+        self.size_threshold = size_threshold
+
+        if not self.table.exists():
+            self.table.create(
+                {
+                    "id": str,
+                    "value": bytes,
+                },
+                pk="id",
+            )
+
+        super().__init__(data) # 内部で__setitem__を呼んで処理するので、ここでは何もしない
+
+    def _setprocess(self, value):
+        id = self.PREFIX + str(uuid.uuid4())
+        self.table.insert({"id": id, "value": pickle.dumps(value)})
+        return str(id)
+
+    def _getprocess(self, id):
+        result = self.table.get(id)
+        if result:
+            return pickle.loads(result["value"])
+        raise NotFoundError(f"id {id} not found")
+
+    def __setitem__(self, key, value):
+        if len(pickle.dumps(value)) >= self.size_threshold:
+            value = self._setprocess(value)
+        return super().__setitem__(key, value)
+
+    def __getitem__(self, key):
+        value = super().__getitem__(key)
+        if type(value) == str:
+            if value.startswith(self.PREFIX):
+                return self._getprocess(value)
+        return value
+
+    def __repr__(self):
+        self.repr_data = UserDict()
+        for key, value in self.items():
+            if type(value) == str:
+                if value.startswith(self.PREFIX):
+                    value = self._getprocess(value)
+            self.repr_data[key] = value
+        return repr(self.repr_data)
+
+class SwapList(UserList):
+    PREFIX = "swaplist_"
+    PATH = "swapcollections_cache.db"
+    
+    def __init__(self, data=None, size_threshold: int = 1048576):
+        self.db = Database(self.PATH)
+        self.table = self.db["LIST"]
+        self.size_threshold = size_threshold
+
+        if not self.table.exists():
+            self.table.create(
+                {
+                    "id": str,
+                    "value": bytes,
+                },
+                pk="id",
+            )
+
+        tmp_data = []
+        for i in (data or []):
+            if len(pickle.dumps(i)) >= self.size_threshold:
+                i = self._setprocess(i)
+            tmp_data.append(i)
+
+        super().__init__(tmp_data)
+
+    def _setprocess(self, value):
+        id = self.PREFIX + str(uuid.uuid4())
+        self.table.insert({"id": id, "value": pickle.dumps(value)})
+        return id
+
+    def _getprocess(self, id):
+        result = self.table.get(id)
+        if result:
+            return pickle.loads(result["value"])
+        raise NotFoundError(f"id {id} not found")
+
+    def __setitem__(self, i, value):
+        if isinstance(i, slice):
+            processed = []
+            for v in value:
+                if len(pickle.dumps(v)) >= self.size_threshold:
+                    v = self._setprocess(v)
+                processed.append(v)
+            return super().__setitem__(i, processed)
+        if len(pickle.dumps(value)) >= self.size_threshold:
+            value = self._setprocess(value)
+        return super().__setitem__(i, value)
+
+    def __eq__(self, other):
+        try:
+            return list(self) == list(other)
+        except TypeError:
+            return NotImplemented
+    
+    def __getitem__(self, i):
+        value = super().__getitem__(i)
+        if type(value) == str:            
+            if value.startswith(self.PREFIX):                
+                return self._getprocess(value)
+        return value
+    
+    def _resolve(self, value):
+        """Return the original value if it's a spill ID, otherwise return as-is."""
+        if isinstance(value, str) and value.startswith(self.PREFIX):
+            return self._getprocess(value)
+        return value
+
+    def __contains__(self, item):
+        for i in range(len(self)):
+            if self[i] == item:
+                return True
+        return False
+
+    def pop(self, i=-1):
+        resolved = self[i]
+        self.data.pop(i)
+        return resolved
+
+    def index(self, item, *args):
+        start = args[0] if len(args) > 0 else 0
+        stop = args[1] if len(args) > 1 else len(self)
+        for i in range(start, stop):
+            if self[i] == item:
+                return i
+        raise ValueError(f"{item!r} is not in list")
+
+    def count(self, item):
+        cnt = 0
+        for i in range(len(self)):
+            if self[i] == item:
+                cnt += 1
+        return cnt
+
+    def remove(self, item):
+        idx = self.index(item)
+        del self[idx]
+
+    def sort(self, *args, **kwds):
+        resolved = [self._resolve(v) for v in self.data]
+        resolved.sort(*args, **kwds)
+        self.data.clear()
+        for value in resolved:
+            if len(pickle.dumps(value)) >= self.size_threshold:
+                value = self._setprocess(value)
+            self.data.append(value)
+
+    def append(self, value):
+        if len(pickle.dumps(value)) >= self.size_threshold:
+            value = self._setprocess(value)
+        return super().append(value)
+
+    def extend(self, value):
+        tmp_value = []
+        for i in value:
+            if len(pickle.dumps(i)) >= self.size_threshold:
+                i = self._setprocess(i)
+            tmp_value.append(i)
+        return super().extend(tmp_value)
+
+    def __repr__(self):
+        self.repr_data = UserList()
+        for i in self:
+            if type(i) == str:
+                if i.startswith(self.PREFIX):
+                    i = self._getprocess(i)
+            self.repr_data.append(i)
+        return repr(self.repr_data)
+
+if __name__ == "__main__":
+
+    class TestObj:
+        def __init__(self, a, b):
+            self.a = a
+            self.b = b
+            self.c = a + b
+        
+        def test(self):
+            return self.c
+
+    testobj = TestObj(1, 2)
+
+    tmp = SwapList(
+        [
+            "hello", 
+            {"hello": "world"},
+            testobj,
+            1977,
+        ],
+        size_threshold=1,
+    )
+
+    tmp.append("hello")
+    tmp.extend([{"hello": "world"}, testobj, 1977])
+    tmp[2:4] = ["hello", {"hello": "world"}, testobj, 1977]
+    print(tmp)
+
+
+    for i in tmp:
+        print(i)
+
+    # tmp2 = SwapList(
+    #     [
+    #         "small",
+    #         {"hello": "world"},
+    #         "large",
+    #         {"hello": "world", "hello": "world", "hello": "world"},
+    #     ],
+    #     size_threshold=1,
+    # )
+
+    # print(tmp2)
+    # print(tmp2[0])
+    # print(tmp2[1])
+    # print(tmp2[2])    
+    # print(tmp2[3])

swapcollection-1.0.0/swapcollection.egg-info/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: swapcollection
+Version: 1.0.0
+Summary: A library that automatically offloads large pickle-able objects stored in a dict/list to SQLite, reducing memory usage. It can be used with the same syntax as a regular dict/list.
+Author: llinghyami
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/llinghyami/swapcollection
+Project-URL: Repository, https://github.com/llinghyami/swapcollection.git
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqlite-utils
+Dynamic: license-file
+
+> **English** · [日本語 (Japanese)](README.md)
+
+# swapcollection
+
+A library that automatically offloads large objects stored in a
+`dict`/`list` to SQLite, reducing memory usage. It can be used with
+the same syntax as a regular `dict`/`list`.
+
+## Quick Start
+
+```bash
+pip install swapcollection
+```
+
+```python
+from swapcollection import SwapDict, SwapList
+
+# Spill values >= 1 MB (pickle size) to SQLite
+data = SwapDict(size_threshold=1024)
+
+data["small"] = b"hello"
+data["large"] = b"x" * 100_000
+
+# Access transparently — same as a normal dict
+print(data["small"])
+print(len(data["large"]))
+```
+
+## How It Works
+
+`SwapDict` / `SwapList` automatically offloads values whose pickled size
+meets or exceeds `size_threshold` to a SQLite database. Any pickle-able
+object (not just `bytes`) is eligible.
+
+| Value | Storage |
+|---|---|
+| Pickle size < `size_threshold` | In-memory |
+| Pickle size >= `size_threshold` | SQLite (transparent) |
+
+Spilled values are retrieved, updated, and deleted using the same `dict`/`list`
+interface — no special API calls needed.
+
+## Caveats
+
+- `size_threshold` is the pickle-size threshold in bytes; values >= this size
+  are spilled.
+- `size_threshold` cannot be changed after instantiation.
+- Any pickle-able object type (not only `bytes`) can be spilled.
+- Deleting the SQLite database will cause errors on previously spilled items.
+
+## License
+
+MIT

swapcollection-1.0.0/swapcollection.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.en.md
+README.md
+pyproject.toml
+swapcollection/__init__.py
+swapcollection/swapcollection.py
+swapcollection.egg-info/PKG-INFO
+swapcollection.egg-info/SOURCES.txt
+swapcollection.egg-info/dependency_links.txt
+swapcollection.egg-info/requires.txt
+swapcollection.egg-info/top_level.txt
+tests/test_eq_comprehensive.py
+tests/test_swapcollection.py

swapcollection-1.0.0/swapcollection.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

swapcollection-1.0.0/swapcollection.egg-info/requires.txt

@@ -0,0 +1 @@
+sqlite-utils

swapcollection-1.0.0/swapcollection.egg-info/top_level.txt

@@ -0,0 +1 @@
+swapcollection