lfss 0.8.3__tar.gz → 0.9.0__tar.gz

{lfss-0.8.3 → lfss-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lfss
-Version: 0.8.3
+Version: 0.9.0
 Summary: Lightweight file storage service
 Home-page: https://github.com/MenxLi/lfss
 Author: li, mengxun
@@ -24,9 +24,17 @@ Description-Content-Type: text/markdown
 # Lightweight File Storage Service (LFSS)
 [![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/)
 
-My experiment on a lightweight and high-performance file/object storage service.  
+My experiment on a lightweight and high-performance file/object storage service...  
+
+**Highlights:**
+
+- User storage limit and access control.
+- Pagination and sorted file listing for vast number of files.  
+- High performance: high concurrency, near-native speed on stress tests.
+- Support range requests, so you can stream large files / resume download.
+
 It stores small files and metadata in sqlite, large files in the filesystem.  
-Tested on 2 million files, and it works fine... 
+Tested on 2 million files, and it is still fast.
 
 Usage: 
 ```sh
@@ -45,8 +53,8 @@ lfss-panel --open
 Or, you can start a web server at `/frontend` and open `index.html` in your browser. 
 
 The API usage is simple, just `GET`, `PUT`, `DELETE` to the `/<username>/file/url` path.  
-Authentication is done via `Authorization` header with the value `Bearer <token>`, or through the `token` query parameter.  
-You can refer to `frontend` as an application example, and `frontend/api.js` or `lfss/api/connector.py` for the API usage.
+Authentication via `Authorization` header with the value `Bearer <token>`, or through the `token` query parameter.  
+You can refer to `frontend` as an application example, `lfss/api/connector.py` for more APIs. 
 
 By default, the service exposes all files to the public for `GET` requests, 
 but file-listing is restricted to the user's own files.  

{lfss-0.8.3 → lfss-0.9.0}/Readme.md

@@ -1,9 +1,17 @@
 # Lightweight File Storage Service (LFSS)
 [![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/)
 
-My experiment on a lightweight and high-performance file/object storage service.  
+My experiment on a lightweight and high-performance file/object storage service...  
+
+**Highlights:**
+
+- User storage limit and access control.
+- Pagination and sorted file listing for vast number of files.  
+- High performance: high concurrency, near-native speed on stress tests.
+- Support range requests, so you can stream large files / resume download.
+
 It stores small files and metadata in sqlite, large files in the filesystem.  
-Tested on 2 million files, and it works fine... 
+Tested on 2 million files, and it is still fast.
 
 Usage: 
 ```sh
@@ -22,8 +30,8 @@ lfss-panel --open
 Or, you can start a web server at `/frontend` and open `index.html` in your browser. 
 
 The API usage is simple, just `GET`, `PUT`, `DELETE` to the `/<username>/file/url` path.  
-Authentication is done via `Authorization` header with the value `Bearer <token>`, or through the `token` query parameter.  
-You can refer to `frontend` as an application example, and `frontend/api.js` or `lfss/api/connector.py` for the API usage.
+Authentication via `Authorization` header with the value `Bearer <token>`, or through the `token` query parameter.  
+You can refer to `frontend` as an application example, `lfss/api/connector.py` for more APIs. 
 
 By default, the service exposes all files to the public for `GET` requests, 
 but file-listing is restricted to the user's own files.  

lfss-0.9.0/docs/Permission.md

@@ -0,0 +1,58 @@
+
+# Permission System
+There are two user roles in the system: Admin and Normal User ("users" are like "buckets" to some extent).  
+A user have all permissions of the files and subpaths under its path (starting with `/<user>/`).  
+Admins have all permissions of all files and paths.
+
+> **path** ends with `/` and **file** does not end with `/`.
+
+## Peers
+The user can have multiple peer users. The peer user can have read or write access to the user's path, depending on the access level set when adding the peer user.  
+The peer user can list the files under the user's path.  
+If the peer user only has read access (peer-r), then the peer user can only `GET` files under the user's path.  
+If the peer user has write access (peer-w), then the peer user can `GET`/`PUT`/`POST`/`DELETE` files under the user's path.  
+
+## Ownership
+A file is owned by the user who created it, may not necessarily be the user under whose path the file is stored (admin/write-peer can create files under any user's path).
+
+# Non-peer and public access
+
+**NOTE:** below discussion is based on the assumption that the user is not a peer of the path owner, or is guest user (public access).
+
+## File access with `GET` permission
+
+### File access
+For accessing file content, the user must have `GET` permission of the file, which is determined by the `permission` field of both the owner and the file.   
+
+There are four types of permissions: `unset`, `public`, `protected`, `private`.
+Non-admin users can access files based on:   
+
+- If the file is `public`, then all users can access it.
+- If the file is `protected`, then only the logged-in user can access it.  
+- If the file is `private`, then only the owner can access it.
+- If the file is `unset`, then the file's permission is inherited from the owner's permission.
+- If both the owner and the file have `unset` permission, then the file is `public`.
+
+## File creation with `PUT`/`POST` permission
+`PUT`/`POST` permission is not allowed for non-peer users.
+
+## File `DELETE` and moving permissions
+- Non-login user don't have `DELETE`/move permission.
+- Every user can have `DELETE` permission that they own.
+- User can move files if they have write access to the destination path.
+
+## Path-listing
+Path-listing is not allowed for these users.
+
+# Summary
+
+| Permission | Admin | User | Peer-r | Peer-w | Owner (not the user) | Non-peer user / Guest  |
+|------------|-------|------|--------|--------|----------------------|------------------------|
+| GET        | Yes   | Yes  | Yes    | Yes    | Yes                  | Depends on file        |
+| PUT/POST   | Yes   | Yes  | No     | Yes    | Yes                  | No                     |
+| DELETE file| Yes   | Yes  | No     | Yes    | Yes                  | No                     | 
+| DELETE path| Yes   | Yes  | No     | Yes    | N/A                  | No                     | 
+| move       | Yes   | Yes  | No     | Yes    | Dep. on destination  | No                     |
+| list       | Yes   | Yes  | Yes    | Yes    | No if not peer       | No                     |
+
+> Capitilized methods are HTTP methods, N/A means not applicable.

{lfss-0.8.3 → lfss-0.9.0}/frontend/api.js

@@ -45,6 +45,17 @@ export const permMap = {
     3: 'private'
 }
 
+async function fmtFailedResponse(res){
+    const raw = await res.text();
+    const json = raw ? JSON.parse(raw) : {};
+    const txt = JSON.stringify(json.detail || json || "No message");
+    const maxWords = 32;
+    if (txt.length > maxWords){
+        return txt.slice(0, maxWords) + '...';
+    }
+    return txt;
+}
+
 export default class Connector {
 
     constructor(){
@@ -79,7 +90,7 @@ export default class Connector {
             body: fileBytes
         });
         if (res.status != 200 && res.status != 201){
-            throw new Error(`Failed to upload file, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to upload file, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return (await res.json()).url;
     }
@@ -111,7 +122,7 @@ export default class Connector {
         });
                 
         if (res.status != 200 && res.status != 201){
-            throw new Error(`Failed to upload file, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to upload file, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return (await res.json()).url;
     }
@@ -133,7 +144,7 @@ export default class Connector {
             body: JSON.stringify(data)
         });
         if (res.status != 200 && res.status != 201){
-            throw new Error(`Failed to upload object, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to upload object, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return (await res.json()).url;
     }
@@ -147,7 +158,7 @@ export default class Connector {
             }, 
         });
         if (res.status == 200) return;
-        throw new Error(`Failed to delete file, status code: ${res.status}, message: ${await res.json()}`);
+        throw new Error(`Failed to delete file, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
     }
 
     /**
@@ -211,7 +222,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to count files, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to count files, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return (await res.json()).count;
     }
@@ -250,7 +261,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to list files, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to list files, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return await res.json();
     }
@@ -270,7 +281,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to count directories, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to count directories, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return (await res.json()).count;
     }
@@ -309,7 +320,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to list directories, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to list directories, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
         return await res.json();
     }
@@ -347,7 +358,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to set permission, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to set permission, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
     }
 
@@ -369,7 +380,7 @@ export default class Connector {
             },
         });
         if (res.status != 200){
-            throw new Error(`Failed to move file, status code: ${res.status}, message: ${await res.json()}`);
+            throw new Error(`Failed to move file, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
         }
     }
 

{lfss-0.8.3 → lfss-0.9.0}/frontend/thumb.js

@@ -39,10 +39,14 @@ function getIconSVGFromMimeType(mimeType){
         return ICON_ZIP;
     }
     if ([
-        "text/html", "application/xhtml+xml", "application/xml", "text/css", "application/javascript", "text/javascript", "application/json", "text/x-python", "text/x-java-source",
-        "application/x-httpd-php", "text/x-ruby", "text/x-perl", "application/x-sh", "application/sql", "text/x-c", "text/x-c++", "text/x-csharp", "text/x-go", "text/x-haskell",
-        "text/x-lua", "text/x-markdown", "application/wasm", "application/x-tcl", "text/x-yaml", "application/x-latex", "application/x-tex", "text/x-scss", "application/x-lisp",
-        "application/x-rust", "application/x-ruby", "text/x-asm"
+        "text/html", "application/xhtml+xml", "application/xml", "text/css", "text/x-scss", "application/javascript", "text/javascript",
+        "application/json", "text/x-yaml", "text/x-markdown", "application/wasm", 
+        "text/x-ruby", "application/x-ruby", "text/x-perl", "application/x-lisp", 
+        "text/x-haskell", "text/x-lua", "application/x-tcl", 
+        "text/x-python", "text/x-java-source", "text/x-go", "application/x-rust", "text/x-asm", 
+        "application/sql", "text/x-c", "text/x-c++", "text/x-csharp", 
+        "application/x-httpd-php", "application/x-sh", "application/x-shellscript", 
+        "application/x-latex", "application/x-tex", 
     ].includes(mimeType)){
         return ICON_CODE;
     }

{lfss-0.8.3 → lfss-0.9.0}/lfss/api/connector.py

@@ -17,14 +17,20 @@ _default_token = os.environ.get('LFSS_TOKEN', '')
 
 class Connector:
     class Session:
-        def __init__(self, connector: Connector, pool_size: int = 10):
+        def __init__(
+            self, connector: Connector, pool_size: int = 10, 
+            retry: int = 1, backoff_factor: float = 0.5, status_forcelist: list[int] = [503]
+            ):
             self.connector = connector
             self.pool_size = pool_size
+            self.retry_adapter = requests.adapters.Retry(
+                total=retry, backoff_factor=backoff_factor, status_forcelist=status_forcelist, 
+            )
         def open(self):
             self.close()
             if self.connector._session is None:
                 s = requests.Session()
-                adapter = requests.adapters.HTTPAdapter(pool_connections=self.pool_size, pool_maxsize=self.pool_size)
+                adapter = requests.adapters.HTTPAdapter(pool_connections=self.pool_size, pool_maxsize=self.pool_size, max_retries=self.retry_adapter)
                 s.mount('http://', adapter)
                 s.mount('https://', adapter)
                 self.connector._session = s
@@ -48,9 +54,9 @@ class Connector:
         }
         self._session: Optional[requests.Session] = None
     
-    def session(self, pool_size: int = 10):
+    def session( self, pool_size: int = 10, **kwargs):
         """ avoid creating a new session for each request.  """
-        return self.Session(self, pool_size)
+        return self.Session(self, pool_size, **kwargs)
     
     def _fetch_factory(
         self, method: Literal['GET', 'POST', 'PUT', 'DELETE'], 
@@ -114,7 +120,6 @@ class Connector:
         
         if isinstance(file, str):
             assert os.path.exists(file), "File does not exist on disk"
-            fsize = os.path.getsize(file)
 
         with open(file, 'rb') if isinstance(file, str) else SpooledTemporaryFile(max_size=1024*1024*32) as fp:
 
@@ -124,7 +129,6 @@ class Connector:
                 fp.seek(0)
 
             # https://stackoverflow.com/questions/12385179/
-            print(f"Uploading {fsize} bytes")
             response = self._fetch_factory('POST', path, search_params={
                 'permission': int(permission),
                 'conflict': conflict

{lfss-0.8.3 → lfss-0.9.0}/lfss/cli/cli.py

@@ -1,24 +1,18 @@
-from lfss.api import Connector, upload_directory, upload_file, download_file, download_directory
 from pathlib import Path
 import argparse, typing
+from lfss.api import Connector, upload_directory, upload_file, download_file, download_directory
 from lfss.src.datatype import FileReadPermission, FileSortKey, DirSortKey
 from lfss.src.utils import decode_uri_compnents
 from . import catch_request_error, line_sep
 
 def parse_permission(s: str) -> FileReadPermission:
-    if s.lower() == "public":
-        return FileReadPermission.PUBLIC
-    if s.lower() == "protected":
-        return FileReadPermission.PROTECTED
-    if s.lower() == "private":
-        return FileReadPermission.PRIVATE
-    if s.lower() == "unset":
-        return FileReadPermission.UNSET
+    for p in FileReadPermission:
+        if p.name.lower() == s.lower():
+            return p
     raise ValueError(f"Invalid permission {s}")
 
 def parse_arguments():
     parser = argparse.ArgumentParser(description="Command line interface, please set LFSS_ENDPOINT and LFSS_TOKEN environment variables.")
-    parser.add_argument("-v", "--verbose", action="store_true", help="Verbose output")
 
     sp = parser.add_subparsers(dest="command", required=True)
 
@@ -26,6 +20,7 @@ def parse_arguments():
     sp_upload = sp.add_parser("upload", help="Upload files")
     sp_upload.add_argument("src", help="Source file or directory", type=str)
     sp_upload.add_argument("dst", help="Destination url path", type=str)
+    sp_upload.add_argument("-v", "--verbose", action="store_true", help="Verbose output")
     sp_upload.add_argument("-j", "--jobs", type=int, default=1, help="Number of concurrent uploads")
     sp_upload.add_argument("--interval", type=float, default=0, help="Interval between files, only works with directory upload")
     sp_upload.add_argument("--conflict", choices=["overwrite", "abort", "skip", "skip-ahead"], default="abort", help="Conflict resolution")
@@ -36,6 +31,7 @@ def parse_arguments():
     sp_download = sp.add_parser("download", help="Download files")
     sp_download.add_argument("src", help="Source url path", type=str)
     sp_download.add_argument("dst", help="Destination file or directory", type=str)
+    sp_download.add_argument("-v", "--verbose", action="store_true", help="Verbose output")
     sp_download.add_argument("-j", "--jobs", type=int, default=1, help="Number of concurrent downloads")
     sp_download.add_argument("--interval", type=float, default=0, help="Interval between files, only works with directory download")
     sp_download.add_argument("--overwrite", action="store_true", help="Overwrite existing files")

{lfss-0.8.3 → lfss-0.9.0}/lfss/cli/user.py

@@ -2,9 +2,16 @@ import argparse, asyncio, os
 from contextlib import asynccontextmanager
 from .cli import parse_permission, FileReadPermission
 from ..src.utils import parse_storage_size, fmt_storage_size
+from ..src.datatype import AccessLevel
 from ..src.database import Database, FileReadPermission, transaction, UserConn, unique_cursor, FileConn
 from ..src.connection_pool import global_entrance
 
+def parse_access_level(s: str) -> AccessLevel:
+    for p in AccessLevel:
+        if p.name.lower() == s.lower():
+            return p
+    raise ValueError(f"Invalid access level {s}")
+
 @global_entrance(1)
 async def _main():
     parser = argparse.ArgumentParser()
@@ -31,11 +38,16 @@ async def _main():
     sp_set.add_argument('-a', '--admin', type=parse_bool, default=None)
     sp_set.add_argument('--permission', type=parse_permission, default=None)
     sp_set.add_argument('--max-storage', type=parse_storage_size, default=None)
-    
+
     sp_list = sp.add_parser('list')
     sp_list.add_argument("username", nargs='*', type=str, default=None)
     sp_list.add_argument("-l", "--long", action="store_true")
     
+    sp_peer = sp.add_parser('set-peer')
+    sp_peer.add_argument('src_username', type=str)
+    sp_peer.add_argument('dst_username', type=str)
+    sp_peer.add_argument('--level', type=parse_access_level, default=AccessLevel.READ, help="Access level")
+    
     args = parser.parse_args()
     db = await Database().init()
 
@@ -72,6 +84,16 @@ async def _main():
             assert user is not None
             print('User updated, credential:', user.credential)
     
+    if args.subparser_name == 'set-peer':
+        async with get_uconn() as uconn:
+            src_user = await uconn.get_user(args.src_username)
+            dst_user = await uconn.get_user(args.dst_username)
+            if src_user is None or dst_user is None:
+                print('User not found')
+                exit(1)
+            await uconn.set_peer_level(src_user.id, dst_user.id, args.level)
+            print(f"Peer set: [{src_user.username}] now have [{args.level.name}] access to [{dst_user.username}]")
+    
     if args.subparser_name == 'list':
         async with get_uconn() as uconn:
             term_width = os.get_terminal_size().columns
@@ -86,6 +108,11 @@ async def _main():
                         user_size_used = await fconn.user_size(user.id)
                     print('- Credential: ', user.credential)
                     print(f'- Storage: {fmt_storage_size(user_size_used)} / {fmt_storage_size(user.max_storage)}')
+                    for p in AccessLevel:
+                        if p > AccessLevel.NONE:
+                            usernames = [x.username for x in await uconn.list_peer_users(user.id, p)]
+                            if usernames:
+                                print(f'- Peers [{p.name}]: {", ".join(usernames)}')
         
 def main():
     asyncio.run(_main())

{lfss-0.8.3 → lfss-0.9.0}/lfss/sql/init.sql

@@ -27,6 +27,15 @@ CREATE TABLE IF NOT EXISTS usize (
     size INTEGER DEFAULT 0
 );
 
+CREATE TABLE IF NOT EXISTS upeer (
+    src_user_id INTEGER NOT NULL,
+    dst_user_id INTEGER NOT NULL,
+    access_level INTEGER DEFAULT 0,
+    PRIMARY KEY(src_user_id, dst_user_id),
+    FOREIGN KEY(src_user_id) REFERENCES user(id),
+    FOREIGN KEY(dst_user_id) REFERENCES user(id)
+); 
+
 CREATE INDEX IF NOT EXISTS idx_fmeta_url ON fmeta(url);
 
 CREATE INDEX IF NOT EXISTS idx_user_username ON user(username);

{lfss-0.8.3 → lfss-0.9.0}/lfss/src/connection_pool.py

@@ -8,6 +8,7 @@ from functools import wraps
 from typing import Callable, Awaitable
 
 from .log import get_logger
+from .error import DatabaseLockedError
 from .config import DATA_HOME
 
 async def execute_sql(conn: aiosqlite.Connection | aiosqlite.Cursor, name: str):
@@ -28,7 +29,7 @@ async def get_connection(read_only: bool = False) -> aiosqlite.Connection:
 
     conn = await aiosqlite.connect(
         get_db_uri(DATA_HOME / 'index.db', read_only=read_only), 
-        timeout = 60, uri = True
+        timeout = 20, uri = True
         )
     async with conn.cursor() as c:
         await c.execute(
@@ -46,7 +47,7 @@ class SqlConnection:
 
 class SqlConnectionPool:
     _r_sem: Semaphore
-    _w_sem: Lock | Semaphore
+    _w_sem: Lock
     def __init__(self):
         self._readers: list[SqlConnection] = []
         self._writer: None | SqlConnection = None
@@ -65,6 +66,17 @@ class SqlConnectionPool:
             self._readers.append(SqlConnection(conn))
         self._r_sem = Semaphore(n_read)
     
+    def status(self):   # debug
+        assert self._writer
+        assert len(self._readers) == self.n_read
+        n_free_readers = sum([1 for c in self._readers if c.is_available])
+        n_free_writers = 1 if self._writer.is_available else 0
+        n_free_r_sem = self._r_sem._value
+        n_free_w_sem = 1 - self._w_sem.locked()
+        assert n_free_readers == n_free_r_sem, f"{n_free_readers} != {n_free_r_sem}"
+        assert n_free_writers == n_free_w_sem, f"{n_free_writers} != {n_free_w_sem}"
+        return f"Readers: {n_free_readers}/{self.n_read}, Writers: {n_free_writers}/{1}"
+    
     @property
     def n_read(self):
         return len(self._readers)
@@ -142,6 +154,10 @@ async def unique_cursor(is_write: bool = False):
             connection_obj = await g_pool.get()
             try:
                 yield await connection_obj.conn.cursor()
+            except Exception as e:
+                if 'database is locked' in str(e):
+                    raise DatabaseLockedError from e
+                raise e
             finally:
                 await g_pool.release(connection_obj)
     else:
@@ -149,10 +165,13 @@ async def unique_cursor(is_write: bool = False):
             connection_obj = await g_pool.get(w=True)
             try:
                 yield await connection_obj.conn.cursor()
+            except Exception as e:
+                if 'database is locked' in str(e):
+                    raise DatabaseLockedError from e
+                raise e
             finally:
                 await g_pool.release(connection_obj)
 
-# todo: add exclusive transaction option
 @asynccontextmanager
 async def transaction():
     async with unique_cursor(is_write=True) as cur: