lfss 0.7.2__py3-none-any.whl → 0.7.4__py3-none-any.whl

Readme.md

@@ -2,8 +2,8 @@
 [![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/)
 
 My experiment on a lightweight file/object storage service.  
-It stores small files and metadata in sqlite, large files in the filesystem, and serves them through a simple REST API.  
-Tested on 2 million files, and it works fine... thanks to the sqlite database!
+It stores small files and metadata in sqlite, large files in the filesystem.  
+Tested on 2 million files, and it works fine... 
 
 Usage: 
 ```sh

docs/Permission.md

@@ -1,22 +1,12 @@
 
 # Permission System
-There are two roles in the system: Admin and User (you can treat then as buckets).  
+There are two roles in the system: Admin and User ('user' are like 'bucket' to some extent).
 
-## `PUT` and `DELETE` permissions
-Non-login user don't have `PUT/DELETE` permissions.  
-Every user can have `PUT/DELETE` permissions of files under its own `/<user>/` path.  
-The admin can have `PUT/DELETE` permissions of files of all users.
+## File access with `GET` permission
+The `GET` is used to access the file (if path is not ending with `/`), or to list the files under a path (if path is ending with `/`).  
 
-## `GET` permissions
-The `GET` is used to access the file (if path is not ending with `/`), or to list the files under a path (if path is ending with `/`). 
-
-### Path-listing
-- Non-login users cannot list any files.
-- All users can list the files under their own path 
-- Admins can list the files under other users' path. 
-
-### File-access
-For accessing file content, the user must have `GET` permission of the file, which is determined by the `permission` field of both the owner and the file. 
+### File access
+For accessing file content, the user must have `GET` permission of the file, which is determined by the `permission` field of both the owner and the file.   
 (Note: The owner of the file is the user who created the file, may not necessarily be the user under whose path the file is stored.)   
 
 There are four types of permissions: `unset`, `public`, `protected`, `private`.
@@ -26,4 +16,29 @@ Non-admin users can access files based on:
 - If the file is `protected`, then only the logged-in user can access it.  
 - If the file is `private`, then only the owner can access it.
 - If the file is `unset`, then the file's permission is inherited from the owner's permission.
-- If both the owner and the file have `unset` permission, then the file is `public`.
+- If both the owner and the file have `unset` permission, then the file is `public`.
+
+### Meta-data access
+- Non-login users can't access any file-meta.
+- All users can access the file-meta of files under their own path.
+- For files under other users' path, the file-meta is determined in a way same as file access.
+- Admins can access the path-meta of all users.
+- All users can access the path-meta of their own path.
+
+### Path-listing
+- Non-login users cannot list any files.
+- All users can list the files under their own path 
+- Admins can list the files under other users' path. 
+
+## File creation with `PUT` permission
+The `PUT` is used to create a file. 
+- Non-login user don't have `PUT` permission.  
+- Every user can have `PUT` permission of files under its own `/<user>/` path.  
+- The admin can have `PUT` permission of files of all users.
+
+## `DELETE` and moving permissions
+- Non-login user don't have `DELETE`/move permission.
+- Every user can have `DELETE`/move permission that they own.
+- The admin can have `DELETE` permission of files of all users
+(The admin can't move files of other users, because move does not change the owner of the file. 
+If move is allowed, then its equivalent to create file on behalf of other users.)

lfss/cli/cli.py

@@ -1,22 +1,32 @@
-from lfss.client import Connector, upload_directory
-from lfss.src.database import FileReadPermission
+from lfss.client import Connector, upload_directory, upload_file, download_file, download_directory
+from lfss.src.datatype import FileReadPermission
 from pathlib import Path
 import argparse
 
 def parse_arguments():
     parser = argparse.ArgumentParser(description="Command line interface, please set LFSS_ENDPOINT and LFSS_TOKEN environment variables.")
+    parser.add_argument("-v", "--verbose", action="store_true", help="Verbose output")
 
     sp = parser.add_subparsers(dest="command", required=True)
 
     # upload
     sp_upload = sp.add_parser("upload", help="Upload files")
     sp_upload.add_argument("src", help="Source file or directory", type=str)
-    sp_upload.add_argument("dst", help="Destination path", type=str)
+    sp_upload.add_argument("dst", help="Destination url path", type=str)
     sp_upload.add_argument("-j", "--jobs", type=int, default=1, help="Number of concurrent uploads")
     sp_upload.add_argument("--interval", type=float, default=0, help="Interval between files, only works with directory upload")
     sp_upload.add_argument("--conflict", choices=["overwrite", "abort", "skip", "skip-ahead"], default="abort", help="Conflict resolution")
     sp_upload.add_argument("--permission", type=FileReadPermission, default=FileReadPermission.UNSET, help="File permission")
-    sp_upload.add_argument("--retries", type=int, default=0, help="Number of retries, only works with directory upload")
+    sp_upload.add_argument("--retries", type=int, default=0, help="Number of retries")
+
+    # download
+    sp_download = sp.add_parser("download", help="Download files")
+    sp_download.add_argument("src", help="Source url path", type=str)
+    sp_download.add_argument("dst", help="Destination file or directory", type=str)
+    sp_download.add_argument("-j", "--jobs", type=int, default=1, help="Number of concurrent downloads")
+    sp_download.add_argument("--interval", type=float, default=0, help="Interval between files, only works with directory download")
+    sp_download.add_argument("--overwrite", action="store_true", help="Overwrite existing files")
+    sp_download.add_argument("--retries", type=int, default=0, help="Number of retries")
 
     return parser.parse_args()
 
@@ -28,9 +38,9 @@ def main():
         if src_path.is_dir():
             failed_upload = upload_directory(
                 connector, args.src, args.dst, 
-                verbose=True, 
+                verbose=args.verbose,
                 n_concurrent=args.jobs, 
-                n_reties=args.retries, 
+                n_retries=args.retries, 
                 interval=args.interval,
                 conflict=args.conflict,
                 permission=args.permission
@@ -40,13 +50,46 @@ def main():
                 for path in failed_upload:
                     print(f"  {path}")
         else:
-            with open(args.src, 'rb') as f:
-                connector.put(
-                    args.dst, 
-                    f.read(), 
-                    conflict=args.conflict,
-                    permission=args.permission
-                    )
+            success = upload_file(
+                connector, 
+                file_path = args.src, 
+                dst_url = args.dst, 
+                verbose=args.verbose,
+                n_retries=args.retries, 
+                interval=args.interval,
+                conflict=args.conflict,
+                permission=args.permission
+            )
+            if not success:
+                print("Failed to upload.")
+    
+    elif args.command == "download":
+        is_dir = args.src.endswith("/")
+        if is_dir:
+            failed_download = download_directory(
+                connector, args.src, args.dst, 
+                verbose=args.verbose,
+                n_concurrent=args.jobs, 
+                n_retries=args.retries, 
+                interval=args.interval,
+                overwrite=args.overwrite
+            )
+            if failed_download:
+                print("Failed to download:")
+                for path in failed_download:
+                    print(f"  {path}")
+        else:
+            success = download_file(
+                connector, 
+                src_url = args.src, 
+                file_path = args.dst, 
+                verbose=args.verbose,
+                n_retries=args.retries, 
+                interval=args.interval,
+                overwrite=args.overwrite
+            )
+            if not success:
+                print("Failed to download.")
     else:
         raise NotImplementedError(f"Command {args.command} not implemented.")
     

lfss/cli/user.py

@@ -45,6 +45,7 @@ async def _main():
     sp_list.add_argument("-l", "--long", action="store_true")
     
     args = parser.parse_args()
+    db = await Database().init()
 
     @asynccontextmanager
     async def get_uconn():
@@ -65,7 +66,6 @@ async def _main():
             print('User not found')
             exit(1)
         else:
-            db = await Database().init()
             await db.delete_user(user.id)
         print('User deleted')
     

lfss/client/__init__.py

@@ -1,14 +1,45 @@
-import os, time
+import os, time, pathlib
 from threading import Lock
 from concurrent.futures import ThreadPoolExecutor
 from .api import Connector
 
+def upload_file(
+    connector: Connector, 
+    file_path: str, 
+    dst_url: str, 
+    n_retries: int = 0, 
+    interval: float = 0, 
+    verbose: bool = False, 
+    **put_kwargs
+    ):
+    this_try = 0
+    while this_try <= n_retries:
+        try:
+            with open(file_path, 'rb') as f:
+                blob = f.read()
+            connector.put(dst_url, blob, **put_kwargs)
+            break
+        except Exception as e:
+            if isinstance(e, KeyboardInterrupt):
+                raise e
+            if verbose:
+                print(f"Error uploading {file_path}: {e}, retrying...")
+            this_try += 1
+        finally:
+            time.sleep(interval)
+
+    if this_try > n_retries:
+        if verbose:
+            print(f"Failed to upload {file_path} after {n_retries} retries.")
+        return False
+    return True
+
 def upload_directory(
     connector: Connector, 
     directory: str, 
     path: str, 
     n_concurrent: int = 1,
-    n_reties: int = 0,
+    n_retries: int = 0,
     interval: float = 0,
     verbose: bool = False,
     **put_kwargs
@@ -16,6 +47,7 @@ def upload_directory(
     assert path.endswith('/'), "Path must end with a slash."
     if path.startswith('/'):
         path = path[1:]
+    directory = str(directory)
     
     _counter = 0
     _counter_lock = Lock()
@@ -30,31 +62,94 @@ def upload_directory(
             if verbose:
                 print(f"[{this_count}] Uploading {file_path} to {dst_path}")
 
-        this_try = 0
-        with open(file_path, 'rb') as f:
-            blob = f.read()
-
-        while this_try <= n_reties:
-            try:
-                connector.put(dst_path, blob, **put_kwargs)
-                break
-            except Exception as e:
-                if isinstance(e, KeyboardInterrupt):
-                    raise e
-                if verbose:
-                    print(f"[{this_count}] Error uploading {file_path}: {e}, retrying...")
-                this_try += 1
-            finally:
-                time.sleep(interval)
-
-        if this_try > n_reties:
+        if not upload_file(
+            connector, file_path, dst_path, 
+            n_retries=n_retries, interval=interval, verbose=verbose, **put_kwargs
+            ):
             faild_files.append(file_path)
-            if verbose:
-                print(f"[{this_count}] Failed to upload {file_path} after {n_reties} retries.")
 
     with ThreadPoolExecutor(n_concurrent) as executor:
         for root, dirs, files in os.walk(directory):
             for file in files:
                 executor.submit(put_file, os.path.join(root, file))
 
-    return faild_files
+    return faild_files
+
+def download_file(
+    connector: Connector, 
+    src_url: str, 
+    file_path: str, 
+    n_retries: int = 0, 
+    interval: float = 0, 
+    verbose: bool = False, 
+    overwrite: bool = False
+    ):
+    this_try = 0
+    while this_try <= n_retries:
+        if not overwrite and os.path.exists(file_path):
+            if verbose:
+                print(f"File {file_path} already exists, skipping download.")
+            return True
+        try:
+            blob = connector.get(src_url)
+            if not blob:
+                return False
+            pathlib.Path(file_path).parent.mkdir(parents=True, exist_ok=True)
+            with open(file_path, 'wb') as f:
+                f.write(blob)
+            break
+        except Exception as e:
+            if isinstance(e, KeyboardInterrupt):
+                raise e
+            if verbose:
+                print(f"Error downloading {src_url}: {e}, retrying...")
+            this_try += 1
+        finally:
+            time.sleep(interval)
+
+    if this_try > n_retries:
+        if verbose:
+            print(f"Failed to download {src_url} after {n_retries} retries.")
+        return False
+    return True
+
+def download_directory(
+    connector: Connector,
+    src_path: str,
+    directory: str,
+    n_concurrent: int = 1,
+    n_retries: int = 0,
+    interval: float = 0,
+    verbose: bool = False,
+    overwrite: bool = False
+    ) -> list[str]:
+
+    directory = str(directory)
+
+    if not src_path.endswith('/'):
+        src_path += '/'
+    if not directory.endswith(os.sep):
+        directory += os.sep
+    
+    _counter = 0
+    _counter_lock = Lock()
+    failed_files = []
+    def get_file(src_url):
+        nonlocal _counter, failed_files
+        with _counter_lock:
+            _counter += 1
+            this_count = _counter
+            dst_path = f"{directory}{os.path.relpath(src_url, src_path)}"
+            if verbose:
+                print(f"[{this_count}] Downloading {src_url} to {dst_path}")
+
+        if not download_file(
+            connector, src_url, dst_path, 
+            n_retries=n_retries, interval=interval, verbose=verbose, overwrite=overwrite
+            ):
+            failed_files.append(src_url)
+        
+    with ThreadPoolExecutor(n_concurrent) as executor:
+        for file in connector.list_path(src_path).files:
+            executor.submit(get_file, file.url)
+    return failed_files

lfss/client/api.py

@@ -5,6 +5,7 @@ import urllib.parse
 from lfss.src.datatype import (
     FileReadPermission, FileRecord, DirectoryRecord, UserRecord, PathContents
     )
+from lfss.src.utils import ensure_uri_compnents
 
 _default_endpoint = os.environ.get('LFSS_ENDPOINT', 'http://localhost:8000')
 _default_token = os.environ.get('LFSS_TOKEN', '')
@@ -17,12 +18,13 @@ class Connector:
             "token": token
         }
     
-    def _fetch(
+    def _fetch_factory(
         self, method: Literal['GET', 'POST', 'PUT', 'DELETE'], 
         path: str, search_params: dict = {}
     ):
         if path.startswith('/'):
             path = path[1:]
+        path = ensure_uri_compnents(path)
         def f(**kwargs):
             url = f"{self.config['endpoint']}/{path}" + "?" + urllib.parse.urlencode(search_params)
             headers: dict = kwargs.pop('headers', {})
@@ -46,7 +48,7 @@ class Connector:
             else:
                 return {'status': 'skipped', 'path': path}
 
-        response = self._fetch('PUT', path, search_params={
+        response = self._fetch_factory('PUT', path, search_params={
             'permission': int(permission),
             'conflict': conflict
             })(
@@ -68,7 +70,7 @@ class Connector:
             else:
                 return {'status': 'skipped', 'path': path}
 
-        response = self._fetch('PUT', path, search_params={
+        response = self._fetch_factory('PUT', path, search_params={
             'permission': int(permission),
             'conflict': conflict
             })(
@@ -79,7 +81,7 @@ class Connector:
     
     def _get(self, path: str) -> Optional[requests.Response]:
         try:
-            response = self._fetch('GET', path)()
+            response = self._fetch_factory('GET', path)()
         except requests.exceptions.HTTPError as e:
             if e.response.status_code == 404:
                 return None
@@ -100,12 +102,12 @@ class Connector:
     
     def delete(self, path: str):
         """Deletes the file at the specified path."""
-        self._fetch('DELETE', path)()
+        self._fetch_factory('DELETE', path)()
     
     def get_metadata(self, path: str) -> Optional[FileRecord | DirectoryRecord]:
         """Gets the metadata for the file at the specified path."""
         try:
-            response = self._fetch('GET', '_api/meta', {'path': path})()
+            response = self._fetch_factory('GET', '_api/meta', {'path': path})()
             if path.endswith('/'):
                 return DirectoryRecord(**response.json())
             else:
@@ -117,22 +119,24 @@ class Connector:
     
     def list_path(self, path: str) -> PathContents:
         assert path.endswith('/')
-        response = self._fetch('GET', path)()
-        return PathContents(**response.json())
+        response = self._fetch_factory('GET', path)()
+        dirs = [DirectoryRecord(**d) for d in response.json()['dirs']]
+        files = [FileRecord(**f) for f in response.json()['files']]
+        return PathContents(dirs=dirs, files=files)
 
     def set_file_permission(self, path: str, permission: int | FileReadPermission):
         """Sets the file permission for the specified path."""
-        self._fetch('POST', '_api/meta', {'path': path, 'perm': int(permission)})(
+        self._fetch_factory('POST', '_api/meta', {'path': path, 'perm': int(permission)})(
             headers={'Content-Type': 'application/www-form-urlencoded'}
         )
         
     def move(self, path: str, new_path: str):
         """Move file or directory to a new path."""
-        self._fetch('POST', '_api/meta', {'path': path, 'new_path': new_path})(
+        self._fetch_factory('POST', '_api/meta', {'path': path, 'new_path': new_path})(
             headers = {'Content-Type': 'application/www-form-urlencoded'}
         )
         
     def whoami(self) -> UserRecord:
         """Gets information about the current user."""
-        response = self._fetch('GET', '_api/whoami')()
+        response = self._fetch_factory('GET', '_api/whoami')()
         return UserRecord(**response.json())

lfss/src/config.py

@@ -1,5 +1,5 @@
 from pathlib import Path
-import os
+import os, hashlib
 
 __default_dir = '.storage_data'
 
@@ -14,4 +14,4 @@ LARGE_BLOB_DIR.mkdir(exist_ok=True)
 # https://sqlite.org/fasterthanfs.html
 LARGE_FILE_BYTES = 8 * 1024 * 1024   # 8MB
 MAX_FILE_BYTES = 512 * 1024 * 1024   # 512MB
-MAX_BUNDLE_BYTES = 512 * 1024 * 1024   # 512MB
+MAX_BUNDLE_BYTES = 512 * 1024 * 1024   # 512MB

lfss/src/connection_pool.py

@@ -139,6 +139,7 @@ async def unique_cursor(is_write: bool = False):
             finally:
                 await g_pool.release(connection_obj)
 
+# todo: add exclusive transaction option
 @asynccontextmanager
 async def transaction():
     async with unique_cursor(is_write=True) as cur:

lfss/src/database.py

@@ -3,7 +3,7 @@ from typing import Optional, overload, Literal, AsyncIterable
 from abc import ABC
 
 import urllib.parse
-import hashlib, uuid
+import uuid
 import zipfile, io, asyncio
 
 import aiosqlite, aiofiles
@@ -13,12 +13,9 @@ from .connection_pool import execute_sql, unique_cursor, transaction
 from .datatype import UserRecord, FileReadPermission, FileRecord, DirectoryRecord, PathContents
 from .config import LARGE_BLOB_DIR
 from .log import get_logger
-from .utils import decode_uri_compnents
+from .utils import decode_uri_compnents, hash_credential
 from .error import *
 
-def hash_credential(username, password):
-    return hashlib.sha256((username + password).encode()).hexdigest()
-
 class DBObjectBase(ABC):
     logger = get_logger('database', global_instance=True)
     _cur: aiosqlite.Cursor
@@ -153,16 +150,6 @@ class FileConn(DBObjectBase):
             return []
         return [self.parse_record(r) for r in res]
     
-    async def get_user_file_records(self, owner_id: int) -> list[FileRecord]:
-        await self.cur.execute("SELECT * FROM fmeta WHERE owner_id = ?", (owner_id, ))
-        res = await self.cur.fetchall()
-        return [self.parse_record(r) for r in res]
-    
-    async def get_path_file_records(self, url: str) -> list[FileRecord]:
-        await self.cur.execute("SELECT * FROM fmeta WHERE url LIKE ?", (url + '%', ))
-        res = await self.cur.fetchall()
-        return [self.parse_record(r) for r in res]
-
     async def list_root(self, *usernames: str) -> list[DirectoryRecord]:
         """
         Efficiently list users' directories, if usernames is empty, list all users' directories.
@@ -340,25 +327,27 @@ class FileConn(DBObjectBase):
     async def log_access(self, url: str):
         await self.cur.execute("UPDATE fmeta SET access_time = CURRENT_TIMESTAMP WHERE url = ?", (url, ))
     
-    async def delete_file_record(self, url: str):
-        file_record = await self.get_file_record(url)
-        if file_record is None: return
-        await self.cur.execute("DELETE FROM fmeta WHERE url = ?", (url, ))
+    async def delete_file_record(self, url: str) -> Optional[FileRecord]:
+        res = await self.cur.execute("DELETE FROM fmeta WHERE url = ? RETURNING *", (url, ))
+        row = await res.fetchone()
+        if row is None:
+            raise FileNotFoundError(f"File {url} not found")
+        file_record = FileRecord(*row)
         await self._user_size_dec(file_record.owner_id, file_record.file_size)
         self.logger.info(f"Deleted fmeta {url}")
+        return file_record
     
-    async def delete_user_file_records(self, owner_id: int):
+    async def delete_user_file_records(self, owner_id: int) -> list[FileRecord]:
         cursor = await self.cur.execute("SELECT * FROM fmeta WHERE owner_id = ?", (owner_id, ))
         res = await cursor.fetchall()
-        await self.cur.execute("DELETE FROM fmeta WHERE owner_id = ?", (owner_id, ))
         await self.cur.execute("DELETE FROM usize WHERE user_id = ?", (owner_id, ))
-        self.logger.info(f"Deleted {len(res)} files for user {owner_id}") # type: ignore
+        res = await self.cur.execute("DELETE FROM fmeta WHERE owner_id = ? RETURNING *", (owner_id, ))
+        ret = [self.parse_record(r) for r in await res.fetchall()]
+        self.logger.info(f"Deleted {len(ret)} file(s) for user {owner_id}") # type: ignore
+        return ret
     
-    async def delete_path_records(self, path: str):
+    async def delete_path_records(self, path: str, under_user_id: Optional[int] = None) -> list[FileRecord]:
         """Delete all records with url starting with path"""
-        cursor = await self.cur.execute("SELECT * FROM fmeta WHERE url LIKE ?", (path + '%', ))
-        all_f_rec = await cursor.fetchall()
-        
         # update user size
         cursor = await self.cur.execute("SELECT DISTINCT owner_id FROM fmeta WHERE url LIKE ?", (path + '%', ))
         res = await cursor.fetchall()
@@ -368,8 +357,16 @@ class FileConn(DBObjectBase):
             if size is not None:
                 await self._user_size_dec(r[0], size[0])
         
-        await self.cur.execute("DELETE FROM fmeta WHERE url LIKE ?", (path + '%', ))
-        self.logger.info(f"Deleted {len(all_f_rec)} files for path {path}") # type: ignore
+        # if any new records are created here, the size update may be inconsistent
+        # but it's not a big deal...
+        
+        if under_user_id is None:
+            res = await self.cur.execute("DELETE FROM fmeta WHERE url LIKE ? RETURNING *", (path + '%', ))
+        else:
+            res = await self.cur.execute("DELETE FROM fmeta WHERE url LIKE ? AND owner_id = ? RETURNING *", (path + '%', under_user_id))
+        all_f_rec = await res.fetchall()
+        self.logger.info(f"Deleted {len(all_f_rec)} file(s) for path {path}") # type: ignore
+        return [self.parse_record(r) for r in all_f_rec]
     
     async def set_file_blob(self, file_id: str, blob: bytes):
         await self.cur.execute("INSERT OR REPLACE INTO blobs.fdata (file_id, data) VALUES (?, ?)", (file_id, blob))
@@ -546,28 +543,37 @@ class Database:
 
         return blob
 
-    async def delete_file(self, url: str) -> Optional[FileRecord]:
+    async def delete_file(self, url: str, assure_user: Optional[UserRecord] = None) -> Optional[FileRecord]:
         validate_url(url)
 
         async with transaction() as cur:
             fconn = FileConn(cur)
-            r = await fconn.get_file_record(url)
+            r = await fconn.delete_file_record(url)
             if r is None:
                 return None
+            if assure_user is not None:
+                if r.owner_id != assure_user.id:
+                    # will rollback
+                    raise PermissionDeniedError(f"Permission denied: {assure_user.username} cannot delete file {url}")
             f_id = r.file_id
-            await fconn.delete_file_record(url)
             if r.external:
                 await fconn.delete_file_blob_external(f_id)
             else:
                 await fconn.delete_file_blob(f_id)
             return r
     
-    async def move_file(self, old_url: str, new_url: str):
+    async def move_file(self, old_url: str, new_url: str, ensure_user: Optional[UserRecord] = None):
         validate_url(old_url)
         validate_url(new_url)
 
         async with transaction() as cur:
             fconn = FileConn(cur)
+            r = await fconn.get_file_record(old_url)
+            if r is None:
+                raise FileNotFoundError(f"File {old_url} not found")
+            if ensure_user is not None:
+                if r.owner_id != ensure_user.id:
+                    raise PermissionDeniedError(f"Permission denied: {ensure_user.username} cannot move file {old_url}")
             await fconn.move_file(old_url, new_url)
     
     async def move_path(self, user: UserRecord, old_url: str, new_url: str):
@@ -615,16 +621,16 @@ class Database:
             await fconn.delete_file_blob_external(external_ids[i])
             
 
-    async def delete_path(self, url: str):
+    async def delete_path(self, url: str, under_user: Optional[UserRecord] = None) -> Optional[list[FileRecord]]:
         validate_url(url, is_file=False)
+        user_id = under_user.id if under_user is not None else None
 
         async with transaction() as cur:
             fconn = FileConn(cur)
-            records = await fconn.get_path_file_records(url)
+            records = await fconn.delete_path_records(url, user_id)
             if not records:
                 return None
             await self.__batch_delete_file_blobs(fconn, records)
-            await fconn.delete_path_records(url)
             return records
     
     async def delete_user(self, u: str | int):
@@ -633,12 +639,17 @@ class Database:
             if user is None:
                 return
 
-            fconn = FileConn(cur)
-            records = await fconn.get_user_file_records(user.id)
-            await self.__batch_delete_file_blobs(fconn, records)
-            await fconn.delete_user_file_records(user.id)
+            # no new files can be added since profile deletion
             uconn = UserConn(cur)
             await uconn.delete_user(user.username)
+
+            fconn = FileConn(cur)
+            records = await fconn.delete_user_file_records(user.id)
+            await self.__batch_delete_file_blobs(fconn, records)
+
+            # make sure the user's directory is deleted, 
+            # may contain admin's files, but delete them all
+            await fconn.delete_path_records(user.username + '/')
     
     async def iter_path(self, top_url: str, urls: Optional[list[str]]) -> AsyncIterable[tuple[FileRecord, bytes | AsyncIterable[bytes]]]:
         async with unique_cursor() as cur:

lfss/src/server.py

@@ -76,6 +76,11 @@ async def get_current_user(
         raise HTTPException(status_code=401, detail="Invalid token")
     return user
 
+async def registered_user(user: UserRecord = Depends(get_current_user)):
+    if user.id == 0:
+        raise HTTPException(status_code=401, detail="Permission denied")
+    return user
+
 app = FastAPI(docs_url=None, redoc_url=None, lifespan=lifespan)
 app.add_middleware(
     CORSMiddleware,
@@ -186,11 +191,8 @@ async def put_file(
     path: str, 
     conflict: Literal["overwrite", "skip", "abort"] = "abort",
     permission: int = 0,
-    user: UserRecord = Depends(get_current_user)):
+    user: UserRecord = Depends(registered_user)):
     path = ensure_uri_compnents(path)
-    if user.id == 0:
-        logger.debug("Reject put request from DECOY_USER")
-        raise HTTPException(status_code=401, detail="Permission denied")
     if not path.startswith(f"{user.username}/") and not user.is_admin:
         logger.debug(f"Reject put request from {user.username} to {path}")
         raise HTTPException(status_code=403, detail="Permission denied")
@@ -217,6 +219,8 @@ async def put_file(
             }, content=json.dumps({"url": path}))
         # remove the old file
         exists_flag = True
+        if not user.is_admin and not file_record.owner_id == user.id:
+            raise HTTPException(status_code=403, detail="Permission denied, cannot overwrite other's file")
         await db.delete_file(path)
     
     # check content-type
@@ -267,19 +271,17 @@ async def put_file(
 
 @router_fs.delete("/{path:path}")
 @handle_exception
-async def delete_file(path: str, user: UserRecord = Depends(get_current_user)):
+async def delete_file(path: str, user: UserRecord = Depends(registered_user)):
     path = ensure_uri_compnents(path)
-    if user.id == 0:
-        raise HTTPException(status_code=401, detail="Permission denied")
     if not path.startswith(f"{user.username}/") and not user.is_admin:
         raise HTTPException(status_code=403, detail="Permission denied")
     
     logger.info(f"DELETE {path}, user: {user.username}")
 
     if path.endswith("/"):
-        res = await db.delete_path(path)
+        res = await db.delete_path(path, user if not user.is_admin else None)
     else:
-        res = await db.delete_file(path)
+        res = await db.delete_file(path, user if not user.is_admin else None)
 
     await db.record_user_activity(user.username)
     if res:
@@ -291,21 +293,23 @@ router_api = APIRouter(prefix="/_api")
 
 @router_api.get("/bundle")
 @handle_exception
-async def bundle_files(path: str, user: UserRecord = Depends(get_current_user)):
+async def bundle_files(path: str, user: UserRecord = Depends(registered_user)):
     logger.info(f"GET bundle({path}), user: {user.username}")
-    if user.id == 0:
-        raise HTTPException(status_code=401, detail="Permission denied")
     path = ensure_uri_compnents(path)
     assert path.endswith("/") or path == ""
 
     if not path == "" and path[0] == "/":   # adapt to both /path and path
         path = path[1:]
     
-    owner_records_cache = {}     # cache owner records, ID -> UserRecord
+    owner_records_cache: dict[int, UserRecord] = {}     # cache owner records, ID -> UserRecord
     async def is_access_granted(file_record: FileRecord):
         owner_id = file_record.owner_id
         owner = owner_records_cache.get(owner_id, None)
         if owner is None:
+            async with unique_cursor() as conn:
+                uconn = UserConn(conn)
+                owner = await uconn.get_user_by_id(owner_id)
+                assert owner is not None, "Owner not found"
             owner_records_cache[owner_id] = owner
             
         allow_access, _ = check_user_permission(user, owner, file_record)
@@ -334,16 +338,35 @@ async def bundle_files(path: str, user: UserRecord = Depends(get_current_user)):
 
 @router_api.get("/meta")
 @handle_exception
-async def get_file_meta(path: str, user: UserRecord = Depends(get_current_user)):
+async def get_file_meta(path: str, user: UserRecord = Depends(registered_user)):
+    """
+    Permission:
+        for file:
+            if file is under user's path, return the meta, 
+            else, determine by the permission same as get_file
+        for path:
+            if path is under user's path, return the meta, else return 403
+    """
     logger.info(f"GET meta({path}), user: {user.username}")
     path = ensure_uri_compnents(path)
+    is_file = not path.endswith("/")
     async with unique_cursor() as conn:
         fconn = FileConn(conn)
-        get_fn = fconn.get_file_record if not path.endswith("/") else fconn.get_path_record
-        record = await get_fn(path)
-
-    if not record:
-        raise HTTPException(status_code=404, detail="Path not found")
+        if is_file:
+            record = await fconn.get_file_record(path)
+            if not record:
+                raise HTTPException(status_code=404, detail="File not found")
+            if not path.startswith(f"{user.username}/") and not user.is_admin:
+                uconn = UserConn(conn)
+                owner = await uconn.get_user_by_id(record.owner_id)
+                assert owner is not None, "Owner not found"
+                is_allowed, reason = check_user_permission(user, owner, record)
+                if not is_allowed:
+                    raise HTTPException(status_code=403, detail=reason)
+        else:
+            record = await fconn.get_path_record(path)
+            if not path.startswith(f"{user.username}/") and not user.is_admin:
+                raise HTTPException(status_code=403, detail="Permission denied")
     return record
 
 @router_api.post("/meta")
@@ -352,10 +375,8 @@ async def update_file_meta(
     path: str, 
     perm: Optional[int] = None, 
     new_path: Optional[str] = None,
-    user: UserRecord = Depends(get_current_user)
+    user: UserRecord = Depends(registered_user)
     ):
-    if user.id == 0:
-        raise HTTPException(status_code=401, detail="Permission denied")
     path = ensure_uri_compnents(path)
     if path.startswith("/"):
         path = path[1:]
@@ -374,7 +395,7 @@ async def update_file_meta(
         if new_path is not None:
             new_path = ensure_uri_compnents(new_path)
             logger.info(f"Update path of {path} to {new_path}")
-            await db.move_file(path, new_path)
+            await db.move_file(path, new_path, user)
     
     # directory
     else:
@@ -389,9 +410,7 @@ async def update_file_meta(
     
 @router_api.get("/whoami")
 @handle_exception
-async def whoami(user: UserRecord = Depends(get_current_user)):
-    if user.id == 0:
-        raise HTTPException(status_code=401, detail="Login required")
+async def whoami(user: UserRecord = Depends(registered_user)):
     user.credential = "__HIDDEN__"
     return user
 

lfss/src/utils.py

@@ -2,6 +2,10 @@ import datetime
 import urllib.parse
 import asyncio
 import functools
+import hashlib
+
+def hash_credential(username: str, password: str):
+    return hashlib.sha256((username + password).encode()).hexdigest()
 
 def encode_uri_compnents(path: str):
     """

{lfss-0.7.2.dist-info → lfss-0.7.4.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lfss
-Version: 0.7.2
+Version: 0.7.4
 Summary: Lightweight file storage service
 Home-page: https://github.com/MenxLi/lfss
 Author: li, mengxun
@@ -22,8 +22,8 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/)
 
 My experiment on a lightweight file/object storage service.  
-It stores small files and metadata in sqlite, large files in the filesystem, and serves them through a simple REST API.  
-Tested on 2 million files, and it works fine... thanks to the sqlite database!
+It stores small files and metadata in sqlite, large files in the filesystem.  
+Tested on 2 million files, and it works fine... 
 
 Usage: 
 ```sh

{lfss-0.7.2.dist-info → lfss-0.7.4.dist-info}/RECORD

@@ -1,6 +1,6 @@
-Readme.md,sha256=t5dninN6gFzve_NySxqGrEkmAYhF1h1RhaMdCOeCILA,1348
+Readme.md,sha256=vsPotlwPAaHI5plh4aaszpi3rr7ZGDn7-wLdEYTWQ0k,1275
 docs/Known_issues.md,sha256=rfdG3j1OJF-59S9E06VPyn0nZKbW-ybPxkoZ7MEZWp8,81
-docs/Permission.md,sha256=EY1Y4tT5PDdHDb2pXsVgMAJXxkUihTZfPT0_z7Q53FA,1485
+docs/Permission.md,sha256=X0VNfBKU52f93QYqcVyiBFJ3yURiSkhIo9S_5fdSgzM,2265
 frontend/api.js,sha256=-ouhsmucEunAK3m1H__MqffQkXAjoeVEfM15BvqfIZs,7677
 frontend/index.html,sha256=VPJDs2LG8ep9kjlsKzjWzpN9vc1VGgdvOUlNTZWyQoQ,2088
 frontend/popup.css,sha256=VzkjG1ZTLxhHMtTyobnlvqYmVsTmdbJJed2Pu1cc06c,1007
@@ -9,25 +9,25 @@ frontend/scripts.js,sha256=hQ8m3L7P-LplLqrPUWD6pBo4C_tCUl2XZKRNtkWBy8I,21155
 frontend/styles.css,sha256=wly8O-zF4EUgV12Tv1bATSfmJsLITv2u3_SiyXVaxv4,4096
 frontend/utils.js,sha256=Ts4nlef8pkrEgpwX-uQwAhWvwxlIzex8ijDLNCa22ps,2372
 lfss/cli/balance.py,sha256=heOgwH6oNnfYsKJfA4VxWKdEXPstdVbbRXWxcDqLIS0,4176
-lfss/cli/cli.py,sha256=bJOeEyri_XVWUvnjohsw_oPYKp-bELxLrg5sVWOpKQA,2259
+lfss/cli/cli.py,sha256=Yup3xIVEQPu10uM8dq1bvre1fK5ngweQHxXZsgQq4Hc,4187
 lfss/cli/panel.py,sha256=iGdVmdWYjA_7a78ZzWEB_3ggIOBeUKTzg6F5zLaB25c,1401
 lfss/cli/serve.py,sha256=bO3GT0kuylMGN-7bZWP4e71MlugGZ_lEMkYaYld_Ntg,985
-lfss/cli/user.py,sha256=Bu2IOchLdClBqjBqVeDck3kE0UaKWpL6mG5SPihBorc,3498
-lfss/client/__init__.py,sha256=YttaGTBup7mxnW0CtxdZPF0HWga2cGM4twz9MXJIrts,1827
-lfss/client/api.py,sha256=ICqpcyvSf-9QmYNv9EQ5fA_MViSuLxSNn-CIBNqWkW8,5414
+lfss/cli/user.py,sha256=h-USWF6lB0Ztm9vwQznqsghKJ5INq5mBmaQeX2D5F-w,3490
+lfss/client/__init__.py,sha256=R9erioMInKIPZ0rXu1J-4mezbwyGMjig18532ATFP5s,4545
+lfss/client/api.py,sha256=aun5HWVNPBeJK6x0_iaM8gVcE3wX6yaqX0TsfsfifSw,5728
 lfss/sql/init.sql,sha256=C-JtQAlaOjESI8uoF1Y_9dKukEVSw5Ll-7yA3gG-XHU,1210
 lfss/sql/pragma.sql,sha256=uENx7xXjARmro-A3XAK8OM8v5AxDMdCCRj47f86UuXg,206
 lfss/src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-lfss/src/config.py,sha256=G-tOdk6RrE62_d-QoD6WXD4l70YriBazUFdumTz3P20,529
-lfss/src/connection_pool.py,sha256=qBtXr2j_O7vGI2Mv0HVsza35T8nljIx85jLlyEv1SgE,4918
-lfss/src/database.py,sha256=tGcHfxThmRTmtLIlEWdJtAe1tzP64wHSC1JcbQdRU-8,31226
+lfss/src/config.py,sha256=z0aVOW8yGgKSryhQCWTf2RY3iHmKMcKIZ-HiosTnPRs,539
+lfss/src/connection_pool.py,sha256=69QMJ4gRQ62qi39t0JKdvIaWRBrbU9S7slutIpCc30A,4959
+lfss/src/database.py,sha256=VKTLJSUFImF3pzMUaqGWEh1H06Yk3VgJa78bw5rAvG8,32147
 lfss/src/datatype.py,sha256=BLS7vuuKnFZQg0nrKeP9SymqUhcN6HwPgejU0yBd_Ak,1622
 lfss/src/error.py,sha256=imbhwnbhnI3HLhkbfICROe3F0gleKrOk4XnqHJDOtuI,285
 lfss/src/log.py,sha256=qNE04sHoZ2rMbQ5dR2zT7Xaz1KVAfYp5hKpWJX42S9g,5244
-lfss/src/server.py,sha256=9mnBlcHJruwOV-aoHa1P146F4khpVDmif7rq58yZX3U,15306
+lfss/src/server.py,sha256=4GYDAa9Fx4H4fYh_He1u1zlIro3zaQmaEBNKpPO2Q2E,16374
 lfss/src/stat.py,sha256=hTMtQyM_Ukmhc33Bb9FGCfBMIX02KrGHQg8nL7sC8sU,2082
-lfss/src/utils.py,sha256=8VkrtpSmurbMiX7GyK-n7Grvzy3uwSJXHdONEsuLCDI,2272
-lfss-0.7.2.dist-info/METADATA,sha256=pJcPBmMpKwNujl4CWZKRchT3uXyCPaEvi1mek7SUJUE,2040
-lfss-0.7.2.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
-lfss-0.7.2.dist-info/entry_points.txt,sha256=d_Ri3GXxUW-S0E6q953A8od0YMmUAnZGlJSKS46OiW8,172
-lfss-0.7.2.dist-info/RECORD,,
+lfss/src/utils.py,sha256=ZE3isOS3gafEYw1z8s2ucY08eWQHIN4YUbKYH8F1hEQ,2409
+lfss-0.7.4.dist-info/METADATA,sha256=XeJ4ft1B9KDdfcxyFrFdwUMTYc1QsUAkgqI8v5rwbq4,1967
+lfss-0.7.4.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
+lfss-0.7.4.dist-info/entry_points.txt,sha256=d_Ri3GXxUW-S0E6q953A8od0YMmUAnZGlJSKS46OiW8,172
+lfss-0.7.4.dist-info/RECORD,,