lfss 0.9.1__py3-none-any.whl → 0.9.4__py3-none-any.whl

Readme.md

@@ -34,10 +34,12 @@ The API usage is simple, just `GET`, `PUT`, `DELETE` to the `/<username>/file/ur
 The authentication can be acheived through one of the following methods:
 1. `Authorization` header with the value `Bearer sha256(<username><password>)`.
 2. `token` query parameter with the value `sha256(<username><password>)`.
-3. HTTP Basic Authentication with the username and password. 
+3. HTTP Basic Authentication with the username and password (If WebDAV is enabled).
 
 You can refer to `frontend` as an application example, `lfss/api/connector.py` for more APIs. 
 
 By default, the service exposes all files to the public for `GET` requests, 
 but file-listing is restricted to the user's own files.  
-Please refer to [docs/Permission.md](./docs/Permission.md) for more details on the permission system.
+Please refer to [docs/Permission.md](./docs/Permission.md) for more details on the permission system. 
+
+More can be found in the [docs](./docs) directory.

docs/Changelog.md

@@ -0,0 +1,27 @@
+
+## 0.9
+
+### 0.9.4
+- Decode WebDAV file name. 
+- Allow root-listing for WebDAV.
+- Always return 207 status code for propfind.
+- Refactor debounce utility. 
+
+### 0.9.3
+- Fix empty file getting.
+- HTTP `PUT/POST` default to overwrite the file.
+- Use shared implementations for `PUT`, `GET`, `DELETE` methods.
+- Inherit permission on overwriting `unset` permission files.
+
+### 0.9.2
+- Native copy function.
+- Only enable basic authentication if WebDAV is enabled.
+- `WWW-Authenticate` header is now added to the response when authentication fails.
+
+### 0.9.1
+- Add WebDAV support.
+- Code refactor, use `lfss.eng` and `lfss.svc`.
+
+### 0.9.0
+- User peer access control, now user can share their path with other users.
+- Fix high concurrency database locking on file getting.

docs/Enviroment_variables.md

@@ -0,0 +1,12 @@
+
+# Enviroment variables
+
+**Server**
+- `LFSS_DATA`: The directory to store the data. Default is `.storage_data`.
+- `LFSS_WEBDAV`: Enable WebDAV support. Default is `0`, set to `1` to enable.
+- `LFSS_LARGE_FILE`: The size limit of the file to store in the database. Default is `8m`.
+- `LFSS_DEBUG`: Enable debug mode for more verbose logging. Default is `0`, set to `1` to enable.
+
+**Client**
+- `LFSS_ENDPOINT`: The fallback server endpoint. Default is `http://localhost:8000`.
+- `LFSS_TOKEN`: The fallback token to authenticate. Should be `sha256(<username><password>)`.

docs/Known_issues.md

@@ -1 +1,3 @@
-[Safari 中文输入法回车捕获](https://github.com/anse-app/anse/issues/127)
+[Safari 中文输入法回车捕获](https://github.com/anse-app/anse/issues/127)
+
+[Word 临时文件](https://answers.microsoft.com/en-us/msoffice/forum/all/mac-os-word-temp-sb-folders-created-on-smb-share/40fda56c-c77c-4365-8fa3-eb87ac814207?page=1)

docs/Webdav.md

@@ -15,8 +15,8 @@ Please note:
 2. LFSS not allow creating files in the root directory, however some client such as [Finder](https://sabre.io/dav/clients/finder/) will try to create files in the root directory. Thus, it is safer to mount the user directory only, e.g. `http://localhost:8000/<username>/`.
 3. LFSS not allow directory creation, instead it creates directoy implicitly when a file is uploaded to a non-exist directory. 
    i.e. `PUT http://localhost:8000/<username>/dir/file.txt` will create the `dir` directory if it does not exist. 
-   However, the WebDAV `MKCOL` method requires the directory to be created explicitly, so WebDAV `MKCOL` method instead create a decoy file on the path (`.lfss-keep`), and hide the file from the file listing by `PROPFIND` method. 
+   However, the WebDAV `MKCOL` method requires the directory to be created explicitly, so WebDAV `MKCOL` method instead create a decoy file on the path (`.lfss_keep`), and hide the file from the file listing by `PROPFIND` method. 
    This leads to: 
-   1) You may see a `.lfss-keep` file in the directory with native file listing (e.g. `/_api/list-files`), but it is hidden in WebDAV clients.
-   2) The directory may be deleted if there is no file in it and the `.lfss-keep` file is not created by WebDAV client.  
+   1) You may see a `.lfss_keep` file in the directory with native file listing (e.g. `/_api/list-files`), but it is hidden in WebDAV clients.
+   2) The directory may be deleted if there is no file in it and the `.lfss_keep` file is not created by WebDAV client.  
 

frontend/api.js

@@ -384,6 +384,27 @@ export default class Connector {
         }
     }
 
+    /**
+     * @param {string} srcPath - file path(url)
+     * @param {string} dstPath - new file path(url)
+     */
+    async copy(srcPath, dstPath){
+        if (srcPath.startsWith('/')){ srcPath = srcPath.slice(1); }
+        if (dstPath.startsWith('/')){ dstPath = dstPath.slice(1); }
+        const dst = new URL(this.config.endpoint + '/_api/copy');
+        dst.searchParams.append('src', srcPath);
+        dst.searchParams.append('dst', dstPath);
+        const res = await fetch(dst.toString(), {
+            method: 'POST',
+            headers: {
+                'Authorization': 'Bearer ' + this.config.token,
+                'Content-Type': 'application/www-form-urlencoded'
+            },
+        });
+        if (!(res.status == 200 || res.status == 201)){
+            throw new Error(`Failed to copy file, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
+        }
+    }
 }
 
 /**

frontend/scripts.js

@@ -347,7 +347,7 @@ async function refreshFileList(){
                                 }
                             );
                         }, {
-                            text: 'Enter the destination path: ',
+                            text: 'Enter the destination path (Move): ',
                             placeholder: 'Destination path',
                             value: decodePathURI(dirurl), 
                             select: "last-pathname"
@@ -355,6 +355,30 @@ async function refreshFileList(){
                     });
                     actContainer.appendChild(moveButton);
 
+                    const copyButton = document.createElement('a');
+                    copyButton.textContent = 'Copy';
+                    copyButton.style.cursor = 'pointer';
+                    copyButton.addEventListener('click', () => {
+                        showFloatingWindowLineInput((dstPath) => {
+                            dstPath = encodePathURI(dstPath);
+                            console.log("Copying", dirurl, "to", dstPath);
+                            conn.copy(dirurl, dstPath)
+                                .then(() => {
+                                    refreshFileList();
+                                }, 
+                                (err) => {
+                                    showPopup('Failed to copy path: ' + err, {level: 'error'});
+                                }
+                            );
+                        }, {
+                            text: 'Enter the destination path (Copy): ',
+                            placeholder: 'Destination path',
+                            value: decodePathURI(dirurl), 
+                            select: "last-pathname"
+                        });
+                    });
+                    actContainer.appendChild(copyButton);
+
                     const downloadButton = document.createElement('a');
                     downloadButton.textContent = 'Download';
                     downloadButton.href = conn.config.endpoint + '/_api/bundle?' + 
@@ -478,7 +502,7 @@ async function refreshFileList(){
                                 }
                             );
                         }, {
-                            text: 'Enter the destination path: ',
+                            text: 'Enter the destination path (Move): ',
                             placeholder: 'Destination path',
                             value: decodePathURI(file.url), 
                             select: "last-filename"
@@ -486,6 +510,29 @@ async function refreshFileList(){
                     });
                     actContainer.appendChild(moveButton);
 
+                    const copyButton = document.createElement('a');
+                    copyButton.textContent = 'Copy';
+                    copyButton.style.cursor = 'pointer';
+                    copyButton.addEventListener('click', () => {
+                        showFloatingWindowLineInput((dstPath) => {
+                            dstPath = encodePathURI(dstPath);
+                            conn.copy(file.url, dstPath)
+                                .then(() => {
+                                    refreshFileList();
+                                }, 
+                                (err) => {
+                                    showPopup('Failed to copy file: ' + err, {level: 'error'});
+                                }
+                            );
+                        }, {
+                            text: 'Enter the destination path (Copy): ',
+                            placeholder: 'Destination path',
+                            value: decodePathURI(file.url), 
+                            select: "last-filename"
+                        });
+                    });
+                    actContainer.appendChild(copyButton);
+
                     const downloadBtn = document.createElement('a');
                     downloadBtn.textContent = 'Download';
                     downloadBtn.href = conn.config.endpoint + '/' + file.url + '?download=true&token=' + conn.config.token;

lfss/api/connector.py

@@ -270,6 +270,12 @@ class Connector:
         self._fetch_factory('POST', '_api/meta', {'path': path, 'new_path': new_path})(
             headers = {'Content-Type': 'application/www-form-urlencoded'}
         )
+    
+    def copy(self, src: str, dst: str):
+        """Copy file from src to dst."""
+        self._fetch_factory('POST', '_api/copy', {'src': src, 'dst': dst})(
+            headers = {'Content-Type': 'application/www-form-urlencoded'}
+        )
         
     def whoami(self) -> UserRecord:
         """Gets information about the current user."""

lfss/eng/connection_pool.py

@@ -29,7 +29,7 @@ async def get_connection(read_only: bool = False) -> aiosqlite.Connection:
 
     conn = await aiosqlite.connect(
         get_db_uri(DATA_HOME / 'index.db', read_only=read_only), 
-        timeout = 20, uri = True
+        timeout = 10, uri = True
         )
     async with conn.cursor() as c:
         await c.execute(

lfss/eng/database.py

@@ -161,7 +161,7 @@ class UserConn(DBObjectBase):
     async def list_peer_users(self, src_user: int | str, level: AccessLevel) -> list[UserRecord]:
         """
         List all users that src_user can do [AliasLevel] to, with level >= level, 
-        Note: the returned list does not include src_user and admin users
+        Note: the returned list does not include src_user and is not apporiate for admin (who has all permissions for all users)
         """
         assert int(level) > AccessLevel.NONE, f"Invalid level, {level}"
         match src_user:
@@ -427,8 +427,7 @@ class FileConn(DBObjectBase):
         await self._user_size_inc(user_id, old.file_size)
         self.logger.info(f"Copied file {old_url} to {new_url}")
     
-    # not tested
-    async def copy_path(self, old_url: str, new_url: str, conflict_handler: Literal['skip', 'overwrite'] = 'overwrite', user_id: Optional[int] = None):
+    async def copy_path(self, old_url: str, new_url: str, user_id: Optional[int] = None):
         assert old_url.endswith('/'), "Old path must end with /"
         assert new_url.endswith('/'), "New path must end with /"
         if user_id is None:
@@ -440,11 +439,8 @@ class FileConn(DBObjectBase):
         for r in res:
             old_record = FileRecord(*r)
             new_r = new_url + old_record.url[len(old_url):]
-            if conflict_handler == 'overwrite':
-                await self.cur.execute("DELETE FROM fmeta WHERE url = ?", (new_r, ))
-            elif conflict_handler == 'skip':
-                if (await self.cur.execute("SELECT url FROM fmeta WHERE url = ?", (new_r, ))) is not None:
-                    continue
+            if await (await self.cur.execute("SELECT url FROM fmeta WHERE url = ?", (new_r, ))).fetchone() is not None:
+                raise FileExistsError(f"File {new_r} already exists")
             new_fid = str(uuid.uuid4())
             user_id = old_record.owner_id if user_id is None else user_id
             await self.cur.execute(
@@ -456,6 +452,7 @@ class FileConn(DBObjectBase):
             else:
                 await copy_file(LARGE_BLOB_DIR / old_record.file_id, LARGE_BLOB_DIR / new_fid)
             await self._user_size_inc(user_id, old_record.file_size)
+        self.logger.info(f"Copied path {old_url} to {new_url}")
     
     async def move_file(self, old_url: str, new_url: str):
         old = await self.get_file_record(old_url)
@@ -467,7 +464,7 @@ class FileConn(DBObjectBase):
         await self.cur.execute("UPDATE fmeta SET url = ?, create_time = CURRENT_TIMESTAMP WHERE url = ?", (new_url, old_url))
         self.logger.info(f"Moved file {old_url} to {new_url}")
     
-    async def move_path(self, old_url: str, new_url: str, conflict_handler: Literal['skip', 'overwrite'] = 'overwrite', user_id: Optional[int] = None):
+    async def move_path(self, old_url: str, new_url: str, user_id: Optional[int] = None):
         assert old_url.endswith('/'), "Old path must end with /"
         assert new_url.endswith('/'), "New path must end with /"
         if user_id is None:
@@ -478,11 +475,9 @@ class FileConn(DBObjectBase):
             res = await cursor.fetchall()
         for r in res:
             new_r = new_url + r[0][len(old_url):]
-            if conflict_handler == 'overwrite':
-                await self.cur.execute("DELETE FROM fmeta WHERE url = ?", (new_r, ))
-            elif conflict_handler == 'skip':
-                if (await self.cur.execute("SELECT url FROM fmeta WHERE url = ?", (new_r, ))) is not None:
-                    continue
+            if await (await self.cur.execute("SELECT url FROM fmeta WHERE url = ?", (new_r, ))).fetchone():
+                self.logger.error(f"File {new_r} already exists on move path: {old_url} -> {new_url}")
+                raise FileDuplicateError(f"File {new_r} already exists")
             await self.cur.execute("UPDATE fmeta SET url = ?, create_time = CURRENT_TIMESTAMP WHERE url = ?", (new_r, r[0]))
     
     async def log_access(self, url: str):
@@ -790,6 +785,8 @@ class Database:
             if op_user is not None:
                 if await check_path_permission(old_url, op_user, cursor=cur) < AccessLevel.WRITE:
                     raise PermissionDeniedError(f"Permission denied: {op_user.username} cannot move file {old_url}")
+                if await check_path_permission(new_url, op_user, cursor=cur) < AccessLevel.WRITE:
+                    raise PermissionDeniedError(f"Permission denied: {op_user.username} cannot move file to {new_url}")
             await fconn.move_file(old_url, new_url)
 
             new_mime, _ = mimetypes.guess_type(new_url)
@@ -834,7 +831,7 @@ class Database:
 
         async with transaction() as cur:
             fconn = FileConn(cur)
-            await fconn.move_path(old_url, new_url, 'overwrite', op_user.id)
+            await fconn.move_path(old_url, new_url, op_user.id)
     
     # not tested
     async def copy_path(self, old_url: str, new_url: str, op_user: UserRecord):
@@ -858,7 +855,7 @@ class Database:
         
         async with transaction() as cur:
             fconn = FileConn(cur)
-            await fconn.copy_path(old_url, new_url, 'overwrite', op_user.id)
+            await fconn.copy_path(old_url, new_url, op_user.id)
 
     async def __batch_delete_file_blobs(self, fconn: FileConn, file_records: list[FileRecord], batch_size: int = 512):
         # https://github.com/langchain-ai/langchain/issues/10321

lfss/eng/error.py

@@ -6,13 +6,17 @@ class FileLockedError(LFSSExceptionBase):...
 
 class InvalidOptionsError(LFSSExceptionBase, ValueError):...
 
+class InvalidDataError(LFSSExceptionBase, ValueError):...
+
+class InvalidPathError(LFSSExceptionBase, ValueError):...
+
 class DatabaseLockedError(LFSSExceptionBase, sqlite3.DatabaseError):...
 
 class PathNotFoundError(LFSSExceptionBase, FileNotFoundError):...
 
-class PermissionDeniedError(LFSSExceptionBase, PermissionError):...
+class FileDuplicateError(LFSSExceptionBase, FileExistsError):...
 
-class InvalidPathError(LFSSExceptionBase, ValueError):...
+class PermissionDeniedError(LFSSExceptionBase, PermissionError):...
 
 class StorageExceededError(LFSSExceptionBase):...
 

lfss/eng/thumb.py

@@ -1,5 +1,6 @@
 from lfss.eng.config import THUMB_DB, THUMB_SIZE
 from lfss.eng.database import FileConn
+from lfss.eng.error import *
 from lfss.eng.connection_pool import unique_cursor
 from typing import Optional
 from PIL import Image
@@ -32,7 +33,10 @@ async def _get_cache_thumb(c: aiosqlite.Cursor, path: str, ctime: str) -> Option
     return blob
     
 async def _save_cache_thumb(c: aiosqlite.Cursor, path: str, ctime: str, raw_bytes: bytes) -> bytes:
-    raw_img = Image.open(BytesIO(raw_bytes))
+    try:
+        raw_img = Image.open(BytesIO(raw_bytes))
+    except Exception:
+        raise InvalidDataError('Invalid image data for thumbnail: ' + path)
     raw_img.thumbnail(THUMB_SIZE)
     img = raw_img.convert('RGB')
     bio = BytesIO()

lfss/eng/utils.py

@@ -36,17 +36,41 @@ def ensure_uri_compnents(path: str):
     """ Ensure the path components are safe to use """
     return encode_uri_compnents(decode_uri_compnents(path))
 
-g_debounce_tasks: OrderedDict[str, asyncio.Task] = OrderedDict()
-lock_debounce_task_queue = Lock()
+class TaskManager:
+    def __init__(self):
+        self._tasks: OrderedDict[str, asyncio.Task] = OrderedDict()
+    
+    def push(self, task: asyncio.Task) -> str:
+        tid = uuid4().hex
+        if tid in self._tasks:
+            raise ValueError("Task ID collision")
+        self._tasks[tid] = task
+        return tid
+    
+    def cancel(self, task_id: str):
+        task = self._tasks.pop(task_id, None)
+        if task is not None:
+            task.cancel()
+    
+    def truncate(self):
+        new_tasks = OrderedDict()
+        for tid, task in self._tasks.items():
+            if not task.done():
+                new_tasks[tid] = task
+        self._tasks = new_tasks
+    
+    async def wait_all(self):
+        async def stop_task(task: asyncio.Task):
+            if not task.done():
+                await task
+        await asyncio.gather(*map(stop_task, self._tasks.values()))
+        self._tasks.clear()
+    
+    def __len__(self): return len(self._tasks)
+
+g_debounce_tasks: TaskManager = TaskManager()
 async def wait_for_debounce_tasks():
-    async def stop_task(task: asyncio.Task):
-        task.cancel()
-        try:
-            await task
-        except asyncio.CancelledError:
-            pass
-    await asyncio.gather(*map(stop_task, g_debounce_tasks.values()))
-    g_debounce_tasks.clear()
+    await g_debounce_tasks.wait_all()
 
 def debounce_async(delay: float = 0.1, max_wait: float = 1.):
     """ 
@@ -54,7 +78,8 @@ def debounce_async(delay: float = 0.1, max_wait: float = 1.):
     ensuring execution at least once every `max_wait` seconds. 
     """
     def debounce_wrap(func):
-        task_record: tuple[str, asyncio.Task] | None = None
+        # task_record: tuple[str, asyncio.Task] | None = None
+        prev_task_id = None
         fn_execution_lock = Lock()
         last_execution_time = 0
 
@@ -67,12 +92,11 @@ def debounce_async(delay: float = 0.1, max_wait: float = 1.):
 
         @functools.wraps(func)
         async def wrapper(*args, **kwargs):
-            nonlocal task_record, last_execution_time
+            nonlocal prev_task_id, last_execution_time
 
-            async with lock_debounce_task_queue:
-                if task_record is not None:
-                    task_record[1].cancel()
-                    g_debounce_tasks.pop(task_record[0], None)
+            if prev_task_id is not None:
+                g_debounce_tasks.cancel(prev_task_id)
+                prev_task_id = None
             
             async with fn_execution_lock:
                 if time.monotonic() - last_execution_time > max_wait:
@@ -81,14 +105,12 @@ def debounce_async(delay: float = 0.1, max_wait: float = 1.):
                     return
 
             task = asyncio.create_task(delayed_func(*args, **kwargs))
-            task_uid = uuid4().hex
-            task_record = (task_uid, task)
-            async with lock_debounce_task_queue:
-                g_debounce_tasks[task_uid] = task
-                if len(g_debounce_tasks) > 2048:
-                    # finished tasks are not removed from the dict
-                    # so we need to clear it periodically
-                    await wait_for_debounce_tasks()
+            prev_task_id = g_debounce_tasks.push(task)
+            if len(g_debounce_tasks) > 1024:
+                # finished tasks are not removed from the dict
+                # so we need to clear it periodically
+                g_debounce_tasks.truncate()
+
         return wrapper
     return debounce_wrap
 

lfss/svc/app.py

@@ -1,9 +1,9 @@
+from .app_base import ENABLE_WEBDAV
 from .app_native import *
-import os
 
 # order matters
 app.include_router(router_api)
-if os.environ.get("LFSS_WEBDAV", "0") == "1":
+if ENABLE_WEBDAV:
     from .app_dav import *
     app.include_router(router_dav)
 app.include_router(router_fs)

lfss/svc/app_base.py

@@ -1,4 +1,4 @@
-import asyncio, time
+import asyncio, time, os
 from contextlib import asynccontextmanager
 from typing import Optional
 from functools import wraps
@@ -17,6 +17,7 @@ from ..eng.error import *
 from ..eng.config import DEBUG_MODE
 from .request_log import RequestDB
 
+ENABLE_WEBDAV = os.environ.get("LFSS_WEBDAV", "0") == "1"
 logger = get_logger("server", term_level="DEBUG")
 logger_failed_request = get_logger("failed_requests", term_level="INFO")
 db = Database()
@@ -46,12 +47,14 @@ def handle_exception(fn):
             if isinstance(e, StorageExceededError): raise HTTPException(status_code=413, detail=str(e))
             if isinstance(e, PermissionError): raise HTTPException(status_code=403, detail=str(e))
             if isinstance(e, InvalidPathError): raise HTTPException(status_code=400, detail=str(e))
+            if isinstance(e, InvalidOptionsError): raise HTTPException(status_code=400, detail=str(e))
+            if isinstance(e, InvalidDataError): raise HTTPException(status_code=400, detail=str(e))
             if isinstance(e, FileNotFoundError): raise HTTPException(status_code=404, detail=str(e))
+            if isinstance(e, FileDuplicateError): raise HTTPException(status_code=409, detail=str(e))
             if isinstance(e, FileExistsError): raise HTTPException(status_code=409, detail=str(e))
             if isinstance(e, TooManyItemsError): raise HTTPException(status_code=400, detail=str(e))
             if isinstance(e, DatabaseLockedError): raise HTTPException(status_code=503, detail=str(e))
             if isinstance(e, FileLockedError): raise HTTPException(status_code=423, detail=str(e))
-            if isinstance(e, InvalidOptionsError): raise HTTPException(status_code=400, detail=str(e))
             logger.error(f"Uncaptured error in {fn.__name__}: {e}")
             raise 
     return wrapper
@@ -119,13 +122,13 @@ async def get_current_user(
         uconn = UserConn(conn)
         if h_token:
             user = await uconn.get_user_by_credential(h_token.credentials)
-            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic"})
-        elif b_token:
+            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic" if ENABLE_WEBDAV else "Bearer"})
+        elif ENABLE_WEBDAV and b_token:
             user = await uconn.get_user_by_credential(hash_credential(b_token.username, b_token.password))
-            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic"})
+            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic" if ENABLE_WEBDAV else "Bearer"})
         elif q_token:
             user = await uconn.get_user_by_credential(q_token)
-            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic"})
+            if not user: raise HTTPException(status_code=401, detail="Invalid token", headers={"WWW-Authenticate": "Basic" if ENABLE_WEBDAV else "Bearer"})
         else:
             return DECOY_USER
 
@@ -136,10 +139,9 @@ async def get_current_user(
 
 async def registered_user(user: UserRecord = Depends(get_current_user)):
     if user.id == 0:
-        raise HTTPException(status_code=401, detail="Permission denied", headers={"WWW-Authenticate": "Basic"})
+        raise HTTPException(status_code=401, detail="Permission denied", headers={"WWW-Authenticate": "Basic" if ENABLE_WEBDAV else "Bearer"})
     return user
 
-
 router_api = APIRouter(prefix="/_api")
 router_dav = APIRouter(prefix="")
 router_fs = APIRouter(prefix="")