lfss 0.9.2__py3-none-any.whl → 0.11.4__py3-none-any.whl

Readme.md

@@ -1,5 +1,5 @@
-# Lightweight File Storage Service (LFSS)
-[![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/)
+# Lite File Storage Service (LFSS)
+[![PyPI](https://img.shields.io/pypi/v/lfss)](https://pypi.org/project/lfss/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/lfss)](https://pypi.org/project/lfss/)
 
 My experiment on a lightweight and high-performance file/object storage service...  
 
@@ -32,8 +32,8 @@ Or, you can start a web server at `/frontend` and open `index.html` in your brow
 
 The API usage is simple, just `GET`, `PUT`, `DELETE` to the `/<username>/file/url` path.  
 The authentication can be acheived through one of the following methods:
-1. `Authorization` header with the value `Bearer sha256(<username><password>)`.
-2. `token` query parameter with the value `sha256(<username><password>)`.
+1. `Authorization` header with the value `Bearer sha256(<username>:<password>)`.
+2. `token` query parameter with the value `sha256(<username>:<password>)`.
 3. HTTP Basic Authentication with the username and password (If WebDAV is enabled).
 
 You can refer to `frontend` as an application example, `lfss/api/connector.py` for more APIs. 

docs/Enviroment_variables.md

@@ -4,9 +4,11 @@
 **Server**
 - `LFSS_DATA`: The directory to store the data. Default is `.storage_data`.
 - `LFSS_WEBDAV`: Enable WebDAV support. Default is `0`, set to `1` to enable.
-- `LFSS_LARGE_FILE`: The size limit of the file to store in the database. Default is `8m`.
+- `LFSS_LARGE_FILE`: The size limit of the file to store in the database. Default is `1m`.
 - `LFSS_DEBUG`: Enable debug mode for more verbose logging. Default is `0`, set to `1` to enable.
+- `LFSS_DISABLE_LOGGING`: Disable all file logging. Default is 0; set to `1` to disable file logging. 
+- `LFSS_ORIGIN`: The `Origin` header to allow CORS requests. Use `,` to separate multiple origins. Default is `*`.
 
 **Client**
 - `LFSS_ENDPOINT`: The fallback server endpoint. Default is `http://localhost:8000`.
-- `LFSS_TOKEN`: The fallback token to authenticate. Should be `sha256(<username><password>)`.
+- `LFSS_TOKEN`: The fallback token to authenticate. Should be `sha256(<username>:<password>)`.

docs/Permission.md

@@ -22,16 +22,16 @@ A file is owned by the user who created it, may not necessarily be the user unde
 ## File access with `GET` permission
 
 ### File access
-For accessing file content, the user must have `GET` permission of the file, which is determined by the `permission` field of both the owner and the file.   
+For accessing file content, the user must have `GET` permission of the file, which is determined by the `permission` field of both the path-owner and the file.   
 
 There are four types of permissions: `unset`, `public`, `protected`, `private`.
 Non-admin users can access files based on:   
 
 - If the file is `public`, then all users can access it.
 - If the file is `protected`, then only the logged-in user can access it.  
-- If the file is `private`, then only the owner can access it.
-- If the file is `unset`, then the file's permission is inherited from the owner's permission.
-- If both the owner and the file have `unset` permission, then the file is `public`.
+- If the file is `private`, then only the owner/path-owner can access it.
+- If the file is `unset`, then the file's permission is inherited from the path-owner's permission.
+- If both the path-owner and the file have `unset` permission, then the file is `public`.
 
 ## File creation with `PUT`/`POST` permission
 `PUT`/`POST` permission is not allowed for non-peer users.

docs/Webdav.md

@@ -15,8 +15,8 @@ Please note:
 2. LFSS not allow creating files in the root directory, however some client such as [Finder](https://sabre.io/dav/clients/finder/) will try to create files in the root directory. Thus, it is safer to mount the user directory only, e.g. `http://localhost:8000/<username>/`.
 3. LFSS not allow directory creation, instead it creates directoy implicitly when a file is uploaded to a non-exist directory. 
    i.e. `PUT http://localhost:8000/<username>/dir/file.txt` will create the `dir` directory if it does not exist. 
-   However, the WebDAV `MKCOL` method requires the directory to be created explicitly, so WebDAV `MKCOL` method instead create a decoy file on the path (`.lfss-keep`), and hide the file from the file listing by `PROPFIND` method. 
+   However, the WebDAV `MKCOL` method requires the directory to be created explicitly, so WebDAV `MKCOL` method instead create a decoy file on the path (`.lfss_keep`), and hide the file from the file listing by `PROPFIND` method. 
    This leads to: 
-   1) You may see a `.lfss-keep` file in the directory with native file listing (e.g. `/_api/list-files`), but it is hidden in WebDAV clients.
-   2) The directory may be deleted if there is no file in it and the `.lfss-keep` file is not created by WebDAV client.  
+   1) You may see a `.lfss_keep` file in the directory with native file listing (e.g. `/_api/list-files`), but it is hidden in WebDAV clients.
+   2) The directory may be deleted if there is no file in it and the `.lfss_keep` file is not created by WebDAV client.  
 

docs/changelog.md

@@ -0,0 +1,58 @@
+## 0.11
+
+### 0.11.2
+- Improve frontend directory upload feedback. 
+- Set default large file threashold to 1M. 
+- Increase default concurrent threads. 
+- Use sqlite for logging.
+- Add vacuum logs. 
+- Refactor: use dir for directory path. 
+
+### 0.11.1
+- Rename api `get_meta` function.
+- Frontend support upload directory.  
+- Fix admin put to non-exists user path. 
+
+### 0.11.0
+- Copy file as hard link. 
+- Add vacuum thumb and all.
+- Thumb database use file_id as index. 
+- improve username and url check with regular expression.
+
+## 0.10
+
+### 0.10.0
+- Inherit permission from path owner for `unset` permission files.
+- Add timeout and verify options for client api.
+- Bundle small files in memory.
+
+## 0.9
+
+### 0.9.5
+- Stream bundle path as zip file.
+- Update authentication token hash format (need to reset password).
+
+### 0.9.4
+- Decode WebDAV file name. 
+- Allow root-listing for WebDAV.
+- Always return 207 status code for propfind.
+- Refactor debounce utility. 
+
+### 0.9.3
+- Fix empty file getting.
+- HTTP `PUT/POST` default to overwrite the file.
+- Use shared implementations for `PUT`, `GET`, `DELETE` methods.
+- Inherit permission on overwriting `unset` permission files.
+
+### 0.9.2
+- Native copy function.
+- Only enable basic authentication if WebDAV is enabled.
+- `WWW-Authenticate` header is now added to the response when authentication fails.
+
+### 0.9.1
+- Add WebDAV support.
+- Code refactor, use `lfss.eng` and `lfss.svc`.
+
+### 0.9.0
+- User peer access control, now user can share their path with other users.
+- Fix high concurrency database locking on file getting.

frontend/api.js

@@ -69,6 +69,10 @@ export default class Connector {
     /**
     * @param {string} path - the path to the file (url)
     * @param {File} file - the file to upload
+    * @param {Object} [options] - Optional upload configuration.
+    * @param {'abort' | 'overwrite' | 'skip'} [options.conflict='abort'] - Conflict resolution strategy:  
+    * `'abort'` to cancel and raise 409, `'overwrite'` to replace.
+    * @param {number} [options.permission=0] - Optional permission setting for the file (refer to backend impl).
     * @returns {Promise<string>} - the promise of the request, the url of the file
     */
     async put(path, file, {
@@ -96,8 +100,12 @@ export default class Connector {
     }
 
     /**
-    * @param {string} path - the path to the file (url)
+    * @param {string} path - the path to the file (url), should end with .json
     * @param {File} file - the file to upload
+    * @param {Object} [options] - Optional upload configuration.
+    * @param {'abort' | 'overwrite' | 'skip'} [options.conflict='abort'] - Conflict resolution strategy:  
+    * `'abort'` to cancel and raise 409, `'overwrite'` to replace, `'skip'` to ignore if already exists.
+    * @param {number} [options.permission=0] - Optional permission setting for the file (refer to backend impl).
     * @returns {Promise<string>} - the promise of the request, the url of the file
     */
     async post(path, file, {
@@ -129,13 +137,23 @@ export default class Connector {
 
     /**
     * @param {string} path - the path to the file (url), should end with .json
-    * @param {Objec} data - the data to upload
+    * @param {Object} data - the data to upload
+    * @param {Object} [options] - Optional upload configuration.
+    * @param {'abort' | 'overwrite' | 'skip'} [options.conflict='abort'] - Conflict resolution strategy:  
+    * `'abort'` to cancel and raise 409, `'overwrite'` to replace, `'skip'` to ignore if already exists.
+    * @param {number} [options.permission=0] - Optional permission setting for the file (refer to backend impl).
     * @returns {Promise<string>} - the promise of the request, the url of the file
     */
-    async putJson(path, data){
+    async putJson(path, data, {
+        conflict = "overwrite", 
+        permission = 0
+    } = {}){
         if (!path.endsWith('.json')){ throw new Error('Upload object must end with .json'); }
         if (path.startsWith('/')){ path = path.slice(1); }
-        const res = await fetch(this.config.endpoint + '/' + path, {
+        const dst = new URL(this.config.endpoint + '/' + path);
+        dst.searchParams.append('conflict', conflict);
+        dst.searchParams.append('permission', permission);
+        const res = await fetch(dst.toString(), {
             method: 'PUT',
             headers: {
                 'Authorization': 'Bearer ' + this.config.token,
@@ -149,6 +167,50 @@ export default class Connector {
         return (await res.json()).url;
     }
 
+    /**
+     * @param {string} path - the path to the file (url), should have content type application/json
+     * @returns {Promise<Object>} - return the json object
+    */
+    async getJson(path){
+        if (path.startsWith('/')){ path = path.slice(1); }
+        const res = await fetch(this.config.endpoint + '/' + path, {
+            method: 'GET',
+            headers: {
+             "Authorization": 'Bearer ' + this.config.token
+            },
+        });
+        if (res.status != 200){
+            throw new Error(`Failed to get object, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
+        }
+        return await res.json();
+    }
+
+    /**
+     * @param {string[]} paths - the paths to the files (url), should have content type plain/text, application/json, etc.
+     * @param {Object} [options] - Optional configuration.
+     * @param {boolean} [options.skipContent=false] - If true, skips fetching content and returns a record of <path, ''>.
+     * @returns {Promise<Record<string, string | null>>} - return the mapping of path to text content, non-existing paths will be ignored
+     */
+    async getMultipleText(paths, {
+        skipContent = false
+    } = {}){
+        const url = new URL(this.config.endpoint + '/_api/get-multiple');
+        url.searchParams.append('skip_content', skipContent);
+        for (const path of paths){
+            url.searchParams.append('path', path);
+        }
+        const res = await fetch(url.toString(), {
+            method: 'GET',
+            headers: {
+                "Authorization": 'Bearer ' + this.config.token,
+            }
+        });
+        if (res.status != 200 && res.status != 206){
+            throw new Error(`Failed to get multiple files, status code: ${res.status}, message: ${await fmtFailedResponse(res)}`);
+        }
+        return await res.json();
+    }
+
     async delete(path){
         if (path.startsWith('/')){ path = path.slice(1); }
         const res = await fetch(this.config.endpoint + '/' + path, {

frontend/login.js

@@ -3,7 +3,6 @@ import { createFloatingWindow, showPopup } from "./popup.js";
 
 /**
  * @import { store } from "./state.js";
- * @import { UserRecord } from "./api.js";
  * 
  * Shows the login panel if necessary.
  * @param {store} store - The store object.

frontend/popup.js

@@ -109,7 +109,14 @@ export function showPopup(content = '',  {
 } = {}){
     const popup = document.createElement("div");
     popup.classList.add("popup-window");
-    popup.innerHTML = showTime? `<span>[${new Date().toLocaleTimeString()}]</span> ${content}` : content;
+    /**
+     * @param {string} c
+     * @returns {void}
+     */
+    function setPopupContent(c){
+        popup.innerHTML = showTime? `<span>[${new Date().toLocaleTimeString()}]</span> ${c}` : c;
+    }
+    setPopupContent(content);
     popup.style.width = width;
     const popupHeight = '1rem';
     popup.style.height = popupHeight;
@@ -132,11 +139,19 @@ export function showPopup(content = '',  {
     if (level === "success") popup.style.backgroundColor = "darkgreen";
     document.body.appendChild(popup);
     shownPopups.push(popup);
-    window.setTimeout(() => {
+
+    function closePopup(){
         if (popup.parentNode) document.body.removeChild(popup);
         shownPopups.splice(shownPopups.indexOf(popup), 1);
         for (let i = 0; i < shownPopups.length; i++) {
             shownPopups[i].style.top = `${i * (parseInt(popupHeight) + 2*parseInt(paddingHeight))*1.2 + 0.5}rem`;
         }
-    }, timeout);
+    }
+
+    window.setTimeout(closePopup, timeout);
+    return {
+        elem: popup, 
+        setContent: setPopupContent, 
+        close: closePopup
+    }
 }

frontend/scripts.js

@@ -5,6 +5,7 @@ import { showInfoPanel, showDirInfoPanel } from './info.js';
 import { makeThumbHtml } from './thumb.js';
 import { store } from './state.js';
 import { maybeShowLoginPanel } from './login.js';
+import { forEachFile } from './utils.js';
 
 /** @type {import('./api.js').UserRecord}*/
 let userRecord = null;
@@ -158,55 +159,61 @@ uploadFileNameInput.addEventListener('input', debounce(onFileNameInpuChange, 500
         e.preventDefault();
         e.stopPropagation();
     });
-    window.addEventListener('drop', (e) => {
+    window.addEventListener('drop', async (e) => {
         e.preventDefault();
         e.stopPropagation();
-        const files = e.dataTransfer.files;
-        if (files.length == 1){
-            uploadFileSelector.files = files;
-            uploadFileNameInput.value = files[0].name;
+        const items = e.dataTransfer.items;
+        if (items.length == 1 && items[0].kind === 'file' && items[0].webkitGetAsEntry().isFile){
+            uploadFileSelector.files = e.dataTransfer.files;
+            uploadFileNameInput.value = e.dataTransfer.files[0].name;
             uploadFileNameInput.focus();
+            return;
         }
-        else if (files.length > 1){
-            let dstPath = store.dirpath + uploadFileNameInput.value;
-            if (!dstPath.endsWith('/')){ dstPath += '/'; }
-            if (!confirm(`
+
+        /** @type {[string, File][]} */
+        const uploadInputVal = uploadFileNameInput.value? uploadFileNameInput.value : '';
+        let dstPath = store.dirpath + uploadInputVal;
+        if (!dstPath.endsWith('/')){ dstPath += '/'; }
+
+        if (!confirm(`\
 You are trying to upload multiple files at once. 
 This will directly upload the files to the [${dstPath}] directory without renaming. 
 Note that same name files will be overwritten.
-Are you sure you want to proceed?
-                `)){ return; }
-            
-            let counter = 0;
-            async function uploadFileFn(...args){
-                const [file, path] = args;
-                try{
-                    await uploadFile(conn, path, file, {conflict: 'overwrite'});
-                }
-                catch (err){
-                    showPopup('Failed to upload file [' + file.name + ']: ' + err, {level: 'error', timeout: 5000});
-                }
-                counter += 1;
-                console.log("Uploading file: ", counter, "/", files.length);
+Are you sure you want to proceed?\
+        `)){ return; }
+
+        let counter = 0;
+        let totalCount = 0;
+        const uploadPopup = showPopup('Uploading multiple files...', {level: 'info', timeout: 999999});
+        async function uploadFileFn(path, file){
+            try{
+                await uploadFile(conn, path, file, {conflict: 'overwrite'});
             }
-            
-            let promises = [];
-            for (let i = 0; i < files.length; i++){
-                const file = files[i];
-                const path = dstPath + file.name;
-                promises.push(uploadFileFn(file, path));
+            catch (err){
+                showPopup('Failed to upload file [' + file.name + ']: ' + err, {level: 'error', timeout: 5000});
             }
-            showPopup('Uploading multiple files...', {level: 'info', timeout: 3000});
-            Promise.all(promises).then(
-                () => {
-                    showPopup('Upload success.', {level: 'success', timeout: 3000});
-                    refreshFileList();
-                }, 
-                (err) => {
-                    showPopup('Failed to upload some files: ' + err, {level: 'error', timeout: 5000});
-                }
-            );
+            console.log(`[${counter}/${totalCount}] Uploaded file: ${path}`);
+            uploadPopup.setContent(`Uploading multiple files... [${counter}/${totalCount}]`);
         }
+
+        const promises = await forEachFile(e, async (relPath, filePromiseFn) => {
+            counter += 1;
+            const file = await filePromiseFn();
+            await uploadFileFn(dstPath + relPath, file);
+        });
+        totalCount = promises.length;
+
+        Promise.all(promises).then(
+            () => {
+                window.setTimeout(uploadPopup.close, 3000);
+                showPopup('Upload success.', {level: 'success', timeout: 3000});
+                refreshFileList();
+            }, 
+            (err) => {
+                showPopup('Failed to upload some files: ' + err, {level: 'error', timeout: 5000});
+            }
+        );
+
     });
 }
 

frontend/utils.js

@@ -93,4 +93,101 @@ export function asHtmlText(text){
     anonElem.textContent = text;
     const htmlText = anonElem.innerHTML;
     return htmlText;
-}
+}
+
+/**
+ * Iterates over all files dropped in the event,
+ * including files inside directories, and processes them
+ * using the provided callback with a concurrency limit.
+ *
+ * @param {Event} e The drop event.
+ * @param {(relPath: string, file: () => Promise<File>) => Promise<void>} callback A function
+ *        that receives the relative path and a promise for the File.
+ * @param {number} [maxConcurrent=5] Maximum number of concurrent callback executions.
+ * @returns {Promise<Promise<void>[]>} A promise resolving to an array of callback promises.
+ */
+export async function forEachFile(e, callback, maxConcurrent = 16) {
+    const results = []; // to collect callback promises
+
+    // Concurrency barrier variables.
+    let activeCount = 0;
+    const queue = [];
+
+    /**
+     * Runs the given async task when below the concurrency limit.
+     * If at limit, waits until a slot is free.
+     *
+     * @param {() => Promise<any>} task An async function returning a promise.
+     * @returns {Promise<any>}
+     */
+    async function runWithLimit(task) {
+        // If we reached the concurrency limit, wait for a free slot.
+        if (activeCount >= maxConcurrent) {
+            await new Promise(resolve => queue.push(resolve));
+        }
+        activeCount++;
+        try {
+            return await task();
+        } finally {
+            activeCount--;
+            // If there are waiting tasks, allow the next one to run.
+            if (queue.length) {
+                queue.shift()();
+            }
+        }
+    }
+
+    /**
+     * Recursively traverses a file system entry.
+     *
+     * @param {FileSystemEntry} entry The entry (file or directory).
+     * @param {string} path The current relative path.
+     */
+    async function traverse(entry, path) {
+        if (entry.isFile) {
+            // Wrap file retrieval in a promise.
+            const filePromiseFn = () =>
+                new Promise((resolve, reject) => entry.file(resolve, reject));
+            // Use the concurrency barrier for the callback invocation.
+            results.push(runWithLimit(() => callback(path + entry.name, filePromiseFn)));
+        } else if (entry.isDirectory) {
+            const reader = entry.createReader();
+
+            async function readAllEntries(reader) {
+                const entries = [];
+                while (true) {
+                const chunk = await new Promise((resolve, reject) => {
+                    reader.readEntries(resolve, reject);
+                });
+                if (chunk.length === 0) break;
+                entries.push(...chunk);
+                }
+                return entries;
+            }
+
+            const entries = await readAllEntries(reader);
+            await Promise.all(
+                entries.map(ent => traverse(ent, path + entry.name + '/'))
+            );
+        }
+    }
+
+    // Process using DataTransfer items if available.
+    if (e.dataTransfer && e.dataTransfer.items) {
+        await Promise.all(
+        Array.from(e.dataTransfer.items).map(async item => {
+            const entry = item.webkitGetAsEntry && item.webkitGetAsEntry();
+            if (entry) {
+            await traverse(entry, '');
+            }
+        })
+        );
+    } else if (e.dataTransfer && e.dataTransfer.files) {
+        // Fallback for browsers that support only dataTransfer.files.
+        Array.from(e.dataTransfer.files).forEach(file => {
+        results.push(runWithLimit(() => callback(file.name, Promise.resolve(file))));
+        });
+    }
+    return results;
+}
+

lfss/api/__init__.py

@@ -113,7 +113,7 @@ def download_file(
                 print(f"File {file_path} already exists, skipping download.")
             return True, error_msg
         try:
-            fmeta = connector.get_metadata(src_url)
+            fmeta = connector.get_meta(src_url)
             if fmeta is None:
                 error_msg = "File not found."
                 return False, error_msg
@@ -170,14 +170,15 @@ def download_directory(
     _counter = 0
     _counter_lock = Lock()
     failed_items: list[tuple[str, str]] = []
+    file_count = 0
     def get_file(c, src_url):
-        nonlocal _counter, failed_items
+        nonlocal _counter, failed_items, file_count, verbose
         with _counter_lock:
             _counter += 1
             this_count = _counter
             dst_path = f"{directory}{os.path.relpath(decode_uri_compnents(src_url), decode_uri_compnents(src_path))}"
             if verbose:
-                print(f"[{this_count}] Downloading {src_url} to {dst_path}")
+                print(f"[{this_count}/{file_count}] Downloading {src_url} to {dst_path}")
 
         if not (res:=download_file(
             c, src_url, dst_path, 
@@ -185,11 +186,13 @@ def download_directory(
             ))[0]:
             failed_items.append((src_url, res[1]))
         
-    batch_size = 10000
+    batch_size = 10_000
     file_list: list[FileRecord] = []
     with connector.session(n_concurrent) as c:
         file_count = c.count_files(src_path, flat=True)
         for offset in range(0, file_count, batch_size):
+            if verbose:
+                print(f"Retrieving file list... ({offset}/{file_count})", end='\r')
             file_list.extend(c.list_files(
                 src_path, offset=offset, limit=batch_size, flat=True
             ))