copyparty 1.14.4__tar.gz → 1.15.1__tar.gz

{copyparty-1.14.4 → copyparty-1.15.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: copyparty
-Version: 1.14.4
+Version: 1.15.1
 Summary:   Portable file server with accelerated resumable uploads, deduplication, WebDAV, FTP, zeroconf, media indexer, video thumbnails, audio transcoding, and write-only folders
 Author-email: ed <copyparty@ocv.me>
 License: MIT
@@ -119,7 +119,8 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [smb server](#smb-server) - unsafe, slow, not recommended for wan
     * [browser ux](#browser-ux) - tweaking the ui
     * [opengraph](#opengraph) - discord and social-media embeds
-    * [file indexing](#file-indexing) - enables dedup and music search ++
+    * [file deduplication](#file-deduplication) - enable symlink-based upload deduplication
+    * [file indexing](#file-indexing) - enable music search, upload-undo, and better dedup
         * [exclude-patterns](#exclude-patterns) - to save some time
         * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
         * [periodic rescan](#periodic-rescan) - filesystem monitoring
@@ -1209,9 +1210,41 @@ NOTE: because discord (and maybe others) strip query args such as `?raw` in open
 if you want to entirely replace the copyparty response with your own jinja2 template, give the template filepath to `--og-tpl` or volflag `og_tpl` (all members of `HttpCli` are available through the `this` object)
 
 
+## file deduplication
+
+enable symlink-based upload deduplication  globally with `--dedup` or per-volume with volflag `dedup`
+
+when someone tries to upload a file that already exists on the server, the upload will be politely declined and a symlink is created instead, pointing to the nearest copy on disk, thus reducinc disk space usage
+
+**warning:** when enabling dedup, you should also:
+* enable indexing with `-e2dsa` or volflag `e2dsa` (see [file indexing](#file-indexing) section below); strongly recommended
+* ...and/or `--hardlink-only` to use hardlink-based deduplication instead of symlinks; see explanation below
+
+it will not be safe to rename/delete files if you only enable dedup and none of the above; if you enable indexing then it is not *necessary* to also do hardlinks (but you may still want to)
+
+by default, deduplication is done based on symlinks (symbolic links); these are tiny files which are pointers to the nearest full copy of the file
+
+you can choose to use hardlinks instead of softlinks, globally with `--hardlink-only` or volflag `hardlinkonly`;
+
+advantages of using hardlinks:
+* hardlinks are more compatible with other software; they behave entirely like regular files
+* you can safely move and rename files using other file managers
+  * symlinks need to be managed by copyparty to ensure the destinations remain correct
+
+advantages of using symlinks (default):
+* each symlink can have its own last-modified timestamp, but a single timestamp is shared by all hardlinks
+* symlinks make it more obvious to other software that the file is not a regular file, so this can be less dangerous
+  * hardlinks look like regular files, so other software may assume they are safe to edit without affecting the other copies
+
+**warning:** if you edit the contents of a deduplicated file, then you will also edit all other copies of that file! This is especially surprising with hardlinks, because they look like regular files, but that same file exists in multiple locations
+
+global-option `--xlink` / volflag `xlink` additionally enables deduplication across volumes, but this is probably buggy and not recommended
+
+
+
 ## file indexing
 
-enables dedup and music search ++
+enable music search, upload-undo, and better dedup
 
 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volflags, or a mix of both.
 
@@ -1225,7 +1258,6 @@ through arguments:
 * `-e2v` verfies file integrity at startup, comparing hashes from the db
 * `-e2vu` patches the database with the new hashes from the filesystem
 * `-e2vp` panics and kills copyparty instead
-* `--xlink` enables deduplication across volumes
 
 the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts`, `d2v` for disabling:
 * `-v ~/music::r:c,e2ds,e2tsr` does a full reindex of everything on startup
@@ -1238,7 +1270,6 @@ note:
 * upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
-* deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
 
 ### exclude-patterns
 
@@ -1978,6 +2009,8 @@ below are some tweaks roughly ordered by usefulness:
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
 * `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
   * and also makes thumbnails load faster, regardless of e2d/e2t
+* `--dedup` enables deduplication and thus avoids writing to the HDD if someone uploads a dupe
+* `--safe-dedup 1` makes deduplication much faster during upload by skipping verification of file contents; safe if there is no other software editing/moving the files in the volumes
 * `--no-hash .` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
 * if your volumes are on a network-disk such as NFS / SMB / s3, specifying larger values for `--iobuf` and/or `--s-rd-sz` and/or `--s-wr-sz` may help; try setting all of them to `524288` or `1048576` or `4194304`
 * `--no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1` minimizes the number of threads; can help in some eccentric environments (like the vscode debugger)
@@ -2032,7 +2065,7 @@ safety profiles:
   * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
     * however note if you edit one file it will also affect the other copies
   * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--nih` removes the server hostname from directory listings
+  * `-nih` removes the server hostname from directory listings
 
 * option `-sss` is a shortcut for the above plus:
   * `--no-dav` disables webdav support

{copyparty-1.14.4 → copyparty-1.15.1}/README.md

@@ -65,7 +65,8 @@ turn almost any device into a file server with resumable uploads/downloads using
     * [smb server](#smb-server) - unsafe, slow, not recommended for wan
     * [browser ux](#browser-ux) - tweaking the ui
     * [opengraph](#opengraph) - discord and social-media embeds
-    * [file indexing](#file-indexing) - enables dedup and music search ++
+    * [file deduplication](#file-deduplication) - enable symlink-based upload deduplication
+    * [file indexing](#file-indexing) - enable music search, upload-undo, and better dedup
         * [exclude-patterns](#exclude-patterns) - to save some time
         * [filesystem guards](#filesystem-guards) - avoid traversing into other filesystems
         * [periodic rescan](#periodic-rescan) - filesystem monitoring
@@ -1155,9 +1156,41 @@ NOTE: because discord (and maybe others) strip query args such as `?raw` in open
 if you want to entirely replace the copyparty response with your own jinja2 template, give the template filepath to `--og-tpl` or volflag `og_tpl` (all members of `HttpCli` are available through the `this` object)
 
 
+## file deduplication
+
+enable symlink-based upload deduplication  globally with `--dedup` or per-volume with volflag `dedup`
+
+when someone tries to upload a file that already exists on the server, the upload will be politely declined and a symlink is created instead, pointing to the nearest copy on disk, thus reducinc disk space usage
+
+**warning:** when enabling dedup, you should also:
+* enable indexing with `-e2dsa` or volflag `e2dsa` (see [file indexing](#file-indexing) section below); strongly recommended
+* ...and/or `--hardlink-only` to use hardlink-based deduplication instead of symlinks; see explanation below
+
+it will not be safe to rename/delete files if you only enable dedup and none of the above; if you enable indexing then it is not *necessary* to also do hardlinks (but you may still want to)
+
+by default, deduplication is done based on symlinks (symbolic links); these are tiny files which are pointers to the nearest full copy of the file
+
+you can choose to use hardlinks instead of softlinks, globally with `--hardlink-only` or volflag `hardlinkonly`;
+
+advantages of using hardlinks:
+* hardlinks are more compatible with other software; they behave entirely like regular files
+* you can safely move and rename files using other file managers
+  * symlinks need to be managed by copyparty to ensure the destinations remain correct
+
+advantages of using symlinks (default):
+* each symlink can have its own last-modified timestamp, but a single timestamp is shared by all hardlinks
+* symlinks make it more obvious to other software that the file is not a regular file, so this can be less dangerous
+  * hardlinks look like regular files, so other software may assume they are safe to edit without affecting the other copies
+
+**warning:** if you edit the contents of a deduplicated file, then you will also edit all other copies of that file! This is especially surprising with hardlinks, because they look like regular files, but that same file exists in multiple locations
+
+global-option `--xlink` / volflag `xlink` additionally enables deduplication across volumes, but this is probably buggy and not recommended
+
+
+
 ## file indexing
 
-enables dedup and music search ++
+enable music search, upload-undo, and better dedup
 
 file indexing relies on two database tables, the up2k filetree (`-e2d`) and the metadata tags (`-e2t`), stored in `.hist/up2k.db`. Configuration can be done through arguments, volflags, or a mix of both.
 
@@ -1171,7 +1204,6 @@ through arguments:
 * `-e2v` verfies file integrity at startup, comparing hashes from the db
 * `-e2vu` patches the database with the new hashes from the filesystem
 * `-e2vp` panics and kills copyparty instead
-* `--xlink` enables deduplication across volumes
 
 the same arguments can be set as volflags, in addition to `d2d`, `d2ds`, `d2t`, `d2ts`, `d2v` for disabling:
 * `-v ~/music::r:c,e2ds,e2tsr` does a full reindex of everything on startup
@@ -1184,7 +1216,6 @@ note:
 * upload-times can be displayed in the file listing by enabling the `.up_at` metadata key, either globally with `-e2d -mte +.up_at` or per-volume with volflags `e2d,mte=+.up_at` (will have a ~17% performance impact on directory listings)
 * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
-* deduplication is possible on windows if you run copyparty as administrator (not saying you should!)
 
 ### exclude-patterns
 
@@ -1924,6 +1955,8 @@ below are some tweaks roughly ordered by usefulness:
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
 * `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
   * and also makes thumbnails load faster, regardless of e2d/e2t
+* `--dedup` enables deduplication and thus avoids writing to the HDD if someone uploads a dupe
+* `--safe-dedup 1` makes deduplication much faster during upload by skipping verification of file contents; safe if there is no other software editing/moving the files in the volumes
 * `--no-hash .` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
 * if your volumes are on a network-disk such as NFS / SMB / s3, specifying larger values for `--iobuf` and/or `--s-rd-sz` and/or `--s-wr-sz` may help; try setting all of them to `524288` or `1048576` or `4194304`
 * `--no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1` minimizes the number of threads; can help in some eccentric environments (like the vscode debugger)
@@ -1978,7 +2011,7 @@ safety profiles:
   * `--hardlink` creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
     * however note if you edit one file it will also affect the other copies
   * `--vague-403` returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
-  * `--nih` removes the server hostname from directory listings
+  * `-nih` removes the server hostname from directory listings
 
 * option `-sss` is a shortcut for the above plus:
   * `--no-dav` disables webdav support

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/__main__.py

@@ -986,9 +986,10 @@ def add_upload(ap):
     ap2.add_argument("--reg-cap", metavar="N", type=int, default=38400, help="max number of uploads to keep in memory when running without \033[33m-e2d\033[0m; roughly 1 MiB RAM per 600")
     ap2.add_argument("--no-fpool", action="store_true", help="disable file-handle pooling -- instead, repeatedly close and reopen files during upload (bad idea to enable this on windows and/or cow filesystems)")
     ap2.add_argument("--use-fpool", action="store_true", help="force file-handle pooling, even when it might be dangerous (multiprocessing, filesystems lacking sparse-files support, ...)")
-    ap2.add_argument("--hardlink", action="store_true", help="prefer hardlinks instead of symlinks when possible (within same filesystem) (volflag=hardlink)")
-    ap2.add_argument("--never-symlink", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=neversymlink)")
-    ap2.add_argument("--no-dedup", action="store_true", help="disable symlink/hardlink creation; copy file contents instead (volflag=copydupes)")
+    ap2.add_argument("--dedup", action="store_true", help="enable symlink-based upload deduplication (volflag=dedup)")
+    ap2.add_argument("--safe-dedup", metavar="N", type=int, default=50, help="how careful to be when deduplicating files; [\033[32m1\033[0m] = just verify the filesize, [\033[32m50\033[0m] = verify file contents have not been altered (volflag=safededup)")
+    ap2.add_argument("--hardlink", action="store_true", help="enable hardlink-based dedup; will fallback on symlinks when that is impossible (across filesystems) (volflag=hardlink)")
+    ap2.add_argument("--hardlink-only", action="store_true", help="do not fallback to symlinks when a hardlink cannot be made (volflag=hardlinkonly)")
     ap2.add_argument("--no-dupe", action="store_true", help="reject duplicate files during upload; only matches within the same volume (volflag=nodupe)")
     ap2.add_argument("--no-snap", action="store_true", help="disable snapshots -- forget unfinished uploads on shutdown; don't create .hist/up2k.snap files -- abandoned/interrupted uploads must be cleaned up manually")
     ap2.add_argument("--snap-wri", metavar="SEC", type=int, default=300, help="write upload state to ./hist/up2k.snap every \033[33mSEC\033[0m seconds; allows resuming incomplete uploads after a server crash")
@@ -1060,6 +1061,7 @@ def add_cert(ap, cert_path):
 
 
 def add_auth(ap):
+    ses_db = os.path.join(E.cfg, "sessions.db")
     ap2 = ap.add_argument_group('IdP / identity provider / user authentication options')
     ap2.add_argument("--idp-h-usr", metavar="HN", type=u, default="", help="bypass the copyparty authentication checks and assume the request-header \033[33mHN\033[0m contains the username of the requesting user (for use with authentik/oauth/...)\n\033[1;31mWARNING:\033[0m if you enable this, make sure clients are unable to specify this header themselves; must be washed away and replaced by a reverse-proxy")
     ap2.add_argument("--idp-h-grp", metavar="HN", type=u, default="", help="assume the request-header \033[33mHN\033[0m contains the groupname of the requesting user; can be referenced in config files for group-based access control")
@@ -1067,6 +1069,9 @@ def add_auth(ap):
     ap2.add_argument("--idp-gsep", metavar="RE", type=u, default="|:;+,", help="if there are multiple groups in \033[33m--idp-h-grp\033[0m, they are separated by one of the characters in \033[33mRE\033[0m")
     ap2.add_argument("--no-bauth", action="store_true", help="disable basic-authentication support; do not accept passwords from the 'Authenticate' header at all. NOTE: This breaks support for the android app")
     ap2.add_argument("--bauth-last", action="store_true", help="keeps basic-authentication enabled, but only as a last-resort; if a cookie is also provided then the cookie wins")
+    ap2.add_argument("--ses-db", metavar="PATH", type=u, default=ses_db, help="where to store the sessions database (if you run multiple copyparty instances, make sure they use different DBs)")
+    ap2.add_argument("--ses-len", metavar="CHARS", type=int, default=20, help="session key length; default is 120 bits ((20//4)*4*6)")
+    ap2.add_argument("--no-ses", action="store_true", help="disable sessions; use plaintext passwords in cookies")
 
 
 def add_chpw(ap):
@@ -1279,6 +1284,7 @@ def add_logging(ap):
     ap2.add_argument("--ansi", action="store_true", help="force colors; overrides environment-variable NO_COLOR")
     ap2.add_argument("--no-logflush", action="store_true", help="don't flush the logfile after each write; tiny bit faster")
     ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
+    ap2.add_argument("--log-utc", action="store_true", help="do not use local timezone; assume the TZ env-var is UTC (tiny bit faster)")
     ap2.add_argument("--log-tdec", metavar="N", type=int, default=3, help="timestamp resolution / number of timestamp decimals")
     ap2.add_argument("--log-badpwd", metavar="N", type=int, default=1, help="log failed login attempt passwords: 0=terse, 1=plaintext, 2=hashed")
     ap2.add_argument("--log-conn", action="store_true", help="debug: print tcp-server msgs")
@@ -1337,7 +1343,7 @@ def add_transcoding(ap):
 def add_db_general(ap, hcores):
     noidx = APPLESAN_TXT if MACOS else ""
     ap2 = ap.add_argument_group('general db options')
-    ap2.add_argument("-e2d", action="store_true", help="enable up2k database, making files searchable + enables upload deduplication")
+    ap2.add_argument("-e2d", action="store_true", help="enable up2k database; this enables file search, upload-undo, improves deduplication")
     ap2.add_argument("-e2ds", action="store_true", help="scan writable folders for new files on startup; sets \033[33m-e2d\033[0m")
     ap2.add_argument("-e2dsa", action="store_true", help="scans all folders on startup; sets \033[33m-e2ds\033[0m")
     ap2.add_argument("-e2v", action="store_true", help="verify file integrity; rehash all files and compare with db")
@@ -1350,7 +1356,7 @@ def add_db_general(ap, hcores):
     ap2.add_argument("--re-dhash", action="store_true", help="force a cache rebuild on startup; enable this once if it gets out of sync (should never be necessary)")
     ap2.add_argument("--no-forget", action="store_true", help="never forget indexed files, even when deleted from disk -- makes it impossible to ever upload the same file twice -- only useful for offloading uploads to a cloud service or something (volflag=noforget)")
     ap2.add_argument("--dbd", metavar="PROFILE", default="wal", help="database durability profile; sets the tradeoff between robustness and speed, see \033[33m--help-dbd\033[0m (volflag=dbd)")
-    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (volflag=xlink)")
+    ap2.add_argument("--xlink", action="store_true", help="on upload: check all volumes for dupes, not just the target volume (probably buggy, not recommended) (volflag=xlink)")
     ap2.add_argument("--hash-mt", metavar="CORES", type=int, default=hcores, help="num cpu cores to use for file hashing; set 0 or 1 for single-core hashing")
     ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="rescan filesystem for changes every \033[33mSEC\033[0m seconds; 0=off (volflag=scan)")
     ap2.add_argument("--db-act", metavar="SEC", type=float, default=10.0, help="defer any scheduled volume reindexing until \033[33mSEC\033[0m seconds after last db write (uploads, renames, ...)")
@@ -1463,6 +1469,7 @@ def run_argparse(
 )  :
     ap = argparse.ArgumentParser(
         formatter_class=formatter,
+        usage=argparse.SUPPRESS,
         prog="copyparty",
         description="http file sharing hub v{} ({})".format(S_VERSION, S_BUILD_DT),
     )
@@ -1613,6 +1620,7 @@ def main(argv  = None, rsrc  = None)  :
         ("--hdr-au-usr", "--idp-h-usr"),
         ("--idp-h-sep", "--idp-gsep"),
         ("--th-no-crop", "--th-crop=n"),
+        ("--never-symlink", "--hardlink-only"),
     ]
     for dk, nk in deprecated:
         idx = -1
@@ -1637,7 +1645,7 @@ def main(argv  = None, rsrc  = None)  :
             argv.extend(["--qr"])
             if ANYWIN or not os.geteuid():
                 # win10 allows symlinks if admin; can be unexpected
-                argv.extend(["-p80,443,3923", "--ign-ebind", "--no-dedup"])
+                argv.extend(["-p80,443,3923", "--ign-ebind"])
     except:
         pass
 

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (1, 14, 4)
-CODENAME = "one step forward"
-BUILD_DT = (2024, 9, 2)
+VERSION = (1, 15, 1)
+CODENAME = "fill the drives"
+BUILD_DT = (2024, 9, 9)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/authsrv.py

@@ -833,8 +833,10 @@ class AuthSrv(object):
 
         # fwd-decl
         self.vfs = VFS(log_func, "", "", AXS(), {})
-        self.acct   = {}
-        self.iacct   = {}
+        self.acct   = {}  # uname->pw
+        self.iacct   = {}  # pw->uname
+        self.ases   = {}  # uname->session
+        self.sesa   = {}  # session->uname
         self.defpw   = {}
         self.grps   = {}
         self.re_pwd  = None
@@ -1884,6 +1886,11 @@ class AuthSrv(object):
             if len(zs) == 3:  # fc5 => ffcc55
                 vol.flags["tcolor"] = "".join([x * 2 for x in zs])
 
+            if vol.flags.get("neversymlink"):
+                vol.flags["hardlinkonly"] = True  # was renamed
+            if vol.flags.get("hardlinkonly"):
+                vol.flags["hardlink"] = True
+
             for k1, k2 in IMPLICATIONS:
                 if k1 in vol.flags:
                     vol.flags[k2] = True
@@ -1988,9 +1995,6 @@ class AuthSrv(object):
                     for x in drop:
                         vol.flags.pop(x)
 
-            if vol.flags.get("neversymlink") and not vol.flags.get("hardlink"):
-                vol.flags["copydupes"] = True
-
             # verify tags mentioned by -mt[mp] are used by -mte
             local_mtp = {}
             local_only_mtp = {}
@@ -2069,6 +2073,8 @@ class AuthSrv(object):
 
         have_e2d = False
         have_e2t = False
+        have_dedup = False
+        unsafe_dedup = []
         t = "volumes and permissions:\n"
         for zv in vfs.all_vols.values():
             if not self.warn_anonwrite or verbosity < 5:
@@ -2101,6 +2107,11 @@ class AuthSrv(object):
             if "e2t" in zv.flags:
                 have_e2t = True
 
+            if "dedup" in zv.flags:
+                have_dedup = True
+                if "e2d" not in zv.flags and "hardlink" not in zv.flags:
+                    unsafe_dedup.append("/" + zv.vpath)
+
             t += "\n"
 
         if self.warn_anonwrite and verbosity > 4:
@@ -2113,10 +2124,17 @@ class AuthSrv(object):
                     self.log("\n\033[{}\033[0m\n".format(t))
 
                 if not have_e2t:
-                    t = "hint: argument -e2ts enables multimedia indexing (artist/title/...)"
+                    t = "hint: enable multimedia indexing (artist/title/...) with argument -e2ts"
                     self.log(t, 6)
             else:
-                t = "hint: argument -e2dsa enables searching, upload-undo, and better deduplication"
+                t = "hint: enable searching and upload-undo with argument -e2dsa"
+                self.log(t, 6)
+
+            if unsafe_dedup:
+                t = "WARNING: symlink-based deduplication is enabled for some volumes, but without indexing. Please enable -e2dsa and/or --hardlink to avoid problems when moving/renaming files. Affected volumes: %s"
+                self.log(t % (", ".join(unsafe_dedup)), 3)
+            elif not have_dedup:
+                t = "hint: enable upload deduplication with --dedup (but see readme for consequences)"
                 self.log(t, 6)
 
             zv, _ = vfs.get("/", "*", False, False)
@@ -2158,8 +2176,11 @@ class AuthSrv(object):
         self.grps = grps
         self.iacct = {v: k for k, v in acct.items()}
 
+        self.load_sessions()
+
         self.re_pwd = None
         pwds = [re.escape(x) for x in self.iacct.keys()]
+        pwds.extend(list(self.sesa))
         if pwds:
             if self.ah.on:
                 zs = r"(\[H\] pw:.*|[?&]pw=)([^&]+)"
@@ -2234,6 +2255,72 @@ class AuthSrv(object):
             cur.close()
             db.close()
 
+    def load_sessions(self, quiet=False)  :
+        # mutex me
+        if self.args.no_ses:
+            self.ases = {}
+            self.sesa = {}
+            return
+
+        import sqlite3
+
+        ases = {}
+        blen = (self.args.ses_len // 4) * 4  # 3 bytes in 4 chars
+        blen = (blen * 3) // 4  # bytes needed for ses_len chars
+
+        db = sqlite3.connect(self.args.ses_db)
+        cur = db.cursor()
+
+        for uname, sid in cur.execute("select un, si from us"):
+            if uname in self.acct:
+                ases[uname] = sid
+
+        n = []
+        q = "insert into us values (?,?,?)"
+        for uname in self.acct:
+            if uname not in ases:
+                sid = ub64enc(os.urandom(blen)).decode("utf-8")
+                cur.execute(q, (uname, sid, int(time.time())))
+                ases[uname] = sid
+                n.append(uname)
+
+        if n:
+            db.commit()
+
+        cur.close()
+        db.close()
+
+        self.ases = ases
+        self.sesa = {v: k for k, v in ases.items()}
+        if n and not quiet:
+            t = ", ".join(n[:3])
+            if len(n) > 3:
+                t += "..."
+            self.log("added %d new sessions (%s)" % (len(n), t))
+
+    def forget_session(self, broker , uname )  :
+        with self.mutex:
+            self._forget_session(uname)
+
+        if broker:
+            broker.ask("_reload_sessions").get()
+
+    def _forget_session(self, uname )  :
+        if self.args.no_ses:
+            return
+
+        import sqlite3
+
+        db = sqlite3.connect(self.args.ses_db)
+        cur = db.cursor()
+        cur.execute("delete from us where un = ?", (uname,))
+        db.commit()
+        cur.close()
+        db.close()
+
+        self.sesa.pop(self.ases.get(uname, ""), "")
+        self.ases.pop(uname, "")
+
     def chpw(self, broker , uname, pw)   :
         if not self.args.chpw:
             return False, "feature disabled in server config"
@@ -2253,7 +2340,7 @@ class AuthSrv(object):
         if hpw == self.acct[uname]:
             return False, "that's already your password my dude"
 
-        if hpw in self.iacct:
+        if hpw in self.iacct or hpw in self.sesa:
             return False, "password is taken"
 
         with self.mutex:

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/broker_mp.py

@@ -72,6 +72,10 @@ class BrokerMp(object):
         for _, proc in enumerate(self.procs):
             proc.q_pend.put((0, "reload", []))
 
+    def reload_sessions(self)  :
+        for _, proc in enumerate(self.procs):
+            proc.q_pend.put((0, "reload_sessions", []))
+
     def collector(self, proc )  :
         """receive message from hub in other process"""
         while True:

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/broker_mpw.py

@@ -88,6 +88,10 @@ class MpWorker(BrokerCli):
                 self.asrv.reload()
                 self.logw("mpw.asrv reloaded")
 
+            elif dest == "reload_sessions":
+                with self.asrv.mutex:
+                    self.asrv.load_sessions()
+
             elif dest == "listen":
                 self.httpsrv.listen(args[0], args[1])
 

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/broker_thr.py

@@ -30,6 +30,7 @@ class BrokerThr(BrokerCli):
         self.iphash = HMaccas(os.path.join(self.args.E.cfg, "iphash"), 8)
         self.httpsrv = HttpSrv(self, None)
         self.reload = self.noop
+        self.reload_sessions = self.noop
 
     def shutdown(self)  :
         # self.log("broker", "shutting down")

{copyparty-1.14.4 → copyparty-1.15.1}/copyparty/cfg.py

@@ -12,8 +12,7 @@ def vf_bmap()   :
         "dav_auth": "davauth",
         "dav_rt": "davrt",
         "ed": "dots",
-        "never_symlink": "neversymlink",
-        "no_dedup": "copydupes",
+        "hardlink_only": "hardlinkonly",
         "no_dupe": "nodupe",
         "no_forget": "noforget",
         "no_pipe": "nopipe",
@@ -23,6 +22,7 @@ def vf_bmap()   :
         "no_athumb": "dathumb",
     }
     for k in (
+        "dedup",
         "dotsrch",
         "e2d",
         "e2ds",
@@ -58,6 +58,7 @@ def vf_vmap()   :
         "no_hash": "nohash",
         "no_idx": "noidx",
         "re_maxage": "scan",
+        "safe_dedup": "safededup",
         "th_convt": "convt",
         "th_size": "thsize",
         "th_crop": "crop",
@@ -129,10 +130,11 @@ permdescs = {
 
 flagcats = {
     "uploads, general": {
+        "dedup": "enable symlink-based file deduplication",
+        "hardlink": "enable hardlink-based file deduplication,\nwith fallback on symlinks when that is impossible",
+        "hardlinkonly": "dedup with hardlink only, never symlink;\nmake a full copy if hardlink is impossible",
+        "safededup": "verify on-disk data before using it for dedup",
         "nodupe": "rejects existing files (instead of symlinking them)",
-        "hardlink": "does dedup with hardlinks instead of symlinks",
-        "neversymlink": "disables symlink fallback; full copy instead",
-        "copydupes": "disables dedup, always saves full copies of dupes",
         "sparse": "force use of sparse files, mainly for s3-backed storage",
         "daw": "enable full WebDAV write support (dangerous);\nPUT-operations will now \033[1;31mOVERWRITE\033[0;35m existing files",
         "nosub": "forces all uploads into the top folder of the vfs",
@@ -159,7 +161,7 @@ flagcats = {
         "lifetime=3600": "uploads are deleted after 1 hour",
     },
     "database, general": {
-        "e2d": "enable database; makes files searchable + enables upload dedup",
+        "e2d": "enable database; makes files searchable + enables upload-undo",
         "e2ds": "scan writable folders for new files on startup; also sets -e2d",
         "e2dsa": "scans all folders for new files on startup; also sets -e2d",
         "e2t": "enable multimedia indexing; makes it possible to search for tags",
@@ -177,7 +179,7 @@ flagcats = {
         "noforget": "don't forget files when deleted from disk",
         "fat32": "avoid excessive reindexing on android sdcardfs",
         "dbd=[acid|swal|wal|yolo]": "database speed-durability tradeoff",
-        "xlink": "cross-volume dupe detection / linking",
+        "xlink": "cross-volume dupe detection / linking (dangerous)",
         "xdev": "do not descend into other filesystems",
         "xvol": "do not follow symlinks leaving the volume root",
         "dotsrch": "show dotfiles in search results",