overleaf-sync 2.0.0a2__tar.gz → 2.0.0a4__tar.gz

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: overleaf-sync
-Version: 2.0.0a2
+Version: 2.0.0a4
 Summary: Overleaf Two-Way Sync Tool
 Keywords: overleaf,sync,latex,tex
 Author-email: Moritz Glöckl <moritzgloeckl@users.noreply.github.com>
@@ -78,11 +78,20 @@ Keep the `.olauth` file save, as it can be used to log in into your account.
 On a server without a display, or if the login window doesn't work for your account (e.g. single sign-on via ORCID or your institution), log in with a session cookie instead:
 
 1. Log in to Overleaf in your normal browser.
-2. Open the developer tools → Application (Firefox: Storage) → Cookies → `https://www.overleaf.com` and copy the value of `overleaf_session2`.
+2. Open the developer tools → Application (Firefox: Storage) → Cookies → `https://www.overleaf.com` and copy the value of `overleaf_session2` (on a self-hosted server: the cookie `overleaf.sid` of your server).
 3. Run `ols login --cookie` and paste the value at the (hidden) prompt, or provide it via the `OLSYNC_COOKIE` environment variable.
 
 This works without installing the `gui` extra.
 
+### Self-hosted Overleaf servers
+overleaf-sync also works with self-hosted Overleaf servers. It is tested against Overleaf Community Edition; Server Pro shares its code and should work too, but is untested. Pass the server's URL when logging in:
+
+```
+moritz@github:~/test$ ols login --server https://overleaf.example.com
+```
+
+The server is stored together with the login, so all other commands (`ols`, `ols list`, `ols download`) use it automatically. You can also set it with `--server` on any command or with the `OLSYNC_SERVER` environment variable. `ols login --cookie --server ...` works too; self-hosted servers name the session cookie `overleaf.sid` (`sharelatex.sid` before version 5).
+
 ### Listing all projects
 ```
 moritz@github:~/test$ ols list [--store-path -v/--verbose]
@@ -106,7 +115,9 @@ Use `ols download` to compile and download your project's PDF. Specify a downloa
 moritz@github:~/test$ ols [-l/--local-only -r/--remote-only --store-path -p/--path -i/--olignore]
 ```
 
-Just calling `ols` will two-way sync your project. When there are changes both locally, and remotely you will be asked which file to keep. Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
+Just calling `ols` will two-way sync your project. overleaf-sync remembers what each file looked like after the last sync (in a local `.olsync-state.json` file, which is never uploaded), so it knows on which side a file changed: files changed locally are uploaded, files changed on Overleaf are downloaded. If a file was changed on both sides, or differs on the first sync, you are asked which version to keep (or to skip it). Nothing is overwritten silently.
+
+Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
 
 The option `--store-path` specifies the path of the cookie file created by the `login` command. If you did not change its path, you do not need to specify this argument. The `-p/--path` option allows you to specify a different sync folder than the one you're calling `ols` from. The `-i/--olignore` option allows you to specify the path of an `.olignore` file. It uses `fnmatch` internally, so it may have some similarity to `.gitignore` but doesn't work exactly the same. For example, if you wish to exclude a specific folder named `out`, you need to specify it as `out/*`. See [here](https://docs.python.org/3/library/fnmatch.html) for more information.
 

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/README.md

@@ -49,11 +49,20 @@ Keep the `.olauth` file save, as it can be used to log in into your account.
 On a server without a display, or if the login window doesn't work for your account (e.g. single sign-on via ORCID or your institution), log in with a session cookie instead:
 
 1. Log in to Overleaf in your normal browser.
-2. Open the developer tools → Application (Firefox: Storage) → Cookies → `https://www.overleaf.com` and copy the value of `overleaf_session2`.
+2. Open the developer tools → Application (Firefox: Storage) → Cookies → `https://www.overleaf.com` and copy the value of `overleaf_session2` (on a self-hosted server: the cookie `overleaf.sid` of your server).
 3. Run `ols login --cookie` and paste the value at the (hidden) prompt, or provide it via the `OLSYNC_COOKIE` environment variable.
 
 This works without installing the `gui` extra.
 
+### Self-hosted Overleaf servers
+overleaf-sync also works with self-hosted Overleaf servers. It is tested against Overleaf Community Edition; Server Pro shares its code and should work too, but is untested. Pass the server's URL when logging in:
+
+```
+moritz@github:~/test$ ols login --server https://overleaf.example.com
+```
+
+The server is stored together with the login, so all other commands (`ols`, `ols list`, `ols download`) use it automatically. You can also set it with `--server` on any command or with the `OLSYNC_SERVER` environment variable. `ols login --cookie --server ...` works too; self-hosted servers name the session cookie `overleaf.sid` (`sharelatex.sid` before version 5).
+
 ### Listing all projects
 ```
 moritz@github:~/test$ ols list [--store-path -v/--verbose]
@@ -77,7 +86,9 @@ Use `ols download` to compile and download your project's PDF. Specify a downloa
 moritz@github:~/test$ ols [-l/--local-only -r/--remote-only --store-path -p/--path -i/--olignore]
 ```
 
-Just calling `ols` will two-way sync your project. When there are changes both locally, and remotely you will be asked which file to keep. Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
+Just calling `ols` will two-way sync your project. overleaf-sync remembers what each file looked like after the last sync (in a local `.olsync-state.json` file, which is never uploaded), so it knows on which side a file changed: files changed locally are uploaded, files changed on Overleaf are downloaded. If a file was changed on both sides, or differs on the first sync, you are asked which version to keep (or to skip it). Nothing is overwritten silently.
+
+Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
 
 The option `--store-path` specifies the path of the cookie file created by the `login` command. If you did not change its path, you do not need to specify this argument. The `-p/--path` option allows you to specify a different sync folder than the one you're calling `ols` from. The `-i/--olignore` option allows you to specify the path of an `.olignore` file. It uses `fnmatch` internally, so it may have some similarity to `.gitignore` but doesn't work exactly the same. For example, if you wish to exclude a specific folder named `out`, you need to specify it as `out/*`. See [here](https://docs.python.org/3/library/fnmatch.html) for more information.
 

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/olsync/__init__.py

@@ -1,3 +1,3 @@
 """Overleaf Two-Way Sync Tool"""
 
-__version__ = '2.0.0a2'
+__version__ = '2.0.0a4'

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/olsync/olbrowserlogin.py

@@ -13,13 +13,14 @@ from PySide6.QtWidgets import *
 from PySide6.QtWebEngineWidgets import *
 from PySide6.QtWebEngineCore import QWebEngineProfile, QWebEngineSettings, QWebEnginePage
 
-# Where to get the CSRF Token and where to send the login request to
-LOGIN_URL = "https://www.overleaf.com/login"
-PROJECT_URL = "https://www.overleaf.com/project"  # The dashboard URL
+BASE_URL = "https://www.overleaf.com"  # The default Overleaf server
+LOGIN_PATH = "/login"  # Where the user logs in
+PROJECT_PATH = "/project"  # The dashboard, reached after a successful login
 # JS snippet to extract the csrfToken
 JAVASCRIPT_CSRF_EXTRACTOR = "document.getElementsByName('ol-csrfToken')[0].content"
-# Name of the cookies we want to extract
-COOKIE_NAMES = ["overleaf_session2", "GCLB"]
+# Name of the cookies we want to extract: overleaf.com's session and load balancer cookies, and the session cookie of
+# self-hosted servers (Community Edition; "sharelatex.sid" before version 5)
+COOKIE_NAMES = ["overleaf_session2", "GCLB", "overleaf.sid", "sharelatex.sid"]
 
 
 class OlBrowserLoginWindow(QMainWindow):
@@ -28,9 +29,11 @@ class OlBrowserLoginWindow(QMainWindow):
     Opens a browser window to securely login the user and returns relevant login data.
     """
 
-    def __init__(self, *args, **kwargs):
+    def __init__(self, base_url=BASE_URL, *args, **kwargs):
         super(OlBrowserLoginWindow, self).__init__(*args, **kwargs)
 
+        self._base_url = base_url.rstrip("/")
+
         self.webview = QWebEngineView()
 
         self._cookies = {}
@@ -46,7 +49,7 @@ class OlBrowserLoginWindow(QMainWindow):
 
         webpage = QWebEnginePage(self.profile, self)
         self.webview.setPage(webpage)
-        self.webview.load(QUrl.fromUserInput(LOGIN_URL))
+        self.webview.load(QUrl.fromUserInput(self._base_url + LOGIN_PATH))
         self.webview.loadFinished.connect(self.handle_load_finished)
 
         self.setCentralWidget(self.webview)
@@ -58,7 +61,7 @@ class OlBrowserLoginWindow(QMainWindow):
             self._login_success = True
             QCoreApplication.quit()
 
-        if self.webview.url().toString() == PROJECT_URL:
+        if self.webview.url().toString().rstrip("/") == self._base_url + PROJECT_PATH:
             self.webview.page().runJavaScript(
                 JAVASCRIPT_CSRF_EXTRACTOR, 0, callback
             )
@@ -81,14 +84,14 @@ class OlBrowserLoginWindow(QMainWindow):
         return self._login_success
 
 
-def login():
+def login(base_url=BASE_URL):
     from PySide6.QtCore import QLoggingCategory
     QLoggingCategory.setFilterRules('''\
     qt.webenginecontext.info=false
     ''')
 
     app = QApplication([])
-    ol_browser_login_window = OlBrowserLoginWindow()
+    ol_browser_login_window = OlBrowserLoginWindow(base_url)
     ol_browser_login_window.show()
     app.exec()
 

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/olsync/olclient.py

@@ -12,6 +12,7 @@ import requests as reqs
 from bs4 import BeautifulSoup
 import json
 import ssl
+import sys
 import uuid
 import websocket
 from socketIO_client import SocketIO
@@ -24,18 +25,61 @@ import time
 if not hasattr(websocket, "SSLError"):
     websocket.SSLError = ssl.SSLError
 
-# Where to get the CSRF Token and where to send the login request to
-LOGIN_URL = "https://www.overleaf.com/login"
-PROJECT_URL = "https://www.overleaf.com/project"  # The dashboard URL
-# The URL to download all the files in zip format
-DOWNLOAD_URL = "https://www.overleaf.com/project/{}/download/zip"
-UPLOAD_URL = "https://www.overleaf.com/project/{}/upload"  # The URL to upload files
-FOLDER_URL = "https://www.overleaf.com/project/{}/folder"  # The URL to create folders
-DELETE_URL = "https://www.overleaf.com/project/{}/{}/{}"  # The URL to delete files (entity type: doc or file)
-COMPILE_URL = "https://www.overleaf.com/project/{}/compile?enable_pdf_caching=true"  # The URL to compile the project
-BASE_URL = "https://www.overleaf.com"  # The Overleaf Base URL
+BASE_URL = "https://www.overleaf.com"  # The default Overleaf server; self-hosted servers are passed as base_url
+# Paths relative to the server's base URL
+LOGIN_PATH = "/login"  # Where to get the CSRF Token and where to send the login request to
+PROJECT_PATH = "/project"  # The dashboard
+DOWNLOAD_PATH = "/project/{}/download/zip"  # Download all the files in zip format
+UPLOAD_PATH = "/project/{}/upload"  # Upload files
+FOLDER_PATH = "/project/{}/folder"  # Create folders
+DELETE_PATH = "/project/{}/{}/{}"  # Delete files (entity type: doc or file)
+COMPILE_PATH = "/project/{}/compile?enable_pdf_caching=true"  # Compile the project
+SOCKET_IO_PATH = "/socket.io/1/"  # Real-time service handshake
 PATH_SEP = "/"  # Use hardcoded path separator for both windows and posix system
 
+# Overleaf answers with HTTP 429 (without Retry-After) when a rate limit is reached, e.g. 30 dashboard loads or 10 zip
+# downloads per minute and project. Its limits use one-minute windows, so wait this many seconds before each retry.
+RATE_LIMIT_WAITS = (5, 10, 20, 30)
+
+
+class RateLimitError(reqs.HTTPError):
+    """Overleaf's rate limit was still reached after waiting and retrying"""
+
+
+def _request(method, url, **kwargs):
+    """requests.request that waits and retries when Overleaf's rate limit is reached"""
+    for wait in RATE_LIMIT_WAITS:
+        r = reqs.request(method, url, **kwargs)
+        if r.status_code != 429:
+            return r
+        print(f"Overleaf's rate limit was reached, retrying in {wait}s ...", file=sys.stderr)
+        time.sleep(wait)
+        # Rewind uploaded files, the rejected request has already read them
+        for value in (kwargs.get("files") or {}).values():
+            file = value[1] if isinstance(value, tuple) else value
+            if hasattr(file, "seek"):
+                file.seek(0)
+    r = reqs.request(method, url, **kwargs)
+    if r.status_code == 429:
+        raise RateLimitError("Overleaf's rate limit was reached (too many requests in a short time). "
+                             "Please wait a minute and try again.", response=r)
+    return r
+
+
+class RealtimeSocket(SocketIO):
+    """
+    socketIO-client's SocketIO, minus its cleanup quirk: SocketIO.__del__ calls disconnect(), which opens a brand-new
+    connection when the old one is already closed (and, with wait_for_connection=True, retries that forever).
+    """
+
+    def __del__(self):
+        try:
+            if self.connected:
+                self.disconnect()
+        except Exception:
+            pass
+
+
 class OverleafClient(object):
     """
     Overleaf API Wrapper
@@ -51,9 +95,17 @@ class OverleafClient(object):
                 if all(p.get(k) == v for k, v in more_attrs.items()):
                     yield p
 
-    def __init__(self, cookie=None, csrf=None):
+    def __init__(self, cookie=None, csrf=None, base_url=BASE_URL):
         self._cookie = cookie  # Store the cookie for authenticated requests
         self._csrf = csrf  # Store the CSRF token since it is needed for some requests
+        self._base_url = base_url.rstrip("/")  # overleaf.com or a self-hosted server
+
+    @property
+    def base_url(self):
+        return self._base_url
+
+    def _url(self, path):
+        return self._base_url + path
 
     def login(self, username, password):
         """
@@ -63,7 +115,7 @@ class OverleafClient(object):
         Returns: Dict of cookie and CSRF
         """
 
-        get_login = reqs.get(LOGIN_URL)
+        get_login = reqs.get(self._url(LOGIN_PATH))
         self._csrf = BeautifulSoup(get_login.content, 'html.parser').find(
             'input', {'name': '_csrf'}).get('value')
         login_json = {
@@ -71,7 +123,7 @@ class OverleafClient(object):
             "email": username,
             "password": password
         }
-        post_login = reqs.post(LOGIN_URL, json=login_json,
+        post_login = reqs.post(self._url(LOGIN_PATH), json=login_json,
                                cookies=get_login.cookies)
 
         # On a successful authentication the Overleaf API returns a new authenticated cookie.
@@ -84,7 +136,7 @@ class OverleafClient(object):
             self._cookie['GCLB'] = get_login.cookies['GCLB']
 
             # CSRF changes after making the login request, new CSRF token will be on the projects page
-            projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
+            projects_page = reqs.get(self._url(PROJECT_PATH), cookies=self._cookie)
             self._csrf = BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-csrfToken'}) \
                 .get('content')
 
@@ -114,7 +166,7 @@ class OverleafClient(object):
         Get the CSRF token from the project dashboard, e.g. after logging in with a cookie copied from a browser
         Returns: the CSRF token, or None if the session cookie is not (or no longer) valid
         """
-        projects_page = reqs.get(PROJECT_URL, cookies=self._cookie, allow_redirects=False)
+        projects_page = _request("GET", self._url(PROJECT_PATH), cookies=self._cookie, allow_redirects=False)
         if not projects_page.ok or projects_page.is_redirect:
             return None
         csrf_tag = BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-csrfToken'})
@@ -127,7 +179,7 @@ class OverleafClient(object):
         Get all of a user's active projects (= not archived and not trashed)
         Returns: List of project objects
         """
-        projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
+        projects_page = _request("GET", self._url(PROJECT_PATH), cookies=self._cookie)
         json_content = OverleafClient._extract_projects(projects_page.content)
         return list(OverleafClient.filter_projects(json_content))
 
@@ -138,7 +190,7 @@ class OverleafClient(object):
         Returns: project object
         """
 
-        projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
+        projects_page = _request("GET", self._url(PROJECT_PATH), cookies=self._cookie)
         json_content = OverleafClient._extract_projects(projects_page.content)
         return next(OverleafClient.filter_projects(json_content, {"name": project_name}), None)
 
@@ -148,7 +200,7 @@ class OverleafClient(object):
         Params: project_id, the id of the project
         Returns: bytes string (zip file)
         """
-        r = reqs.get(DOWNLOAD_URL.format(project_id),
+        r = _request("GET", self._url(DOWNLOAD_PATH.format(project_id)),
                      stream=True, cookies=self._cookie)
         return r.content
 
@@ -171,7 +223,7 @@ class OverleafClient(object):
         headers = {
             "X-Csrf-Token": self._csrf
         }
-        r = reqs.post(FOLDER_URL.format(project_id),
+        r = _request("POST", self._url(FOLDER_PATH.format(project_id)),
                       cookies=self._cookie, headers=headers, json=params)
 
         if r.ok:
@@ -218,8 +270,8 @@ class OverleafClient(object):
         # (Built from whatever cookies are present rather than hardcoded names:
         # a session may not carry every cookie -- this previously raised
         # KeyError here even though overleaf_session2 alone is enough to connect.)
-        handshake = reqs.get(
-            "{}/socket.io/1/".format(BASE_URL),
+        handshake = _request("GET", 
+            self._url(SOCKET_IO_PATH),
             params={"t": int(time.time() * 1000)},
             cookies=self._cookie,
         )
@@ -237,10 +289,12 @@ class OverleafClient(object):
         # handshake"); confirmed it rejects even when projectId is only on
         # the WebSocket upgrade URL -- it must be on the initial
         # /socket.io/1/ GET specifically.
-        socket_io = SocketIO(
-            BASE_URL,
+        socket_io = RealtimeSocket(
+            self._base_url,
             params={'t': int(time.time()), 'projectId': project_id},
-            headers={'Cookie': cookie}
+            headers={'Cookie': cookie},
+            # Fail instead of retrying forever (and hanging the CLI) when the real-time service is unreachable
+            wait_for_connection=False
         )
 
         # Wait until we connect to the socket
@@ -255,15 +309,16 @@ class OverleafClient(object):
         # .on() never registers -- it would return immediately without
         # reading anything. Wait in short slices instead, stopping as soon as
         # the event has actually been processed.
-        socket_io.on('joinProjectResponse', set_project_infos)
-        for _ in range(20):
-            if project_infos is not None:
-                break
-            socket_io.wait(seconds=1)
-
-        # Disconnect from the socket if still connected
-        if socket_io.connected:
-            socket_io.disconnect()
+        try:
+            socket_io.on('joinProjectResponse', set_project_infos)
+            for _ in range(20):
+                if project_infos is not None:
+                    break
+                socket_io.wait(seconds=1)
+        finally:
+            # Disconnect from the socket if still connected
+            if socket_io.connected:
+                socket_io.disconnect()
 
         return project_infos
 
@@ -323,7 +378,7 @@ class OverleafClient(object):
         }
 
         # Upload the file to the predefined folder
-        r = reqs.post(UPLOAD_URL.format(project_id), cookies=self._cookie, params=params, files=files)
+        r = _request("POST", self._url(UPLOAD_PATH.format(project_id)), cookies=self._cookie, params=params, files=files)
 
         # status_code is an int; comparing to str(200) was always False here,
         # so this reported failure on every call regardless of the real
@@ -367,7 +422,7 @@ class OverleafClient(object):
             "X-Csrf-Token": self._csrf
         }
 
-        r = reqs.delete(DELETE_URL.format(project_id, entity_type, file['_id']), cookies=self._cookie,
+        r = _request("DELETE", self._url(DELETE_PATH.format(project_id, entity_type, file['_id'])), cookies=self._cookie,
                         headers=headers, json={})
 
         # status_code is an int; comparing to str(204) was always False
@@ -396,7 +451,7 @@ class OverleafClient(object):
             "stopOnFirstError": False
         }
 
-        r = reqs.post(COMPILE_URL.format(project_id), cookies=self._cookie, headers=headers, json=body)
+        r = _request("POST", self._url(COMPILE_PATH.format(project_id)), cookies=self._cookie, headers=headers, json=body)
 
         if not r.ok:
             raise reqs.HTTPError()
@@ -416,7 +471,7 @@ class OverleafClient(object):
         if compile_result.get("clsiServerId"):
             params["clsiserverid"] = compile_result["clsiServerId"]
 
-        download_req = reqs.get(BASE_URL + pdf_file['url'], cookies=self._cookie, headers=headers, params=params)
+        download_req = _request("GET", self._url(pdf_file['url']), cookies=self._cookie, headers=headers, params=params)
 
         if download_req.ok:
             return pdf_file['path'], download_req.content

{overleaf_sync-2.0.0a2 → overleaf_sync-2.0.0a4}/olsync/olsync.py

@@ -23,10 +23,21 @@ from pathlib import Path
 
 try:
     # Import for pip installation / wheel
-    from olsync.olclient import OverleafClient
+    from olsync.olclient import BASE_URL, OverleafClient, RateLimitError
+    from olsync import syncstate
 except ImportError:
     # Import for development
-    from olclient import OverleafClient
+    from olclient import BASE_URL, OverleafClient, RateLimitError
+    import syncstate
+
+# Session cookie names: overleaf.com, and self-hosted servers (Community Edition, "sharelatex.sid" before version 5)
+OVERLEAF_COM_SESSION_COOKIES = ["overleaf_session2"]
+SELF_HOSTED_SESSION_COOKIES = ["overleaf.sid", "sharelatex.sid"]
+
+server_option = click.option(
+    '--server', 'server', default=None, envvar='OLSYNC_SERVER', metavar='URL',
+    help="URL of a self-hosted Overleaf server (default: the server used at login, or https://www.overleaf.com). "
+         "Can also be set with OLSYNC_SERVER.")
 
 
 def configure_console():
@@ -54,20 +65,14 @@ def configure_console():
 @click.option('-i', '--olignore', 'olignore_path', default=".olignore", type=click.Path(exists=False),
               help="Path to the .olignore file relative to sync path (ignored if syncing from remote to local). See "
                    "fnmatch / unix filename pattern matching for information on how to use it.")
+@server_option
 @click.option('-v', '--verbose', 'verbose', is_flag=True, help="Enable extended error logging.")
 @click.version_option(package_name='overleaf-sync')
 @click.pass_context
-def main(ctx, local, remote, project_name, cookie_path, sync_path, olignore_path, verbose):
+def main(ctx, local, remote, project_name, cookie_path, sync_path, olignore_path, server, verbose):
     configure_console()
     if ctx.invoked_subcommand is None:
-        if not os.path.isfile(cookie_path):
-            raise click.ClickException(
-                "Persisted Overleaf cookie not found. Please login or check store path.")
-
-        with open(cookie_path, 'rb') as f:
-            store = pickle.load(f)
-
-        overleaf_client = OverleafClient(store["cookie"], store["csrf"])
+        overleaf_client = load_client(cookie_path, server)
 
         # Change the current directory to the specified sync path
         os.chdir(sync_path)
@@ -95,9 +100,13 @@ def main(ctx, local, remote, project_name, cookie_path, sync_path, olignore_path
             "Project could not be downloaded.",
             verbose)
 
-        sync = not (local or remote)
+        if not (local or remote):
+            two_way_sync(overleaf_client, project, project_infos, zip_file, olignore_path, verbose)
+            return
+
+        sync = False  # -l and -r below sync in one direction only
 
-        if remote or sync:
+        if remote:
             remote_files = zip_file.namelist()
             deleted_files = [] if sync else [f for f in olignore_keep_list(olignore_path) if f not in remote_files]
             sync_func(
@@ -114,8 +123,8 @@ def main(ctx, local, remote, project_name, cookie_path, sync_path, olignore_path
                 from_name="remote",
                 to_name="local",
                 verbose=verbose)
-        if local or sync:
-            # Computed once (#60), and after the remote -> local pass so newly downloaded files are included
+        if local:
+            # Computed once (#60)
             local_files = olignore_keep_list(olignore_path)
             deleted_files = [] if sync else [f for f in zip_file.namelist() if f not in local_files]
             sync_func(
@@ -138,26 +147,31 @@ def main(ctx, local, remote, project_name, cookie_path, sync_path, olignore_path
 @click.option('--path', 'cookie_path', default=".olauth", type=click.Path(exists=False),
               help="Path to store the persisted Overleaf cookie.")
 @click.option('--cookie', 'use_cookie', is_flag=True,
-              help="Log in without the login window, using the `overleaf_session2` cookie from a browser that is "
-                   "logged in to Overleaf (asked for at a hidden prompt, or read from OLSYNC_COOKIE). Useful on "
-                   "servers without a display and for single sign-on accounts.")
+              help="Log in without the login window, using the session cookie from a browser that is logged in to "
+                   "Overleaf (`overleaf_session2` on overleaf.com, `overleaf.sid` on self-hosted servers), asked for "
+                   "at a hidden prompt or read from OLSYNC_COOKIE. Useful on servers without a display and for single "
+                   "sign-on accounts.")
+@server_option
 @click.option('-v', '--verbose', 'verbose', is_flag=True, help="Enable extended error logging.")
-def login(cookie_path, use_cookie, verbose):
+@click.pass_context
+def login(ctx, cookie_path, use_cookie, server, verbose):
+    server = resolve_login_server(ctx, server)
     if os.path.isfile(cookie_path) and not click.confirm(
             'Persisted Overleaf cookie already exist. Do you want to override it?'):
         return
     if use_cookie:
+        cookie_names = OVERLEAF_COM_SESSION_COOKIES if server == BASE_URL else SELF_HOSTED_SESSION_COOKIES
         # Never accept the cookie as a command line argument, it would end up in the shell history
-        cookie = os.environ.get("OLSYNC_COOKIE") or click.prompt("Value of the overleaf_session2 cookie",
+        cookie = os.environ.get("OLSYNC_COOKIE") or click.prompt(f"Value of the {cookie_names[0]} cookie",
                                                                  hide_input=True)
-        execute_action(lambda: cookie_login_handler(cookie.strip(), cookie_path), "Login",
+        execute_action(lambda: cookie_login_handler(cookie.strip(), cookie_path, server, cookie_names), "Login",
                        "Login successful. Cookie persisted as `" + click.format_filename(
                            cookie_path) + "`. You may now sync your project.",
                        "Login failed: the cookie is not valid (expired, or copied incompletely).", verbose)
         return
     olbrowserlogin = load_browser_login()
     click.clear()
-    execute_action(lambda: login_handler(olbrowserlogin, cookie_path), "Login",
+    execute_action(lambda: login_handler(olbrowserlogin, cookie_path, server), "Login",
                    "Login successful. Cookie persisted as `" + click.format_filename(
                        cookie_path) + "`. You may now sync your project.",
                    "Login failed. Please try again.", verbose)
@@ -166,8 +180,10 @@ def login(cookie_path, use_cookie, verbose):
 @main.command(name='list')
 @click.option('--store-path', 'cookie_path', default=".olauth", type=click.Path(exists=False),
               help="Relative path to load the persisted Overleaf cookie.")
+@server_option
 @click.option('-v', '--verbose', 'verbose', is_flag=True, help="Enable extended error logging.")
-def list_projects(cookie_path, verbose):
+@click.pass_context
+def list_projects(ctx, cookie_path, server, verbose):
     def query_projects():
         for index, p in enumerate(sorted(overleaf_client.all_projects(), key=lambda x: x['lastUpdated'], reverse=True)):
             if not index:
@@ -175,14 +191,7 @@ def list_projects(cookie_path, verbose):
             click.echo(f"{dateutil.parser.isoparse(p['lastUpdated']).strftime('%m/%d/%Y, %H:%M:%S')} - {p['name']}")
         return True
 
-    if not os.path.isfile(cookie_path):
-        raise click.ClickException(
-            "Persisted Overleaf cookie not found. Please login or check store path.")
-
-    with open(cookie_path, 'rb') as f:
-        store = pickle.load(f)
-
-    overleaf_client = OverleafClient(store["cookie"], store["csrf"])
+    overleaf_client = load_client(cookie_path, server or ctx.find_root().params.get("server"))
 
     click.clear()
     execute_action(query_projects, "Querying all projects",
@@ -196,8 +205,10 @@ def list_projects(cookie_path, verbose):
 @click.option('--download-path', 'download_path', default=".", type=click.Path(exists=True))
 @click.option('--store-path', 'cookie_path', default=".olauth", type=click.Path(exists=False),
               help="Relative path to load the persisted Overleaf cookie.")
+@server_option
 @click.option('-v', '--verbose', 'verbose', is_flag=True, help="Enable extended error logging.")
-def download_pdf(project_name, download_path, cookie_path, verbose):
+@click.pass_context
+def download_pdf(ctx, project_name, download_path, cookie_path, server, verbose):
     def download_project_pdf():
         nonlocal project_name
         project_name = project_name or os.path.basename(os.getcwd())
@@ -217,14 +228,7 @@ def download_pdf(project_name, download_path, cookie_path, verbose):
 
         return True
 
-    if not os.path.isfile(cookie_path):
-        raise click.ClickException(
-            "Persisted Overleaf cookie not found. Please login or check store path.")
-
-    with open(cookie_path, 'rb') as f:
-        store = pickle.load(f)
-
-    overleaf_client = OverleafClient(store["cookie"], store["csrf"])
+    overleaf_client = load_client(cookie_path, server or ctx.find_root().params.get("server"))
 
     click.clear()
 
@@ -262,23 +266,46 @@ def load_browser_login():
     return olbrowserlogin
 
 
-def login_handler(olbrowserlogin, path):
-    store = olbrowserlogin.login()
+def resolve_login_server(ctx, server):
+    """--server on the login command, else on the group (both also read OLSYNC_SERVER), else overleaf.com"""
+    return (server or ctx.find_root().params.get("server") or BASE_URL).rstrip("/")
+
+
+def load_client(cookie_path, server=None):
+    """
+    Create an OverleafClient from the persisted login. The server is --server/OLSYNC_SERVER if given, otherwise the
+    server stored at login (auth files from before 2.0.0a3 have none and mean overleaf.com).
+    """
+    if not os.path.isfile(cookie_path):
+        raise click.ClickException(
+            "Persisted Overleaf cookie not found. Please login or check store path.")
+
+    with open(cookie_path, 'rb') as f:
+        store = pickle.load(f)
+
+    return OverleafClient(store["cookie"], store["csrf"], base_url=server or store.get("server") or BASE_URL)
+
+
+def login_handler(olbrowserlogin, path, server):
+    store = olbrowserlogin.login(server)
     if store is None:
         return False
+    store["server"] = server
     with open(path, 'wb+') as f:
         pickle.dump(store, f)
     return True
 
 
-def cookie_login_handler(cookie, path):
-    cookies = {"overleaf_session2": cookie}
-    csrf = OverleafClient(cookies).fetch_csrf()
-    if csrf is None:
-        return False
-    with open(path, 'wb+') as f:
-        pickle.dump({"cookie": cookies, "csrf": csrf}, f)
-    return True
+def cookie_login_handler(cookie, path, server, cookie_names):
+    # Self-hosted servers name the session cookie differently depending on the version, so try each name
+    for cookie_name in cookie_names:
+        cookies = {cookie_name: cookie}
+        csrf = OverleafClient(cookies, base_url=server).fetch_csrf()
+        if csrf is not None:
+            with open(path, 'wb+') as f:
+                pickle.dump({"cookie": cookies, "csrf": csrf, "server": server}, f)
+            return True
+    return False
 
 
 def delete_file(path):
@@ -305,6 +332,93 @@ def write_file(path, content):
         f.write(content)
 
 
+def two_way_sync(overleaf_client, project, project_infos, zip_file, olignore_path, verbose=False):
+    """
+    Two-way sync that decides per file, based on the state of the last sync (see syncstate.py), instead of the
+    project-wide lastUpdated timestamp. Changes on one side are synced; files changed on both sides, or differing
+    without sync history, are only overwritten after asking.
+    """
+    def read(path):
+        with open(path, 'rb') as f:
+            return f.read()
+
+    server = overleaf_client.base_url
+    remote_content = {name: zip_file.read(name) for name in zip_file.namelist() if not name.endswith("/")}
+    # Local candidates: files allowed by .olignore, plus local copies of remote files (to compare them)
+    local_paths = set(olignore_keep_list(olignore_path)) | {name for name in remote_content if os.path.isfile(name)}
+    local = {path: syncstate.file_hash(read(path)) for path in local_paths}
+    remote = {name: syncstate.file_hash(content) for name, content in remote_content.items()}
+    state = syncstate.load_state(".", server, project["id"])
+    plan = syncstate.plan_two_way(local, remote, state)
+
+    click.echo("\nSyncing files between [local] and [remote]")
+    click.echo('=' * 40)
+
+    downloads, uploads, skipped = list(plan["download"]), list(plan["upload"]), []
+    for path in plan["conflict"]:
+        reason = "was changed locally and on Overleaf since the last sync" if path in state \
+            else "differs between local and Overleaf, and there is no sync history for it yet"
+        choice = click.prompt(
+            '\n-> Conflict: <%s> %s.\nKeep the [l]ocal version, keep the [r]emote version or [s]kip?' % (path, reason),
+            default="s", type=click.Choice(['l', 'r', 's']))
+        {"l": uploads, "r": downloads, "s": skipped}[choice].append(path)
+
+    failed = []
+    for path in downloads:
+        try:
+            write_file(path, remote_content[path])
+        except OSError:
+            if verbose:
+                print(traceback.format_exc())
+            failed.append(path)
+    for path in uploads:
+        try:
+            with open(path, 'rb') as f:
+                ok = overleaf_client.upload_file(project["id"], project_infos, path, os.path.getsize(path), f)
+        except Exception:
+            if verbose:
+                print(traceback.format_exc())
+            ok = False
+        if not ok:
+            failed.append(path)
+
+    for title, paths in [("[DOWNLOAD] Following file(s) downloaded from [remote]", downloads),
+                         ("[UPLOAD] Following file(s) uploaded to [remote]", uploads),
+                         ("[SYNC] Following file(s) are up to date", plan["in_sync"]),
+                         ("[SKIP] Following conflicting file(s) have not been synced", skipped),
+                         ("[ERROR] Following file(s) could not be synced", failed)]:
+        click.echo("\n" + title)
+        for path in paths:
+            click.echo("\t%s" % path)
+
+    # Record what both sides look like now. After uploads, read the project again, so that changes Overleaf makes
+    # to uploaded files (e.g. line endings) are not mistaken for remote edits next time.
+    if set(uploads) - set(failed):
+        try:
+            fresh = zipfile.ZipFile(io.BytesIO(overleaf_client.download_project(project["id"])))
+            remote_content = {name: fresh.read(name) for name in fresh.namelist() if not name.endswith("/")}
+        except Exception:
+            if verbose:
+                print(traceback.format_exc())
+            for path in set(uploads) - set(failed):
+                remote_content[path] = read(path)
+    new_state = {}
+    for path in sorted(set(local_paths) | set(remote_content) | set(downloads)):
+        if path in skipped or path in failed:
+            if path in state:
+                new_state[path] = state[path]
+        elif path in remote_content and os.path.isfile(path):
+            new_state[path] = {"local": syncstate.file_hash(read(path)),
+                               "remote": syncstate.file_hash(remote_content[path])}
+    syncstate.save_state(".", server, project["id"], new_state)
+
+    click.echo("")
+    if failed:
+        raise click.ClickException("%d file(s) could not be synced, see above" % len(failed))
+    click.echo("✅  Synced files between [local] and [remote]")
+    click.echo("")
+
+
 def sync_func(files_from, deleted_files, create_file_at_to, delete_file_at_to, create_file_at_from, from_exists_in_to,
               from_equal_to_to, from_newer_than_to, from_name,
               to_name, verbose=False):
@@ -415,6 +529,10 @@ def execute_action(action, progress_message, success_message, fail_message, verb
     with yaspin(text=progress_message, color="green") as spinner:
         try:
             success = action()
+        except RateLimitError as e:
+            # A clear reason is more helpful than the generic failure message
+            spinner.fail("💥 ")
+            raise click.ClickException(str(e))
         except:
             if verbose_error_logging:
                 print(traceback.format_exc())

overleaf_sync-2.0.0a4/olsync/syncstate.py

@@ -0,0 +1,98 @@
+"""Sync state for two-way sync"""
+##################################################
+# MIT License
+##################################################
+# File: syncstate.py
+# Description: Per-file sync state, so two-way sync can tell which side changed a file
+# Author: Moritz Glöckl
+# License: MIT
+##################################################
+
+import hashlib
+import json
+import os
+import tempfile
+from pathlib import Path
+
+# Stored in the sync folder. Dotfiles are never uploaded (glob skips them), so it stays local.
+STATE_FILE = ".olsync-state.json"
+STATE_VERSION = 1
+
+
+def file_hash(content):
+    """
+    Content hash used to detect changes
+    Params: content, the file content (bytes)
+    Returns: hex digest
+    """
+    return hashlib.sha256(content).hexdigest()
+
+
+def load_state(sync_dir, server, project_id):
+    """
+    Load the per-file state of the last successful sync of this project
+    Params: sync_dir, server, project_id
+    Returns: {path: {"local": hash, "remote": hash}}, empty if there is no (valid) state for this server and project
+    """
+    try:
+        with open(Path(sync_dir) / STATE_FILE) as f:
+            state = json.load(f)
+    except (OSError, ValueError):
+        return {}
+    if not isinstance(state, dict) or state.get("version") != STATE_VERSION \
+            or state.get("server") != server or state.get("project_id") != project_id:
+        return {}
+    return state.get("files", {})
+
+
+def save_state(sync_dir, server, project_id, files):
+    """
+    Save the per-file state atomically, so an interrupted sync never leaves a broken state file
+    Params: sync_dir, server, project_id, files ({path: {"local": hash, "remote": hash}})
+    """
+    state = {"version": STATE_VERSION, "server": server, "project_id": project_id, "files": files}
+    fd, tmp_path = tempfile.mkstemp(dir=sync_dir, prefix=STATE_FILE + ".", suffix=".tmp")
+    try:
+        with os.fdopen(fd, "w") as f:
+            json.dump(state, f, indent=1, sort_keys=True)
+        os.replace(tmp_path, Path(sync_dir) / STATE_FILE)
+    except BaseException:
+        os.unlink(tmp_path)
+        raise
+
+
+def plan_two_way(local, remote, state):
+    """
+    Decide per file what a two-way sync has to do
+    Params:
+    local: {path: hash} of the local files
+    remote: {path: hash} of the files on Overleaf
+    state: {path: {"local": hash, "remote": hash}} from the last successful sync
+    Returns: {"upload": [...], "download": [...], "conflict": [...], "in_sync": [...]} (sorted paths)
+    """
+    plan = {"upload": [], "download": [], "conflict": [], "in_sync": []}
+    for path in sorted(set(local) | set(remote)):
+        local_hash, remote_hash, last = local.get(path), remote.get(path), state.get(path)
+        if remote_hash is None:
+            action = "upload"
+        elif local_hash is None:
+            action = "download"
+        elif local_hash == remote_hash:
+            action = "in_sync"
+        elif last is None:
+            # No sync history for this file: never guess which version is newer
+            action = "conflict"
+        else:
+            local_changed = local_hash != last.get("local")
+            remote_changed = remote_hash != last.get("remote")
+            if local_changed and remote_changed:
+                action = "conflict"
+            elif local_changed:
+                action = "upload"
+            elif remote_changed:
+                action = "download"
+            else:
+                # Differs, but neither side changed since the last sync (e.g. Overleaf normalised the file)
+                action = "in_sync"
+        plan[action].append(path)
+    return plan