overleaf-sync 1.1.6__tar.gz → 2.0.0a1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- overleaf_sync-2.0.0a1/PKG-INFO +139 -0
- {overleaf-sync-1.1.6 → overleaf_sync-2.0.0a1}/README.md +16 -5
- {overleaf-sync-1.1.6 → overleaf_sync-2.0.0a1}/olsync/__init__.py +1 -1
- {overleaf-sync-1.1.6 → overleaf_sync-2.0.0a1}/olsync/olbrowserlogin.py +10 -5
- overleaf_sync-2.0.0a1/olsync/olclient.py +404 -0
- {overleaf-sync-1.1.6 → overleaf_sync-2.0.0a1}/olsync/olsync.py +151 -10
- overleaf_sync-2.0.0a1/pyproject.toml +60 -0
- overleaf-sync-1.1.6/.gitignore +0 -118
- overleaf-sync-1.1.6/PKG-INFO +0 -7
- overleaf-sync-1.1.6/olsync/olclient.py +0 -238
- overleaf-sync-1.1.6/pyproject.toml +0 -30
- overleaf-sync-1.1.6/requirements.txt +0 -11
- overleaf-sync-1.1.6/setup.py +0 -39
- {overleaf-sync-1.1.6 → overleaf_sync-2.0.0a1}/LICENSE +0 -0
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: overleaf-sync
|
|
3
|
+
Version: 2.0.0a1
|
|
4
|
+
Summary: Overleaf Two-Way Sync Tool
|
|
5
|
+
Keywords: overleaf,sync,latex,tex
|
|
6
|
+
Author-email: Moritz Glöckl <moritzgloeckl@users.noreply.github.com>
|
|
7
|
+
Requires-Python: >=3.11
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Intended Audience :: Science/Research
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Requires-Dist: requests >=2.31,<3
|
|
14
|
+
Requires-Dist: beautifulsoup4 >=4.12,<5
|
|
15
|
+
Requires-Dist: yaspin >=3,<4
|
|
16
|
+
Requires-Dist: python-dateutil >=2.8,<3
|
|
17
|
+
Requires-Dist: click >=8.1,<9
|
|
18
|
+
Requires-Dist: socketIO-client ==0.5.7.2
|
|
19
|
+
Requires-Dist: websocket-client >=1,<2
|
|
20
|
+
Requires-Dist: pytest >=8 ; extra == "dev"
|
|
21
|
+
Requires-Dist: pytest-cov >=5 ; extra == "dev"
|
|
22
|
+
Requires-Dist: responses >=0.25 ; extra == "dev"
|
|
23
|
+
Requires-Dist: ruff >=0.6 ; extra == "dev"
|
|
24
|
+
Requires-Dist: PySide6 >=6.5,<7 ; extra == "gui"
|
|
25
|
+
Project-URL: Homepage, https://github.com/moritzgloeckl/overleaf-sync
|
|
26
|
+
Project-URL: Issues, https://github.com/moritzgloeckl/overleaf-sync/issues
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Provides-Extra: gui
|
|
29
|
+
|
|
30
|
+
# Overleaf-Sync
|
|
31
|
+
### Easy Overleaf Two-Way Synchronization
|
|
32
|
+
|
|
33
|
+
   
|
|
34
|
+
|
|
35
|
+
This tool provides an easy way to synchronize Overleaf projects from and to your local computer. No paid account necessary.
|
|
36
|
+
|
|
37
|
+
----
|
|
38
|
+
|
|
39
|
+
## Features
|
|
40
|
+
- Sync your locally modified `.tex` (and other) files to your Overleaf projects
|
|
41
|
+
- Sync your remotely modified `.tex` (and other) files to computer
|
|
42
|
+
- Works with free Overleaf account
|
|
43
|
+
- No Git or Dropbox required
|
|
44
|
+
- Does not steal or store your login credentials (works with a persisted cookie, logging in is done on the original Overleaf website)
|
|
45
|
+
|
|
46
|
+
## How To Use
|
|
47
|
+
### Install
|
|
48
|
+
The package is available via [PyPI](https://pypi.org/project/overleaf-sync/) and needs Python 3.11 or newer. Install it including the login window (`gui` extra):
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
moritz@github:~/test$ pip3 install "overleaf-sync[gui]"
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Depending on your local Python installation, you might need to use `pip` instead of `pip3`. [pipx](https://pipx.pypa.io/) works too: `pipx install "overleaf-sync[gui]"`.
|
|
55
|
+
|
|
56
|
+
> **2.0 pre-release:** Overleaf changed its website in a way that broke version 1.2.0. The fixes are available as a pre-release: `pip3 install --pre "overleaf-sync[gui]"`. Please report any problems in the [issues](https://github.com/moritzgloeckl/overleaf-sync/issues).
|
|
57
|
+
|
|
58
|
+
### Prerequisites
|
|
59
|
+
- Create your project on [Overleaf](https://www.overleaf.com/project), for example a project named `test`. Overleaf-sync is not able to create projects (yet).
|
|
60
|
+
- Create a folder, preferably with the same name as the project (`test`) on your computer.
|
|
61
|
+
- Execute the script from that folder (`test`).
|
|
62
|
+
- If you do not specify the project name, overleaf-sync uses the current folder's name as the project name.
|
|
63
|
+
|
|
64
|
+
### Usage
|
|
65
|
+
#### Login
|
|
66
|
+
```
|
|
67
|
+
moritz@github:~/test$ ols login [--path]
|
|
68
|
+
Login successful. Cookie persisted as `.olauth`. You may now sync your project.
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Logging in will be handled by a mini web browser opening on your device (using Qt via PySide6, from the `gui` extra). You can then enter your username and password securely on the official Overleaf website. You might get asked to solve a CAPTCHA in the process. Your credentials are sent to Overleaf over HTTPS.
|
|
72
|
+
|
|
73
|
+
It then stores your *cookie* (**not** your login credentials) in a hidden file called `.olauth` in the same folder you run the command from. It is possible to store the cookie elsewhere using the `--path` option. The cookie file will not be synced to or from Overleaf.
|
|
74
|
+
|
|
75
|
+
Keep the `.olauth` file save, as it can be used to log in into your account.
|
|
76
|
+
|
|
77
|
+
### Listing all projects
|
|
78
|
+
```
|
|
79
|
+
moritz@github:~/test$ ols list [--store-path -v/--verbose]
|
|
80
|
+
10/31/2021, 01:23:45 - Project A
|
|
81
|
+
09/21/2020, 01:23:45 - Project B
|
|
82
|
+
08/11/2019, 01:23:45 - Project C
|
|
83
|
+
07/01/2018, 01:23:45 - Project D
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Use `ols list` to conveniently list all projects in your account available for syncing.
|
|
87
|
+
|
|
88
|
+
### Downloading project's PDF
|
|
89
|
+
```
|
|
90
|
+
moritz@github:~/test$ ols download [--name --download-path --store-path -v/--verbose]
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Use `ols download` to compile and download your project's PDF. Specify a download path if you do not want to store the PDF file in the current folder. Currently only downloads the first PDF file it finds.
|
|
94
|
+
|
|
95
|
+
### Syncing
|
|
96
|
+
```
|
|
97
|
+
moritz@github:~/test$ ols [-l/--local-only -r/--remote-only --store-path -p/--path -i/--olignore]
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Just calling `ols` will two-way sync your project. When there are changes both locally, and remotely you will be asked which file to keep. Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
|
|
101
|
+
|
|
102
|
+
The option `--store-path` specifies the path of the cookie file created by the `login` command. If you did not change its path, you do not need to specify this argument. The `-p/--path` option allows you to specify a different sync folder than the one you're calling `ols` from. The `-i/--olignore` option allows you to specify the path of an `.olignore` file. It uses `fnmatch` internally, so it may have some similarity to `.gitignore` but doesn't work exactly the same. For example, if you wish to exclude a specific folder named `out`, you need to specify it as `out/*`. See [here](https://docs.python.org/3/library/fnmatch.html) for more information.
|
|
103
|
+
|
|
104
|
+
Sample Output:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
Project queried successfully.
|
|
108
|
+
✅ Querying project
|
|
109
|
+
Project downloaded successfully.
|
|
110
|
+
✅ Downloading project
|
|
111
|
+
|
|
112
|
+
Syncing files from remote to local
|
|
113
|
+
====================
|
|
114
|
+
|
|
115
|
+
[SYNCING] report.tex
|
|
116
|
+
report.tex does not exist on local. Creating file.
|
|
117
|
+
|
|
118
|
+
[SYNCING] other-report.tex
|
|
119
|
+
other-report.tex does not exist on local. Creating file.
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
✅ Syncing files from remote to local
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
## Known Bugs
|
|
126
|
+
- When modifying a file on Overleaf and immediately syncing afterwards, the tool might not detect the changes. Please allow 1-2 minutes after modifying a file on Overleaf before syncing it to your local computer.
|
|
127
|
+
|
|
128
|
+
## Contributing
|
|
129
|
+
|
|
130
|
+
All pull requests and change/feature requests are welcome.
|
|
131
|
+
|
|
132
|
+
## Disclaimer
|
|
133
|
+
THE AUTHOR OF THIS SOFTWARE AND THIS SOFTWARE IS NOT ENDORSED BY, DIRECTLY AFFILIATED WITH, MAINTAINED, AUTHORIZED, OR SPONSORED BY OVERLEAF OR WRITELATEX LIMITED. ALL PRODUCT AND COMPANY NAMES ARE THE REGISTERED TRADEMARKS OF THEIR ORIGINAL OWNERS. THE USE OF ANY TRADE NAME OR TRADEMARK IS FOR IDENTIFICATION AND REFERENCE PURPOSES ONLY AND DOES NOT IMPLY ANY ASSOCIATION WITH THE TRADEMARK HOLDER OF THEIR PRODUCT BRAND.
|
|
134
|
+
|
|
135
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
136
|
+
|
|
137
|
+
THIS SOFTWARE WAS DESIGNED TO BE USED ONLY FOR RESEARCH PURPOSES. THIS SOFTWARE COMES WITH NO WARRANTIES OF ANY KIND WHATSOEVER. USE IT AT YOUR OWN RISK! IF THESE TERMS ARE NOT ACCEPTABLE, YOU AREN'T ALLOWED TO USE THE CODE.
|
|
138
|
+
|
|
139
|
+
|
|
@@ -16,13 +16,15 @@ This tool provides an easy way to synchronize Overleaf projects from and to your
|
|
|
16
16
|
|
|
17
17
|
## How To Use
|
|
18
18
|
### Install
|
|
19
|
-
The package is available via [PyPI](https://pypi.org/project/overleaf-sync/).
|
|
19
|
+
The package is available via [PyPI](https://pypi.org/project/overleaf-sync/) and needs Python 3.11 or newer. Install it including the login window (`gui` extra):
|
|
20
20
|
|
|
21
21
|
```
|
|
22
|
-
moritz@github:~/test$ pip3 install overleaf-sync
|
|
22
|
+
moritz@github:~/test$ pip3 install "overleaf-sync[gui]"
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
|
|
25
|
+
Depending on your local Python installation, you might need to use `pip` instead of `pip3`. [pipx](https://pipx.pypa.io/) works too: `pipx install "overleaf-sync[gui]"`.
|
|
26
|
+
|
|
27
|
+
> **2.0 pre-release:** Overleaf changed its website in a way that broke version 1.2.0. The fixes are available as a pre-release: `pip3 install --pre "overleaf-sync[gui]"`. Please report any problems in the [issues](https://github.com/moritzgloeckl/overleaf-sync/issues).
|
|
26
28
|
|
|
27
29
|
### Prerequisites
|
|
28
30
|
- Create your project on [Overleaf](https://www.overleaf.com/project), for example a project named `test`. Overleaf-sync is not able to create projects (yet).
|
|
@@ -37,7 +39,7 @@ moritz@github:~/test$ ols login [--path]
|
|
|
37
39
|
Login successful. Cookie persisted as `.olauth`. You may now sync your project.
|
|
38
40
|
```
|
|
39
41
|
|
|
40
|
-
Logging in will be handled by a mini web browser opening on your device (using
|
|
42
|
+
Logging in will be handled by a mini web browser opening on your device (using Qt via PySide6, from the `gui` extra). You can then enter your username and password securely on the official Overleaf website. You might get asked to solve a CAPTCHA in the process. Your credentials are sent to Overleaf over HTTPS.
|
|
41
43
|
|
|
42
44
|
It then stores your *cookie* (**not** your login credentials) in a hidden file called `.olauth` in the same folder you run the command from. It is possible to store the cookie elsewhere using the `--path` option. The cookie file will not be synced to or from Overleaf.
|
|
43
45
|
|
|
@@ -54,12 +56,21 @@ moritz@github:~/test$ ols list [--store-path -v/--verbose]
|
|
|
54
56
|
|
|
55
57
|
Use `ols list` to conveniently list all projects in your account available for syncing.
|
|
56
58
|
|
|
59
|
+
### Downloading project's PDF
|
|
60
|
+
```
|
|
61
|
+
moritz@github:~/test$ ols download [--name --download-path --store-path -v/--verbose]
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Use `ols download` to compile and download your project's PDF. Specify a download path if you do not want to store the PDF file in the current folder. Currently only downloads the first PDF file it finds.
|
|
65
|
+
|
|
57
66
|
### Syncing
|
|
58
67
|
```
|
|
59
68
|
moritz@github:~/test$ ols [-l/--local-only -r/--remote-only --store-path -p/--path -i/--olignore]
|
|
60
69
|
```
|
|
61
70
|
|
|
62
|
-
Just calling `ols` will two-way sync your project. When there are changes both locally and remotely you will be asked which file to keep. Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively.
|
|
71
|
+
Just calling `ols` will two-way sync your project. When there are changes both locally, and remotely you will be asked which file to keep. Using the `-l` or `-r` option you can specify to either sync local project files to Overleaf only or Overleaf files to local ones only respectively. When using these options you can also sync deleted files. If a file has been deleted it can either be deleted on the target (remote when `-l`, local when `-r`) as well, restored on the source (local when `-l`, remote when `-r`) or ignored.
|
|
72
|
+
|
|
73
|
+
The option `--store-path` specifies the path of the cookie file created by the `login` command. If you did not change its path, you do not need to specify this argument. The `-p/--path` option allows you to specify a different sync folder than the one you're calling `ols` from. The `-i/--olignore` option allows you to specify the path of an `.olignore` file. It uses `fnmatch` internally, so it may have some similarity to `.gitignore` but doesn't work exactly the same. For example, if you wish to exclude a specific folder named `out`, you need to specify it as `out/*`. See [here](https://docs.python.org/3/library/fnmatch.html) for more information.
|
|
63
74
|
|
|
64
75
|
Sample Output:
|
|
65
76
|
|
|
@@ -6,12 +6,12 @@
|
|
|
6
6
|
# Description: Overleaf Browser Login Utility
|
|
7
7
|
# Author: Moritz Glöckl
|
|
8
8
|
# License: MIT
|
|
9
|
-
# Version: 1.1.6
|
|
10
9
|
##################################################
|
|
11
10
|
|
|
12
|
-
from
|
|
13
|
-
from
|
|
14
|
-
from
|
|
11
|
+
from PySide6.QtCore import *
|
|
12
|
+
from PySide6.QtWidgets import *
|
|
13
|
+
from PySide6.QtWebEngineWidgets import *
|
|
14
|
+
from PySide6.QtWebEngineCore import QWebEngineProfile, QWebEngineSettings, QWebEnginePage
|
|
15
15
|
|
|
16
16
|
# Where to get the CSRF Token and where to send the login request to
|
|
17
17
|
LOGIN_URL = "https://www.overleaf.com/login"
|
|
@@ -60,7 +60,7 @@ class OlBrowserLoginWindow(QMainWindow):
|
|
|
60
60
|
|
|
61
61
|
if self.webview.url().toString() == PROJECT_URL:
|
|
62
62
|
self.webview.page().runJavaScript(
|
|
63
|
-
JAVASCRIPT_CSRF_EXTRACTOR, callback
|
|
63
|
+
JAVASCRIPT_CSRF_EXTRACTOR, 0, callback
|
|
64
64
|
)
|
|
65
65
|
|
|
66
66
|
def handle_cookie_added(self, cookie):
|
|
@@ -82,6 +82,11 @@ class OlBrowserLoginWindow(QMainWindow):
|
|
|
82
82
|
|
|
83
83
|
|
|
84
84
|
def login():
|
|
85
|
+
from PySide6.QtCore import QLoggingCategory
|
|
86
|
+
QLoggingCategory.setFilterRules('''\
|
|
87
|
+
qt.webenginecontext.info=false
|
|
88
|
+
''')
|
|
89
|
+
|
|
85
90
|
app = QApplication([])
|
|
86
91
|
ol_browser_login_window = OlBrowserLoginWindow()
|
|
87
92
|
ol_browser_login_window.show()
|
|
@@ -0,0 +1,404 @@
|
|
|
1
|
+
"""Overleaf Client"""
|
|
2
|
+
##################################################
|
|
3
|
+
# MIT License
|
|
4
|
+
##################################################
|
|
5
|
+
# File: olclient.py
|
|
6
|
+
# Description: Overleaf API Wrapper
|
|
7
|
+
# Author: Moritz Glöckl
|
|
8
|
+
# License: MIT
|
|
9
|
+
##################################################
|
|
10
|
+
|
|
11
|
+
import requests as reqs
|
|
12
|
+
from bs4 import BeautifulSoup
|
|
13
|
+
import json
|
|
14
|
+
import ssl
|
|
15
|
+
import uuid
|
|
16
|
+
import websocket
|
|
17
|
+
from socketIO_client import SocketIO
|
|
18
|
+
import time
|
|
19
|
+
|
|
20
|
+
# socketIO_client/transports.py has `except websocket.SSLError`, an attribute
|
|
21
|
+
# recent websocket-client (1.9.0+) no longer exposes; that raises AttributeError
|
|
22
|
+
# and masks whatever the actual connection error is. Keep this shim for as long
|
|
23
|
+
# as we depend on socketIO-client.
|
|
24
|
+
if not hasattr(websocket, "SSLError"):
|
|
25
|
+
websocket.SSLError = ssl.SSLError
|
|
26
|
+
|
|
27
|
+
# Where to get the CSRF Token and where to send the login request to
|
|
28
|
+
LOGIN_URL = "https://www.overleaf.com/login"
|
|
29
|
+
PROJECT_URL = "https://www.overleaf.com/project" # The dashboard URL
|
|
30
|
+
# The URL to download all the files in zip format
|
|
31
|
+
DOWNLOAD_URL = "https://www.overleaf.com/project/{}/download/zip"
|
|
32
|
+
UPLOAD_URL = "https://www.overleaf.com/project/{}/upload" # The URL to upload files
|
|
33
|
+
FOLDER_URL = "https://www.overleaf.com/project/{}/folder" # The URL to create folders
|
|
34
|
+
DELETE_URL = "https://www.overleaf.com/project/{}/doc/{}" # The URL to delete files
|
|
35
|
+
COMPILE_URL = "https://www.overleaf.com/project/{}/compile?enable_pdf_caching=true" # The URL to compile the project
|
|
36
|
+
BASE_URL = "https://www.overleaf.com" # The Overleaf Base URL
|
|
37
|
+
PATH_SEP = "/" # Use hardcoded path separator for both windows and posix system
|
|
38
|
+
|
|
39
|
+
class OverleafClient(object):
|
|
40
|
+
"""
|
|
41
|
+
Overleaf API Wrapper
|
|
42
|
+
Supports login, querying all projects, querying a specific project, downloading a project and
|
|
43
|
+
uploading a file to a project.
|
|
44
|
+
"""
|
|
45
|
+
|
|
46
|
+
@staticmethod
|
|
47
|
+
def filter_projects(json_content, more_attrs=None):
|
|
48
|
+
more_attrs = more_attrs or {}
|
|
49
|
+
for p in json_content:
|
|
50
|
+
if not p.get("archived") and not p.get("trashed"):
|
|
51
|
+
if all(p.get(k) == v for k, v in more_attrs.items()):
|
|
52
|
+
yield p
|
|
53
|
+
|
|
54
|
+
def __init__(self, cookie=None, csrf=None):
|
|
55
|
+
self._cookie = cookie # Store the cookie for authenticated requests
|
|
56
|
+
self._csrf = csrf # Store the CSRF token since it is needed for some requests
|
|
57
|
+
|
|
58
|
+
def login(self, username, password):
|
|
59
|
+
"""
|
|
60
|
+
WARNING - DEPRECATED - Not working as Overleaf introduced captchas
|
|
61
|
+
Login to the Overleaf Service with a username and a password
|
|
62
|
+
Params: username, password
|
|
63
|
+
Returns: Dict of cookie and CSRF
|
|
64
|
+
"""
|
|
65
|
+
|
|
66
|
+
get_login = reqs.get(LOGIN_URL)
|
|
67
|
+
self._csrf = BeautifulSoup(get_login.content, 'html.parser').find(
|
|
68
|
+
'input', {'name': '_csrf'}).get('value')
|
|
69
|
+
login_json = {
|
|
70
|
+
"_csrf": self._csrf,
|
|
71
|
+
"email": username,
|
|
72
|
+
"password": password
|
|
73
|
+
}
|
|
74
|
+
post_login = reqs.post(LOGIN_URL, json=login_json,
|
|
75
|
+
cookies=get_login.cookies)
|
|
76
|
+
|
|
77
|
+
# On a successful authentication the Overleaf API returns a new authenticated cookie.
|
|
78
|
+
# If the cookie is different than the cookie of the GET request the authentication was successful
|
|
79
|
+
if post_login.status_code == 200 and get_login.cookies["overleaf_session2"] != post_login.cookies[
|
|
80
|
+
"overleaf_session2"]:
|
|
81
|
+
self._cookie = post_login.cookies
|
|
82
|
+
|
|
83
|
+
# Enrich cookie with GCLB cookie from GET request above
|
|
84
|
+
self._cookie['GCLB'] = get_login.cookies['GCLB']
|
|
85
|
+
|
|
86
|
+
# CSRF changes after making the login request, new CSRF token will be on the projects page
|
|
87
|
+
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
|
|
88
|
+
self._csrf = BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-csrfToken'}) \
|
|
89
|
+
.get('content')
|
|
90
|
+
|
|
91
|
+
return {"cookie": self._cookie, "csrf": self._csrf}
|
|
92
|
+
|
|
93
|
+
@staticmethod
|
|
94
|
+
def _extract_projects(page_content):
|
|
95
|
+
"""
|
|
96
|
+
Extract the project list from the Overleaf dashboard page.
|
|
97
|
+
|
|
98
|
+
Overleaf used to embed the list as a bare JSON array in a
|
|
99
|
+
<meta name="ol-projects"> tag. It now embeds
|
|
100
|
+
{"totalSize": N, "projects": [...]} in a
|
|
101
|
+
<meta name="ol-prefetchedProjectsBlob"> tag instead. Support both so
|
|
102
|
+
this keeps working if Overleaf reverts or if an older cookie/page is
|
|
103
|
+
cached somewhere.
|
|
104
|
+
"""
|
|
105
|
+
soup = BeautifulSoup(page_content, 'html.parser')
|
|
106
|
+
tag = soup.find('meta', {'name': 'ol-prefetchedProjectsBlob'}) or soup.find('meta', {'name': 'ol-projects'})
|
|
107
|
+
json_content = json.loads(tag.get('content'))
|
|
108
|
+
if isinstance(json_content, dict):
|
|
109
|
+
json_content = json_content.get('projects', [])
|
|
110
|
+
return json_content
|
|
111
|
+
|
|
112
|
+
def all_projects(self):
|
|
113
|
+
"""
|
|
114
|
+
Get all of a user's active projects (= not archived and not trashed)
|
|
115
|
+
Returns: List of project objects
|
|
116
|
+
"""
|
|
117
|
+
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
|
|
118
|
+
json_content = OverleafClient._extract_projects(projects_page.content)
|
|
119
|
+
return list(OverleafClient.filter_projects(json_content))
|
|
120
|
+
|
|
121
|
+
def get_project(self, project_name):
|
|
122
|
+
"""
|
|
123
|
+
Get a specific project by project_name
|
|
124
|
+
Params: project_name, the name of the project
|
|
125
|
+
Returns: project object
|
|
126
|
+
"""
|
|
127
|
+
|
|
128
|
+
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
|
|
129
|
+
json_content = OverleafClient._extract_projects(projects_page.content)
|
|
130
|
+
return next(OverleafClient.filter_projects(json_content, {"name": project_name}), None)
|
|
131
|
+
|
|
132
|
+
def download_project(self, project_id):
|
|
133
|
+
"""
|
|
134
|
+
Download project in zip format
|
|
135
|
+
Params: project_id, the id of the project
|
|
136
|
+
Returns: bytes string (zip file)
|
|
137
|
+
"""
|
|
138
|
+
r = reqs.get(DOWNLOAD_URL.format(project_id),
|
|
139
|
+
stream=True, cookies=self._cookie)
|
|
140
|
+
return r.content
|
|
141
|
+
|
|
142
|
+
def create_folder(self, project_id, parent_folder_id, folder_name):
|
|
143
|
+
"""
|
|
144
|
+
Create a new folder in a project
|
|
145
|
+
|
|
146
|
+
Params:
|
|
147
|
+
project_id: the id of the project
|
|
148
|
+
parent_folder_id: the id of the parent folder, root is the project_id
|
|
149
|
+
folder_name: how the folder will be named
|
|
150
|
+
|
|
151
|
+
Returns: folder id or None
|
|
152
|
+
"""
|
|
153
|
+
|
|
154
|
+
params = {
|
|
155
|
+
"parent_folder_id": parent_folder_id,
|
|
156
|
+
"name": folder_name
|
|
157
|
+
}
|
|
158
|
+
headers = {
|
|
159
|
+
"X-Csrf-Token": self._csrf
|
|
160
|
+
}
|
|
161
|
+
r = reqs.post(FOLDER_URL.format(project_id),
|
|
162
|
+
cookies=self._cookie, headers=headers, json=params)
|
|
163
|
+
|
|
164
|
+
if r.ok:
|
|
165
|
+
return json.loads(r.content)
|
|
166
|
+
elif r.status_code == 400:
|
|
167
|
+
# Folder already exists. status_code is an int; comparing to
|
|
168
|
+
# str(400) was always False, so this branch never actually
|
|
169
|
+
# fired -- a real folder-exists response fell through to the
|
|
170
|
+
# HTTPError below instead of silently continuing.
|
|
171
|
+
return
|
|
172
|
+
else:
|
|
173
|
+
raise reqs.HTTPError()
|
|
174
|
+
|
|
175
|
+
def get_project_infos(self, project_id):
|
|
176
|
+
"""
|
|
177
|
+
Get detailed project infos about the project
|
|
178
|
+
|
|
179
|
+
Params:
|
|
180
|
+
project_id: the id of the project
|
|
181
|
+
|
|
182
|
+
Returns: project details
|
|
183
|
+
"""
|
|
184
|
+
project_infos = None
|
|
185
|
+
|
|
186
|
+
# Callback for the joinProjectResponse event the server pushes right
|
|
187
|
+
# after a successful connect+auto-join (see below) -- delivered as a
|
|
188
|
+
# plain named event, not as an ack to an explicit emit, and shaped as
|
|
189
|
+
# {"publicId": ..., "project": {...}}. Downstream code (upload_file,
|
|
190
|
+
# delete_file) indexes project_infos['rootFolder'][0][...] directly,
|
|
191
|
+
# so unwrap to the inner "project" dict here rather than returning
|
|
192
|
+
# the wrapper.
|
|
193
|
+
def set_project_infos(data):
|
|
194
|
+
nonlocal project_infos
|
|
195
|
+
project_infos = data.get('project')
|
|
196
|
+
|
|
197
|
+
# The Socket.IO handshake below (done again internally by the SocketIO()
|
|
198
|
+
# constructor) has Overleaf issue a *fresh* load-balancer affinity cookie
|
|
199
|
+
# (currently named GCLB, see olbrowserlogin.COOKIE_NAMES and commit
|
|
200
|
+
# 0cc61e7) on its response -- but socketIO_client calls requests.get()
|
|
201
|
+
# directly rather than through a Session, so that Set-Cookie is silently
|
|
202
|
+
# dropped, and the WebSocket upgrade that follows then 502s without it.
|
|
203
|
+
# Perform the same handshake ourselves first, purely to capture that
|
|
204
|
+
# cookie, and merge it into the header used for the real connection.
|
|
205
|
+
# (Built from whatever cookies are present rather than hardcoded names:
|
|
206
|
+
# a session may not carry every cookie -- this previously raised
|
|
207
|
+
# KeyError here even though overleaf_session2 alone is enough to connect.)
|
|
208
|
+
handshake = reqs.get(
|
|
209
|
+
"{}/socket.io/1/".format(BASE_URL),
|
|
210
|
+
params={"t": int(time.time() * 1000)},
|
|
211
|
+
cookies=self._cookie,
|
|
212
|
+
)
|
|
213
|
+
cookie_dict = dict(self._cookie)
|
|
214
|
+
cookie_dict.update(handshake.cookies.get_dict())
|
|
215
|
+
cookie = "; ".join(
|
|
216
|
+
"{}={}".format(name, value) for name, value in cookie_dict.items()
|
|
217
|
+
)
|
|
218
|
+
|
|
219
|
+
# projectId must be a query parameter on THIS handshake -- the one
|
|
220
|
+
# socketIO_client performs internally when SocketIO() connects, not
|
|
221
|
+
# the pre-fetch above. Overleaf now binds the session to a project at
|
|
222
|
+
# handshake time and rejects the connection otherwise
|
|
223
|
+
# (connectionRejected: "missing/bad ?projectId=... query flag on
|
|
224
|
+
# handshake"); confirmed it rejects even when projectId is only on
|
|
225
|
+
# the WebSocket upgrade URL -- it must be on the initial
|
|
226
|
+
# /socket.io/1/ GET specifically.
|
|
227
|
+
socket_io = SocketIO(
|
|
228
|
+
BASE_URL,
|
|
229
|
+
params={'t': int(time.time()), 'projectId': project_id},
|
|
230
|
+
headers={'Cookie': cookie}
|
|
231
|
+
)
|
|
232
|
+
|
|
233
|
+
# Wait until we connect to the socket
|
|
234
|
+
socket_io.on('connect', lambda: None)
|
|
235
|
+
socket_io.wait_for_callbacks()
|
|
236
|
+
|
|
237
|
+
# With projectId on the handshake, the server auto-joins and pushes
|
|
238
|
+
# the result as a joinProjectResponse event -- no explicit joinProject
|
|
239
|
+
# emit needed (confirmed live: the emit's ack callback never fires
|
|
240
|
+
# since the server responds with a named event, not an ack).
|
|
241
|
+
# wait_for_callbacks() only waits for a pending *ack* callback, which
|
|
242
|
+
# .on() never registers -- it would return immediately without
|
|
243
|
+
# reading anything. Wait in short slices instead, stopping as soon as
|
|
244
|
+
# the event has actually been processed.
|
|
245
|
+
socket_io.on('joinProjectResponse', set_project_infos)
|
|
246
|
+
for _ in range(20):
|
|
247
|
+
if project_infos is not None:
|
|
248
|
+
break
|
|
249
|
+
socket_io.wait(seconds=1)
|
|
250
|
+
|
|
251
|
+
# Disconnect from the socket if still connected
|
|
252
|
+
if socket_io.connected:
|
|
253
|
+
socket_io.disconnect()
|
|
254
|
+
|
|
255
|
+
return project_infos
|
|
256
|
+
|
|
257
|
+
def upload_file(self, project_id, project_infos, file_name, file_size, file):
|
|
258
|
+
"""
|
|
259
|
+
Upload a file to the project
|
|
260
|
+
|
|
261
|
+
Params:
|
|
262
|
+
project_id: the id of the project
|
|
263
|
+
file_name: how the file will be named
|
|
264
|
+
file_size: the size of the file in bytes
|
|
265
|
+
file: the file itself
|
|
266
|
+
|
|
267
|
+
Returns: True on success, False on fail
|
|
268
|
+
"""
|
|
269
|
+
|
|
270
|
+
# Set the folder_id to the id of the root folder
|
|
271
|
+
folder_id = project_infos['rootFolder'][0]['_id']
|
|
272
|
+
|
|
273
|
+
# The file name contains path separators, check folders
|
|
274
|
+
if PATH_SEP in file_name:
|
|
275
|
+
local_folders = file_name.split(PATH_SEP)[:-1] # Remove last item since this is the file name
|
|
276
|
+
current_overleaf_folder = project_infos['rootFolder'][0]['folders'] # Set the current remote folder
|
|
277
|
+
|
|
278
|
+
for local_folder in local_folders:
|
|
279
|
+
exists_on_remote = False
|
|
280
|
+
for remote_folder in current_overleaf_folder:
|
|
281
|
+
# Check if the folder exists on remote, continue with the new folder structure
|
|
282
|
+
if local_folder.lower() == remote_folder['name'].lower():
|
|
283
|
+
exists_on_remote = True
|
|
284
|
+
folder_id = remote_folder['_id']
|
|
285
|
+
current_overleaf_folder = remote_folder['folders']
|
|
286
|
+
break
|
|
287
|
+
# Create the folder if it doesn't exist
|
|
288
|
+
if not exists_on_remote:
|
|
289
|
+
new_folder = self.create_folder(project_id, folder_id, local_folder)
|
|
290
|
+
current_overleaf_folder.append(new_folder)
|
|
291
|
+
folder_id = new_folder['_id']
|
|
292
|
+
current_overleaf_folder = new_folder['folders']
|
|
293
|
+
params = {
|
|
294
|
+
"folder_id": folder_id,
|
|
295
|
+
"_csrf": self._csrf,
|
|
296
|
+
"qquuid": str(uuid.uuid4()),
|
|
297
|
+
"qqfilename": file_name,
|
|
298
|
+
"qqtotalfilesize": file_size,
|
|
299
|
+
}
|
|
300
|
+
files = {
|
|
301
|
+
# Overleaf's upload endpoint now rejects the request with 422
|
|
302
|
+
# {"success":false,"error":"invalid_filename"} unless the
|
|
303
|
+
# filename is also present as a plain multipart field, not just
|
|
304
|
+
# in qqfilename above. Confirmed live: identical request without
|
|
305
|
+
# this field gets 422; with it, 200 and the file is actually
|
|
306
|
+
# updated. It must be the bare file name: Overleaf rejects names
|
|
307
|
+
# containing "/" (SafePath), the folder is chosen via folder_id.
|
|
308
|
+
"name": (None, file_name.split(PATH_SEP)[-1]),
|
|
309
|
+
"qqfile": file
|
|
310
|
+
}
|
|
311
|
+
|
|
312
|
+
# Upload the file to the predefined folder
|
|
313
|
+
r = reqs.post(UPLOAD_URL.format(project_id), cookies=self._cookie, params=params, files=files)
|
|
314
|
+
|
|
315
|
+
# status_code is an int; comparing to str(200) was always False here,
|
|
316
|
+
# so this reported failure on every call regardless of the real
|
|
317
|
+
# result. Harmless today only because callers don't check the return
|
|
318
|
+
# value, but worth fixing since it's the only signal this method
|
|
319
|
+
# gives back.
|
|
320
|
+
return r.status_code == 200 and json.loads(r.content)["success"]
|
|
321
|
+
|
|
322
|
+
def delete_file(self, project_id, project_infos, file_name):
|
|
323
|
+
"""
|
|
324
|
+
Deletes a project's file
|
|
325
|
+
|
|
326
|
+
Params:
|
|
327
|
+
project_id: the id of the project
|
|
328
|
+
file_name: how the file will be named
|
|
329
|
+
|
|
330
|
+
Returns: True on success, False on fail
|
|
331
|
+
"""
|
|
332
|
+
|
|
333
|
+
file = None
|
|
334
|
+
|
|
335
|
+
# The file name contains path separators, check folders
|
|
336
|
+
if PATH_SEP in file_name:
|
|
337
|
+
local_folders = file_name.split(PATH_SEP)[:-1] # Remove last item since this is the file name
|
|
338
|
+
current_overleaf_folder = project_infos['rootFolder'][0]['folders'] # Set the current remote folder
|
|
339
|
+
|
|
340
|
+
for local_folder in local_folders:
|
|
341
|
+
for remote_folder in current_overleaf_folder:
|
|
342
|
+
if local_folder.lower() == remote_folder['name'].lower():
|
|
343
|
+
file = next((v for v in remote_folder['docs'] if v['name'] == file_name.split(PATH_SEP)[-1]),
|
|
344
|
+
None)
|
|
345
|
+
current_overleaf_folder = remote_folder['folders']
|
|
346
|
+
break
|
|
347
|
+
# File is in root folder
|
|
348
|
+
else:
|
|
349
|
+
file = next((v for v in project_infos['rootFolder'][0]['docs'] if v['name'] == file_name), None)
|
|
350
|
+
|
|
351
|
+
# File not found!
|
|
352
|
+
if file is None:
|
|
353
|
+
return False
|
|
354
|
+
|
|
355
|
+
headers = {
|
|
356
|
+
"X-Csrf-Token": self._csrf
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
r = reqs.delete(DELETE_URL.format(project_id, file['_id']), cookies=self._cookie, headers=headers, json={})
|
|
360
|
+
|
|
361
|
+
# status_code is an int; comparing to str(204) was always False
|
|
362
|
+
# regardless of the real result, same bug class as upload_file and
|
|
363
|
+
# create_folder above.
|
|
364
|
+
return r.status_code == 204
|
|
365
|
+
|
|
366
|
+
def download_pdf(self, project_id):
|
|
367
|
+
"""
|
|
368
|
+
Compiles and returns a project's PDF
|
|
369
|
+
|
|
370
|
+
Params:
|
|
371
|
+
project_id: the id of the project
|
|
372
|
+
|
|
373
|
+
Returns: PDF file name and content on success
|
|
374
|
+
"""
|
|
375
|
+
headers = {
|
|
376
|
+
"X-Csrf-Token": self._csrf
|
|
377
|
+
}
|
|
378
|
+
|
|
379
|
+
body = {
|
|
380
|
+
"check": "silent",
|
|
381
|
+
"draft": False,
|
|
382
|
+
"incrementalCompilesEnabled": True,
|
|
383
|
+
"rootDoc_id": "",
|
|
384
|
+
"stopOnFirstError": False
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
r = reqs.post(COMPILE_URL.format(project_id), cookies=self._cookie, headers=headers, json=body)
|
|
388
|
+
|
|
389
|
+
if not r.ok:
|
|
390
|
+
raise reqs.HTTPError()
|
|
391
|
+
|
|
392
|
+
compile_result = json.loads(r.content)
|
|
393
|
+
|
|
394
|
+
if compile_result["status"] != "success":
|
|
395
|
+
raise reqs.HTTPError()
|
|
396
|
+
|
|
397
|
+
pdf_file = next(v for v in compile_result['outputFiles'] if v['type'] == 'pdf')
|
|
398
|
+
|
|
399
|
+
download_req = reqs.get(BASE_URL + pdf_file['url'], cookies=self._cookie, headers=headers)
|
|
400
|
+
|
|
401
|
+
if download_req.ok:
|
|
402
|
+
return pdf_file['path'], download_req.content
|
|
403
|
+
|
|
404
|
+
return None
|