browserfetch 0.6.2__tar.gz → 0.8.0__tar.gz

{browserfetch-0.6.2 → browserfetch-0.8.0}/PKG-INFO

@@ -1,71 +1,71 @@
-Metadata-Version: 2.1
-Name: browserfetch
-Version: 0.6.2
-Summary: fetch in Python using your browser! 
-License: GNU General Public License v3 (GPLv3)
-Project-URL: Homepage, https://github.com/5j9/browserfetch
-Keywords: browser,fetch,python,cookies
-Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
-Requires-Python: >=3.11
-Description-Content-Type: text/x-rst
-License-File: LICENSE
-Requires-Dist: aiohttp
-Requires-Dist: pyperclip
-
-Fetch using your browser.
-
-Let the browser manage cookies for you.
-
-⚠️ This project is a very simple implementation. Not tested thoroughly. Consider it a proof of concept.
-
-Usage
------
-1. You'll run a Python script containing some code like this:
-
-.. code-block:: python
-
-    from asyncio import gather, new_event_loop
-
-    from browserfetch import fetch, get, post, run_server
-
-
-    async def main():
-        response1, response2, reponse3 = await gather(
-            get('https://example.com/path1', params={'a': 1}),
-            fetch('https://example.com/image.png'),
-            post('https://example.com/path2', data={'a': 1}),
-        )
-        # do stuff with retrieved responses
-
-
-    loop = new_event_loop()
-    loop.create_task(start_server())
-    loop.run_until_complete(main())
-
-
-2. Open your browser, goto http://example.com (perhaps solve a captcha and log in).
-3. Copy the contents of `browserfetch.js`_ file and paste it in browser's console. (You can use a browser extensions like violentmonkey_/tampermonkey_ to do this step for you.)
-
-That's it! Your Python script starts handling requests.
-The browser tab should remain open of-coarse.
-
-The server can handle multiple websocket connections from different websites simultaneously.
-
-How it works
-------------
-``browserfetch`` communicates with your browser using a websocket. The ``fetch`` function just passes the request to browser and it is the browser that handles the actual request. Response data is sent back to Python using the same WebSocket connection.
-
-Motivations
------------
-* `browser_cookie3 stopped working on Chrome-based browsers`_. There is a workaround: ShadowCopy, but it requires admin privilege.
-* Another issue with browser_cookie's approach is that it retrieves cookies from cookie files, but these files are not updated instantly. Thus, you might have to wait or retry a few times before you can successfully access newly set cookies.
-* ShadowCopying and File access are slow and inefficient operations.
-
-Downsides
----------
-* Setting up ``browserfetch`` is more cumbersome since it requires running a Python server and also injecting a small script into the webpage. Using ``browser_cookie3`` might be a better choice if there are many websites that you need to communicate with.
-
-.. _`browser_cookie3 stopped working on Chrome-based browsers`: https://github.com/borisbabic/browser_cookie3/issues/180
-.. _tampermonkey: https://github.com/Tampermonkey/tampermonkey
-.. _violentmonkey: https://github.com/violentmonkey/violentmonkey
-.. _browserfetch.js: https://github.com/5j9/browserfetch/blob/master/browserfetch/browserfetch.js
+Metadata-Version: 2.1
+Name: browserfetch
+Version: 0.8.0
+Summary: fetch in Python using your browser! 
+Keywords: browser,fetch,python,cookies
+Requires-Python: >=3.11
+Description-Content-Type: text/x-rst
+Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
+Requires-Dist: aiohttp
+Requires-Dist: pyperclip
+Project-URL: Homepage, https://github.com/5j9/browserfetch
+
+Fetch using your browser.
+
+Let the browser manage cookies for you.
+
+⚠️ Incomplete. Not tested thoroughly. Consider using `Playwright`_, especially for more complex scenarios.
+
+Usage
+-----
+1. You'll run a Python script containing some code like this:
+
+.. code-block:: python
+
+    from asyncio import gather, new_event_loop
+
+    from browserfetch import fetch, get, post, run_server
+
+
+    async def main():
+        response1, response2, reponse3 = await gather(
+            get('https://example.com/path1', params={'a': 1}),
+            fetch('https://example.com/image.png'),
+            post('https://example.com/path2', data={'a': 1}),
+        )
+        # do stuff with retrieved responses
+
+
+    loop = new_event_loop()
+    loop.create_task(start_server())
+    loop.run_until_complete(main())
+
+
+2. Open your browser, goto http://example.com (perhaps solve a captcha and log in).
+3. Copy the contents of `browserfetch.js`_ file and paste it in browser's console. (You can use a browser extensions like violentmonkey_/tampermonkey_ to do this step for you.)
+
+That's it! Your Python script starts handling requests.
+The browser tab should remain open of-coarse.
+
+The server can handle multiple websocket connections from different websites simultaneously.
+
+How it works
+------------
+``browserfetch`` communicates with your browser using a websocket. The ``fetch`` function just passes the request to browser and it is the browser that handles the actual request. Response data is sent back to Python using the same WebSocket connection.
+
+Motivations
+-----------
+* `browser_cookie3 stopped working on Chrome-based browsers`_. There is a workaround: ShadowCopy, but it requires admin privilege.
+* Another issue with browser_cookie's approach is that it retrieves cookies from cookie files, but these files are not updated instantly. Thus, you might have to wait or retry a few times before you can successfully access newly set cookies.
+* ShadowCopying and File access are slow and inefficient operations.
+
+Downsides
+---------
+* Setting up ``browserfetch`` is more cumbersome since it requires running a Python server and also injecting a small script into the webpage. Using ``browser_cookie3`` might be a better choice if there are many websites that you need to communicate with.
+
+.. _playwright: https://playwright.dev/python/docs/intro
+.. _`browser_cookie3 stopped working on Chrome-based browsers`: https://github.com/borisbabic/browser_cookie3/issues/180
+.. _tampermonkey: https://github.com/Tampermonkey/tampermonkey
+.. _violentmonkey: https://github.com/violentmonkey/violentmonkey
+.. _browserfetch.js: https://github.com/5j9/browserfetch/blob/master/browserfetch/browserfetch.js
+

{browserfetch-0.6.2 → browserfetch-0.8.0}/README.rst

@@ -2,7 +2,7 @@ Fetch using your browser.
 
 Let the browser manage cookies for you.
 
-⚠️ This project is a very simple implementation. Not tested thoroughly. Consider it a proof of concept.
+⚠️ Incomplete. Not tested thoroughly. Consider using `Playwright`_, especially for more complex scenarios.
 
 Usage
 -----
@@ -51,6 +51,7 @@ Downsides
 ---------
 * Setting up ``browserfetch`` is more cumbersome since it requires running a Python server and also injecting a small script into the webpage. Using ``browser_cookie3`` might be a better choice if there are many websites that you need to communicate with.
 
+.. _playwright: https://playwright.dev/python/docs/intro
 .. _`browser_cookie3 stopped working on Chrome-based browsers`: https://github.com/borisbabic/browser_cookie3/issues/180
 .. _tampermonkey: https://github.com/Tampermonkey/tampermonkey
 .. _violentmonkey: https://github.com/violentmonkey/violentmonkey

{browserfetch-0.6.2 → browserfetch-0.8.0}/browserfetch/__init__.py

@@ -1,7 +1,14 @@
-__version__ = '0.6.2'
+__version__ = '0.8.0'
 
 import atexit
-from asyncio import CancelledError, Event, Task, get_running_loop, wait_for
+from asyncio import (
+    AbstractEventLoop,
+    CancelledError,
+    Event,
+    Task,
+    get_running_loop,
+    wait_for,
+)
 from collections import defaultdict
 from dataclasses import dataclass
 from json import dumps, loads
@@ -15,9 +22,9 @@ from aiohttp.web_runner import AppRunner, TCPSite
 
 logger = getLogger(__name__)
 # maps host to its host_ready event or its websocket
-hosts: dict[
-    str, Event | WebSocketResponse | ClientWebSocketResponse
-] = defaultdict(Event)
+hosts: dict[str, Event | WebSocketResponse | ClientWebSocketResponse] = (
+    defaultdict(Event)
+)
 # maps response event id to its response event or response dict
 responses: dict[int, Event | dict] = {}
 
@@ -106,6 +113,7 @@ async def receive_responses(ws: WebSocketResponse | ClientWebSocketResponse):
 
 
 routes = RouteTableDef()
+PROTOCOL = '2'
 
 
 @routes.get('/ws')
@@ -113,7 +121,10 @@ async def _(request):
     ws = WebSocketResponse()
     await ws.prepare(request)
 
-    host = await ws.receive_str()
+    host, _, version = (await ws.receive_str()).partition(' ')
+    assert (
+        version == PROTOCOL
+    ), f'JavaScript protocol version: {version}, expected: {PROTOCOL}'
     logger.info('registering host %s', host)
 
     ws_or_e = hosts[host]
@@ -165,10 +176,29 @@ async def relay_client(server_host, server_port):
             try:
                 await receive_responses(ws)
             except TypeError:
-                logger.info('relay server WebSocket was closed')
+                logger.info('relay WebSocket was closed')
+                hosts.default_factory = Event
+                for host, ws_or_e in hosts.items():
+                    if isinstance(ws_or_e, Event):
+                        ws_or_e.clear()
+                    else:
+                        hosts[host] = Event()
+                await start_server(host=_host, port=_port)
                 return
 
 
+async def evaluate(
+    string: str,
+    host: str,
+    timeout: int | float = 95,
+):
+    """Evaluate string in browser context and return JSON.stringify(result)."""
+    d = await _request(
+        host, {'action': 'eval', 'string': string, 'timeout': timeout}, None
+    )
+    return d['result']
+
+
 async def fetch(
     url: str,
     *,
@@ -195,6 +225,7 @@ async def fetch(
     d = await _request(
         host,
         {
+            'action': 'fetch',
             'url': url,
             'options': options,
             'timeout': timeout,
@@ -268,12 +299,13 @@ app.add_routes(routes)
 app_runner = AppRunner(app)
 
 
-def shutdown_server(loop):
+def _shutdown_server(loop: AbstractEventLoop):
     logger.info('waiting for app_runner.cleanup()')
     loop.run_until_complete(app_runner.cleanup())
 
 
-def shutdown_relay_client(loop, task: Task):
+def _cancel_relay_task(loop: AbstractEventLoop, task: Task):
+    logger.info('cancelling relay task')
     task.cancel()
     try:
         loop.run_until_complete(task)
@@ -282,10 +314,13 @@ def shutdown_relay_client(loop, task: Task):
 
 
 _server = False
+_host = '127.0.0.1'
+_port = 9404
 
 
-async def start_server(*, host='127.0.0.1', port=9404):
-    global _server
+async def start_server(*, host=_host, port=_port):
+    global _server, _host, _port
+    _host, _port = host, port
     loop = get_running_loop()
     await app_runner.setup()
     site = TCPSite(app_runner, host, port)
@@ -298,8 +333,8 @@ async def start_server(*, host='127.0.0.1', port=9404):
             e,
         )
         relay_task = loop.create_task(relay_client(host, port))
-        atexit.register(shutdown_relay_client, loop, relay_task)
+        atexit.register(_cancel_relay_task, loop, relay_task)
     else:
         _server = True
-        atexit.register(shutdown_server, loop)
+        atexit.register(_shutdown_server, loop)
         logger.info('server started at http://%s:%s', host, port)

{browserfetch-0.6.2 → browserfetch-0.8.0}/browserfetch/__main__.py

@@ -20,5 +20,5 @@ def main():
             copy(f.read())
 
 
-if __name__ == "__main__":
+if __name__ == '__main__':
     main()

browserfetch-0.8.0/browserfetch/browserfetch.js

@@ -0,0 +1,120 @@
+// ==UserScript==
+// @name        browserfetch
+// @namespace   https://github.com/5j9/browserfetch
+// @match       https://example.com/
+// @grant       GM_registerMenuCommand
+// ==/UserScript==
+(() => {
+    /**
+     * @param {Blob} body 
+     * @param {j} Object
+     * @returns {Promise<Blob>}
+     */
+    async function doFetch(req, body) {
+        var returnData, response;
+        var options = req['options'] || {};
+
+        if (req['timeout']) {
+            options.signal = AbortSignal.timeout(req['timeout'] * 1000);
+        }
+
+        if (body !== null) {
+            options.body = body;
+        }
+
+        try {
+            var r = await fetch(req['url'], options);
+            returnData = {
+                'event_id': req['event_id'],
+                'headers': Object.fromEntries([...r.headers]),
+                'ok': r.ok,
+                'redirected': r.redirected,
+                'status': r.status,
+                'status_text': r.statusText,
+                'type': r.type,
+                'url': r.url
+            };
+            response = await r.blob();
+        } catch (err) {
+            returnData = {
+                'event_id': req['event_id'],
+                'error': err.toString()
+            };
+            response = "";
+        };
+        return new Blob([new TextEncoder().encode(JSON.stringify(returnData)), "\0", response]);
+    }
+
+    /**
+     * 
+     * @param {String} s 
+     * @returns {Promise<Uint8Array>}
+     */
+    async function doEval(req) {
+        var evalled, resp;
+        try {
+            evalled = eval(req['string']);
+            resp = {'result': evalled, 'event_id': req['event_id']};
+        } catch (err) {
+            resp = {'result': err.toString(), 'event_id': req['event_id']};
+        }
+        return new TextEncoder().encode(JSON.stringify(resp));
+    }
+
+    /**
+     * 
+     * @param {ArrayBuffer} d 
+     * @returns {Array.<{binaryPart: Uint8Array, jsonPart: Object}>}
+     */
+    function parseData(d) {
+        var blob, jArray;
+        var dArray = new Uint8Array(d);
+        var nullIndex = dArray.indexOf(0);
+        if (nullIndex === -1) {
+            blob = null;
+            jArray = dArray;
+        } else {
+            blob = dArray.slice(nullIndex + 1);
+            jArray = dArray.slice(0, nullIndex)
+        }
+
+        return [blob, JSON.parse(new TextDecoder().decode(jArray))]
+    }
+
+    function connect() {
+        var protocol = '2'
+        var ws = new WebSocket("ws://127.0.0.1:9404/ws");
+        ws.binaryType = "arraybuffer";
+
+        ws.onopen = () => {
+            ws.send(location.host + ' ' + protocol);
+        }
+
+        ws.onclose = () => {
+            console.error('browserfetch: WebSocket was closed; will retry in 5 seconds');
+            setTimeout(connect, 5000);
+        };
+
+        ws.onmessage = async (evt) => {
+            var result, j, b;
+            [b, j] = parseData(evt.data);
+            switch (j['action']) {
+                case 'fetch':
+                    result = await doFetch(j, b);
+                    break;
+                case 'eval':
+                    result = await doEval(j);
+            }
+            ws.send(result);
+        }
+    };
+
+    if (window.GM_registerMenuCommand) {
+        GM_registerMenuCommand(
+            'connect to browserfetch',
+            connect
+        );
+    } else {
+        connect();
+    }
+})();

{browserfetch-0.6.2 → browserfetch-0.8.0}/pyproject.toml

@@ -1,14 +1,6 @@
 [build-system]
-requires = [
-    "setuptools>=66.1.0",
-    "wheel",
-]
-build-backend = "setuptools.build_meta"
-
-[tool.isort]
-profile = "black"
-line_length = 79
-combine_as_imports = true
+requires = ['flit_core >=3.2,<4']
+build-backend = 'flit_core.buildapi'
 
 [project]
 name = "browserfetch"
@@ -38,13 +30,18 @@ content-type = "text/x-rst"
 [project.urls]
 Homepage = "https://github.com/5j9/browserfetch"
 
-[tool.setuptools]
-packages = ["browserfetch"]
-zip-safe = false
-include-package-data = false
-
-[tool.setuptools.dynamic.version]
-attr = "browserfetch.__version__"
-
-[tool.setuptools.package-data]
-browserfetch = ["*.js"]
+[tool.ruff]
+line-length = 79
+format.quote-style = 'single'
+lint.isort.combine-as-imports = true
+lint.extend-select = [
+    'I',  # isort
+    'UP',  # pyupgrade
+]
+lint.ignore = [
+    'UP027',  # list comprehensions are faster than generator expressions
+    'E721',  # Do not compare types, use `isinstance()`
+]
+
+[tool.pytest.ini_options]
+addopts = '--quiet --tb=short'

browserfetch-0.6.2/browserfetch/browserfetch.js

@@ -1,75 +0,0 @@
-// ==UserScript==
-// @name        browserfetch
-// @namespace   https://github.com/5j9/browserfetch
-// @match       https://example.com/
-// @grant       GM_registerMenuCommand
-// ==/UserScript==
-(() => {
-    function connect() {
-        var ws = new WebSocket("ws://127.0.0.1:9404/ws");
-        ws.binaryType = "arraybuffer";
-
-        ws.onopen = () => {
-            ws.send(location.host);
-        }
-
-        ws.onclose = function () {
-            console.error('browserfetch: WebSocket was closed; will retry in 5 seconds');
-            setTimeout(connect, 5000);
-        };
-
-        ws.onmessage = async (evt) => {
-            var returnData, responseBlob, body, jArray;
-            var requestArray = new Uint8Array(evt.data);
-            var nullIndex = requestArray.indexOf(0);
-            if (nullIndex === -1) {
-                body = null;
-                jArray = requestArray;
-            } else {
-                body = requestArray.slice(nullIndex + 1);
-                jArray = requestArray.slice(0, nullIndex)
-            }
-            var j = JSON.parse(new TextDecoder().decode(jArray));
-            var options = j['options'] || {};
-
-            if (j['timeout']) {
-                options.signal = AbortSignal.timeout(j['timeout'] * 1000);
-            }
-
-            if (body !== null) {
-                options.body = body;
-            }
-
-            try {
-                var r = await fetch(j['url'], options);
-                returnData = {
-                    'event_id': j['event_id'],
-                    'headers': Object.fromEntries([...r.headers]),
-                    'ok': r.ok,
-                    'redirected': r.redirected,
-                    'status': r.status,
-                    'status_text': r.statusText,
-                    'type': r.type,
-                    'url': r.url
-                };
-                responseBlob = await r.blob();
-            } catch (err) {
-                returnData = {
-                    'event_id': j['event_id'],
-                    'error': err.toString()
-                };
-                responseBlob = "";
-            };
-            ws.send(new Blob([new TextEncoder().encode(JSON.stringify(returnData)), "\0", responseBlob]));
-        }
-    };
-
-    if (window.GM_registerMenuCommand) {
-        GM_registerMenuCommand(
-            'connect to browserfetch',
-            connect
-        );
-    } else {
-        connect();
-    }
-})();

browserfetch-0.6.2/browserfetch.egg-info/PKG-INFO

@@ -1,71 +0,0 @@
-Metadata-Version: 2.1
-Name: browserfetch
-Version: 0.6.2
-Summary: fetch in Python using your browser! 
-License: GNU General Public License v3 (GPLv3)
-Project-URL: Homepage, https://github.com/5j9/browserfetch
-Keywords: browser,fetch,python,cookies
-Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
-Requires-Python: >=3.11
-Description-Content-Type: text/x-rst
-License-File: LICENSE
-Requires-Dist: aiohttp
-Requires-Dist: pyperclip
-
-Fetch using your browser.
-
-Let the browser manage cookies for you.
-
-⚠️ This project is a very simple implementation. Not tested thoroughly. Consider it a proof of concept.
-
-Usage
------
-1. You'll run a Python script containing some code like this:
-
-.. code-block:: python
-
-    from asyncio import gather, new_event_loop
-
-    from browserfetch import fetch, get, post, run_server
-
-
-    async def main():
-        response1, response2, reponse3 = await gather(
-            get('https://example.com/path1', params={'a': 1}),
-            fetch('https://example.com/image.png'),
-            post('https://example.com/path2', data={'a': 1}),
-        )
-        # do stuff with retrieved responses
-
-
-    loop = new_event_loop()
-    loop.create_task(start_server())
-    loop.run_until_complete(main())
-
-
-2. Open your browser, goto http://example.com (perhaps solve a captcha and log in).
-3. Copy the contents of `browserfetch.js`_ file and paste it in browser's console. (You can use a browser extensions like violentmonkey_/tampermonkey_ to do this step for you.)
-
-That's it! Your Python script starts handling requests.
-The browser tab should remain open of-coarse.
-
-The server can handle multiple websocket connections from different websites simultaneously.
-
-How it works
-------------
-``browserfetch`` communicates with your browser using a websocket. The ``fetch`` function just passes the request to browser and it is the browser that handles the actual request. Response data is sent back to Python using the same WebSocket connection.
-
-Motivations
------------
-* `browser_cookie3 stopped working on Chrome-based browsers`_. There is a workaround: ShadowCopy, but it requires admin privilege.
-* Another issue with browser_cookie's approach is that it retrieves cookies from cookie files, but these files are not updated instantly. Thus, you might have to wait or retry a few times before you can successfully access newly set cookies.
-* ShadowCopying and File access are slow and inefficient operations.
-
-Downsides
----------
-* Setting up ``browserfetch`` is more cumbersome since it requires running a Python server and also injecting a small script into the webpage. Using ``browser_cookie3`` might be a better choice if there are many websites that you need to communicate with.
-
-.. _`browser_cookie3 stopped working on Chrome-based browsers`: https://github.com/borisbabic/browser_cookie3/issues/180
-.. _tampermonkey: https://github.com/Tampermonkey/tampermonkey
-.. _violentmonkey: https://github.com/violentmonkey/violentmonkey
-.. _browserfetch.js: https://github.com/5j9/browserfetch/blob/master/browserfetch/browserfetch.js

browserfetch-0.6.2/browserfetch.egg-info/SOURCES.txt

@@ -1,12 +0,0 @@
-LICENSE
-README.rst
-pyproject.toml
-browserfetch/__init__.py
-browserfetch/__main__.py
-browserfetch/browserfetch.js
-browserfetch.egg-info/PKG-INFO
-browserfetch.egg-info/SOURCES.txt
-browserfetch.egg-info/dependency_links.txt
-browserfetch.egg-info/not-zip-safe
-browserfetch.egg-info/requires.txt
-browserfetch.egg-info/top_level.txt

browserfetch-0.6.2/browserfetch.egg-info/dependency_links.txt

@@ -1 +0,0 @@
-

browserfetch-0.6.2/browserfetch.egg-info/not-zip-safe

@@ -1 +0,0 @@
-

browserfetch-0.6.2/browserfetch.egg-info/requires.txt

@@ -1,2 +0,0 @@
-aiohttp
-pyperclip

browserfetch-0.6.2/browserfetch.egg-info/top_level.txt

@@ -1 +0,0 @@
-browserfetch

browserfetch-0.6.2/setup.cfg

@@ -1,4 +0,0 @@
-[egg_info]
-tag_build = 
-tag_date = 0
-