httpdbg 0.32.1__tar.gz → 1.1.0__tar.gz

{httpdbg-0.32.1 → httpdbg-1.1.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.2
 Name: httpdbg
-Version: 0.32.1
-Summary: A very simple tool to debug HTTP(S) client requests
+Version: 1.1.0
+Summary: A very simple tool to debug HTTP(S) client and server requests.
 Author-email: cle-b <cle@tictac.pm>
 License: Apache-2.0
 Project-URL: Source, https://github.com/cle-b/httpdbg/
@@ -26,7 +26,7 @@ License-File: LICENSE
 
 # httpdbg
 
-`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client requests in a Python program.
+`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client and server requests in a Python program.
 
 To use it, execute your program using the `pyhttpdbg` command instead of `python` and that's it. Open a browser to `http://localhost:4909` to view the requests:
 
@@ -42,6 +42,8 @@ pip install httpdbg
 
 ## usage
 
+By default, both client and server requests are recorded. To record only client requests, use the `--only-client` command line argument.
+
 ### interactive console
 
 Open an interactive console using the command `pyhttpdbg`.
@@ -89,8 +91,32 @@ pyhttpdbg -m pytest [arg1 --arg2 ...]
 
 In that case, the requests will be grouped by test, and any requests made within a fixture or the setup/teardown methods will be identified by a tag.
 
-## Initiators
 
+### HTTP server
+
+You can trace all HTTP requests received by your HTTP server.
+
+```console
+pyhttpdbg -m flask --app demoflask run
+.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
+  httpdbg - HTTP(S) requests available at http://localhost:4909/
+.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
+ * Serving Flask app 'demoflask'
+ * Debug mode: off
+WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
+ * Running on http://127.0.0.1:5000
+Press CTRL+C to quit
+127.0.0.1 - - [14/Mar/2025 17:47:42] "GET / HTTP/1.1" 200 -
+127.0.0.1 - - [14/Mar/2025 17:47:50] "GET /hello/Ben HTTP/1.1" 200 -
+127.0.0.1 - - [14/Mar/2025 17:47:56] "GET /oups HTTP/1.1" 404 -
+```
+
+![httpdbg 1.1.0 - pyhttpdbg -m flask --app demoflask run](https://github.com/cle-b/httpdbg/blob/main/ui_server.png?raw=true)
+
+This feature is new and not fully implemented. See [limitations](#limitations) for more details.
+
+## initiators and groups
+ 
 An initiator is the function/method that is at the origin of the HTTP requests. By default, we already support some packages but you can add your own initiators. 
 
 To add a new package in the list of initiators, you can use the `-i` command line argument:
@@ -101,17 +127,34 @@ pyhttpdbg -i api_client_pck --script my_script.py
 
 You can use any package as an initiator, this is not limited to HTTP requests.
 
-### Already supported packages
+### supported HTTP client
 
-| packages       | status                              | 
-|----------------|-------------------------------------|
-| requests       | supported                           |
-| urllib3        | supported                           |
-| httpx          | supported                           |
-| aiohttp        | supported                           |
-| pytest         | supported                           |
-| unittest       | supported                           |
-| _your_package_ | yes, with the arg _-i your_package_ |
+The initiator is the highest-level method used to make the request through the client. The group and the initiator are identical. If a request to a URL generates multiple requests, as in the case of redirection, all these requests will be grouped together.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| requests       | supported                           | yes       | yes       |
+| urllib3        | supported                           | yes       | yes       |
+| httpx          | supported                           | yes       | yes       |
+| aiohttp        | supported                           | yes       | yes       |
+| _your_package_ | yes, with the arg _-i your_package_ | yes       | yes       |
+
+### supported HTTP server
+
+The initiator is the low-level socket method that receives the data. The group is the endpoint method used to handle the request. A client request made within the endpoint method will be included in this group.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| flask          | supported                           | no        | yes       |
+
+### supported test framework
+
+The requests are grouped by tests.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| pytest         | supported                           | no        | yes       |
+| unittest       | supported                           | no        | yes       |
 
 ## configuration
 
@@ -120,7 +163,9 @@ No configuration is necessary to start but some few settings are available for p
 ### command line
 
 ```console
-usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version] [--initiator INITIATOR] [--keep-up | --force-quit]
+usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version]
+                 [--initiator INITIATOR] [--only-client]
+                 [--keep-up | --force-quit]                 
                  [--console | --module MODULE | --script SCRIPT]
 
 httdbg - a very simple tool to debug HTTP(S) client requests
@@ -132,6 +177,7 @@ options:
   --version, -v         print the httpdbg version
   --initiator INITIATOR, -i INITIATOR
                         add a new initiator (package)
+  --only-client         record only HTTP client requests
   --keep-up, -k         keep the server up even if the requests have been read
   --force-quit, -q      stop the server even if the requests have not been read
   --console             run a python console (default)
@@ -167,6 +213,12 @@ The requests:
  * are still available in the web page even if the python process stopped (except if you force quit before the requests have been loaded by the web page).
  * are automatically cleaned if a new execution is detected.
 
+## limitations
+
+Theoretically, if your HTTP client or server uses a standard Python socket, the HTTP requests will be recorded.
+
+Support for recording requests on the server side is new in `httpdbg 1.0.0` and is currently limited. For example, it doesn't work with `FastAPI`, which uses `uvloop`, but it does work with `Uvicorn` if the loop is `asyncio`. Work is in progress to improve this.
+
 ## documentation
 
 https://httpdbg.readthedocs.io

{httpdbg-0.32.1 → httpdbg-1.1.0}/README.md

@@ -1,6 +1,6 @@
 # httpdbg
 
-`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client requests in a Python program.
+`httpdbg` is a tool for Python developers to easily debug the HTTP(S) client and server requests in a Python program.
 
 To use it, execute your program using the `pyhttpdbg` command instead of `python` and that's it. Open a browser to `http://localhost:4909` to view the requests:
 
@@ -16,6 +16,8 @@ pip install httpdbg
 
 ## usage
 
+By default, both client and server requests are recorded. To record only client requests, use the `--only-client` command line argument.
+
 ### interactive console
 
 Open an interactive console using the command `pyhttpdbg`.
@@ -63,8 +65,32 @@ pyhttpdbg -m pytest [arg1 --arg2 ...]
 
 In that case, the requests will be grouped by test, and any requests made within a fixture or the setup/teardown methods will be identified by a tag.
 
-## Initiators
 
+### HTTP server
+
+You can trace all HTTP requests received by your HTTP server.
+
+```console
+pyhttpdbg -m flask --app demoflask run
+.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
+  httpdbg - HTTP(S) requests available at http://localhost:4909/
+.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
+ * Serving Flask app 'demoflask'
+ * Debug mode: off
+WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
+ * Running on http://127.0.0.1:5000
+Press CTRL+C to quit
+127.0.0.1 - - [14/Mar/2025 17:47:42] "GET / HTTP/1.1" 200 -
+127.0.0.1 - - [14/Mar/2025 17:47:50] "GET /hello/Ben HTTP/1.1" 200 -
+127.0.0.1 - - [14/Mar/2025 17:47:56] "GET /oups HTTP/1.1" 404 -
+```
+
+![httpdbg 1.1.0 - pyhttpdbg -m flask --app demoflask run](https://github.com/cle-b/httpdbg/blob/main/ui_server.png?raw=true)
+
+This feature is new and not fully implemented. See [limitations](#limitations) for more details.
+
+## initiators and groups
+ 
 An initiator is the function/method that is at the origin of the HTTP requests. By default, we already support some packages but you can add your own initiators. 
 
 To add a new package in the list of initiators, you can use the `-i` command line argument:
@@ -75,17 +101,34 @@ pyhttpdbg -i api_client_pck --script my_script.py
 
 You can use any package as an initiator, this is not limited to HTTP requests.
 
-### Already supported packages
+### supported HTTP client
 
-| packages       | status                              | 
-|----------------|-------------------------------------|
-| requests       | supported                           |
-| urllib3        | supported                           |
-| httpx          | supported                           |
-| aiohttp        | supported                           |
-| pytest         | supported                           |
-| unittest       | supported                           |
-| _your_package_ | yes, with the arg _-i your_package_ |
+The initiator is the highest-level method used to make the request through the client. The group and the initiator are identical. If a request to a URL generates multiple requests, as in the case of redirection, all these requests will be grouped together.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| requests       | supported                           | yes       | yes       |
+| urllib3        | supported                           | yes       | yes       |
+| httpx          | supported                           | yes       | yes       |
+| aiohttp        | supported                           | yes       | yes       |
+| _your_package_ | yes, with the arg _-i your_package_ | yes       | yes       |
+
+### supported HTTP server
+
+The initiator is the low-level socket method that receives the data. The group is the endpoint method used to handle the request. A client request made within the endpoint method will be included in this group.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| flask          | supported                           | no        | yes       |
+
+### supported test framework
+
+The requests are grouped by tests.
+
+| packages       | status                              | initiator | group     |
+|----------------|-------------------------------------|-----------|-----------|
+| pytest         | supported                           | no        | yes       |
+| unittest       | supported                           | no        | yes       |
 
 ## configuration
 
@@ -94,7 +137,9 @@ No configuration is necessary to start but some few settings are available for p
 ### command line
 
 ```console
-usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version] [--initiator INITIATOR] [--keep-up | --force-quit]
+usage: pyhttpdbg [-h] [--host HOST] [--port PORT] [--version]
+                 [--initiator INITIATOR] [--only-client]
+                 [--keep-up | --force-quit]                 
                  [--console | --module MODULE | --script SCRIPT]
 
 httdbg - a very simple tool to debug HTTP(S) client requests
@@ -106,6 +151,7 @@ options:
   --version, -v         print the httpdbg version
   --initiator INITIATOR, -i INITIATOR
                         add a new initiator (package)
+  --only-client         record only HTTP client requests
   --keep-up, -k         keep the server up even if the requests have been read
   --force-quit, -q      stop the server even if the requests have not been read
   --console             run a python console (default)
@@ -141,6 +187,12 @@ The requests:
  * are still available in the web page even if the python process stopped (except if you force quit before the requests have been loaded by the web page).
  * are automatically cleaned if a new execution is detected.
 
+## limitations
+
+Theoretically, if your HTTP client or server uses a standard Python socket, the HTTP requests will be recorded.
+
+Support for recording requests on the server side is new in `httpdbg 1.0.0` and is currently limited. For example, it doesn't work with `FastAPI`, which uses `uvloop`, but it does work with `Uvicorn` if the loop is `asyncio`. Work is in progress to improve this.
+
 ## documentation
 
 https://httpdbg.readthedocs.io

{httpdbg-0.32.1 → httpdbg-1.1.0}/httpdbg/__init__.py

@@ -3,6 +3,6 @@ from httpdbg.hooks.all import httprecord
 from httpdbg.records import HTTPRecords
 
 
-__version__ = "0.32.1"
+__version__ = "1.1.0"
 
 __all__ = ["httprecord", "HTTPRecords"]

{httpdbg-0.32.1 → httpdbg-1.1.0}/httpdbg/__main__.py

@@ -29,7 +29,8 @@ def pyhttpdbg(params, subparams, test_mode=False):
     sys.path.insert(0, "")  # to mimic the default python command behavior
 
     with httpdbg_srv(params.host, params.port) as records:
-        with httprecord(records, params.initiator):
+        records.server = not params.only_client
+        with httprecord(records, params.initiator, server=records.server):
             if params.module:
                 run_module(subparams)
             elif params.script:

{httpdbg-0.32.1 → httpdbg-1.1.0}/httpdbg/args.py

@@ -36,6 +36,12 @@ def read_args(args: List[str]) -> Tuple[argparse.Namespace, List[str]]:
         "--initiator", "-i", action="append", help="add a new initiator (package)"
     )
 
+    parser.add_argument(
+        "--only-client",
+        action="store_true",
+        help="record only HTTP client requests",
+    )
+
     server_state = parser.add_mutually_exclusive_group()
 
     server_state.add_argument(

httpdbg-1.1.0/httpdbg/hooks/all.py

@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+from contextlib import contextmanager
+from typing import Generator
+from typing import List
+from typing import Tuple
+from typing import Union
+
+from httpdbg.hooks.aiohttp import hook_aiohttp
+from httpdbg.hooks.external import watcher_external
+from httpdbg.hooks.generic import hook_generic
+from httpdbg.hooks.http import hook_http
+from httpdbg.hooks.httpx import hook_httpx
+from httpdbg.hooks.pytest import hook_pytest
+from httpdbg.hooks.requests import hook_requests
+from httpdbg.hooks.socket import hook_socket
+from httpdbg.hooks.unittest import hook_unittest
+from httpdbg.hooks.urllib3 import hook_urllib3
+from httpdbg.hooks.flask import hook_flask
+from httpdbg.records import HTTPRecords
+
+
+@contextmanager
+def httprecord(
+    records: HTTPRecords = None,
+    initiators: Union[List[str], None] = None,
+    client: bool = True,
+    server: bool = False,
+    ignore: Union[List[Tuple[str, int]], None] = None,
+) -> Generator[HTTPRecords, None, None]:
+    if records is None:
+        records = HTTPRecords(client=client, server=server, ignore=ignore)
+
+    with watcher_external(records, initiators, server):
+        with hook_flask(records):
+            with hook_socket(records):
+                with hook_http(records):
+                    with hook_httpx(records):
+                        with hook_requests(records):
+                            with hook_urllib3(records):
+                                with hook_aiohttp(records):
+                                    with hook_pytest(records):
+                                        with hook_unittest(records):
+                                            with hook_generic(records, initiators):
+                                                yield records

{httpdbg-0.32.1 → httpdbg-1.1.0}/httpdbg/hooks/external.py

@@ -4,10 +4,11 @@ from typing import Generator
 import glob
 import os
 import pickle
-import shutil
 import tempfile
 import time
 import threading
+from typing import List
+from typing import Union
 
 from httpdbg.env import HTTPDBG_MULTIPROCESS_DIR
 from httpdbg.log import logger
@@ -15,7 +16,11 @@ from httpdbg.records import HTTPRecords
 
 
 @contextmanager
-def watcher_external(records: HTTPRecords) -> Generator[HTTPRecords, None, None]:
+def watcher_external(
+    records: HTTPRecords,
+    initiators: Union[List[str], None] = None,
+    server: bool = False,
+) -> Generator[HTTPRecords, None, None]:
     if HTTPDBG_MULTIPROCESS_DIR not in os.environ:
         with tempfile.TemporaryDirectory(prefix="httpdbg_") as httpdbg_multiprocess_dir:
 
@@ -31,7 +36,20 @@ def watcher_external(records: HTTPRecords) -> Generator[HTTPRecords, None, None]
             )
             sitecustomize = os.path.join(httpdbg_multiprocess_dir, "sitecustomize.py")
 
-            shutil.copy(template, sitecustomize)
+            with open(template) as f:
+                template_content = f.read()
+
+            if initiators:
+                template_content = template_content.replace(
+                    "HTTPDBG_INITIATORS = []", f"HTTPDBG_INITIATORS = {initiators}", 1
+                )
+
+            template_content = template_content.replace(
+                "HTTPDBG_RECORD_SERVER = False", f"HTTPDBG_RECORD_SERVER = {server}", 1
+            )
+
+            with open(sitecustomize, "w") as f:
+                f.write(template_content)
 
             if "PYTHONPATH" in os.environ:
                 os.environ["PYTHONPATH"] = (

httpdbg-1.1.0/httpdbg/hooks/flask.py

@@ -0,0 +1,134 @@
+# -*- coding: utf-8 -*-
+from collections.abc import Callable
+from contextlib import contextmanager
+from functools import wraps
+from typing import Generator
+
+from httpdbg.hooks.utils import getcallargs
+from httpdbg.hooks.utils import decorate
+from httpdbg.hooks.utils import undecorate
+from httpdbg.initiator import httpdbg_endpoint
+from httpdbg.records import HTTPRecords
+
+
+def set_hook_flask_endpoint(records: HTTPRecords, method: Callable):
+
+    @wraps(
+        method
+    )  # to avoid AssertionError: View function mapping is overwriting an existing endpoint function: hook
+    def hook(*args, **kwargs):
+
+        with httpdbg_endpoint(
+            records,
+            method,
+            *args,
+            **kwargs,
+        ):
+            ret = method(*args, **kwargs)
+        return ret
+
+    return hook
+
+
+# we must not apply the hook more than once on a mapped endpoint function
+# AssertionError: View function mapping is overwriting an existing endpoint function: xxxx
+already_mapped = {}
+
+
+def set_hook_flask_add_url_rule(records: HTTPRecords, method: Callable):
+
+    def hook(*args, **kwargs):
+
+        callargs = getcallargs(method, *args, **kwargs)
+
+        param = "view_func"
+
+        if param in callargs:
+            original_view_func = callargs[param]
+            if original_view_func not in already_mapped:
+                callargs[param] = decorate(
+                    records, callargs[param], set_hook_flask_endpoint
+                )
+                already_mapped[original_view_func] = callargs[param]
+            else:
+                callargs[param] = already_mapped[original_view_func]
+
+            args = [callargs[param] if x == original_view_func else x for x in args]
+            if param in kwargs:
+                kwargs[param] = callargs[param]
+
+        return method(*args, **kwargs)
+
+    return hook
+
+
+def set_hook_flask_register_error_handler(records: HTTPRecords, method: Callable):
+
+    def hook(*args, **kwargs):
+
+        callargs = getcallargs(method, *args, **kwargs)
+
+        param = "f"
+
+        if param in callargs:
+            original_view_func = callargs[param]
+            if original_view_func not in already_mapped:
+                callargs[param] = decorate(
+                    records, callargs[param], set_hook_flask_endpoint
+                )
+                already_mapped[original_view_func] = callargs[param]
+            else:
+                callargs[param] = already_mapped[original_view_func]
+
+            args = [callargs[param] if x == original_view_func else x for x in args]
+            if param in kwargs:
+                kwargs[param] = callargs[param]
+
+        return method(*args, **kwargs)
+
+    return hook
+
+
+@contextmanager
+def hook_flask(records: HTTPRecords) -> Generator[None, None, None]:
+    hooks = False
+    try:
+        import flask
+
+        flask.Flask.add_url_rule = decorate(
+            records, flask.Flask.add_url_rule, set_hook_flask_add_url_rule
+        )
+
+        flask.Flask.register_error_handler = decorate(
+            records,
+            flask.Flask.register_error_handler,
+            set_hook_flask_register_error_handler,
+        )
+        flask.Flask.handle_exception = decorate(
+            records, flask.Flask.handle_exception, set_hook_flask_endpoint
+        )
+        flask.Flask.handle_http_exception = decorate(
+            records, flask.Flask.handle_http_exception, set_hook_flask_endpoint
+        )
+        flask.Flask.handle_user_exception = decorate(
+            records, flask.Flask.handle_user_exception, set_hook_flask_endpoint
+        )
+
+        hooks = True
+    except ImportError:
+        pass
+
+    yield
+
+    if hooks:
+        flask.Flask.add_url_rule = undecorate(flask.Flask.add_url_rule)
+        flask.Flask.register_error_handler = undecorate(
+            flask.Flask.register_error_handler
+        )
+        flask.Flask.handle_exception = undecorate(flask.Flask.handle_exception)
+        flask.Flask.handle_http_exception = undecorate(
+            flask.Flask.handle_http_exception
+        )
+        flask.Flask.handle_user_exception = undecorate(
+            flask.Flask.handle_user_exception
+        )

httpdbg-1.1.0/httpdbg/hooks/http.py

@@ -0,0 +1,47 @@
+# -*- coding: utf-8 -*-
+from collections.abc import Callable
+from contextlib import contextmanager
+from typing import Generator
+
+
+from httpdbg.hooks.utils import decorate
+from httpdbg.hooks.utils import undecorate
+from httpdbg.initiator import httpdbg_group
+from httpdbg.records import HTTPRecords
+
+
+def set_hook_for_http_server_handle_one_request(records: HTTPRecords, method: Callable):
+    def hook(*args, **kwargs):
+
+        # the group label/full_label will be updated when the endpoint method will be called
+        with httpdbg_group(records, "one request", ""):
+            ret = method(*args, **kwargs)
+
+        return ret
+
+    return hook
+
+
+@contextmanager
+def hook_http(records: HTTPRecords) -> Generator[None, None, None]:
+    hooks = False
+    try:
+        import http
+        import http.server
+
+        http.server.BaseHTTPRequestHandler.handle_one_request = decorate(
+            records,
+            http.server.BaseHTTPRequestHandler.handle_one_request,
+            set_hook_for_http_server_handle_one_request,
+        )
+
+        hooks = True
+    except ImportError:
+        pass
+
+    yield
+
+    if hooks:
+        http.server.BaseHTTPRequestHandler.handle_one_request = undecorate(
+            http.server.BaseHTTPRequestHandler.handle_one_request
+        )

{httpdbg-0.32.1 → httpdbg-1.1.0}/httpdbg/hooks/pytest.py

@@ -34,7 +34,7 @@ def set_hook_for_pytest_runtest(records: HTTPRecords, method: Callable):
         if "item" in callargs:
             full_label = getattr(callargs["item"], "nodeid", "unknown test")
             label = getattr(callargs["item"], "name", "unknown test")
-            with httpdbg_group(records, label, full_label):
+            with httpdbg_group(records, label, full_label, updatable=False):
                 return method(*args, **kwargs)
         else:
             return method(*args, **kwargs)