fastapi-gae-logging 0.0.1__tar.gz → 0.0.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,162 @@
1
+ Metadata-Version: 2.1
2
+ Name: fastapi-gae-logging
3
+ Version: 0.0.3
4
+ Summary: Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine. Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
5
+ Home-page: https://github.com/chrisK824/fastapi-gae-logging
6
+ Author: Chris Karvouniaris
7
+ Author-email: christos.karvouniaris247@gmail.com
8
+ License: MIT
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.7
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: fastapi>=0.112.2
15
+ Requires-Dist: google-cloud-logging>=3.11.2
16
+
17
+ # fastapi-gae-logging
18
+ Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine to ease out logs analysis and monitoring through Google Cloud Log Explorer.
19
+
20
+ ## What problem does this package solve? Why do we need this?
21
+ When deploying FastAPI applications on Google App Engine, I encountered several challenges with logging and observability even when using the official package for this purpose, `google-cloud-logging`.
22
+
23
+ - **Scattered Logs**: Logs generated during a single request lifecycle were scattered across different log entries, making it difficult to trace the entire flow of a request, especially when troubleshooting issues.
24
+
25
+ - **Log Severity Mismatch**: The severity level of logs was not properly propagated throughout the request lifecycle. This meant that if an error occurred at any point in the request, the earlier logs did not reflect the severity of the final outcome, making it harder to identify problematic requests.
26
+
27
+ - **Payload Logging Issues**: Capturing and logging request payloads was cumbersome, requiring extra logging in the handlers and extra deployments. This led to incomplete logs, making it harder to reproduce issues or analyze request content.
28
+
29
+ - **Inconsistent Log Structures**: The default logging setup lacked a consistent structure, which made it challenging to filter, search, and analyze logs in the Google Cloud Log Explorer.
30
+
31
+ ## So what does it do?
32
+ The `fastapi-gae-logging` module addresses these problems by:
33
+
34
+ - **Grouping Logs by Request**: All logs generated during a request's lifecycle are grouped together, allowing for a complete view of the request flow in the Google Cloud Log Explorer. This makes it much easier to trace and troubleshoot issues.
35
+
36
+ - **Log Level Propagation**: The maximum log level observed during a request's lifecycle is propagated, ensuring that logs associated with a failed request reflect the appropriate severity. This improves the accuracy and utility of log searches based on severity.
37
+
38
+ - **Structured Payload Logging**: Request payloads are captured and logged in a structured format, even for non-dictionary JSON payloads. This ensures that all relevant request data is available for analysis, improving the ability to diagnose issues.
39
+
40
+
41
+ ## Install
42
+ `pip install fastapi-gae-logging`
43
+
44
+ ## Features:
45
+
46
+ - **Request Logs Grouping**: Groups logs from the same request lifecycle to simplify log analysis using Google Cloud Log Explorer. The logger name for grouping can be customized and defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
47
+ - **Request Maximum Log Level Propagation**: Propagates the maximum log level throughout the request lifecycle, making it easier to search logs based on the severity of an issue.
48
+ - **Request Payload Logging**:
49
+ - Logs the payload of a request as part of the parent log in request-grouped logs.
50
+ - Assumes the payload is a valid JSON - basically what Python's `json.loads()` can parse - otherwise, it is discarded.
51
+ - Logs the payload as a dictionary using the `google-cloud-logging`'s `log_struct` method. If the payload is not a dictionary, then it is wrapped in a dummy keyword to construct one. The dummy keyword follows the format: `<original_type_of_payload>__payload_wrapper`. For example, if the payload is a list, which still a valid JSON which Python's `json.loads()` will parse just fine, then the payload list will be wrapped in a virtual dictionary under the key `list_payload_wrapper`. The dictionary is logged alongside the parent log in request-grouped logs and ends up in the `jsonPayload` field.
52
+
53
+
54
+ ## API
55
+
56
+ - `FastAPIGAELoggingHandler` class: A custom Cloud Logging Handler for FastAPI applications to use with the official google-cloud-logging library.
57
+
58
+
59
+ ## Example of usage
60
+
61
+ ```python
62
+ from fastapi import FastAPI
63
+ from fastapi.responses import JSONResponse
64
+ from fastapi.exceptions import HTTPException
65
+ import logging
66
+ import os
67
+
68
+ app = FastAPI()
69
+
70
+ if os.getenv('GAE_ENV', '').startswith('standard'):
71
+ import google.cloud.logging
72
+ from google.cloud.logging_v2.handlers import setup_logging
73
+ from fastapi_gae_logging import FastAPIGAELoggingHandler
74
+
75
+ client = google.cloud.logging.Client()
76
+ gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
77
+ setup_logging(handler=gae_log_handler)
78
+
79
+ logging.getLogger().setLevel(logging.DEBUG)
80
+
81
+ @app.get("/info")
82
+ def info():
83
+ logging.debug("this is a debug")
84
+ logging.info("this is an info")
85
+ return JSONResponse(
86
+ content={"message": "info"}
87
+ )
88
+
89
+ @app.get("/warning")
90
+ async def warning():
91
+ logging.debug("this is a debug")
92
+ logging.info("this is an info")
93
+ logging.warning("this is a warning")
94
+ return JSONResponse(
95
+ content={"message": "warning"}
96
+ )
97
+
98
+ @app.get("/error")
99
+ def error():
100
+ logging.debug("this is a debug")
101
+ logging.info("this is an info")
102
+ logging.warning("this is a warning")
103
+ logging.error("this is an error")
104
+ return JSONResponse(
105
+ content={"message": "error"}
106
+ )
107
+
108
+ @app.get("/exception")
109
+ def exception():
110
+ logging.debug("this is a debug")
111
+ logging.info("this is an info")
112
+ logging.error("this is an error")
113
+ raise ValueError("This is a value error")
114
+
115
+ @app.get("/http_exception")
116
+ def http_exception():
117
+ logging.debug("this is a debug")
118
+ logging.info("this is an info")
119
+ raise HTTPException(
120
+ status_code=404,
121
+ detail={"error": "Resource not found"}
122
+ )
123
+
124
+ @app.post("/post_payload")
125
+ def post_payload(payload: Any = Body(None)):
126
+ logging.debug("this is an debug")
127
+ logging.info(payload)
128
+ return JSONResponse(content={"mirror_response": payload}, status_code=200)
129
+
130
+ ```
131
+
132
+ ## How it looks in Google Cloud Log Explorer
133
+
134
+ ### Logger selection
135
+ ![alt text](<logger_selection.jpg>)
136
+
137
+ ### Groupped logs with propagated log severity to the parent log
138
+
139
+ ![alt text](<groupped_logs.jpg>)
140
+
141
+ ### Grouped logs in request with payload
142
+ ![alt text](<request_with_payload.jpg>)
143
+
144
+ ## Dependencies
145
+ This tool is built upon the following packages:
146
+
147
+ - `fastapi`: A modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints.
148
+ - `google-cloud-logging`: Google Cloud Logging API client library for logging and managing logs in Google Cloud Platform.
149
+
150
+
151
+ ## Implementation Concept
152
+ - **Middleware Integration**: A custom middleware integrates into FastAPI to intercept requests and log data after processing.The custom middleware is added to the FastAPI application during the initialization of the FastAPIGAELoggingHandler.
153
+ - **Context Management**: Uses context variables to manage request-specific data and metadata such as request payload, Google Cloud trace ID, start time of the incoming request and the maximum log level observed during the request lifecycle.
154
+ - **Log Interception**: A logging filter intercepts log records, injecting trace information and adjusting the maximum log level based on observed log severity.
155
+ - **Cloud Logging**: Utilizes Google Cloud Logging to group logs by request and propagate the maximum log level, enhancing observability and troubleshooting.
156
+ - **Structured Logging**: Parent log of the request-response lifecycle is structured and sent to Google Cloud Logging with additional context, such as the request method, URL, and user agent after the request has been processed and served.
157
+
158
+
159
+ ## Roadmap
160
+ - Allow for opting out of the dictionary enforcement of logged payload.
161
+ - Allow for opting out of request payload logging at all.
162
+ - Explore other fields in Google Cloud Logs. Investigate and consider utilizing additional fields available in Google Cloud Logs that may allow for more goodies.
@@ -0,0 +1,146 @@
1
+ # fastapi-gae-logging
2
+ Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine to ease out logs analysis and monitoring through Google Cloud Log Explorer.
3
+
4
+ ## What problem does this package solve? Why do we need this?
5
+ When deploying FastAPI applications on Google App Engine, I encountered several challenges with logging and observability even when using the official package for this purpose, `google-cloud-logging`.
6
+
7
+ - **Scattered Logs**: Logs generated during a single request lifecycle were scattered across different log entries, making it difficult to trace the entire flow of a request, especially when troubleshooting issues.
8
+
9
+ - **Log Severity Mismatch**: The severity level of logs was not properly propagated throughout the request lifecycle. This meant that if an error occurred at any point in the request, the earlier logs did not reflect the severity of the final outcome, making it harder to identify problematic requests.
10
+
11
+ - **Payload Logging Issues**: Capturing and logging request payloads was cumbersome, requiring extra logging in the handlers and extra deployments. This led to incomplete logs, making it harder to reproduce issues or analyze request content.
12
+
13
+ - **Inconsistent Log Structures**: The default logging setup lacked a consistent structure, which made it challenging to filter, search, and analyze logs in the Google Cloud Log Explorer.
14
+
15
+ ## So what does it do?
16
+ The `fastapi-gae-logging` module addresses these problems by:
17
+
18
+ - **Grouping Logs by Request**: All logs generated during a request's lifecycle are grouped together, allowing for a complete view of the request flow in the Google Cloud Log Explorer. This makes it much easier to trace and troubleshoot issues.
19
+
20
+ - **Log Level Propagation**: The maximum log level observed during a request's lifecycle is propagated, ensuring that logs associated with a failed request reflect the appropriate severity. This improves the accuracy and utility of log searches based on severity.
21
+
22
+ - **Structured Payload Logging**: Request payloads are captured and logged in a structured format, even for non-dictionary JSON payloads. This ensures that all relevant request data is available for analysis, improving the ability to diagnose issues.
23
+
24
+
25
+ ## Install
26
+ `pip install fastapi-gae-logging`
27
+
28
+ ## Features:
29
+
30
+ - **Request Logs Grouping**: Groups logs from the same request lifecycle to simplify log analysis using Google Cloud Log Explorer. The logger name for grouping can be customized and defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
31
+ - **Request Maximum Log Level Propagation**: Propagates the maximum log level throughout the request lifecycle, making it easier to search logs based on the severity of an issue.
32
+ - **Request Payload Logging**:
33
+ - Logs the payload of a request as part of the parent log in request-grouped logs.
34
+ - Assumes the payload is a valid JSON - basically what Python's `json.loads()` can parse - otherwise, it is discarded.
35
+ - Logs the payload as a dictionary using the `google-cloud-logging`'s `log_struct` method. If the payload is not a dictionary, then it is wrapped in a dummy keyword to construct one. The dummy keyword follows the format: `<original_type_of_payload>__payload_wrapper`. For example, if the payload is a list, which still a valid JSON which Python's `json.loads()` will parse just fine, then the payload list will be wrapped in a virtual dictionary under the key `list_payload_wrapper`. The dictionary is logged alongside the parent log in request-grouped logs and ends up in the `jsonPayload` field.
36
+
37
+
38
+ ## API
39
+
40
+ - `FastAPIGAELoggingHandler` class: A custom Cloud Logging Handler for FastAPI applications to use with the official google-cloud-logging library.
41
+
42
+
43
+ ## Example of usage
44
+
45
+ ```python
46
+ from fastapi import FastAPI
47
+ from fastapi.responses import JSONResponse
48
+ from fastapi.exceptions import HTTPException
49
+ import logging
50
+ import os
51
+
52
+ app = FastAPI()
53
+
54
+ if os.getenv('GAE_ENV', '').startswith('standard'):
55
+ import google.cloud.logging
56
+ from google.cloud.logging_v2.handlers import setup_logging
57
+ from fastapi_gae_logging import FastAPIGAELoggingHandler
58
+
59
+ client = google.cloud.logging.Client()
60
+ gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
61
+ setup_logging(handler=gae_log_handler)
62
+
63
+ logging.getLogger().setLevel(logging.DEBUG)
64
+
65
+ @app.get("/info")
66
+ def info():
67
+ logging.debug("this is a debug")
68
+ logging.info("this is an info")
69
+ return JSONResponse(
70
+ content={"message": "info"}
71
+ )
72
+
73
+ @app.get("/warning")
74
+ async def warning():
75
+ logging.debug("this is a debug")
76
+ logging.info("this is an info")
77
+ logging.warning("this is a warning")
78
+ return JSONResponse(
79
+ content={"message": "warning"}
80
+ )
81
+
82
+ @app.get("/error")
83
+ def error():
84
+ logging.debug("this is a debug")
85
+ logging.info("this is an info")
86
+ logging.warning("this is a warning")
87
+ logging.error("this is an error")
88
+ return JSONResponse(
89
+ content={"message": "error"}
90
+ )
91
+
92
+ @app.get("/exception")
93
+ def exception():
94
+ logging.debug("this is a debug")
95
+ logging.info("this is an info")
96
+ logging.error("this is an error")
97
+ raise ValueError("This is a value error")
98
+
99
+ @app.get("/http_exception")
100
+ def http_exception():
101
+ logging.debug("this is a debug")
102
+ logging.info("this is an info")
103
+ raise HTTPException(
104
+ status_code=404,
105
+ detail={"error": "Resource not found"}
106
+ )
107
+
108
+ @app.post("/post_payload")
109
+ def post_payload(payload: Any = Body(None)):
110
+ logging.debug("this is an debug")
111
+ logging.info(payload)
112
+ return JSONResponse(content={"mirror_response": payload}, status_code=200)
113
+
114
+ ```
115
+
116
+ ## How it looks in Google Cloud Log Explorer
117
+
118
+ ### Logger selection
119
+ ![alt text](<logger_selection.jpg>)
120
+
121
+ ### Groupped logs with propagated log severity to the parent log
122
+
123
+ ![alt text](<groupped_logs.jpg>)
124
+
125
+ ### Grouped logs in request with payload
126
+ ![alt text](<request_with_payload.jpg>)
127
+
128
+ ## Dependencies
129
+ This tool is built upon the following packages:
130
+
131
+ - `fastapi`: A modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints.
132
+ - `google-cloud-logging`: Google Cloud Logging API client library for logging and managing logs in Google Cloud Platform.
133
+
134
+
135
+ ## Implementation Concept
136
+ - **Middleware Integration**: A custom middleware integrates into FastAPI to intercept requests and log data after processing.The custom middleware is added to the FastAPI application during the initialization of the FastAPIGAELoggingHandler.
137
+ - **Context Management**: Uses context variables to manage request-specific data and metadata such as request payload, Google Cloud trace ID, start time of the incoming request and the maximum log level observed during the request lifecycle.
138
+ - **Log Interception**: A logging filter intercepts log records, injecting trace information and adjusting the maximum log level based on observed log severity.
139
+ - **Cloud Logging**: Utilizes Google Cloud Logging to group logs by request and propagate the maximum log level, enhancing observability and troubleshooting.
140
+ - **Structured Logging**: Parent log of the request-response lifecycle is structured and sent to Google Cloud Logging with additional context, such as the request method, URL, and user agent after the request has been processed and served.
141
+
142
+
143
+ ## Roadmap
144
+ - Allow for opting out of the dictionary enforcement of logged payload.
145
+ - Allow for opting out of request payload logging at all.
146
+ - Explore other fields in Google Cloud Logs. Investigate and consider utilizing additional fields available in Google Cloud Logs that may allow for more goodies.
@@ -3,6 +3,7 @@ import time
3
3
  import contextvars
4
4
  import os
5
5
  import re
6
+ import json
6
7
  from typing import Optional, Dict, Any
7
8
  from fastapi import FastAPI
8
9
  from starlette.exceptions import HTTPException
@@ -119,9 +120,14 @@ class GAERequestLogger:
119
120
  if request.method in ('POST', 'PUT', 'PATCH', 'DELETE'):
120
121
  try:
121
122
  payload = await request.json()
122
- except Exception:
123
+ except json.JSONDecodeError:
123
124
  pass
124
125
 
126
+ if not isinstance(payload, dict):
127
+ payload = {
128
+ f"{type(payload).__name__}_payload_wrapper": payload
129
+ }
130
+
125
131
  self.logger.log_struct(
126
132
  info=payload,
127
133
  resource=self.resource,
@@ -143,7 +149,17 @@ class FastAPIGAELoggingMiddleware:
143
149
 
144
150
  async def __call__(self, scope: Scope, receive: Receive, send: Send):
145
151
  if scope["type"] == "http":
146
- request = Request(scope, receive=receive)
152
+
153
+ # https://stackoverflow.com/questions/64115628/get-starlette-request-body-in-the-middleware-context
154
+ # Mock function that returns a cached copy of the request
155
+ # so that anyone can ask for the body aftewards from the request object
156
+
157
+ receive_cached_ = await receive()
158
+
159
+ async def receive_cached():
160
+ return receive_cached_
161
+
162
+ request = Request(scope, receive=receive_cached)
147
163
 
148
164
  gae_request_context.set({
149
165
  'trace': request.headers.get('X-Cloud-Trace-Context'),
@@ -151,9 +167,12 @@ class FastAPIGAELoggingMiddleware:
151
167
  'max_log_level': logging.NOTSET
152
168
  })
153
169
 
170
+ # start with the default response in case of generic error
154
171
  response = Response(status_code=500)
155
172
 
156
- async def send_wrapper(message: Dict[str, Any]) -> None:
173
+ # intercept the sent message to get the response status and body
174
+ # for later use
175
+ async def send_spoof_wrapper(message: Dict[str, Any]) -> None:
157
176
  if message["type"] == "http.response.start":
158
177
  response.status_code = message["status"]
159
178
  elif message["type"] == "http.response.body":
@@ -161,7 +180,7 @@ class FastAPIGAELoggingMiddleware:
161
180
  await send(message)
162
181
 
163
182
  try:
164
- await self.app(scope, receive, send_wrapper)
183
+ await self.app(scope, receive_cached, send_spoof_wrapper)
165
184
  except Exception as e:
166
185
  if not isinstance(e, HTTPException):
167
186
  logging.exception(e)
@@ -0,0 +1,162 @@
1
+ Metadata-Version: 2.1
2
+ Name: fastapi-gae-logging
3
+ Version: 0.0.3
4
+ Summary: Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine. Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
5
+ Home-page: https://github.com/chrisK824/fastapi-gae-logging
6
+ Author: Chris Karvouniaris
7
+ Author-email: christos.karvouniaris247@gmail.com
8
+ License: MIT
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.7
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: fastapi>=0.112.2
15
+ Requires-Dist: google-cloud-logging>=3.11.2
16
+
17
+ # fastapi-gae-logging
18
+ Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine to ease out logs analysis and monitoring through Google Cloud Log Explorer.
19
+
20
+ ## What problem does this package solve? Why do we need this?
21
+ When deploying FastAPI applications on Google App Engine, I encountered several challenges with logging and observability even when using the official package for this purpose, `google-cloud-logging`.
22
+
23
+ - **Scattered Logs**: Logs generated during a single request lifecycle were scattered across different log entries, making it difficult to trace the entire flow of a request, especially when troubleshooting issues.
24
+
25
+ - **Log Severity Mismatch**: The severity level of logs was not properly propagated throughout the request lifecycle. This meant that if an error occurred at any point in the request, the earlier logs did not reflect the severity of the final outcome, making it harder to identify problematic requests.
26
+
27
+ - **Payload Logging Issues**: Capturing and logging request payloads was cumbersome, requiring extra logging in the handlers and extra deployments. This led to incomplete logs, making it harder to reproduce issues or analyze request content.
28
+
29
+ - **Inconsistent Log Structures**: The default logging setup lacked a consistent structure, which made it challenging to filter, search, and analyze logs in the Google Cloud Log Explorer.
30
+
31
+ ## So what does it do?
32
+ The `fastapi-gae-logging` module addresses these problems by:
33
+
34
+ - **Grouping Logs by Request**: All logs generated during a request's lifecycle are grouped together, allowing for a complete view of the request flow in the Google Cloud Log Explorer. This makes it much easier to trace and troubleshoot issues.
35
+
36
+ - **Log Level Propagation**: The maximum log level observed during a request's lifecycle is propagated, ensuring that logs associated with a failed request reflect the appropriate severity. This improves the accuracy and utility of log searches based on severity.
37
+
38
+ - **Structured Payload Logging**: Request payloads are captured and logged in a structured format, even for non-dictionary JSON payloads. This ensures that all relevant request data is available for analysis, improving the ability to diagnose issues.
39
+
40
+
41
+ ## Install
42
+ `pip install fastapi-gae-logging`
43
+
44
+ ## Features:
45
+
46
+ - **Request Logs Grouping**: Groups logs from the same request lifecycle to simplify log analysis using Google Cloud Log Explorer. The logger name for grouping can be customized and defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
47
+ - **Request Maximum Log Level Propagation**: Propagates the maximum log level throughout the request lifecycle, making it easier to search logs based on the severity of an issue.
48
+ - **Request Payload Logging**:
49
+ - Logs the payload of a request as part of the parent log in request-grouped logs.
50
+ - Assumes the payload is a valid JSON - basically what Python's `json.loads()` can parse - otherwise, it is discarded.
51
+ - Logs the payload as a dictionary using the `google-cloud-logging`'s `log_struct` method. If the payload is not a dictionary, then it is wrapped in a dummy keyword to construct one. The dummy keyword follows the format: `<original_type_of_payload>__payload_wrapper`. For example, if the payload is a list, which still a valid JSON which Python's `json.loads()` will parse just fine, then the payload list will be wrapped in a virtual dictionary under the key `list_payload_wrapper`. The dictionary is logged alongside the parent log in request-grouped logs and ends up in the `jsonPayload` field.
52
+
53
+
54
+ ## API
55
+
56
+ - `FastAPIGAELoggingHandler` class: A custom Cloud Logging Handler for FastAPI applications to use with the official google-cloud-logging library.
57
+
58
+
59
+ ## Example of usage
60
+
61
+ ```python
62
+ from fastapi import FastAPI
63
+ from fastapi.responses import JSONResponse
64
+ from fastapi.exceptions import HTTPException
65
+ import logging
66
+ import os
67
+
68
+ app = FastAPI()
69
+
70
+ if os.getenv('GAE_ENV', '').startswith('standard'):
71
+ import google.cloud.logging
72
+ from google.cloud.logging_v2.handlers import setup_logging
73
+ from fastapi_gae_logging import FastAPIGAELoggingHandler
74
+
75
+ client = google.cloud.logging.Client()
76
+ gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
77
+ setup_logging(handler=gae_log_handler)
78
+
79
+ logging.getLogger().setLevel(logging.DEBUG)
80
+
81
+ @app.get("/info")
82
+ def info():
83
+ logging.debug("this is a debug")
84
+ logging.info("this is an info")
85
+ return JSONResponse(
86
+ content={"message": "info"}
87
+ )
88
+
89
+ @app.get("/warning")
90
+ async def warning():
91
+ logging.debug("this is a debug")
92
+ logging.info("this is an info")
93
+ logging.warning("this is a warning")
94
+ return JSONResponse(
95
+ content={"message": "warning"}
96
+ )
97
+
98
+ @app.get("/error")
99
+ def error():
100
+ logging.debug("this is a debug")
101
+ logging.info("this is an info")
102
+ logging.warning("this is a warning")
103
+ logging.error("this is an error")
104
+ return JSONResponse(
105
+ content={"message": "error"}
106
+ )
107
+
108
+ @app.get("/exception")
109
+ def exception():
110
+ logging.debug("this is a debug")
111
+ logging.info("this is an info")
112
+ logging.error("this is an error")
113
+ raise ValueError("This is a value error")
114
+
115
+ @app.get("/http_exception")
116
+ def http_exception():
117
+ logging.debug("this is a debug")
118
+ logging.info("this is an info")
119
+ raise HTTPException(
120
+ status_code=404,
121
+ detail={"error": "Resource not found"}
122
+ )
123
+
124
+ @app.post("/post_payload")
125
+ def post_payload(payload: Any = Body(None)):
126
+ logging.debug("this is an debug")
127
+ logging.info(payload)
128
+ return JSONResponse(content={"mirror_response": payload}, status_code=200)
129
+
130
+ ```
131
+
132
+ ## How it looks in Google Cloud Log Explorer
133
+
134
+ ### Logger selection
135
+ ![alt text](<logger_selection.jpg>)
136
+
137
+ ### Groupped logs with propagated log severity to the parent log
138
+
139
+ ![alt text](<groupped_logs.jpg>)
140
+
141
+ ### Grouped logs in request with payload
142
+ ![alt text](<request_with_payload.jpg>)
143
+
144
+ ## Dependencies
145
+ This tool is built upon the following packages:
146
+
147
+ - `fastapi`: A modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints.
148
+ - `google-cloud-logging`: Google Cloud Logging API client library for logging and managing logs in Google Cloud Platform.
149
+
150
+
151
+ ## Implementation Concept
152
+ - **Middleware Integration**: A custom middleware integrates into FastAPI to intercept requests and log data after processing.The custom middleware is added to the FastAPI application during the initialization of the FastAPIGAELoggingHandler.
153
+ - **Context Management**: Uses context variables to manage request-specific data and metadata such as request payload, Google Cloud trace ID, start time of the incoming request and the maximum log level observed during the request lifecycle.
154
+ - **Log Interception**: A logging filter intercepts log records, injecting trace information and adjusting the maximum log level based on observed log severity.
155
+ - **Cloud Logging**: Utilizes Google Cloud Logging to group logs by request and propagate the maximum log level, enhancing observability and troubleshooting.
156
+ - **Structured Logging**: Parent log of the request-response lifecycle is structured and sent to Google Cloud Logging with additional context, such as the request method, URL, and user agent after the request has been processed and served.
157
+
158
+
159
+ ## Roadmap
160
+ - Allow for opting out of the dictionary enforcement of logged payload.
161
+ - Allow for opting out of request payload logging at all.
162
+ - Explore other fields in Google Cloud Logs. Investigate and consider utilizing additional fields available in Google Cloud Logs that may allow for more goodies.
@@ -0,0 +1,2 @@
1
+ fastapi>=0.112.2
2
+ google-cloud-logging>=3.11.2
@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
2
2
 
3
3
  setup(
4
4
  name="fastapi-gae-logging",
5
- version="0.0.1",
5
+ version="0.0.3",
6
6
  description="Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine. \
7
7
  Groups logs coming from the same request lifecycle and propagates the maximum log level \
8
8
  throughout the request lifecycle using middleware and context management.",
@@ -14,7 +14,7 @@ setup(
14
14
  packages=find_packages(),
15
15
  install_requires=[
16
16
  "fastapi>=0.112.2",
17
- "google-cloud-logging>=3.10.0",
17
+ "google-cloud-logging>=3.11.2",
18
18
  ],
19
19
  python_requires=">=3.7",
20
20
  license="MIT",
@@ -1,112 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: fastapi-gae-logging
3
- Version: 0.0.1
4
- Summary: Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine. Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
5
- Home-page: https://github.com/chrisK824/fastapi-gae-logging
6
- Author: Chris Karvouniaris
7
- Author-email: christos.karvouniaris247@gmail.com
8
- License: MIT
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.7
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Requires-Dist: fastapi>=0.112.2
15
- Requires-Dist: google-cloud-logging>=3.10.0
16
-
17
- # fastapi-gae-logging
18
- Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine.
19
- Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
20
-
21
- ## Install
22
- `pip install fastapi-gae-logging`
23
-
24
- ## Features:
25
-
26
- - **Request Logs Grouping**: Groups logs coming from the same request lifecycle to ease out log analysis using the Google Cloud Log Explorer. Grouping logger name can be customised and it defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
27
- - **Request Maximum Log Level propagation**: Propagates the maximum log level throughout the request lifecycle to ease out log searching based on severity of an issue.
28
-
29
- ## API
30
- - Custom Cloud Logging Handler to use with official library `google-cloud-logging`: `FastAPIGAELoggingHandler`
31
-
32
-
33
- ## Example
34
-
35
- ```python
36
- from fastapi import FastAPI
37
- from fastapi.responses import JSONResponse
38
- from fastapi.exceptions import HTTPException
39
- import logging
40
- import os
41
-
42
-
43
- app = FastAPI()
44
-
45
- if os.getenv('GAE_ENV', '').startswith('standard'):
46
- import google.cloud.logging
47
- from google.cloud.logging_v2.handlers import setup_logging
48
- from fastapi_gae_logging import FastAPIGAELoggingHandler
49
-
50
- client = google.cloud.logging.Client()
51
- gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
52
- setup_logging(handler=gae_log_handler)
53
-
54
-
55
- logging.getLogger().setLevel(logging.DEBUG)
56
-
57
-
58
- @app.get("/info")
59
- def info():
60
- logging.debug("this is a debug")
61
- logging.info("this is an info")
62
- return JSONResponse(
63
- content={
64
- "message": "info"
65
- }
66
- )
67
-
68
-
69
- @app.get("/warning")
70
- async def warning():
71
- logging.debug("this is a debug")
72
- logging.info("this is an info")
73
- logging.warning("this is a warning")
74
- return JSONResponse(
75
- content={
76
- "message": "warning"
77
- }
78
- )
79
-
80
-
81
- @app.get("/error")
82
- def error():
83
- logging.debug("this is a debug")
84
- logging.info("this is an info")
85
- logging.warning("this is a warning")
86
- logging.error("this is an error")
87
- return JSONResponse(
88
- content={
89
- "message": "error"
90
- }
91
- )
92
-
93
-
94
- @app.get("/exception")
95
- def exception():
96
- logging.debug("this is a debug")
97
- logging.info("this is an info")
98
- logging.error("this is an error")
99
- raise ValueError("This is a value error")
100
-
101
-
102
- @app.get("/http_exception")
103
- def http_exception():
104
- logging.debug("this is an debug")
105
- logging.info("this is an info")
106
- raise HTTPException(
107
- status_code=404,
108
- detail={
109
- "error": "Resource not found"
110
- }
111
- )
112
- ```
@@ -1,96 +0,0 @@
1
- # fastapi-gae-logging
2
- Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine.
3
- Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
4
-
5
- ## Install
6
- `pip install fastapi-gae-logging`
7
-
8
- ## Features:
9
-
10
- - **Request Logs Grouping**: Groups logs coming from the same request lifecycle to ease out log analysis using the Google Cloud Log Explorer. Grouping logger name can be customised and it defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
11
- - **Request Maximum Log Level propagation**: Propagates the maximum log level throughout the request lifecycle to ease out log searching based on severity of an issue.
12
-
13
- ## API
14
- - Custom Cloud Logging Handler to use with official library `google-cloud-logging`: `FastAPIGAELoggingHandler`
15
-
16
-
17
- ## Example
18
-
19
- ```python
20
- from fastapi import FastAPI
21
- from fastapi.responses import JSONResponse
22
- from fastapi.exceptions import HTTPException
23
- import logging
24
- import os
25
-
26
-
27
- app = FastAPI()
28
-
29
- if os.getenv('GAE_ENV', '').startswith('standard'):
30
- import google.cloud.logging
31
- from google.cloud.logging_v2.handlers import setup_logging
32
- from fastapi_gae_logging import FastAPIGAELoggingHandler
33
-
34
- client = google.cloud.logging.Client()
35
- gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
36
- setup_logging(handler=gae_log_handler)
37
-
38
-
39
- logging.getLogger().setLevel(logging.DEBUG)
40
-
41
-
42
- @app.get("/info")
43
- def info():
44
- logging.debug("this is a debug")
45
- logging.info("this is an info")
46
- return JSONResponse(
47
- content={
48
- "message": "info"
49
- }
50
- )
51
-
52
-
53
- @app.get("/warning")
54
- async def warning():
55
- logging.debug("this is a debug")
56
- logging.info("this is an info")
57
- logging.warning("this is a warning")
58
- return JSONResponse(
59
- content={
60
- "message": "warning"
61
- }
62
- )
63
-
64
-
65
- @app.get("/error")
66
- def error():
67
- logging.debug("this is a debug")
68
- logging.info("this is an info")
69
- logging.warning("this is a warning")
70
- logging.error("this is an error")
71
- return JSONResponse(
72
- content={
73
- "message": "error"
74
- }
75
- )
76
-
77
-
78
- @app.get("/exception")
79
- def exception():
80
- logging.debug("this is a debug")
81
- logging.info("this is an info")
82
- logging.error("this is an error")
83
- raise ValueError("This is a value error")
84
-
85
-
86
- @app.get("/http_exception")
87
- def http_exception():
88
- logging.debug("this is an debug")
89
- logging.info("this is an info")
90
- raise HTTPException(
91
- status_code=404,
92
- detail={
93
- "error": "Resource not found"
94
- }
95
- )
96
- ```
@@ -1,112 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: fastapi-gae-logging
3
- Version: 0.0.1
4
- Summary: Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine. Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
5
- Home-page: https://github.com/chrisK824/fastapi-gae-logging
6
- Author: Chris Karvouniaris
7
- Author-email: christos.karvouniaris247@gmail.com
8
- License: MIT
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.7
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Requires-Dist: fastapi>=0.112.2
15
- Requires-Dist: google-cloud-logging>=3.10.0
16
-
17
- # fastapi-gae-logging
18
- Custom Cloud Logging handler for FastAPI applications deployed in Google App Engine.
19
- Groups logs coming from the same request lifecycle and propagates the maximum log level throughout the request lifecycle using middleware and context management.
20
-
21
- ## Install
22
- `pip install fastapi-gae-logging`
23
-
24
- ## Features:
25
-
26
- - **Request Logs Grouping**: Groups logs coming from the same request lifecycle to ease out log analysis using the Google Cloud Log Explorer. Grouping logger name can be customised and it defaults to the Google Cloud Project ID with '-request-logger' as a suffix.
27
- - **Request Maximum Log Level propagation**: Propagates the maximum log level throughout the request lifecycle to ease out log searching based on severity of an issue.
28
-
29
- ## API
30
- - Custom Cloud Logging Handler to use with official library `google-cloud-logging`: `FastAPIGAELoggingHandler`
31
-
32
-
33
- ## Example
34
-
35
- ```python
36
- from fastapi import FastAPI
37
- from fastapi.responses import JSONResponse
38
- from fastapi.exceptions import HTTPException
39
- import logging
40
- import os
41
-
42
-
43
- app = FastAPI()
44
-
45
- if os.getenv('GAE_ENV', '').startswith('standard'):
46
- import google.cloud.logging
47
- from google.cloud.logging_v2.handlers import setup_logging
48
- from fastapi_gae_logging import FastAPIGAELoggingHandler
49
-
50
- client = google.cloud.logging.Client()
51
- gae_log_handler = FastAPIGAELoggingHandler(app=app, client=client)
52
- setup_logging(handler=gae_log_handler)
53
-
54
-
55
- logging.getLogger().setLevel(logging.DEBUG)
56
-
57
-
58
- @app.get("/info")
59
- def info():
60
- logging.debug("this is a debug")
61
- logging.info("this is an info")
62
- return JSONResponse(
63
- content={
64
- "message": "info"
65
- }
66
- )
67
-
68
-
69
- @app.get("/warning")
70
- async def warning():
71
- logging.debug("this is a debug")
72
- logging.info("this is an info")
73
- logging.warning("this is a warning")
74
- return JSONResponse(
75
- content={
76
- "message": "warning"
77
- }
78
- )
79
-
80
-
81
- @app.get("/error")
82
- def error():
83
- logging.debug("this is a debug")
84
- logging.info("this is an info")
85
- logging.warning("this is a warning")
86
- logging.error("this is an error")
87
- return JSONResponse(
88
- content={
89
- "message": "error"
90
- }
91
- )
92
-
93
-
94
- @app.get("/exception")
95
- def exception():
96
- logging.debug("this is a debug")
97
- logging.info("this is an info")
98
- logging.error("this is an error")
99
- raise ValueError("This is a value error")
100
-
101
-
102
- @app.get("/http_exception")
103
- def http_exception():
104
- logging.debug("this is an debug")
105
- logging.info("this is an info")
106
- raise HTTPException(
107
- status_code=404,
108
- detail={
109
- "error": "Resource not found"
110
- }
111
- )
112
- ```
@@ -1,2 +0,0 @@
1
- fastapi>=0.112.2
2
- google-cloud-logging>=3.10.0