log-collector-async 1.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- log_collector_async-1.1.0/PKG-INFO +804 -0
- log_collector_async-1.1.0/README.md +760 -0
- log_collector_async-1.1.0/log_collector/__init__.py +13 -0
- log_collector_async-1.1.0/log_collector/async_client.py +651 -0
- log_collector_async-1.1.0/log_collector_async.egg-info/PKG-INFO +804 -0
- log_collector_async-1.1.0/log_collector_async.egg-info/SOURCES.txt +13 -0
- log_collector_async-1.1.0/log_collector_async.egg-info/dependency_links.txt +1 -0
- log_collector_async-1.1.0/log_collector_async.egg-info/requires.txt +8 -0
- log_collector_async-1.1.0/log_collector_async.egg-info/top_level.txt +2 -0
- log_collector_async-1.1.0/setup.cfg +4 -0
- log_collector_async-1.1.0/setup.py +53 -0
- log_collector_async-1.1.0/tests/__init__.py +1 -0
- log_collector_async-1.1.0/tests/test_async_client.py +292 -0
- log_collector_async-1.1.0/tests/test_integration.py +372 -0
- log_collector_async-1.1.0/tests/test_performance.py +143 -0
|
@@ -0,0 +1,804 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: log-collector-async
|
|
3
|
+
Version: 1.1.0
|
|
4
|
+
Summary: 비동기 로그 수집 클라이언트
|
|
5
|
+
Home-page: https://github.com/yourusername/log-analysis-system
|
|
6
|
+
Author: Log Analysis System Team
|
|
7
|
+
Author-email: jack1087902@gmail.com
|
|
8
|
+
Project-URL: Bug Tracker, https://github.com/yourusername/log-analysis-system/issues
|
|
9
|
+
Project-URL: Documentation, https://github.com/yourusername/log-analysis-system/blob/main/clients/python/README.md
|
|
10
|
+
Project-URL: Source Code, https://github.com/yourusername/log-analysis-system/tree/main/clients/python
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Topic :: System :: Logging
|
|
22
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
23
|
+
Classifier: Framework :: AsyncIO
|
|
24
|
+
Requires-Python: >=3.8
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
Requires-Dist: aiohttp>=3.8.0
|
|
27
|
+
Requires-Dist: python-dotenv>=0.19.0
|
|
28
|
+
Provides-Extra: dev
|
|
29
|
+
Requires-Dist: pytest>=7.0.0; extra == "dev"
|
|
30
|
+
Requires-Dist: pytest-asyncio>=0.20.0; extra == "dev"
|
|
31
|
+
Requires-Dist: black>=22.0.0; extra == "dev"
|
|
32
|
+
Requires-Dist: flake8>=4.0.0; extra == "dev"
|
|
33
|
+
Dynamic: author
|
|
34
|
+
Dynamic: author-email
|
|
35
|
+
Dynamic: classifier
|
|
36
|
+
Dynamic: description
|
|
37
|
+
Dynamic: description-content-type
|
|
38
|
+
Dynamic: home-page
|
|
39
|
+
Dynamic: project-url
|
|
40
|
+
Dynamic: provides-extra
|
|
41
|
+
Dynamic: requires-dist
|
|
42
|
+
Dynamic: requires-python
|
|
43
|
+
Dynamic: summary
|
|
44
|
+
|
|
45
|
+
# Log Collector - Python Client
|
|
46
|
+
|
|
47
|
+
고성능 비동기 로그 수집 클라이언트 for Python
|
|
48
|
+
|
|
49
|
+
[](https://www.python.org/)
|
|
50
|
+
[](LICENSE)
|
|
51
|
+
|
|
52
|
+
## 📋 Prerequisites
|
|
53
|
+
|
|
54
|
+
Before using this library, ensure you have:
|
|
55
|
+
|
|
56
|
+
- **Python 3.8+** installed
|
|
57
|
+
- **Package manager**: pip
|
|
58
|
+
- **Log server running**: See [Log Save Server Setup](../../services/log-save-server/README.md)
|
|
59
|
+
- **PostgreSQL database**: For log storage (v12+)
|
|
60
|
+
- **Basic async knowledge**: Understanding of threading and queue patterns
|
|
61
|
+
|
|
62
|
+
## 🎯 Why Use This Library?
|
|
63
|
+
|
|
64
|
+
### The Problem
|
|
65
|
+
Traditional logging blocks your application, creating performance bottlenecks:
|
|
66
|
+
- Each log = 1 HTTP request = ~50ms blocked time
|
|
67
|
+
- 100 logs/sec = 5 seconds of blocking per second (impossible!)
|
|
68
|
+
- Application threads wait for network I/O
|
|
69
|
+
- Database connection pool exhaustion
|
|
70
|
+
|
|
71
|
+
### The Solution
|
|
72
|
+
Asynchronous batch logging with zero blocking:
|
|
73
|
+
- ✅ **~0.1ms per log** - App never blocks waiting for network
|
|
74
|
+
- ✅ **Batches 1000 logs** - Single HTTP request instead of 1000
|
|
75
|
+
- ✅ **Background thread** - Separate daemon thread handles transmission
|
|
76
|
+
- ✅ **Auto compression** - gzip reduces bandwidth by ~70%
|
|
77
|
+
- ✅ **Reliable delivery** - Automatic retries with exponential backoff
|
|
78
|
+
- ✅ **Graceful shutdown** - Flushes queue before exit, zero log loss
|
|
79
|
+
|
|
80
|
+
### When to Use This
|
|
81
|
+
- High-traffic applications (>100 requests/sec)
|
|
82
|
+
- Performance-critical paths where blocking is unacceptable
|
|
83
|
+
- Microservices needing centralized structured logging
|
|
84
|
+
- Distributed tracing across services
|
|
85
|
+
- PostgreSQL-based log analysis and querying
|
|
86
|
+
|
|
87
|
+
### When NOT to Use This
|
|
88
|
+
- Low-traffic apps (<10 req/sec) - simple file logging is fine
|
|
89
|
+
- Quick debugging sessions - use print() for speed
|
|
90
|
+
- Need real-time log streaming - use dedicated streaming solutions
|
|
91
|
+
- Cannot run log server infrastructure - use cloud logging services
|
|
92
|
+
|
|
93
|
+
## 🚀 Quick Start (30 seconds)
|
|
94
|
+
|
|
95
|
+
### Step 1: Install
|
|
96
|
+
```bash
|
|
97
|
+
pip install log-collector
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
### Step 2: Use in your app
|
|
101
|
+
```python
|
|
102
|
+
from log_collector import AsyncLogClient
|
|
103
|
+
|
|
104
|
+
# Initialize logger
|
|
105
|
+
logger = AsyncLogClient("http://localhost:8000")
|
|
106
|
+
|
|
107
|
+
# Send logs - non-blocking, ~0.1ms
|
|
108
|
+
logger.info("Hello world!", user_id="123", action="test")
|
|
109
|
+
logger.warn("High memory usage", memory_mb=512)
|
|
110
|
+
logger.error("Database error", error="connection timeout")
|
|
111
|
+
|
|
112
|
+
# Logs are batched and sent automatically every 1 second or 1000 logs
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Step 3: Check logs in database
|
|
116
|
+
```bash
|
|
117
|
+
psql -h localhost -U postgres -d logs_db \
|
|
118
|
+
-c "SELECT * FROM logs ORDER BY created_at DESC LIMIT 5;"
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
**Want more details?** See [Framework Integration](#-feature-2-http-컨텍스트-자동-수집) below.
|
|
122
|
+
|
|
123
|
+
**Want a working example?** Check out [Demo Applications](#-live-demo).
|
|
124
|
+
|
|
125
|
+
## 📺 Live Demo
|
|
126
|
+
|
|
127
|
+
See working examples with full context tracking:
|
|
128
|
+
|
|
129
|
+
### Python + FastAPI
|
|
130
|
+
- **Location**: [tests/demo-app/backend-python/](../../tests/demo-app/backend-python/)
|
|
131
|
+
- **Features**: Login, CRUD operations, error handling, slow API testing
|
|
132
|
+
- **Run**: `python tests/demo-app/backend-python/server.py`
|
|
133
|
+
|
|
134
|
+
### JavaScript + Express
|
|
135
|
+
- **Location**: [tests/demo-app/backend/](../../tests/demo-app/backend/)
|
|
136
|
+
- **Features**: Same features but with JavaScript
|
|
137
|
+
- **Run**: `node tests/demo-app/backend/server.js`
|
|
138
|
+
|
|
139
|
+
### Frontend Integration
|
|
140
|
+
- **Location**: [tests/demo-app/frontend/](../../tests/demo-app/frontend/)
|
|
141
|
+
- **Features**: Browser-based logging with proper CORS setup
|
|
142
|
+
- **Run**: Open `tests/demo-app/frontend/index-python.html` in browser
|
|
143
|
+
|
|
144
|
+
### Quick Demo Setup
|
|
145
|
+
```bash
|
|
146
|
+
# 1. Start log server (in Docker)
|
|
147
|
+
cd services/log-save-server
|
|
148
|
+
docker-compose up
|
|
149
|
+
|
|
150
|
+
# 2. Start backend (Python or JavaScript)
|
|
151
|
+
cd tests/demo-app/backend-python
|
|
152
|
+
python server.py
|
|
153
|
+
|
|
154
|
+
# 3. Open frontend
|
|
155
|
+
open ../frontend/index-python.html
|
|
156
|
+
|
|
157
|
+
# 4. Interact with app, then check logs
|
|
158
|
+
psql -h localhost -U postgres -d logs_db \
|
|
159
|
+
-c "SELECT service, level, message FROM logs ORDER BY created_at DESC LIMIT 10;"
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
## 🔗 Integration with Full System
|
|
163
|
+
|
|
164
|
+
This client is part of a complete log analysis system. See the [main README](../../README.md) for the full picture.
|
|
165
|
+
|
|
166
|
+
### System Architecture
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
[Your App] → [Python Client] → [Log Save Server] → [PostgreSQL] → [Analysis Server] → [Frontend]
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
### Related Components
|
|
173
|
+
|
|
174
|
+
- **Log Save Server**: Receives logs via HTTP POST ([README](../../services/log-save-server/README.md))
|
|
175
|
+
- **Log Analysis Server**: Text-to-SQL with Claude Sonnet 4.5 ([README](../../services/log-analysis-server/README.md))
|
|
176
|
+
- **Frontend Dashboard**: Svelte 5 web interface ([README](../../frontend/README.md))
|
|
177
|
+
- **JavaScript Client**: JavaScript async log collection ([README](../javascript/README.md))
|
|
178
|
+
- **Database Schema**: PostgreSQL 15 with 21 fields ([schema.sql](../../database/schema.sql))
|
|
179
|
+
|
|
180
|
+
### Quick System Setup
|
|
181
|
+
|
|
182
|
+
For a complete local environment with all components:
|
|
183
|
+
|
|
184
|
+
```bash
|
|
185
|
+
# From root directory
|
|
186
|
+
docker-compose up -d
|
|
187
|
+
# Starts: PostgreSQL, Log Save Server, Log Analysis Server, Frontend
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
See [QUICKSTART.md](../../QUICKSTART.md) for detailed setup.
|
|
191
|
+
|
|
192
|
+
## ✨ 주요 기능
|
|
193
|
+
|
|
194
|
+
- ⚡ **비블로킹 로깅** - 앱 블로킹 < 0.1ms
|
|
195
|
+
- 🚀 **배치 전송** - 1000건 or 1초마다 자동 전송
|
|
196
|
+
- 📦 **자동 압축** - gzip 압축으로 네트워크 비용 절감
|
|
197
|
+
- 🔄 **Graceful Shutdown** - 앱 종료 시 큐 자동 flush
|
|
198
|
+
- 🎯 **자동 필드 수집** - 호출 위치, HTTP 컨텍스트, 사용자 컨텍스트 자동 포함
|
|
199
|
+
- 🌐 **웹 프레임워크 통합** - Flask, FastAPI, Django 지원
|
|
200
|
+
- 🔍 **분산 추적** - trace_id로 마이크로서비스 간 요청 추적
|
|
201
|
+
|
|
202
|
+
## 📦 Installation
|
|
203
|
+
|
|
204
|
+
```bash
|
|
205
|
+
pip install log-collector
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
Development dependencies (for testing):
|
|
209
|
+
```bash
|
|
210
|
+
pip install log-collector[dev]
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
## 💡 Basic Usage
|
|
214
|
+
|
|
215
|
+
### Standard Usage
|
|
216
|
+
|
|
217
|
+
```python
|
|
218
|
+
from log_collector import AsyncLogClient
|
|
219
|
+
|
|
220
|
+
# Initialize with options
|
|
221
|
+
logger = AsyncLogClient(
|
|
222
|
+
server_url="http://localhost:8000",
|
|
223
|
+
service="my-service",
|
|
224
|
+
environment="production"
|
|
225
|
+
)
|
|
226
|
+
|
|
227
|
+
# Send logs (non-blocking, batched automatically)
|
|
228
|
+
logger.info("Application started")
|
|
229
|
+
logger.warn("High memory usage detected", memory_mb=512)
|
|
230
|
+
logger.error("Database connection failed", db_host="localhost")
|
|
231
|
+
|
|
232
|
+
# Automatic graceful shutdown on process exit
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
### Environment Variables
|
|
236
|
+
|
|
237
|
+
`.env` file or environment variables:
|
|
238
|
+
```bash
|
|
239
|
+
LOG_SERVER_URL=http://localhost:8000
|
|
240
|
+
SERVICE_NAME=payment-api
|
|
241
|
+
NODE_ENV=production
|
|
242
|
+
SERVICE_VERSION=v1.2.3
|
|
243
|
+
LOG_TYPE=BACKEND
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
```python
|
|
247
|
+
# Auto-load from environment variables
|
|
248
|
+
logger = AsyncLogClient()
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
## 🎯 Feature 1: 자동 호출 위치 추적
|
|
252
|
+
|
|
253
|
+
**모든 로그에 `function_name`, `file_path` 자동 포함!**
|
|
254
|
+
|
|
255
|
+
```python
|
|
256
|
+
def process_payment(amount):
|
|
257
|
+
logger.info("Processing payment", amount=amount)
|
|
258
|
+
# → function_name="process_payment", file_path="/app/payment.py" 자동 포함!
|
|
259
|
+
|
|
260
|
+
# 비활성화도 가능
|
|
261
|
+
logger.log("INFO", "Manual log", auto_caller=False)
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
**PostgreSQL 분석:**
|
|
265
|
+
```sql
|
|
266
|
+
SELECT function_name, COUNT(*) as call_count
|
|
267
|
+
FROM logs
|
|
268
|
+
WHERE created_at > NOW() - INTERVAL '1 hour'
|
|
269
|
+
GROUP BY function_name
|
|
270
|
+
ORDER BY call_count DESC;
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
## 🌐 Feature 2: HTTP 컨텍스트 자동 수집
|
|
274
|
+
|
|
275
|
+
**웹 프레임워크 환경에서 `path`, `method`, `ip` 자동 포함!**
|
|
276
|
+
|
|
277
|
+
### Flask 통합
|
|
278
|
+
|
|
279
|
+
```python
|
|
280
|
+
import time
|
|
281
|
+
import uuid
|
|
282
|
+
from flask import Flask, request, g
|
|
283
|
+
from log_collector import AsyncLogClient
|
|
284
|
+
|
|
285
|
+
app = Flask(__name__)
|
|
286
|
+
logger = AsyncLogClient("http://localhost:8000")
|
|
287
|
+
|
|
288
|
+
@app.before_request
|
|
289
|
+
def setup_log_context():
|
|
290
|
+
"""요청마다 로그 컨텍스트 생성"""
|
|
291
|
+
# 로그 컨텍스트를 g 객체에 저장
|
|
292
|
+
g.log_context = {
|
|
293
|
+
'path': request.path,
|
|
294
|
+
'method': request.method,
|
|
295
|
+
'ip': request.remote_addr,
|
|
296
|
+
'trace_id': request.headers.get('x-trace-id', str(uuid.uuid4()).replace('-', '')[:32])
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
# 사용자 ID가 있으면 추가
|
|
300
|
+
if request.headers.get('x-user-id'):
|
|
301
|
+
g.log_context['user_id'] = request.headers['x-user-id']
|
|
302
|
+
|
|
303
|
+
# 요청 시작 시간 기록
|
|
304
|
+
g.start_time = time.time()
|
|
305
|
+
|
|
306
|
+
# 요청 시작 로그
|
|
307
|
+
logger.info("Request received", **g.log_context)
|
|
308
|
+
|
|
309
|
+
@app.after_request
|
|
310
|
+
def log_response(response):
|
|
311
|
+
"""응답 완료 시 로그"""
|
|
312
|
+
if hasattr(g, 'log_context') and hasattr(g, 'start_time'):
|
|
313
|
+
duration_ms = int((time.time() - g.start_time) * 1000)
|
|
314
|
+
logger.info("Request completed",
|
|
315
|
+
status_code=response.status_code,
|
|
316
|
+
duration_ms=duration_ms,
|
|
317
|
+
**g.log_context)
|
|
318
|
+
return response
|
|
319
|
+
|
|
320
|
+
@app.route('/api/users/<user_id>')
|
|
321
|
+
def get_user(user_id):
|
|
322
|
+
# 라우트 핸들러에서 컨텍스트를 메타데이터로 전달
|
|
323
|
+
logger.info(f"Fetching user {user_id}",
|
|
324
|
+
user_id_param=user_id,
|
|
325
|
+
**g.log_context)
|
|
326
|
+
# → path, method, ip, trace_id 모두 자동 포함!
|
|
327
|
+
return {"user_id": user_id}
|
|
328
|
+
|
|
329
|
+
@app.route('/api/todos', methods=['POST'])
|
|
330
|
+
def create_todo():
|
|
331
|
+
logger.info("Creating todo",
|
|
332
|
+
todo_text=request.json.get('text'),
|
|
333
|
+
**g.log_context)
|
|
334
|
+
# ... handle todo creation
|
|
335
|
+
return {"success": True}
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
### FastAPI 통합
|
|
339
|
+
|
|
340
|
+
```python
|
|
341
|
+
import time
|
|
342
|
+
import uuid
|
|
343
|
+
from fastapi import FastAPI, Request
|
|
344
|
+
from log_collector import AsyncLogClient
|
|
345
|
+
|
|
346
|
+
app = FastAPI()
|
|
347
|
+
logger = AsyncLogClient("http://localhost:8000")
|
|
348
|
+
|
|
349
|
+
@app.middleware("http")
|
|
350
|
+
async def log_context_middleware(request: Request, call_next):
|
|
351
|
+
"""HTTP 컨텍스트 미들웨어"""
|
|
352
|
+
# 요청 시작 시간
|
|
353
|
+
start_time = time.time()
|
|
354
|
+
|
|
355
|
+
# trace_id 생성
|
|
356
|
+
trace_id = request.headers.get("x-trace-id", str(uuid.uuid4()).replace("-", "")[:32])
|
|
357
|
+
|
|
358
|
+
# HTTP 컨텍스트
|
|
359
|
+
log_context = {
|
|
360
|
+
"path": request.url.path,
|
|
361
|
+
"method": request.method,
|
|
362
|
+
"ip": request.client.host if request.client else None,
|
|
363
|
+
"trace_id": trace_id,
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
# 사용자 컨텍스트 추가
|
|
367
|
+
user_id = request.headers.get("x-user-id")
|
|
368
|
+
if user_id:
|
|
369
|
+
log_context["user_id"] = user_id
|
|
370
|
+
|
|
371
|
+
# 요청 컨텍스트를 request.state에 저장
|
|
372
|
+
request.state.log_context = log_context
|
|
373
|
+
request.state.start_time = start_time
|
|
374
|
+
|
|
375
|
+
logger.info("Request received", **log_context)
|
|
376
|
+
|
|
377
|
+
# 요청 처리
|
|
378
|
+
response = await call_next(request)
|
|
379
|
+
|
|
380
|
+
# 응답 완료
|
|
381
|
+
duration_ms = int((time.time() - start_time) * 1000)
|
|
382
|
+
logger.info("Request completed",
|
|
383
|
+
status_code=response.status_code,
|
|
384
|
+
duration_ms=duration_ms,
|
|
385
|
+
**log_context)
|
|
386
|
+
|
|
387
|
+
return response
|
|
388
|
+
|
|
389
|
+
@app.get("/api/users/{user_id}")
|
|
390
|
+
async def get_user(request: Request, user_id: int):
|
|
391
|
+
# 라우트 핸들러에서 컨텍스트를 메타데이터로 전달
|
|
392
|
+
log_ctx = request.state.log_context
|
|
393
|
+
logger.info(f"Fetching user {user_id}",
|
|
394
|
+
user_id_param=user_id,
|
|
395
|
+
**log_ctx)
|
|
396
|
+
# → path, method, ip, trace_id 모두 자동 포함!
|
|
397
|
+
return {"user_id": user_id}
|
|
398
|
+
|
|
399
|
+
@app.post("/api/todos")
|
|
400
|
+
async def create_todo(request: Request, body: dict):
|
|
401
|
+
log_ctx = request.state.log_context
|
|
402
|
+
logger.info("Creating todo",
|
|
403
|
+
todo_text=body.get('text'),
|
|
404
|
+
**log_ctx)
|
|
405
|
+
# ... handle todo creation
|
|
406
|
+
return {"success": True}
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
## 👤 Feature 3: 사용자 컨텍스트 관리
|
|
410
|
+
|
|
411
|
+
**`user_id`, `trace_id`, `session_id` 등을 모든 로그에 자동 포함!**
|
|
412
|
+
|
|
413
|
+
### Context Manager 방식 (권장)
|
|
414
|
+
|
|
415
|
+
```python
|
|
416
|
+
# 특정 블록에만 컨텍스트 적용
|
|
417
|
+
with AsyncLogClient.user_context(
|
|
418
|
+
user_id="user_123",
|
|
419
|
+
trace_id="trace_xyz",
|
|
420
|
+
session_id="sess_abc"
|
|
421
|
+
):
|
|
422
|
+
logger.info("User logged in")
|
|
423
|
+
# → user_id, trace_id, session_id 자동 포함!
|
|
424
|
+
|
|
425
|
+
process_payment()
|
|
426
|
+
logger.info("Payment completed")
|
|
427
|
+
# → 하위 함수에서도 자동으로 컨텍스트 유지!
|
|
428
|
+
|
|
429
|
+
# with 블록 벗어나면 자동 초기화
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
### 중첩 컨텍스트 (자동 병합)
|
|
433
|
+
|
|
434
|
+
```python
|
|
435
|
+
# 외부: tenant_id
|
|
436
|
+
with AsyncLogClient.user_context(tenant_id="tenant_1"):
|
|
437
|
+
logger.info("Tenant operation")
|
|
438
|
+
# → tenant_id="tenant_1"
|
|
439
|
+
|
|
440
|
+
# 내부: user_id 추가
|
|
441
|
+
with AsyncLogClient.user_context(user_id="user_123"):
|
|
442
|
+
logger.info("User operation")
|
|
443
|
+
# → tenant_id="tenant_1", user_id="user_123" 둘 다 포함!
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
### 분산 추적 (Distributed Tracing)
|
|
447
|
+
|
|
448
|
+
```python
|
|
449
|
+
import uuid
|
|
450
|
+
|
|
451
|
+
def handle_request():
|
|
452
|
+
trace_id = str(uuid.uuid4())
|
|
453
|
+
|
|
454
|
+
with AsyncLogClient.user_context(trace_id=trace_id, user_id="user_123"):
|
|
455
|
+
logger.info("Request received")
|
|
456
|
+
call_service_a() # Service A 호출
|
|
457
|
+
call_service_b() # Service B 호출
|
|
458
|
+
logger.info("Request completed")
|
|
459
|
+
# → 모든 로그가 같은 trace_id로 추적 가능!
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
**PostgreSQL 분석:**
|
|
463
|
+
```sql
|
|
464
|
+
-- trace_id로 전체 요청 흐름 추적
|
|
465
|
+
SELECT created_at, service, function_name, message, duration_ms
|
|
466
|
+
FROM logs
|
|
467
|
+
WHERE trace_id = 'your-trace-id'
|
|
468
|
+
ORDER BY created_at;
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
### Set/Clear 방식
|
|
472
|
+
|
|
473
|
+
```python
|
|
474
|
+
# 로그인 시
|
|
475
|
+
AsyncLogClient.set_user_context(
|
|
476
|
+
user_id="user_123",
|
|
477
|
+
session_id="sess_abc"
|
|
478
|
+
)
|
|
479
|
+
|
|
480
|
+
logger.info("User action")
|
|
481
|
+
# → user_id, session_id 자동 포함
|
|
482
|
+
|
|
483
|
+
# 로그아웃 시
|
|
484
|
+
AsyncLogClient.clear_user_context()
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
## 🔧 고급 기능
|
|
488
|
+
|
|
489
|
+
### 타이머 측정
|
|
490
|
+
|
|
491
|
+
```python
|
|
492
|
+
# 수동 타이머
|
|
493
|
+
timer = logger.start_timer()
|
|
494
|
+
result = expensive_operation()
|
|
495
|
+
logger.end_timer(timer, "INFO", "Operation completed")
|
|
496
|
+
# → duration_ms 자동 계산
|
|
497
|
+
|
|
498
|
+
# 함수 래퍼 (동기/비동기 자동 감지)
|
|
499
|
+
result = logger.measure(lambda: expensive_operation())
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
### 에러 추적
|
|
503
|
+
|
|
504
|
+
```python
|
|
505
|
+
try:
|
|
506
|
+
risky_operation()
|
|
507
|
+
except Exception as e:
|
|
508
|
+
logger.error_with_trace("Operation failed", exception=e)
|
|
509
|
+
# → stack_trace, error_type, function_name, file_path 자동 포함!
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
### 수동 Flush
|
|
513
|
+
|
|
514
|
+
```python
|
|
515
|
+
# 중요한 로그를 즉시 전송
|
|
516
|
+
logger.flush()
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
## ⚙️ 설정 옵션
|
|
520
|
+
|
|
521
|
+
```python
|
|
522
|
+
logger = AsyncLogClient(
|
|
523
|
+
server_url="http://localhost:8000",
|
|
524
|
+
service="payment-api",
|
|
525
|
+
environment="production",
|
|
526
|
+
service_version="v1.2.3",
|
|
527
|
+
log_type="BACKEND",
|
|
528
|
+
batch_size=1000, # 배치 크기 (기본: 1000)
|
|
529
|
+
flush_interval=1.0, # Flush 간격 초 (기본: 1.0)
|
|
530
|
+
enable_compression=True # gzip 압축 (기본: True)
|
|
531
|
+
)
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
## 📊 성능
|
|
535
|
+
|
|
536
|
+
- **앱 블로킹**: < 0.1ms per log
|
|
537
|
+
- **처리량**: > 10,000 logs/sec
|
|
538
|
+
- **메모리**: < 10MB (1000건 큐)
|
|
539
|
+
- **압축률**: ~70% (100건 이상 시 자동 압축)
|
|
540
|
+
|
|
541
|
+
## 🧪 테스트
|
|
542
|
+
|
|
543
|
+
```bash
|
|
544
|
+
# 단위 테스트
|
|
545
|
+
pytest tests/
|
|
546
|
+
|
|
547
|
+
# 통합 테스트 (로그 서버 필요)
|
|
548
|
+
pytest tests/test_integration.py
|
|
549
|
+
|
|
550
|
+
# 커버리지
|
|
551
|
+
pytest --cov=log_collector tests/
|
|
552
|
+
```
|
|
553
|
+
|
|
554
|
+
## 📝 로그 레벨
|
|
555
|
+
|
|
556
|
+
```python
|
|
557
|
+
logger.trace("Trace message") # TRACE
|
|
558
|
+
logger.debug("Debug message") # DEBUG
|
|
559
|
+
logger.info("Info message") # INFO
|
|
560
|
+
logger.warn("Warning message") # WARN
|
|
561
|
+
logger.error("Error message") # ERROR
|
|
562
|
+
logger.fatal("Fatal message") # FATAL
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
## 🔍 PostgreSQL 쿼리 예제
|
|
566
|
+
|
|
567
|
+
### 사용자별 로그 조회
|
|
568
|
+
```sql
|
|
569
|
+
SELECT * FROM logs
|
|
570
|
+
WHERE user_id = 'user_123'
|
|
571
|
+
ORDER BY created_at DESC
|
|
572
|
+
LIMIT 100;
|
|
573
|
+
```
|
|
574
|
+
|
|
575
|
+
### 에러 발생률
|
|
576
|
+
```sql
|
|
577
|
+
SELECT
|
|
578
|
+
path,
|
|
579
|
+
method,
|
|
580
|
+
COUNT(*) as total_requests,
|
|
581
|
+
COUNT(CASE WHEN level = 'ERROR' THEN 1 END) as errors,
|
|
582
|
+
ROUND(100.0 * COUNT(CASE WHEN level = 'ERROR' THEN 1 END) / COUNT(*), 2) as error_rate
|
|
583
|
+
FROM logs
|
|
584
|
+
WHERE created_at > NOW() - INTERVAL '1 hour'
|
|
585
|
+
GROUP BY path, method
|
|
586
|
+
ORDER BY error_rate DESC;
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
### 함수별 성능
|
|
590
|
+
```sql
|
|
591
|
+
SELECT
|
|
592
|
+
function_name,
|
|
593
|
+
COUNT(*) as calls,
|
|
594
|
+
AVG(duration_ms) as avg_ms,
|
|
595
|
+
MAX(duration_ms) as max_ms
|
|
596
|
+
FROM logs
|
|
597
|
+
WHERE duration_ms IS NOT NULL
|
|
598
|
+
GROUP BY function_name
|
|
599
|
+
ORDER BY avg_ms DESC;
|
|
600
|
+
```
|
|
601
|
+
|
|
602
|
+
## 🚨 주의사항
|
|
603
|
+
|
|
604
|
+
1. **민감한 정보 포함 금지**
|
|
605
|
+
```python
|
|
606
|
+
# ❌ 절대 안 됨!
|
|
607
|
+
logger.info("Login", password="secret")
|
|
608
|
+
|
|
609
|
+
# ✅ 식별자만 사용
|
|
610
|
+
logger.info("Login successful", user_id="user_123")
|
|
611
|
+
```
|
|
612
|
+
|
|
613
|
+
2. **과도한 로깅 피하기**
|
|
614
|
+
```python
|
|
615
|
+
# ❌ 루프 내부에서 과도한 로깅
|
|
616
|
+
for i in range(10000):
|
|
617
|
+
logger.debug(f"Processing {i}")
|
|
618
|
+
|
|
619
|
+
# ✅ 주요 이벤트만 로깅
|
|
620
|
+
logger.info(f"Batch processing started", count=10000)
|
|
621
|
+
```
|
|
622
|
+
|
|
623
|
+
## 🔧 Troubleshooting
|
|
624
|
+
|
|
625
|
+
### Logs not appearing in database
|
|
626
|
+
|
|
627
|
+
**Symptoms**:
|
|
628
|
+
- `logger.info()` runs without errors
|
|
629
|
+
- No logs visible in PostgreSQL
|
|
630
|
+
- No errors in console
|
|
631
|
+
|
|
632
|
+
**Checklist**:
|
|
633
|
+
1. ✅ **Log server running?**
|
|
634
|
+
```bash
|
|
635
|
+
curl http://localhost:8000/
|
|
636
|
+
# Should return: {"status": "ok"}
|
|
637
|
+
```
|
|
638
|
+
|
|
639
|
+
2. ✅ **PostgreSQL running?**
|
|
640
|
+
```bash
|
|
641
|
+
psql -h localhost -U postgres -d logs_db -c "SELECT 1;"
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
3. ✅ **Schema created?**
|
|
645
|
+
```bash
|
|
646
|
+
psql -h localhost -U postgres -d logs_db -c "\dt"
|
|
647
|
+
# Should show 'logs' table
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
4. ✅ **Batch flushed?**
|
|
651
|
+
- Wait 1 second (default flush interval)
|
|
652
|
+
- OR manually flush: `logger.flush()`
|
|
653
|
+
|
|
654
|
+
5. ✅ **Check server logs**:
|
|
655
|
+
```bash
|
|
656
|
+
cd services/log-save-server
|
|
657
|
+
docker-compose logs -f
|
|
658
|
+
# Look for "Received X logs" messages
|
|
659
|
+
```
|
|
660
|
+
|
|
661
|
+
---
|
|
662
|
+
|
|
663
|
+
### "Connection refused" errors
|
|
664
|
+
|
|
665
|
+
**Symptoms**:
|
|
666
|
+
```
|
|
667
|
+
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
**Cause**: Log server not running
|
|
671
|
+
|
|
672
|
+
**Solution**:
|
|
673
|
+
```bash
|
|
674
|
+
cd services/log-save-server
|
|
675
|
+
docker-compose up -d
|
|
676
|
+
|
|
677
|
+
# Verify it's running
|
|
678
|
+
curl http://localhost:8000/
|
|
679
|
+
```
|
|
680
|
+
|
|
681
|
+
---
|
|
682
|
+
|
|
683
|
+
### High memory usage
|
|
684
|
+
|
|
685
|
+
**Symptoms**:
|
|
686
|
+
- Application memory grows over time
|
|
687
|
+
- Eventually crashes with OOM error
|
|
688
|
+
|
|
689
|
+
**Cause**: Batch size too large or flush interval too long
|
|
690
|
+
|
|
691
|
+
**Solution**: Reduce batching parameters
|
|
692
|
+
```python
|
|
693
|
+
logger = AsyncLogClient(
|
|
694
|
+
"http://localhost:8000",
|
|
695
|
+
batch_size=500, # Reduce from 1000
|
|
696
|
+
flush_interval=0.5 # Reduce from 1.0
|
|
697
|
+
)
|
|
698
|
+
```
|
|
699
|
+
|
|
700
|
+
---
|
|
701
|
+
|
|
702
|
+
### Logs delayed or not sent on app shutdown
|
|
703
|
+
|
|
704
|
+
**Symptoms**:
|
|
705
|
+
- Last few logs before shutdown are missing
|
|
706
|
+
- Queue not flushing properly
|
|
707
|
+
|
|
708
|
+
**Cause**: App exits before background thread flushes
|
|
709
|
+
|
|
710
|
+
**Solution**: Call flush before exit
|
|
711
|
+
```python
|
|
712
|
+
import atexit
|
|
713
|
+
import signal
|
|
714
|
+
|
|
715
|
+
# Auto-flush on normal exit
|
|
716
|
+
atexit.register(logger.flush)
|
|
717
|
+
|
|
718
|
+
# Flush on SIGTERM
|
|
719
|
+
def handle_sigterm(signum, frame):
|
|
720
|
+
logger.flush()
|
|
721
|
+
sys.exit(0)
|
|
722
|
+
|
|
723
|
+
signal.signal(signal.SIGTERM, handle_sigterm)
|
|
724
|
+
|
|
725
|
+
# Or manually before exit
|
|
726
|
+
logger.flush() # Blocks until queue is empty
|
|
727
|
+
```
|
|
728
|
+
|
|
729
|
+
---
|
|
730
|
+
|
|
731
|
+
### Thread daemon warnings on exit
|
|
732
|
+
|
|
733
|
+
**Symptoms**:
|
|
734
|
+
```
|
|
735
|
+
Exception ignored in: <module 'threading' from '/usr/lib/python3.8/threading.py'>
|
|
736
|
+
RuntimeError: can't create new thread at interpreter shutdown
|
|
737
|
+
```
|
|
738
|
+
|
|
739
|
+
**Cause**: Background thread still running during shutdown
|
|
740
|
+
|
|
741
|
+
**Solution**: Call flush to ensure clean shutdown
|
|
742
|
+
```python
|
|
743
|
+
# At the end of your application
|
|
744
|
+
logger.flush()
|
|
745
|
+
```
|
|
746
|
+
|
|
747
|
+
---
|
|
748
|
+
|
|
749
|
+
### UnicodeEncodeError with emojis (Windows)
|
|
750
|
+
|
|
751
|
+
**Symptoms**:
|
|
752
|
+
```
|
|
753
|
+
UnicodeEncodeError: 'cp949' codec can't encode character
|
|
754
|
+
```
|
|
755
|
+
|
|
756
|
+
**Cause**: Windows console encoding issue
|
|
757
|
+
|
|
758
|
+
**Solution**: Set UTF-8 encoding
|
|
759
|
+
```bash
|
|
760
|
+
# Set environment variable before running
|
|
761
|
+
set PYTHONIOENCODING=utf-8
|
|
762
|
+
python your_app.py
|
|
763
|
+
|
|
764
|
+
# Or in code
|
|
765
|
+
import sys
|
|
766
|
+
import io
|
|
767
|
+
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
|
|
768
|
+
```
|
|
769
|
+
|
|
770
|
+
## 📋 Version Compatibility
|
|
771
|
+
|
|
772
|
+
| Component | Minimum Version | Tested Version | Notes |
|
|
773
|
+
|-----------|----------------|----------------|-------|
|
|
774
|
+
| **This Client** | 1.0.0 | 1.0.0 | Current release |
|
|
775
|
+
| **Log Save Server** | 1.0.0 | 1.0.0 | FastAPI 0.104+ |
|
|
776
|
+
| **PostgreSQL** | 12 | 15 | Requires JSONB support |
|
|
777
|
+
| **Log Analysis Server** | 1.0.0 | 1.0.0 | Optional (for Text-to-SQL) |
|
|
778
|
+
| **Python** | 3.8 | 3.11 | Runtime environment |
|
|
779
|
+
|
|
780
|
+
### Breaking Changes
|
|
781
|
+
|
|
782
|
+
- **v1.0.0**: Initial release
|
|
783
|
+
|
|
784
|
+
### Upgrade Guide
|
|
785
|
+
|
|
786
|
+
No upgrades yet. This is the initial release.
|
|
787
|
+
|
|
788
|
+
## 📚 추가 문서
|
|
789
|
+
|
|
790
|
+
- [HTTP-CONTEXT-GUIDE.md](HTTP-CONTEXT-GUIDE.md) - HTTP 컨텍스트 완전 가이드
|
|
791
|
+
- [USER-CONTEXT-GUIDE.md](USER-CONTEXT-GUIDE.md) - 사용자 컨텍스트 완전 가이드
|
|
792
|
+
- [FIELD-AUTO-COLLECTION.md](FIELD-AUTO-COLLECTION.md) - 자동 필드 수집 상세
|
|
793
|
+
|
|
794
|
+
## 🤝 기여
|
|
795
|
+
|
|
796
|
+
기여는 언제나 환영합니다!
|
|
797
|
+
|
|
798
|
+
## 📄 라이선스
|
|
799
|
+
|
|
800
|
+
MIT License - 자유롭게 사용하세요!
|
|
801
|
+
|
|
802
|
+
---
|
|
803
|
+
|
|
804
|
+
**Made with ❤️ by Log Analysis System Team**
|