@flying-pillow/manager 0.1.0-beta.10

package/dist/assets/application/observability/README.md

@@ -0,0 +1,130 @@
+# Flying-Pillow Observability Stack
+
+This document outlines the architecture and configuration of the observability stack for the Flying-Pillow monorepo. The stack is designed to provide comprehensive logging, tracing, and metrics for all services, both TypeScript and Python-based.
+
+## 1. Core Philosophy
+
+The observability stack adheres to the following principles:
+
+- **Centralized Collection:** All telemetry data (Logs, Traces, Metrics) is sent to a central OpenTelemetry (OTel) Collector.
+- **Standardized Format:** All services emit telemetry in the OpenTelemetry Protocol (OTLP) format.
+- **Schema-Driven:** Telemetry data structures are defined in Zod (TypeScript) and Pydantic (Python) schemas, ensuring cross-language consistency. Attribute naming follows the `snake_case` convention to align with OpenTelemetry Semantic Conventions.
+- **Correlation:** Logs are automatically correlated with traces via `trace_id` and `span_id` for seamless debugging across services.
+
+## 2. Architecture Overview
+
+The stack consists of application services, a central OTel Collector, and dedicated backend storage systems for each telemetry signal. All components run in a shared Docker network.
+
+```mermaid
+graph TD
+    subgraph "Application Services"
+        A[app SvelteKit] --> C
+        W[wss Node.js] --> C
+        H[hub FastAPI] --> C
+        P[processor Python/Celery] --> C
+    end
+
+    subgraph "Observability Pipeline"
+        C(OTel Collector)
+        C -- Traces --> J[Jaeger]
+        C -- Metrics --> PR[Prometheus]
+        C -- Logs   --> L[Loki]
+    end
+
+    subgraph "Visualization"
+        G[Grafana]
+        G --> J
+        G --> PR
+        G --> L
+    end
+
+    style C fill:#f9f,stroke:#333,stroke-width:2px
+```
+
+## 3. Data Flow and Configuration
+
+### 3.1. Tracing
+
+- **SDKs:** Services are instrumented using the appropriate OpenTelemetry SDK (`@opentelemetry/sdk-node` for TS, `opentelemetry-sdk` for Python).
+- **Protocol:** OTLP/HTTP.
+- **Destination:** `otel-collector:4318/v1/traces`.
+- **Collector Pipeline:** Receives traces, batches them, and exports them to Jaeger.
+- **Backend:** Jaeger (in-memory `all-in-one` image for development).
+- **Visualization:** Grafana (Jaeger data source).
+
+### 3.2. Metrics
+
+- **SDKs:** Services use the OTel SDKs to collect runtime and custom metrics.
+- **Protocol:** OTLP/HTTP.
+- **Destination:** `otel-collector:4318/v1/metrics`.
+- **Collector Pipeline:** Receives metrics, batches them, and exports them to a Prometheus-compatible endpoint.
+- **Backend:** Prometheus (with persistent volume storage).
+  -. **Visualization:** Grafana (Prometheus data source).
+
+### 3.3. Logging
+
+The logging pipeline is the most sophisticated, designed to produce structured, labeled, and correlated logs.
+
+#### Data Structure
+
+Log records are structured according to schemas defined in `apps/common` and `services/common`. The key components are:
+
+- **Trace Context:** Ambient context for the request (`trace_id`, `organizationId`, `userId`, `role`). Injected automatically by the logger.
+- **Log Metadata:** Developer-provided context (`message`, `function`, `args`, `data`, `error_data`).
+- **Convention:** The public logger API accepts a developer-friendly format, which is transformed internally into a serializable, schema-compliant object before being sent to the pipeline.
+
+#### Detailed Data Flow
+
+```mermaid
+sequenceDiagram
+    participant App as Application Code
+    participant Logger as Type-Safe Logger
+    participant Winston as Winston.js
+    participant OTel as OTel Collector
+    participant Loki
+    participant Grafana
+
+    App->>Logger: logger.error("Message", {error: new Error(), ...});
+    Logger->>Logger: Transforms live Error into serializable `error_data` object.
+    Logger->>Winston: .log("Message", {..., error_data: {...}});
+    Winston->>Winston: Injects Context (trace_id, org_id, etc.).
+    Winston->>OTel: Sends final structured log object via OTLP.
+
+    OTel->>OTel: **transform processor** copies `organizationId`, `role`, etc. to resource attributes.
+    OTel->>OTel: **resource processor** adds `loki.resource.labels` hint.
+    OTel->>Loki: Exports log, creating indexed labels from hints.
+
+    Grafana->>Loki: Queries with `{service_name="app", role="admin"}`.
+    Loki-->>Grafana: Returns matching log streams.
+    Grafana->>Grafana: **| json** parser processes log body.
+    Grafana->>Grafana: **| json attributes="attributes"** parses nested attributes.
+    Grafana->>Grafana: **| line_format** creates clean summary view.
+    Grafana->>Grafana: Displays interactive, nested details on expand.
+```
+
+#### Key Configurations
+
+- **`apps/common/src/system/logging/ServerLogger.ts`:** The TypeScript logger implementation. Responsible for transforming the developer-provided log metadata into a serializable format that avoids conflicts with the pipeline.
+- **`observability/otel-collector/config.yml`:** The heart of the pipeline.
+  - Uses a `transform` processor to copy log record attributes (e.g., `organizationId`, `role`) to the resource attributes.
+  - Uses a `resource` processor to add the `loki.resource.labels` hint, which tells the Loki exporter to create indexed labels from all the enriched resource attributes.
+  - This ensures logs are filterable in Loki by `service_name`, `organizationId`, `role`, `event`, etc.
+- **Grafana Dashboard:**
+  - Uses a multi-stage parser query (`... | json | json attributes="attributes"`) to correctly handle the nested log structure produced by the collector.
+  - Uses `line_format` to create a clean, human-readable summary for each log row.
+  - Uses "Derived Fields" in the Loki data source settings to create a clickable link from the `traceid` field to the corresponding trace in Jaeger.
+
+## 4. How to Use
+
+### For Developers
+
+- **Logging:** Import the singleton `logger` instance. Call methods like `logger.info(message, meta)` and `logger.error(message, meta)`. Follow the type-safe `LogMetaType` and `ErrorMetaType` schemas to provide rich context.
+- **Context:** The `ServerContext` is automatically propagated and its data is injected into all logs. You do not need to manually add `trace_id` or `organizationId` to log calls.
+
+### For Operations / Debugging
+
+- **Grafana:** The primary interface for visualization.
+  - Use the "Flying-Pillow Application Logs" dashboard.
+  - Use the dropdowns to filter logs by `service_name`, `organizationId`, `role`, etc.
+  - Expand log rows to see full, interactive context, including stack traces and function arguments.
+  - Click the link on the `traceid` field to pivot directly to the full distributed trace in Jaeger.