📡 Logging & Telemetry Architecture

Our logging pipeline follows the Twelve-Factor App (Factor XI: Logs) specification[cite: 2]. Applications do not write logs to PostgreSQL or disk files; they emit structured JSON event streams directly to stdout/stderr[cite: 2].

🗺️ Telemetry Ingestion Pipeline

graph TD
    subgraph Emission Layer
        API[apps/api: Go log/slog] -->|JSON to stdout| DockerSocket[/var/run/docker.sock]
        Worker[apps/worker: Python structlog] -->|JSON to stdout| DockerSocket
        DB[(PostgreSQL Container)] -->|Stdout Stream| DockerSocket
    end

    subgraph Collection Tier
        DockerSocket --> Vector[Vector Shipper Container]
        Vector -->|NDJSON Batches :9428| VLogs[VictoriaLogs Engine]
    end

    subgraph Observability UI
        VLogs --> WebUI[VictoriaLogs VMUI: http://localhost:9428/select/vmui]
    end

1. Canonical Log Event Schema

Both Go (log/slog) and Python (structlog) emit matching JSON fields to ensure unified querying[cite: 2]:

2. Distributed Tracing (trace_id)

1. Client Request ──► Go API (Assigns X-Trace-ID)
                         │
                         ▼
2. API writes event with trace_id to PostgreSQL outbox_events table
                         │
                         ▼
3. Python Worker polls event, binds trace_id to structlog contextvars
                         │
                         ▼
4. Searching trace_id in VictoriaLogs returns the full end-to-end lifecycle

• Go Implementation: apps/api/internal/middleware/logging_middleware.go intercepts requests, extracts/generates trace_id, and attaches it to context.Context[cite: 2].
• Python Implementation: apps/worker/src/main.py extracts trace_id from the outbox JSON payload and binds it via structlog.contextvars.bind_contextvars[cite: 2].

3. Log Levels & Dynamic Configuration