Skip to content

🔍 Debugging & Dead Letter Queue (DLQ) Guide

Operational runbook for diagnosing runtime issues, querying VictoriaLogs, and recovering failed asynchronous tasks[cite: 2, 3].

1. VictoriaLogs Web UI (VMUI)

Useful LogsQL Queries

# View all captured logs within the selected time window
*

# Filter by a specific container service
service:dues_postgres
service:dues_vector

# View errors across the entire system
level:ERROR

# Trace an entire payment lifecycle across Go API and Python Worker
trace_id:"550e8400-e29b-41d4-a716-446655440000"

Empty VMUI Results?

Vector attaches to /var/run/docker.sock and collects stdout from Docker containers[cite: 2]. If you run make run-api or make run-worker directly on your host terminal, logs print to your local terminal shell rather than Docker[cite: 2].

2. Dead Letter Queue (DLQ) Runbook

When the asynchronous worker encounters errors during receipt generation or e-invoicing, it applies exponential backoff ($5 \times 2^{\text{retry_count}}$ seconds)[cite: 2, 3]. If an event fails $3$ times (max_retries = 3), it transitions to status = 'DEAD_LETTER'[cite: 2, 3].

PENDING (Retry 0) ──► Fail ──► Sleep 5s (Retry 1) ──► Fail ──► Sleep 10s (Retry 2) ──► Fail ──► DEAD_LETTER[cite: 3]

CLI DLQ Operations (dlq_manager.py)

Run these commands from the repository root[cite: 2]:

# 1. List all events currently stuck in DEAD_LETTER status
make dlq-list
# or: python3 apps/worker/src/dlq_manager.py list

# 2. Replay a specific dead-letter event by its UUID
python3 apps/worker/src/dlq_manager.py replay --id <EVENT_UUID>

# 3. Replay ALL dead-letter events back to PENDING queue
make dlq-replay-all
# or: python3 apps/worker/src/dlq_manager.py replay-all

3. Common Troubleshooting Scenarios

Scenario A: Docker Vector Permission Error

If docker logs dues_vector reports Permission denied on /var/run/docker.sock[cite: 2]: 

sudo chmod 666 /var/run/docker.sock
docker compose -f deploy/docker-compose.yml restart vector

Scenario B: Resetting the Database & Purging Volumes

To wipe corrupt test state and rebuild schemas cleanly[cite: 2]: 

docker compose -f deploy/docker-compose.yml down -v
make up