Three pillars of observability: Logs / Metrics / Traces + OpenTelemetry + SLO

puml.online
  2026-07-30 18:20 2026-07-30 18:20 Created   2026-08-14 21:34:29 2026-08-14 21:34:29 Updated    

When production breaks, “just restart” doesn’t scale. Three-pillar observability (Logs + Metrics + Traces) cuts root-cause analysis from hours to minutes. This is the three-pillar architecture, OpenTelemetry unified collection, Prometheus + Grafana + Tempo/Loki combo, SLO/SLI/SLA design, alert grading.

Three pillars overview

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
@startuml
title "Three Pillars of Observability"

package "Sources" {
  component "App" as App
  component "Service A" as SvcA
  component "Service B" as SvcB
  component "DB" as DB
  component "Queue" as Q
}

package "Collection (OpenTelemetry)" {
  component "OTel SDK" as SDK
  component "OTel Collector" as Collector
}

package "Storage" {
  component "Metrics\n(Prometheus / Mimir)" as Metrics
  component "Logs\n(Loki / Elasticsearch)" as Logs
  component "Traces\n(Tempo / Jaeger)" as Traces
}

package "Visualization" {
  component "Grafana" as Grafana
}

package "Alerting" {
  component "AlertManager" as Alert
  component "PagerDuty" as PD
}

App -> SDK : emit logs, metrics, traces
SvcA -> SDK
SvcB -> SDK
DB -> SDK
Q -> SDK

SDK -> Collector : OTLP
Collector -> Metrics : metrics scrape
Collector -> Logs : logs push
Collector -> Traces : traces push

Metrics --> Grafana
Logs --> Grafana
Traces --> Grafana

Metrics --> Alert
Logs --> Alert
Alert -> PD : page oncall

@enduml

Three pillars:

  • Metrics — numeric time series (cpu / memory / QPS / latency)
  • Logs — discrete events (error / warning / info)
  • Traces — request chains (call relationships / duration)

Pillar 1: Metrics (Prometheus)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
@startuml
title "Prometheus Metrics Flow"

participant "App" as App
participant "Prometheus" as Prom
participant "Grafana" as Graf
participant "AlertManager" as AM

note over App
  Metrics expose /metrics endpoint:
  - http_requests_total{method, path, status}
  - http_request_duration_seconds_bucket{...}
  - process_cpu_seconds_total
end note

App -> Prom : ① scrape /metrics (every 15s)
Prom -> Prom : ② store to TSDB
Prom -> Graf : ③ PromQL query
Graf -> Graf : ④ render dashboard

note over AM
  alert rules:
  - http_error_rate > 5% for 5m
  - p99_latency > 1s for 5m
  - cpu_usage > 80% for 10m
end note

Prom -> AM : ⑤ evaluate alert rules
AM -> AM : ⑥ firing → PagerDuty / Slack
AM -> Graf : ⑦ alert annotation

@enduml

Prometheus four metric types:

Type Use Example
Counter only increases http_requests_total
Gauge current value cpu_usage, queue_size
Histogram distribution request_duration_seconds
Summary percentile similar to Histogram, client-side calc

Histogram auto-calculates p50/p95/p99:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# p99 latency
histogram_quantile(0.99,
  sum by (le, path) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# QPS
sum(rate(http_requests_total[5m]))

# error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Pillar 2: Logs (Loki / ELK)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
@startuml
title "Logs Pipeline"

participant "App" as App
participant "Promtail / Fluentd" as Agent
participant "Loki / Elasticsearch" as Storage
participant "Grafana" as Graf

App -> Agent : ① write stdout (JSON)
Agent -> Agent : ② collect / parse / add labels
Agent -> Storage : ③ push (Loki) / index (ES)

note right of Storage
  Loki:
  - doesn't index full text, only labels
  - cheap, easy to scale
  - suited for cloud native
  
  Elasticsearch:
  - full-text index
  - powerful but expensive
  - suited for complex search
end note

Storage -> Graf : ④ LogQL / KQL query
Graf -> Graf : ⑤ render log panels

note over App : structured log
{ "timestamp": "2026-07-30T10:00:00Z",
  "level": "ERROR",
  "service": "user-service",
  "trace_id": "abc123",
  "message": "DB query failed",
  "error": "connection timeout",
  "stack_trace": "..." }

@enduml

Log best practices:

  • JSON structured — no plain text
  • must include trace_id — link to traces
  • must include service / env / version — labels for filtering
  • never log passwords / tokens — filter sensitive fields
  • sampling — high-traffic services sample 1%

Pillar 3: Traces (OpenTelemetry / Jaeger / Tempo)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
@startuml
title "Distributed Trace - Request Flow"

actor "User" as User
participant "API Gateway" as GW
participant "Auth Service" as Auth
participant "Order Service" as Order
participant "Payment Service" as Pay
participant "DB" as DB
participant "OTel Collector" as OTel

User -> GW : ① GET /api/orders

note over OTel : each request generates trace_id

GW -> OTel : ② trace start (span: gw.request)
GW -> Auth : ③ HTTP /verify
Auth -> OTel : ④ span: auth.verify
Auth --> GW : ⑤ OK
GW -> Order : ⑥ HTTP /orders
Order -> OTel : ⑦ span: order.list
Order -> DB : ⑧ SELECT
DB --> Order : ⑨ orders
Order -> Pay : ⑩ HTTP /verify-payment
Pay -> OTel : ⑪ span: pay.verify
Pay --> Order : ⑫ OK
Order --> GW : ⑬ orders
GW --> User : ⑭ 200

note over OTel
  Trace contains all spans:
  - trace_id: abc123
  - each span has parent_id forming tree
  - each span has start_time + duration
end note

OTel -> OTel : ⑮ collect all spans
OTel -> Tempo : ⑯ push trace

@enduml

Trace key concepts:

  • Trace — all spans for one complete request
  • Span — one call (API call / DB query)
  • trace_id — unique identifier across chain
  • parent_span_id — forms call tree
  • baggage — context passed across services (e.g. user_id)

OpenTelemetry unified collection

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@startuml
title "OpenTelemetry Unified Instrumentation"

participant "App Code" as Code
participant "OTel SDK\n(auto-instrumentation)" as SDK
participant "OTel Collector\n(agent / gateway)" as Collector
participant "Backend" as Backend

note over Code
  // business code doesn't need to care about collection
  // OTel SDK auto-intercepts:
  // - HTTP client/server
  // - gRPC
  // - DB drivers (pg/mysql/redis)
  // - Queue (kafka/sqs)
end note

Code -> SDK : ① auto-intercept (express, pg, redis...)
SDK -> SDK : ② generate spans + metrics + logs
SDK -> Collector : ③ OTLP (gRPC/HTTP)

note over Collector
  Collector processing:
  - batch (merge sends)
  - retry (network failure resend)
  - sampling
  - filter (remove sensitive fields)
  - attributes (add tags)
end note

Collector -> Backend : ④ forward to Prometheus / Tempo / Loki

@enduml

OpenTelemetry Collector config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
  attributes:
    actions:
      - key: environment
        value: production
        action: insert
  filter/logs:
    logs:
      exclude:
        matchers:
          - 'severity_text = "DEBUG"'

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [batch, memory_limiter, filter/logs]
      exporters: [loki]

SLO / SLI / SLA

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
@startuml
title "SLO / SLI / SLA Relationship"

rectangle "SLI (Service Level Indicator)" {
  component "actual measurement" as SLI
  note right
    example:
    - request success rate = success / total
    - p99 latency = 99% requests < X ms
  end note
}

rectangle "SLO (Service Level Objective)" {
  component "internal target" as SLO
  note right
    example:
    - availability ≥ 99.9%
    - p99 latency < 500ms
    - error rate < 0.1%
  end note
}

rectangle "SLA (Service Level Agreement)" {
  component "customer commitment" as SLA
  note right
    example:
    - availability ≥ 99.5% (contract)
    - miss → refund / compensation
  end note
}

SLI --> SLO : "measure vs target"
SLO --> SLA : "internal target ≥ customer commitment"

note right of SLA
  usually:
  SLA looser than SLO (buffer)
  SLO stricter than actual (room to upgrade)
end note

@enduml

SLO in practice:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# slos.yaml
apiVersion: sloth.sloth.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: user-service-slo
spec:
  service: user-service
  labels:
    team: platform
  slos:
    - name: availability
      objective: 99.9
      description: "Service availability over 30 days"
      sli:
        events:
          error_query: sum(rate(http_requests_total{service="user-service",status=~"5.."}[5m]))
          total_query: sum(rate(http_requests_total{service="user-service"}[5m]))
    - name: latency
      objective: 95
      description: "95% of requests complete in < 500ms"
      sli:
        events:
          error_query: sum(rate(http_request_duration_seconds_bucket{service="user-service",le="0.5"}[5m]))
          total_query: sum(rate(http_request_duration_seconds_count{service="user-service"}[5m]))

Alert severity levels

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
@startuml
title "Alert Severity Levels"

rectangle "Severity" {
  rectangle "P0 - Critical\n(immediate response)" {
    component "5xx error rate > 10%\nfull site down\nrevenue impact"
  }
  rectangle "P1 - High\n(within 15 min)" {
    component "5xx error rate 5-10%\ncore feature impacted\np99 latency > 2s"
  }
  rectangle "P2 - Medium\n(within 1 hour)" {
    component "5xx error rate 1-5%\nnon-core feature abnormal"
  }
  rectangle "P3 - Low\n(next business day)" {
    component "performance degraded\ndisk usage high"
  }
}

note bottom of "Severity"
  response mode:
  P0 → phone + SMS + Slack @here
  P1 → Slack + PagerDuty
  P2 → Slack
  P3 → Email / Slack async
end note

@enduml

Alert rules avoiding fatigue:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# alertmanager.yml
groups:
- name: critical
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
      > 0.05
    for: 5m
    labels:
      severity: p1
    annotations:
      summary: "Error rate {{ $value | humanizePercentage }}"
      runbook: "https://wiki/runbooks/high-error-rate"

  - alert: HighLatency
    expr: |
      histogram_quantile(0.99,
        sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
      ) > 1
    for: 5m
    labels:
      severity: p2
    annotations:
      summary: "P99 latency {{ $value }}s for {{ $labels.service }}"

Avoid alert fatigue:

  • every alert must have runbook — link to troubleshooting doc
  • don’t silence alert to silence — fix root cause
  • for duration — not instant jitter alert
  • inhibit rules — A firing → inhibit B (same root cause, no duplicate)

Grafana dashboard architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@startuml
title "Grafana Dashboard Layout"

rectangle "Top Row: RED Metrics" {
  component "Request Rate (QPS)" as RR
  component "Error Rate (%)" as ER
  component "Duration p50/p95/p99" as Dur
}

rectangle "Middle Row: USE Metrics" {
  component "CPU Usage" as CPU
  component "Memory Usage" as Mem
  component "Network IO" as NetIO
}

rectangle "Bottom Row: Business" {
  component "Orders/min" as OrderMin
  component "Signups/min" as SignupMin
  component "Active Users" as AU
}

rectangle "Bottom: Logs Panel" {
  component "Recent Errors (Live tail)" as Logs
}

@enduml

RED:

  • Rate — request rate
  • Errors — error count / rate
  • Duration — latency

USE:

  • Utilization — usage (CPU / memory / disk)
  • Saturation — queue length
  • Errors — error events

Golden Signals (Google SRE):

  • Latency — latency
  • Traffic — traffic
  • Errors — errors
  • Saturation — saturation

Oncall rotation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@startuml
title "Oncall Rotation"

actor "Engineer 1" as E1
actor "Engineer 2" as E2
actor "Engineer 3" as E3

participant "PagerDuty" as PD

note over E1, E3
  schedule:
  - Mon-Fri: E1
  - Sat-Sun: E2
  - backup: E3
  weekly rotation
end note

PD -> E1 : ① alert P0 (Mon 2AM)
E1 -> PD : ② acknowledge
E1 -> E1 : ③ troubleshoot + fix
E1 -> PD : ④ resolve

note right of E1
  escalate:
  - 30 min no response → E2
  - 1 hr no resolve → E3 + manager
end note

@enduml

Oncall best practices:

  • weekly rotation — prevent burnout
  • must have backup — primary out, someone covers
  • postmortem — every P0/P1 write postmortem
  • off-hour rest — day after oncall gets off

Field foot-guns

  • Metrics label explosion — add user_id / email as label, cardinality out of control. labels must be low-cardinality.
  • Logs too many — DEBUG level on in prod, storage explodes. prod INFO / WARN / ERROR, sample DEBUG.
  • Trace 100% sampling — high-traffic service OOMs. tail-based sampling: 5-10% sample, errors all.
  • Trace incomplete — only instrument half the services, chain broken. Full-stack auto-instrument.
  • Alert storm — 100 alerts fire together, don’t know which to look at first. alert grouping + inhibit rules.
  • Alert no runbook — oncall receives alert, doesn’t know next step. Every alert paired with runbook URL.
  • Prometheus single point — single Prometheus dies, all metrics lost. Dual Prometheus + remote_write persistence.
  • Logs without trace_id — want to link trace but logs have no ID. Force business code to include trace_id.
  • Grafana data source chaos — each panel uses different Prometheus. Unified data source + variables.
  • Postmortem no action — write document but no improvement. Postmortem must include action items, tracked.

Decision tree

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Observability starting point?
├─ Personal project → simple logs + uptimerobot
├─ Small team → Prometheus + Loki + Grafana
├─ Medium → + Tempo + AlertManager + PagerDuty
└─ Large → + Mimir + Cortex + custom platform

Storage choice?
├─ Budget tight → Loki + Tempo(Mimir) (cheap)
├─ Strong full-text search → Elasticsearch (expensive)
└─ Multi-cloud → Grafana Cloud (managed)

Alert tooling?
├─ Personal → email / Slack
├─ Team → AlertManager + Slack
└─ 7x24 → PagerDuty + OpsGenie

SLO starting point?
├─ No SLO → start at 99% (3 days downtime / month)
├─ Standard SaaS → 99.9% (43 min / month)
└─ High SLA → 99.99% (4 min / month)

Minimum start: Prometheus + Grafana + simple logs (ELK/Loki). Production: add OpenTelemetry + Tempo + AlertManager + PagerDuty.

Remember: observability is not monitoring — monitoring tells you system is down, observability tells you why. monitor = black-box (down or not), observability = white-box (why down). First lay foundation (Metrics + Logs + Traces), then SLO + alert.

  • Title: Three pillars of observability: Logs / Metrics / Traces + OpenTelemetry + SLO
  • Author: puml.online
  • Created at : 2026-07-30 18:20:00
  • Updated at : 2026-08-14 21:34:29
  • Link: https://puml.online/blog/plantuml-observability-three-pillars-en/
  • License: This work is licensed under CC BY-NC-SA 4.0.
On this page
Three pillars of observability: Logs / Metrics / Traces + OpenTelemetry + SLO
  1. Three pillars overview
  2. Pillar 1: Metrics (Prometheus)
  3. Pillar 2: Logs (Loki / ELK)
  4. Pillar 3: Traces (OpenTelemetry / Jaeger / Tempo)
  5. OpenTelemetry unified collection
  6. SLO / SLI / SLA
  7. Alert severity levels
  8. Grafana dashboard architecture
  9. Oncall rotation
  10. Field foot-guns
  11. Decision tree