可观测性三支柱:Logs / Metrics / Traces + OpenTelemetry + SLO

puml.online
  2026-07-30 18:20 2026-07-30 18:20 创建   2026-08-14 21:34:29 2026-08-14 21:34:29 更新    

生产系统出问题,「重启就好」不是长久之计。三支柱可观测性(Logs + Metrics + Traces)让根因定位从几小时缩短到几分钟。这篇是三支柱架构、OpenTelemetry 统一采集、Prometheus + Grafana + Tempo/Loki 组合、SLO/SLI/SLA 设计、告警分级。

三支柱全景

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
@startuml
title "Three Pillars of Observability"

package "Sources" {
  component "App" as App
  component "Service A" as SvcA
  component "Service B" as SvcB
  component "DB" as DB
  component "Queue" as Q
}

package "Collection (OpenTelemetry)" {
  component "OTel SDK" as SDK
  component "OTel Collector" as Collector
}

package "Storage" {
  component "Metrics\n(Prometheus / Mimir)" as Metrics
  component "Logs\n(Loki / Elasticsearch)" as Logs
  component "Traces\n(Tempo / Jaeger)" as Traces
}

package "Visualization" {
  component "Grafana" as Grafana
}

package "Alerting" {
  component "AlertManager" as Alert
  component "PagerDuty" as PD
}

App -> SDK : emit logs, metrics, traces
SvcA -> SDK
SvcB -> SDK
DB -> SDK
Q -> SDK

SDK -> Collector : OTLP
Collector -> Metrics : metrics scrape
Collector -> Logs : logs push
Collector -> Traces : traces push

Metrics --> Grafana
Logs --> Grafana
Traces --> Grafana

Metrics --> Alert
Logs --> Alert
Alert -> PD : page oncall

@enduml

三支柱:

  • Metrics — 数值时序(cpu / 内存 / QPS / 延迟)
  • Logs — 离散事件(错误 / 警告 / 信息)
  • Traces — 请求链路(调用关系 / 时长)

支柱 1:Metrics(Prometheus)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
@startuml
title "Prometheus Metrics Flow"

participant "App" as App
participant "Prometheus" as Prom
participant "Grafana" as Graf
participant "AlertManager" as AM

note over App
  Metrics 暴露 /metrics endpoint:
  - http_requests_total{method, path, status}
  - http_request_duration_seconds_bucket{...}
  - process_cpu_seconds_total
end note

App -> Prom : ① scrape /metrics (每 15s)
Prom -> Prom : ② 存储到 TSDB
Prom -> Graf : ③ PromQL 查询
Graf -> Graf : ④ 渲染仪表盘

note over AM
  告警规则:
  - http_error_rate > 5% for 5m
  - p99_latency > 1s for 5m
  - cpu_usage > 80% for 10m
end note

Prom -> AM : ⑤ 评估告警规则
AM -> AM : ⑥ firing → PagerDuty / Slack
AM -> Graf : ⑦ alert annotation

@enduml

Prometheus 四种 Metric:

类型 用途 示例
Counter 只增不减 http_requests_total
Gauge 当前值 cpu_usage, queue_size
Histogram 分布 request_duration_seconds
Summary 分位数 类似 Histogram,客户端算

Histogram 自动算 p50/p95/p99:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# p99 延迟
histogram_quantile(0.99,
  sum by (le, path) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# QPS
sum(rate(http_requests_total[5m]))

# 错误率
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

支柱 2:Logs(Loki / ELK)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
@startuml
title "Logs Pipeline"

participant "App" as App
participant "Promtail / Fluentd" as Agent
participant "Loki / Elasticsearch" as Storage
participant "Grafana" as Graf

App -> Agent : ① 写 stdout (JSON)
Agent -> Agent : ② 收集 / 解析 / 加 label
Agent -> Storage : ③ push (Loki) / index (ES)

note right of Storage
  Loki:
  - 不索引全文,只索引 label
  - 便宜,易扩展
  - 适合云原生
  
  Elasticsearch:
  - 全文索引
  - 强大但贵
  - 适合复杂搜索
end note

Storage -> Graf : ④ LogQL / KQL 查询
Graf -> Graf : ⑤ 渲染日志面板

note over App : 结构化日志
{ "timestamp": "2026-07-30T10:00:00Z",
  "level": "ERROR",
  "service": "user-service",
  "trace_id": "abc123",
  "message": "DB query failed",
  "error": "connection timeout",
  "stack_trace": "..." }

@enduml

日志最佳实践:

  • JSON 结构化 — 不要纯文本
  • 必带 trace_id — 关联到 traces
  • 必带 service / env / version — 标签用于过滤
  • 不要记密码 / token — 过滤敏感字段
  • 采样 — 高流量服务采样 1%

支柱 3:Traces(OpenTelemetry / Jaeger / Tempo)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
@startuml
title "Distributed Trace - Request Flow"

actor "User" as User
participant "API Gateway" as GW
participant "Auth Service" as Auth
participant "Order Service" as Order
participant "Payment Service" as Pay
participant "DB" as DB
participant "OTel Collector" as OTel

User -> GW : ① GET /api/orders

note over OTel : 每次请求生成 trace_id

GW -> OTel : ② trace start (span: gw.request)
GW -> Auth : ③ HTTP /verify
Auth -> OTel : ④ span: auth.verify
Auth --> GW : ⑤ OK
GW -> Order : ⑥ HTTP /orders
Order -> OTel : ⑦ span: order.list
Order -> DB : ⑧ SELECT
DB --> Order : ⑨ orders
Order -> Pay : ⑩ HTTP /verify-payment
Pay -> OTel : ⑪ span: pay.verify
Pay --> Order : ⑫ OK
Order --> GW : ⑬ orders
GW --> User : ⑭ 200

note over OTel
  Trace 包含所有 spans:
  - trace_id: abc123
  - 每个 span 有 parent_id 形成树
  - 每个 span 有 start_time + duration
end note

OTel -> OTel : ⑮ 收集所有 spans
OTel -> Tempo : ⑯ push trace

@enduml

Trace 关键概念:

  • Trace — 一次完整请求的所有 spans
  • Span — 一次调用(API call / DB query)
  • trace_id — 全链路唯一标识
  • parent_span_id — 形成调用树
  • baggage — 跨服务传上下文(如 user_id)

OpenTelemetry 统一采集

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@startuml
title "OpenTelemetry Unified Instrumentation"

participant "App Code" as Code
participant "OTel SDK\n(auto-instrumentation)" as SDK
participant "OTel Collector\n(agent / gateway)" as Collector
participant "Backend" as Backend

note over Code
  // 业务代码无需关心采集
  // OTel SDK 自动拦截:
  // - HTTP client/server
  // - gRPC
  // - DB drivers (pg/mysql/redis)
  // - Queue (kafka/sqs)
end note

Code -> SDK : ① 自动拦截(express, pg, redis...)
SDK -> SDK : ② 生成 spans + metrics + logs
SDK -> Collector : ③ OTLP (gRPC/HTTP)

note over Collector
  Collector 处理:
  - batch (合并发送)
  - retry (网络故障重发)
  - sampling (采样)
  - filter (删敏感字段)
  - attributes (加 tag)
end note

Collector -> Backend : ④ 转发到 Prometheus / Tempo / Loki

@enduml

OpenTelemetry Collector 配置:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
  attributes:
    actions:
      - key: environment
        value: production
        action: insert
  filter/logs:
    logs:
      exclude:
        matchers:
          - 'severity_text = "DEBUG"'

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [batch, memory_limiter, filter/logs]
      exporters: [loki]

SLO / SLI / SLA

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
@startuml
title "SLO / SLI / SLA Relationship"

rectangle "SLI (Service Level Indicator)" {
  component "实际测量值" as SLI
  note right
    例:
    - 请求成功率 = 成功请求 / 总请求
    - p99 延迟 = 99% 请求 < X ms
  end note
}

rectangle "SLO (Service Level Objective)" {
  component "内部目标" as SLO
  note right
    例:
    - 可用性 ≥ 99.9%
    - p99 延迟 < 500ms
    - 错误率 < 0.1%
  end note
}

rectangle "SLA (Service Level Agreement)" {
  component "对客户承诺" as SLA
  note right
    例:
    - 可用性 ≥ 99.5%(合同)
    - 达不到退款 / 赔偿
  end note
}

SLI --> SLO : "测量对比"
SLO --> SLA : "内部目标 ≥ 客户承诺"

note right of SLA
  通常:
  SLA 比 SLO 宽松(留 buffer)
  SLO 比实际目标严格(留升级空间)
end note

@enduml

SLO 实战:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# slos.yaml
apiVersion: sloth.sloth.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: user-service-slo
spec:
  service: user-service
  labels:
    team: platform
  slos:
    - name: availability
      objective: 99.9
      description: "Service availability over 30 days"
      sli:
        events:
          error_query: sum(rate(http_requests_total{service="user-service",status=~"5.."}[5m]))
          total_query: sum(rate(http_requests_total{service="user-service"}[5m]))
    - name: latency
      objective: 95
      description: "95% of requests complete in < 500ms"
      sli:
        events:
          error_query: sum(rate(http_request_duration_seconds_bucket{service="user-service",le="0.5"}[5m]))
          total_query: sum(rate(http_request_duration_seconds_count{service="user-service"}[5m]))

告警分级

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
@startuml
title "Alert Severity Levels"

rectangle "Severity" {
  rectangle "P0 - Critical\n(立即响应)" {
    component "5xx 错误率 > 10%\n全站不可用\n影响营收"
  }
  rectangle "P1 - High\n(15 分钟内)" {
    component "5xx 错误率 5-10%\n核心功能受影响\np99 延迟 > 2s"
  }
  rectangle "P2 - Medium\n(1 小时内)" {
    component "5xx 错误率 1-5%\n非核心功能异常"
  }
  rectangle "P3 - Low\n(下个工作日)" {
    component "性能下降\n磁盘使用率高"
  }
}

note bottom of "Severity"
  响应方式:
  P0 → 电话 + 短信 + Slack @here
  P1 → Slack + PagerDuty
  P2 → Slack
  P3 → Email / Slack 异步
end note

@enduml

告警规则避免疲劳:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# alertmanager.yml
groups:
- name: critical
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
      > 0.05
    for: 5m
    labels:
      severity: p1
    annotations:
      summary: "Error rate {{ $value | humanizePercentage }}"
      runbook: "https://wiki/runbooks/high-error-rate"

  - alert: HighLatency
    expr: |
      histogram_quantile(0.99,
        sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
      ) > 1
    for: 5m
    labels:
      severity: p2
    annotations:
      summary: "P99 latency {{ $value }}s for {{ $labels.service }}"

避免告警疲劳:

  • 每个告警必须有 runbook — 链接到排查文档
  • 不能 alert 就关掉 — 而不是屏蔽
  • for 持续时间 — 不是瞬时抖动就 alert
  • 抑制规则 — A firing → 抑制 B(同根源不重复报警)

Grafana 仪表盘架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@startuml
title "Grafana Dashboard Layout"

rectangle "Top Row: RED Metrics" {
  component "Request Rate (QPS)" as RR
  component "Error Rate (%)" as ER
  component "Duration p50/p95/p99" as Dur
}

rectangle "Middle Row: USE Metrics" {
  component "CPU Usage" as CPU
  component "Memory Usage" as Mem
  component "Network IO" as NetIO
}

rectangle "Bottom Row: Business" {
  component "Orders/min" as OrderMin
  component "Signups/min" as SignupMin
  component "Active Users" as AU
}

rectangle "Bottom: Logs Panel" {
  component "Recent Errors (Live tail)" as Logs
}

@enduml

RED:

  • Rate — 请求速率
  • Errors — 错误数 / 错误率
  • Duration — 延迟

USE:

  • Utilization — 使用率(CPU / 内存 / 磁盘)
  • Saturation — 饱和度(队列长度)
  • Errors — 错误事件

黄金信号(Google SRE):

  • Latency — 延迟
  • Traffic — 流量
  • Errors — 错误
  • Saturation — 饱和度

Oncall 值班

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@startuml
title "Oncall Rotation"

actor "Engineer 1" as E1
actor "Engineer 2" as E2
actor "Engineer 3" as E3

participant "PagerDuty" as PD

note over E1, E3
  轮班表:
  - 周一-周五: E1
  - 周六-周日: E2
  - 备份: E3
  每周轮换
end note

PD -> E1 : ① 告警 P0 (周一 2AM)
E1 -> PD : ② acknowledge
E1 -> E1 : ③ 排查 + 修复
E1 -> PD : ④ resolve

note right of E1
  升级:
  - 30 分钟没响应 → E2
  - 1 小时没解决 → E3 + manager
end note

@enduml

Oncall 最佳实践:

  • 每周轮换 — 防止 burnout
  • 必须 backup — 主值出意外有人兜底
  • 事后总结 — 每个 P0/P1 写 postmortem
  • 错峰值班 — 给值班的第二天休

实战踩坑

  • Metrics 标签爆炸 — 加 user_id / email 当 label,cardinality 失控。label 必须低基数。
  • 日志太多 — DEBUG 级别全开,存储爆。生产 INFO / WARN / ERROR,采样 DEBUG。
  • Trace 采样率 100% — 高流量服务 OOM。tail-based sampling: 5-10% 采样,错误全采。
  • Trace 不全 — 只 instrument 一半服务,链路断裂。全栈自动 instrument。
  • 告警风暴 — 100 个 alert 一起响,不知道哪个先看。alert grouping + 抑制规则。
  • 告警没 runbook — oncall 收到 alert 不知道下一步。每个 alert 配 runbook URL。
  • Prometheus 单点 — 单 Prometheus 挂了,所有 metrics 丢。双 Prometheus + remote_write 持久化。
  • 日志没 trace_id — 想关联 trace 但日志没 ID。强制业务代码带 trace_id。
  • Grafana 数据源乱 — 每个 panel 用不同 Prometheus。统一数据源 + 变量。
  • Postmortem 没行动 — 写完文档没改进。Postmortem 必带 action items,跟踪。

决策树

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
可观测性起步?
├─ 个人项目 → 简单日志 + uptimerobot
├─ 小团队 → Prometheus + Loki + Grafana
├─ 中型 → + Tempo + AlertManager + PagerDuty
└─ 大型 → + Mimir + Cortex + 自研平台

存储选型?
├─ 预算紧 → Loki + Tempo(Mimir)(便宜)
├─ 全文搜索强 → Elasticsearch(贵)
└─ 多云 → Grafana Cloud(托管)

告警工具?
├─ 个人 → 邮件 / Slack
├─ 团队 → AlertManager + Slack
└─ 7x24 → PagerDuty + OpsGenie

SLO 起点?
├─ 无 SLO → 先用 99% (3 天 downtime / 月)
├─ 标准 SaaS → 99.9% (43 分钟 / 月)
└─ 高 SLA → 99.99% (4 分钟 / 月)

最小起步:Prometheus + Grafana + 简单日志(ELK/Loki)**。生产加 OpenTelemetry + Tempo + AlertManager + PagerDuty。

记住:可观测性不是监控——监控告诉你系统挂了,可观测性告诉你为什么挂。监控 = 黑盒(挂了没),可观测性 = 白盒(为什么挂)。先打基础(Metrics + Logs + Traces),再 SLO + 告警。

  • 标题: 可观测性三支柱:Logs / Metrics / Traces + OpenTelemetry + SLO
  • 作者: puml.online
  • 创建于 : 2026-07-30 18:20:00
  • 更新于 : 2026-08-14 21:34:29
  • 链接: https://puml.online/blog/plantuml-observability-three-pillars/
  • 版权声明: 本文章采用 CC BY-NC-SA 4.0 进行许可。
目录
可观测性三支柱:Logs / Metrics / Traces + OpenTelemetry + SLO
  1. 三支柱全景
  2. 支柱 1:Metrics(Prometheus)
  3. 支柱 2:Logs(Loki / ELK)
  4. 支柱 3:Traces(OpenTelemetry / Jaeger / Tempo)
  5. OpenTelemetry 统一采集
  6. SLO / SLI / SLA
  7. 告警分级
  8. Grafana 仪表盘架构
  9. Oncall 值班
  10. 实战踩坑
  11. 决策树