k8s系列 – k8s监控搭建（Prometheus+grafana）
本文档记录了k8s中prometheus+alertmanager+grafana的安装流程，点击下载：k8s监控搭建（Prometheus+grafana）.docx 。
正文
以下内容由word文档直接导入，虽然排版差劲一点，但是可以方便大家可以在线查阅。
K8s监控搭建调研
背景介绍
K8S的监控方案淘汰很快，目前v1.12.0版本已经淘汰了heapster，未来的趋势将是prometheus+Grafana+alertmanager的技术栈。
Prometheus工作原理和配置非常的繁杂，学习唯有动手实践，这篇文档目标不是搭一个完美的方案，但应该为未来实施提供重要的基本思路。
下面先手动配置一个prometheus in cluster，演示prometheus基于k8s服务发现的scrape原理。
在理解原理后，我们后续可以选择开源的kube-prometheus项目来安装全套的监控方案，当然这个版本也有很多问题，但至少我们懂原理了能维护了。
手动搭建prometheus
部署node-exporter
以daemonset的方式在每个node上跑一个node-exporter，它会采集机器资源使用情况，供prometheus抓取。
apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: kube-system
labels:
app: node-exporter
annotations:
prometheus.io/should_be_scraped: “true”
prometheus.io/metric_path: “/metrics”
prometheus.io/scrape_port: “9100”
prometheus.io/scrape_scheme: “http”
spec:
clusterIP: None
ports:
– name: metrics
port: 9100
targetPort: 9100
protocol: TCP
selector:
app: node-exporter
type: ClusterIP
—
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
labels:
app: node-exporter
name: node-exporter
namespace: kube-system
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
containers:
– name: node-exporter
image: quay.io/prometheus/node-exporter
args:
– ‘–path.procfs=/host/proc’
– ‘–path.sysfs=/host/sys’
volumeMounts:
– name: proc
mountPath: /host/proc
– name: sys
mountPath: /host/sys
hostNetwork: true
hostPID: true
volumes:
– name: proc
hostPath:
path: /proc
– name: sys
hostPath:
path: /sys
简单说就是node-exporter共享宿主机的环境，以便采集宿主机上的信息。
给它配置service的目的，是与prometheus的采集服务发现有关，就是说prometheus怎么知道node-exporter节点怎么访问呢？ 需要service暴露一些信息，prometheus会监听相关变化以便获知node-exporter服务在哪里，然后调用它们的metrics接口采集数据。
所以service里的annotations是对prometheus有意义的，下面详细分析一下。
Prometheus的配置
我们需要配置prometheus采集什么数据。
我们采集一些k8s集群最基本的Metrics：
- apiserver的metrics，因为apiserver基本了解整个集群的情况。
- kubelet的metrics，因为kubelet负责node上容器的调度。
- kubelet里的cadvisor，因为cadvisor采集每个node上的容器内部信息。
- node-exporter，因为node-exporter采集每个node的宿主机资源。
Prometheus需要部署到k8s集群内，才能顺利和这些进程通讯，我们到k8s中应用的配置文件一般可以通过configmap保存，然后挂载到应用容器里。
那么要给prometheus配置啥呢？
- 首先给它访问apiserver的权限，一般就是申请一个特权serviceaccount，然后把token配置给它。
- 告诉prometheus去哪里采集。我们注意，prometheus并不关心采集到的metrics长什么样，那是使用者关心的事情。Prometheus只需要知道去哪里采集，这些目标地址叫做targets。
Prometheus支持访问apiserver监听k8s集群中endpoints列表，支持监听k8s集群中的nodes列表，还支持pods列表等等，这些东西都在etcd理存着，监听它们很正常。
所以我们要配置一些抓取(scrape)规则，覆盖所有我们需要抓取的targets，同时呢不要误抓到我们不需要的targets。
比如我们要抓所有node上的kubelet，我们可以用node自动发现，prometheus会得到所有的node列表，并且给我一次改写(relabel)的机会，我们改写的目的就是根据这些node的信息，输出我们要去哪里抓取kubelet的metrics。
我们看一下这个job配置：
# Scrape config for nodes (kubelet).
#
# Rather than connecting directly to the node, the scrape is proxied though the
# Kubernetes apiserver. This means it will work if Prometheus is running out of
# cluster, or can’t connect to nodes for some other reason (e.g. because of
# firewalling).
– job_name: ‘kubernetes-nodes’
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
– role: node
relabel_configs:
– action: labelmap
regex: __meta_kubernetes_node_label_(.+)
– target_label: __address__
replacement: kubernetes.default.svc:443
– source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
这个job的目标是抓取所有的kubelet进程，我们知道kubelet在每个node上都有，所以我们用 kubernetes_sd_configs:node服务发现。对于得到的每个node会有一些__meta开头的标签（prometheus的概念），例如符合正则__meta_kubernetes_node_label_(.+) 的标签其实是从k8s node上的标签转化过来的，我们可以对这些标签做判断，做改写等等。
上面配置最重要的，就是把target的抓取地址__address__改成了apiserver的k8s DNS地址kubernetes.default.svc:443 ，把抓取的接口改写成了/api/v1/nodes/${1}/proxy/metrics，也就是通过apiserver代理请求kubelet的metrics接口。 这里假设__meta_kubernetes_node_name是node01，那么target的完整抓取地址就是kubernetes.default.svc:443/api/v1/nodes/node01/proxy/metrics。
上述relabel处理发生在scrape抓取之前，用于基于prometheus已有的发现机制，进行一轮target地址改写，生成真正要抓取的target列表。
我们刚才给node-exporter建过service，并且还写了一些annotations，其实都对应我们的这条抓取规则：
# Example scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# for all or only some endpoints.
– job_name: ‘kubernetes-service-endpoints’
kubernetes_sd_configs:
– role: endpoints
relabel_configs:
# Example relabel to scrape only endpoints that have
# “prometheus.io/should_be_scraped = true” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped]
action: keep
regex: true
# Example relabel to customize metric path based on endpoints
# “prometheus.io/metric_path = <metric path>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metric_path]
action: replace
target_label: __metrics_path__
regex: (.+)
# Example relabel to scrape only single, desired port for the service based
# on endpoints “prometheus.io/scrape_port = <port>” annotation.
– source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Example relabel to configure scrape scheme for all service scrape targets
# based on endpoints “prometheus.io/scrape_scheme = <scheme>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_scheme]
action: replace
target_label: __scheme__
regex: (https?)
– action: labelmap
regex: __meta_kubernetes_service_label_(.+)
– source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
– source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
这次使用了：
kubernetes_sd_configs:
– role: endpoints
发现机制，每次创建service，就会为每个后端暴露的每个端口生成一个endpoints，并且endpoints会继承service的annotations。
下图service的端点包含了如何访问node-exporter的IP和PORT信息：
Prometheus的endpoints发现机制会把k8s endpoints的annotations带到__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped这样的标签里，我们可以relabel的时候做处理，就可以生成访问到node-exporter的target地址了。
__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped这个annotations用来帮助我们区分要不要抓取这个endpoints，只有为true的我们才会keep下来，生成target：
relabel_configs:
# Example relabel to scrape only endpoints that have
# “prometheus.io/should_be_scraped = true” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped]
action: keep
regex: true
所以说，这个job是一个非常通用的方案，基于service的endpoints实现服务发现，基于endpoints的annotations控制是否抓取、在什么端口、什么URI上抓取。
所以，该job的思路可以适用于任何k8s应用的抓取，具备通用型。
手动搭建的过程就不继续赘述了，理解上述原理可以帮助我们更好的使用开源整体解决方案，配置文件完整贴在下面了：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-config
labels:
name: prometheus-server-config
namespace: kube-system
data:
prometheus.yml: |-
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
– static_configs:
– targets:
# – alertmanager:9093
# Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.
rule_files:
# – “first_rules.yml”
# – “second_rules.yml”
# A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# labelmap relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.
# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses endpoints role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port https. This works for single API server deployments as
# well as HA API server deployments.
scrape_configs:
– job_name: ‘kubernetes-apiservers’
kubernetes_sd_configs:
– role: endpoints
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# If your node certificates are self-signed or use a different CA to the
# master CA, then disable certificate verification below. Note that
# certificate verification is an integral part of a secure infrastructure
# so this should only be disabled in a controlled environment. You can
# disable certificate verification by uncommenting the line below.
#
# insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# Keep only the default/kubernetes service endpoints for the https port. This
# will add targets for each API server which Kubernetes adds an endpoint to
# the default/kubernetes service.
relabel_configs:
– source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
# Scrape config for nodes (kubelet).
#
# Rather than connecting directly to the node, the scrape is proxied though the
# Kubernetes apiserver. This means it will work if Prometheus is running out of
# cluster, or can’t connect to nodes for some other reason (e.g. because of
# firewalling).
– job_name: ‘kubernetes-nodes’
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
– role: node
relabel_configs:
– action: labelmap
regex: __meta_kubernetes_node_label_(.+)
– target_label: __address__
replacement: kubernetes.default.svc:443
– source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
# Scrape config for Kubelet cAdvisor.
#
# This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
# (those whose names begin with ‘container_’) have been removed from the
# Kubelet metrics endpoint. This job scrapes the cAdvisor endpoint to
# retrieve those metrics.
#
# In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
# HTTP endpoint; use “replacement: /api/v1/nodes/${1}:4194/proxy/metrics”
# in that case (and ensure cAdvisor’s HTTP server hasn’t been disabled with
# the –cadvisor-port=0 Kubelet flag).
#
# This job is not necessary and should be removed in Kubernetes 1.6 and
# earlier versions, or it will cause the metrics to be scraped twice.
– job_name: ‘kubernetes-cadvisor’
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
– role: node
relabel_configs:
– action: labelmap
regex: __meta_kubernetes_node_label_(.+)
– target_label: __address__
replacement: kubernetes.default.svc:443
– source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
# Example scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# for all or only some endpoints.
– job_name: ‘kubernetes-service-endpoints’
kubernetes_sd_configs:
– role: endpoints
relabel_configs:
# Example relabel to scrape only endpoints that have
# “prometheus.io/should_be_scraped = true” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped]
action: keep
regex: true
# Example relabel to customize metric path based on endpoints
# “prometheus.io/metric_path = <metric path>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metric_path]
action: replace
target_label: __metrics_path__
regex: (.+)
# Example relabel to scrape only single, desired port for the service based
# on endpoints “prometheus.io/scrape_port = <port>” annotation.
– source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Example relabel to configure scrape scheme for all service scrape targets
# based on endpoints “prometheus.io/scrape_scheme = <scheme>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_scheme]
action: replace
target_label: __scheme__
regex: (https?)
– action: labelmap
regex: __meta_kubernetes_service_label_(.+)
– source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
– source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
对于一个搭建起来的prometheus来说，下面的截图表示target自动发现配置正确，相关节点全部被监控了起来：
所以，下面我们把prometheus运行起来：
搭建prometheus
下面Yaml搭建了一个statefulset的单点prometheus，采用hostpath持久化数据以便重启prometheus时不会丢数据。
通过nodeSelector让它运行在node01上，不限制其资源使用。
通过updateStrategy控制有状态服务可以滚动升级，因为我们hostpath持久化了，所以数据得以保留。
建立一个nodePort的service供集群外访问。
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-server
name: prometheus-server
namespace: kube-system
spec:
selector:
app: prometheus-server
type: NodePort
ports:
– name: prometheus
protocol: TCP
port: 9090
targetPort: 9090
—
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: prometheus-server
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
serviceName: prometheus-server
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: prometheus-server
spec:
nodeSelector:
kubernetes.io/hostname: “node01”
containers:
– name: prometheus
image: prom/prometheus
args:
– “–config.file=/prometheus/config/prometheus.yml”
– “–storage.tsdb.path=/prometheus/data”
– “–storage.tsdb.retention=72h”
volumeMounts:
– name: prometheus-config-volume
mountPath: /prometheus/config
– name: prometheus-data-volume
mountPath: /prometheus/data
env:
– name: v
value: “12”
volumes:
– name: prometheus-config-volume
configMap:
name: prometheus-server-config
– name: prometheus-data-volume
hostPath:
path: ‘/home/k8s/prometheus/data’
serviceAccountName: admin-user
生效yaml之前，记得把/home/k8s/prometheus/data 目录建立出来，给与0777的权限，这样容器才能写入到宿主机的该目录内。
生效yaml后，查看一下service的nodePort是30612：
浏览器打开即可： 
修改yaml中的Env，将v修改一下重新提交Yaml，确认重启statefulset数据不会丢失。
历史数据还在：
Prometheus告警
Prometheus支持配置监控表达式，prometheus通过定时的执行表达式，如果表达式成立（有符合条件的时序数据），那么就会触发告警。
Prometheus只负责告警计算，至于如何发送短信还是邮件之类的，会交给alertmanager处理，现在先试验一下prometheus这部分的告警计算。
我准备了下面的表达式，它计算了每个node的cpu平均idle率，小于0.9（即90%）符合条件，将作为输出触发对应node的告警：
avg(rate(node_cpu_seconds_total{mode=”idle”}[5m])) by (instance) < 0.9
更新prometheus-server-config.yml：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-config
labels:
name: prometheus-server-config
namespace: kube-system
data:
alert-rules.yml: |-
groups:
– name: k8s-alert
rules:
– alert: NodeCpuIdelTooLow
expr: ‘avg(rate(node_cpu_seconds_total{mode=”idle”}[5m])) by (instance) < 0.9’
for: 30s
annotations:
summary: “Node {{ $labels.instance }}’s CPU idle is too low”
description: “Node {{ $labels.instance }}’s CPU idle is {{ $value }} for 10 seconds!”
prometheus.yml: |-
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
– static_configs:
– targets:
# – alertmanager:9093
# Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.
rule_files:
– “alert-rules.yml”
- 增加了alert-rules.yml告警配置文件，其中有一个监控项NodeCpuIdleTooLow，如果表达式连续成立30秒就会触发告警，要发送的警告信息可以通过annotations模板生成。
- 在prometheus.yml中配置了rule_files，指向了告警rules文件。
上传新的configmap：apply -f prometheus-server-config.yml。
我们需要重启prometheus生效告警配置，所以我们修改prometheus.yml的podTemplate部分，让env中的v参数变化一下即可：
containers:
– name: prometheus
image: prom/prometheus
args:
– “–config.file=/prometheus/config/prometheus.yml”
– “–storage.tsdb.path=/prometheus/data”
– “–storage.tsdb.retention=72h”
volumeMounts:
– name: prometheus-config-volume
mountPath: /prometheus/config
– name: prometheus-data-volume
mountPath: /prometheus/data
env:
– name: v
value: “12”
apply -f 重启prometheus。
打开prometheus界面：
告警规则展示了出来，一开始处于pending状态表示满足条件，但不足30秒的for时间。 过了一会，发现状态已经是firing了，说明告警已经触发，如果我们配置了alert-manager的话，就会把告警发送出去了。
表达式输出了2条符合条件的时序数据，所以对应2条告警，annotations已经渲染出来了，符合我的期望。
alertmanager
alertmanger被prometheus调用，负责把告警通过邮件/HTTP/微信等各种渠道发送出去。
除了发送之外，它还支持报警消息的分组合并，告警收敛和静默等功能。
这里我只用一个最简单的配置，按prometheus中配置的告警规则名分组合并即可。
配置configmap
还是复用之前的prometheus的configmap，增加alertmanager的配置文件：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-config
labels:
name: prometheus-server-config
namespace: kube-system
data:
alertmanager.yml: |-
global:
smtp_smarthost: ‘smtp.smzdm.com:587’
smtp_from: ‘liangdong@smzdm.com’
smtp_auth_username: ‘liangdong@smzdm.com’
smtp_auth_password: ‘xxxxxx’
route:
group_by: [‘alertname’]
group_wait: 5s
group_interval: 5s
repeat_interval: 1m
receiver: default-receiver
receivers:
– name: ‘default-receiver’
email_configs:
– to: ‘liangdong@smzdm.com’
Route是主要规则部分，将发来的告警时序记录按照label: alertname分组，组内做消息合并和收敛：
Group_wait：组内收到第1条告警后5秒合并发送
Group_interval：组内已有告警的情况下，再来新告警则等待5秒后合并发送。
Repeat_interval：组内已有告警按照1分钟间隔重发。
告警通过smtp发送，但目前出现报错：
rr=”starttls failed: x509: certificate is valid for *.qiye.163.com, not smtp.smzdm.com”
因为我们企业邮箱用的163的，证书也是163的，但我们配的smtp服务器是smzdm.com域名的，所以客户端校验失败。 在下一个版本的alertmanager中将支持配置项不校验服务端证书，暂时先这样吧（alertmanager还支持HTTP webhook等）。
原先prometheus的配置添加了alertmanager服务的地址：
prometheus.yml: |-
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
– static_configs:
– targets:
– localhost:9093
为什么alertmanager通过Localhost:9093就可以访问呢？因为我计划把prometheus+alertmanager放在同一个statefulset的POD内，同一个POD内的container之间共享一个网络namespace。
service服务
alertmanager也有界面，我们继续复用prometheus的service配置，增加一个endpoint即可：
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-server
name: prometheus-server
namespace: kube-system
spec:
selector:
app: prometheus-server
type: NodePort
ports:
– name: prometheus
protocol: TCP
port: 9090
targetPort: 9090
– name: alertmanager
protocol: TCP
port: 9093
targetPort: 9093
也就是说这个service把POD里的2个端口暴露了出来，会有2个nodePort。
Prometheus工作在POD内的9090，alertmanager工作在POD内的9093。
启动alertmanager
在prometheus的POD内，再添加一个container：
– name: alertmanager
image: quay.io/prometheus/alertmanager
args:
– “–config.file=/alertmanager/config/alertmanager.yml”
volumeMounts:
– name: prometheus-config-volume
mountPath: /alertmanager/config
– name: alertmanager-data-volume
mountPath: /alertmanager/data
env:
– name: v
value: “13”
volumes:
– name: prometheus-config-volume
configMap:
name: prometheus-server-config
– name: prometheus-data-volume
hostPath:
path: ‘/home/k8s/prometheus/data’
– name: alertmanager-data-volume
hostPath:
path: ‘/home/k8s/alertmanager/data’
serviceAccountName: admin-user
把alertmanager.yaml配置文件映射进来，同时hostPath一个持久化目录上来，记得给目录改权限0777。
apply -f Prometheus.yaml重新生效，现在可以通过service nodePort访问到alertmanager了：
告警已经从prometheus推送过来了，按照alertname做了group组内的告警合并。
Grafana
Grafana通过请求prometheus执行查询获取时序数据，然后做图表展现。其实我们只需要在prometheus UI里调试好表达式，然后拿到grafana里配置一下图表就可以了。
Grafana的配置文件是ini格式的，同时支持通过环境变量的形式覆盖个别配置。
Grafana还支持通过配置文件预加载dashboard的配置（该功能叫做provisioning）。一般是先启动grafana，然后在UI中配置dashboard并保存，grafana会把UI中配置的dashboard存储到db里，默认是单机sqlite，也可以是mysql这种远程数据库。
生产环境应该用mysql来存dashboard配置提供可靠性，在这里我还是使用sqlite，把grafana的sqlite数据库通过hostPath映射到宿主机上，确保其重启时其db数据不会丢失，但宿主机故障配置就丢了。
安装grafana
安装参考：http://docs.grafana.org/installation/docker/
为了简单，我继续把grafana配置到之前prometheus的statefulset里。
修改prometheus.yaml，给service增加一个endpoint：
– name: grafana
protocol: TCP
port: 3000
targetPort: 3000
再增加一个volumes保存grafana的data：
– name: grafana-data-volume
hostPath:
path: ‘/home/k8s/grafana/data’
serviceAccountName: admin-user
再配置一个container到POD里:
– name: grafana
image: grafana/grafana
volumeMounts:
– name: grafana-data-volume
mountPath: /var/lib/grafana
env:
– name: v
value: “13”
– name: GF_LOG_MODE
value: “console”
这里通过环境变量覆盖了grafana的log配置，只输出到stdout/stderr（docker统一日志文件滚动，不会无限增长），不打日志文件，免得清理起来麻烦。
把hostPath挂载到/var/lib/grafana目录，这是grafana docker镜像配置的默认data存储路径。
生效yaml即可。
配置dashboard
查看一下grafana的service端口：
登录grafana，账号密码：admin admin
先配置一个datasource数据源，即本POD内的prometheus server：
然后添加dashboard。
已经有开源dashboard配置了，所以我们只需要使用它们。
一个是cadvisor采集数据的展现，ID是315：
立即可以看到图表：
另外一个是node-exporter数据的展现，其ID是1860：
我们也可以自己设计dashboard，只要不嫌麻烦就行。
Dashboard原理
以node-exporter的dashboard为例：
可以看到上方有一些选项，这些相当于dashboard的共用变量：
每个下拉框可以选择的内容是通过prometheus表达式计算出来的，比如$node包含了node01和node02，我们一旦选择其中之一$node就被赋值了。
接着，我们可以点某个仪表盘，选择edit看一下表达式：
这个又臭又长的表达式就是发给prometheus的query了，其中$node和$job这些就是我们下拉选择变量了，会替换到query里，发给prometheus。
我们可以参考这些别人写的query，配置alertmanager实现告警，总比自己去琢磨各种metrics怎么算要简单的多。
其他
还有一些东西没有全部实践，比如：
- 通过部署kube-state-metrics，可以抓到k8s各种类型资源的metrics。
- 通过部署blackbox-exporter，可以实现对任意service的HTTP/TCP网络性能metrics。
但整体思路和关键原理经过实践都应有所把握，配置文件见文档末尾。
参考资料
- 后续在网上发现了类似的部署资料，供后续完善参考：
2，prometheus的热加载配置可以基于http接口触发：https://www.robustperception.io/reloading-prometheus-configuration
- 可以考虑是否使用kube-prometheus全栈方案，但目前也不成熟：https://github.com/coreos/kube-prometheus
Prometheus.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-server
name: prometheus-server
namespace: kube-system
spec:
selector:
app: prometheus-server
type: NodePort
ports:
– name: prometheus
protocol: TCP
port: 9090
targetPort: 9090
– name: alertmanager
protocol: TCP
port: 9093
targetPort: 9093
– name: grafana
protocol: TCP
port: 3000
targetPort: 3000
—
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: prometheus-server
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
serviceName: prometheus-server
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: prometheus-server
spec:
nodeSelector:
kubernetes.io/hostname: “node01”
containers:
– name: prometheus
image: prom/prometheus
args:
– “–config.file=/prometheus/config/prometheus.yml”
– “–storage.tsdb.path=/prometheus/data”
– “–storage.tsdb.retention=72h”
volumeMounts:
– name: prometheus-config-volume
mountPath: /prometheus/config
– name: prometheus-data-volume
mountPath: /prometheus/data
env:
– name: v
value: “13”
– name: alertmanager
image: quay.io/prometheus/alertmanager
args:
– “–config.file=/alertmanager/config/alertmanager.yml”
volumeMounts:
– name: prometheus-config-volume
mountPath: /alertmanager/config
– name: alertmanager-data-volume
mountPath: /alertmanager/data
env:
– name: v
value: “13”
– name: grafana
image: grafana/grafana
volumeMounts:
– name: grafana-data-volume
mountPath: /var/lib/grafana
env:
– name: v
value: “13”
– name: GF_LOG_MODE
value: “console”
volumes:
– name: prometheus-config-volume
configMap:
name: prometheus-server-config
– name: prometheus-data-volume
hostPath:
path: ‘/home/k8s/prometheus/data’
– name: alertmanager-data-volume
hostPath:
path: ‘/home/k8s/alertmanager/data’
– name: grafana-data-volume
hostPath:
path: ‘/home/k8s/grafana/data’
serviceAccountName: admin-user
Prometheus-server-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-config
labels:
name: prometheus-server-config
namespace: kube-system
data:
alertmanager.yml: |-
global:
smtp_smarthost: ‘smtp.smzdm.com:587’
smtp_from: ‘liangdong@smzdm.com’
smtp_auth_username: ‘liangdong@smzdm.com’
smtp_auth_password: ‘xxxxxxx’
route:
group_by: [‘alertname’]
group_wait: 5s
group_interval: 5s
repeat_interval: 1m
receiver: default-receiver
receivers:
– name: ‘default-receiver’
email_configs:
– to: ‘liangdong@smzdm.com’
alert-rules.yml: |-
groups:
– name: k8s-alert
rules:
– alert: NodeCpuIdelTooLow
expr: ‘avg(rate(node_cpu_seconds_total{mode=”idle”}[5m])) by (instance) < 0.9’
for: 30s
annotations:
summary: “Node {{ $labels.instance }}’s CPU idle is too low”
description: “Node {{ $labels.instance }}’s CPU idle is {{ $value }} for 10 seconds!”
prometheus.yml: |-
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
– static_configs:
– targets:
– localhost:9093
# Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.
rule_files:
– “alert-rules.yml”
# – “first_rules.yml”
# – “second_rules.yml”
# A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# labelmap relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.
# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses endpoints role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port https. This works for single API server deployments as
# well as HA API server deployments.
scrape_configs:
– job_name: ‘kubernetes-apiservers’
kubernetes_sd_configs:
– role: endpoints
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# If your node certificates are self-signed or use a different CA to the
# master CA, then disable certificate verification below. Note that
# certificate verification is an integral part of a secure infrastructure
# so this should only be disabled in a controlled environment. You can
# disable certificate verification by uncommenting the line below.
#
# insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# Keep only the default/kubernetes service endpoints for the https port. This
# will add targets for each API server which Kubernetes adds an endpoint to
# the default/kubernetes service.
relabel_configs:
– source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
# Scrape config for nodes (kubelet).
#
# Rather than connecting directly to the node, the scrape is proxied though the
# Kubernetes apiserver. This means it will work if Prometheus is running out of
# cluster, or can’t connect to nodes for some other reason (e.g. because of
# firewalling).
– job_name: ‘kubernetes-nodes’
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
– role: node
relabel_configs:
– action: labelmap
regex: __meta_kubernetes_node_label_(.+)
– target_label: __address__
replacement: kubernetes.default.svc:443
– source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
# Scrape config for Kubelet cAdvisor.
#
# This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
# (those whose names begin with ‘container_’) have been removed from the
# Kubelet metrics endpoint. This job scrapes the cAdvisor endpoint to
# retrieve those metrics.
#
# In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
# HTTP endpoint; use “replacement: /api/v1/nodes/${1}:4194/proxy/metrics”
# in that case (and ensure cAdvisor’s HTTP server hasn’t been disabled with
# the –cadvisor-port=0 Kubelet flag).
#
# This job is not necessary and should be removed in Kubernetes 1.6 and
# earlier versions, or it will cause the metrics to be scraped twice.
– job_name: ‘kubernetes-cadvisor’
# Default to scraping over https. If required, just disable this or change to
# http.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
– role: node
relabel_configs:
– action: labelmap
regex: __meta_kubernetes_node_label_(.+)
– target_label: __address__
replacement: kubernetes.default.svc:443
– source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
# Example scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# for all or only some endpoints.
– job_name: ‘kubernetes-service-endpoints’
kubernetes_sd_configs:
– role: endpoints
relabel_configs:
# Example relabel to scrape only endpoints that have
# “prometheus.io/should_be_scraped = true” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_should_be_scraped]
action: keep
regex: true
# Example relabel to customize metric path based on endpoints
# “prometheus.io/metric_path = <metric path>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metric_path]
action: replace
target_label: __metrics_path__
regex: (.+)
# Example relabel to scrape only single, desired port for the service based
# on endpoints “prometheus.io/scrape_port = <port>” annotation.
– source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Example relabel to configure scrape scheme for all service scrape targets
# based on endpoints “prometheus.io/scrape_scheme = <scheme>” annotation.
– source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_scheme]
action: replace
target_label: __scheme__
regex: (https?)
– action: labelmap
regex: __meta_kubernetes_service_label_(.+)
– source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
– source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
node-exporter.yml
apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: kube-system
labels:
app: node-exporter
annotations:
prometheus.io/should_be_scraped: “true”
prometheus.io/metric_path: “/metrics”
prometheus.io/scrape_port: “9100”
prometheus.io/scrape_scheme: “http”
spec:
clusterIP: None
ports:
– name: metrics
port: 9100
targetPort: 9100
protocol: TCP
selector:
app: node-exporter
type: ClusterIP
—
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
labels:
app: node-exporter
name: node-exporter
namespace: kube-system
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
containers:
– name: node-exporter
image: quay.io/prometheus/node-exporter
args:
– ‘–path.procfs=/host/proc’
– ‘–path.sysfs=/host/sys’
volumeMounts:
– name: proc
mountPath: /host/proc
– name: sys
mountPath: /host/sys
hostNetwork: true
hostPID: true
volumes:
– name: proc
hostPath:
path: /proc
– name: sys
hostPath:
path: /sys
如果文章帮助您解决了工作难题，您可以帮我点击屏幕上的任意广告，或者赞助少量费用来支持我的持续创作，谢谢~
1