Skip to main content
Version: Next

Prometheus

system-architecture

本文档主要讲述如何在已有的OpenYurt集群上通过Yurt-Tunnel组件的DNS模式来实现Promethues对边缘端设备的metrics进行采集。

环境要求

  • OpenYurt v0.5.0+

  • CoreDNS v1.6.8+

  • prometheus-operator

    如果你还没有OpenYurt集群,你可以使用 yurtctl工具 来初始化一个OpenYurt集群或将一个Kubernetes集群转换为OpenYurt集群,prometheus-operator的安装可以参考kube-prometheus

1.修改coreDNS配置

OpenYurt会创建yurt-tunnel-nodesConfigMap,其中保存着各节点的对应的DNS记录。

1.将 yurt-tunnel-nodes挂载至coreDNS中

kubectl patch deployment coredns -n kube-system  -p '{"spec": {"template": {"spec": {"volumes": [{"configMap":{"name":"yurt-tunnel-nodes"},"name": "edge"}]}}}}'
kubectl patch deployment coredns -n kube-system -p '{"spec": { "template": { "spec": { "containers": [{"name":"coredns","volumeMounts": [{"mountPath": "/etc/edge", "name": "edge", "readOnly": true }]}]}}}}'

2.修改coreDNS的配置文件

使用 hosts 插件将 yurt-tunnel-nodes中的 dns 记录加载到coreDNS中.

$ kubectl edit configmap coredns -n kube-system
...........
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
hosts /etc/edge/tunnel-nodes { # add hosts plugin
reload 300ms
fallthrough
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}

3.重启coreDNS

 kubectl patch deployment coredns -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","env":[{"name":"RESTART","value":"'$(date +%s)'"}]}]}}}}'

2.Prometheus 配置

prometheus默认使用IP来访问节点的metric地址,我们需要通过prometheus提供的relabel功能将IP改写为节点hostname。promethues-operator使用ServiceMonitor CRD来定义抓取配置,因此需要修改ServiceMonitor来增加relabel功能。

收集kubelet的metrics

在kubelet的ServiceMonitor中增加relabel规则,用__meta_kubernetes_endpoint_address_target_name替换掉节点IP:

$ kubectl edit serviceMonitor kubelet -n monitoring
spec:
endpoint:
..........
relabelings:
- action: replace # add relabel rule
regex: (.*);.*:(.*)
replacement: $1:$2
sourceLabels:
- __meta_kubernetes_endpoint_address_target_name
- __address__
targetLabel: __address__
..........

收集其他metrics(以node-exporter为例)

Yurt-tunnel 默认只转发10250和10255两个端口,如果需要其他端口的映射,可以修改yurt-tunnel-server-cfgConfigMap来添加。以node-exporter 为例, 需要将9100添加到https-proxy-ports中。类似的,如果是添加http端口,则可以修改http-proxy-ports配置。

修改yurt-tunnel-server-cfgConfigMap

kubectl patch configmap yurt-tunnel-server-cfg  -n kube-system  -p '{"data": {"https-proxy-ports":"9100"}}'

在node-exporter的ServiceMonitor中添加relabel规则,用__meta_kubernetes_pod_node_name替换掉节点IP。:

$ kubectl edit servicemonitor  prom-kube-prometheus-stack-node-exporter
spec:
endpoint:
......
relabelings:
- action: replace #add relabel rule
regex: (.*);.*:(.*)
replacement: $1:$2
sourceLabels:
- __meta_kubernetes_pod_node_name
- __address__
targetLabel: __address__
........

参考

Openyurt Yurt-Tunnel DNS模式实践