Skip to main content
Version: v0.5.0

Prometheus

system-architecture

本文档主要讲述如何在已有的OpenYurt集群上通过Yurt-Tunnel组件的DNS模式来实现Promethues对边缘端设备的metrics进行采集。

环境要求

  • OpenYurt v0.5.0+

  • CoreDNS v1.6.8+

  • prometheus-operator

    如果你还没有OpenYurt集群,你可以使用 yurtctl工具 来初始化一个OpenYurt集群或将一个Kubernetes集群转换为OpenYurt集群,prometheus-operator的安装可以参考kube-prometheus

1.修改coreDNS配置

OpenYurt会创建yurt-tunnel-nodesConfigMap,其中保存着各节点的对应的DNS记录。

1.将 yurt-tunnel-nodes挂载至coreDNS中

kubectl patch deployment coredns -n kube-system  -p '{"spec": {"template": {"spec": {"volumes": [{"configMap":{"name":"yurt-tunnel-nodes"},"name": "edge"}]}}}}'
kubectl patch deployment coredns -n kube-system   -p '{"spec": { "template": { "spec": { "containers": [{"name":"coredns","volumeMounts": [{"mountPath": "/etc/edge", "name": "edge", "readOnly": true }]}]}}}}'

2.修改coreDNS的配置文件

使用 hosts 插件将 yurt-tunnel-nodes中的 dns 记录加载到coreDNS中.

$ kubectl edit configmap coredns -n kube-system
...........
 Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        hosts /etc/edge/tunnel-nodes {    # add hosts plugin
            reload 300ms
            fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

3.重启coreDNS

 kubectl patch deployment coredns -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","env":[{"name":"RESTART","value":"'$(date +%s)'"}]}]}}}}'

2.Prometheus 配置

prometheus默认使用IP来访问节点的metric地址,我们需要通过prometheus提供的relabel功能将IP改写为节点hostname。promethues-operator使用ServiceMonitor CRD来定义抓取配置,因此需要修改ServiceMonitor来增加relabel功能。

收集kubelet的metrics

在kubelet的ServiceMonitor中增加relabel规则,用__meta_kubernetes_endpoint_address_target_name替换掉节点IP:

$ kubectl edit serviceMonitor kubelet -n monitoring
spec:
  endpoint:
    ..........
    relabelings:
    - action: replace  # add relabel rule
      regex: (.*);.*:(.*)
      replacement: $1:$2
      sourceLabels:
      - __meta_kubernetes_endpoint_address_target_name
      - __address__
      targetLabel: __address__
    ..........

收集其他metrics(以node-exporter为例)

Yurt-tunnel 默认只转发10250和10255两个端口,如果需要其他端口的映射,可以修改yurt-tunnel-server-cfgConfigMap来添加。以node-exporter 为例, 需要将9100添加到https-proxy-ports中。类似的,如果是添加http端口,则可以修改http-proxy-ports配置。

修改yurt-tunnel-server-cfgConfigMap

kubectl patch configmap yurt-tunnel-server-cfg  -n kube-system  -p '{"data": {"https-proxy-ports":"9100"}}'

在node-exporter的ServiceMonitor中添加relabel规则,用__meta_kubernetes_pod_node_name替换掉节点IP。:

$ kubectl edit servicemonitor  prom-kube-prometheus-stack-node-exporter
spec:
 endpoint:
   ......
   relabelings:
    - action: replace #add relabel rule
      regex: (.*);.*:(.*)
      replacement: $1:$2
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      - __address__
      targetLabel: __address__
    ........

参考

Openyurt Yurt-Tunnel DNS模式实践