HPA

Posted by Vito on January 19, 2024

HPA

  • 通过观察 Pod 的 cpu内存使用率或自定义 metrics 指标进行自动的扩容或缩容 pod 的数量。

  • 通常用于 Deployment,不适用于无法扩/缩容的对象,如 DaemonSet

  • 控制管理器每隔30s(可以通过 –horizontal-pod-autoscaler-sync-period 修改)查询 metrics 的资源使用情况

开启指标服务(便于观察)

  • 下载 metrics-server 组件配置文件
1
2
[root@k8s-master1 ~]# cd /opt/kubernetes/
[root@k8s-master1 kubernetes]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server-components.yaml
  • 修改镜像地址为国内的地址
1
2
3
4
5
6
7
8
9
[root@k8s-master1 kubernetes]# grep image metrics-server-components.yaml 
        image: registry.k8s.io/metrics-server/metrics-server:v0.6.4
        imagePullPolicy: IfNotPresent
 
[root@k8s-master1 kubernetes]# sed -i 's/registry.k8s.io\/metrics-server/registry.cn-hangzhou.aliyuncs.com\/google_containers/g' metrics-server-components.yaml

[root@k8s-master1 kubernetes]# grep image metrics-server-components.yaml 
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.6.4
        imagePullPolicy: IfNotPresent
  • 修改容器的 tls 配置,不验证 tls
    • 在 containers 的 args 参数中增加 --kubelet-insecure-tls 参数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@k8s-master1 kubernetes]# vim metrics-server-components.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
  • 安装组件
1
2
3
4
5
6
7
8
9
10
11
12
13
[root@k8s-master1 kubernetes]# kubectl apply -f metrics-server-components.yaml 
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

[root@k8s-master1 kubernetes]# kubectl get po --all-namespaces | grep metrics
kube-system   metrics-server-b8b756d46-qj7m2                 1/1     Running   0                42s

CPU、内存指标监控

  • 实现 cpu 或内存的监控,首先有个前提条件是该对象必须配置了 resources.requests.cpuresources.requests.memory 才可以,可以配置当 cpu/memory 达到上述配置的百分比后进行扩容或缩容
  • 准备好一个有资源限制的 Deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[root@k8s-master1 ~]# cd test
[root@k8s-master1 test]# vim deploy-vito-web.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: svc-vito-web # Service 的名字
spec:
  selector: # 将服务流量路由到具有与此选择器匹配的 Pod
    app: deploy-vito-nginx
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-vito-web
spec:
  replicas: 2 # 期望副本数
  selector:
    matchLabels:
      app: deploy-vito-nginx
  template: 
    metadata:
      labels:
        app: deploy-vito-nginx
    spec:
      containers:
      - image: nginx:1.7.9
        imagePullPolicy: IfNotPresent
        name: deploy-vito-web-c-nginx
        resources:
          requests: # 最少需要多少资源
            cpu: 10m # 限制 cpu 最少使用 0.01 个核心
            memory: 128Mi # 限制内存最少使用 128 兆
          limits: # 最多可以用多少资源
            cpu: 200m # 限制 cpu 最多使用 0.2 个核心
            memory: 256Mi # 限制内存最多使用 256 兆

[root@k8s-master1 test]# kubectl create -f deploy-vito-web.yaml 
service/svc-vito-web created
deployment.apps/deploy-vito-web created

[root@k8s-master1 test]# kubectl get deploy
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
deploy-vito-web   2/2     2            2           36s
[root@k8s-master1 test]# kubectl get rs
NAME                         DESIRED   CURRENT   READY   AGE
deploy-vito-web-5f74dfcf97   2         2         2       39s
[root@k8s-master1 test]# kubectl get po
NAME                               READY   STATUS    RESTARTS   AGE
deploy-vito-web-5f74dfcf97-46lpj   1/1     Running   0          46s
deploy-vito-web-5f74dfcf97-6crbg   1/1     Running   0          46s
[root@k8s-master1 test]# kubectl get svc
NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP   10.96.0.1     <none>        443/TCP        5d18h
svc-vito-web   NodePort    10.106.0.34   <none>        80:30686/TCP   52s
  • 执行命令 kubectl autoscale deploy deploy-vito-web --cpu-percent=20 --min=2 --max=5
    • 通过 kubectl get hpa 可以获取 HPA 信息
1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master1 test]# kubectl autoscale deploy deploy-vito-web --cpu-percent=20 --min=2 --max=5
horizontalpodautoscaler.autoscaling/deploy-vito-web autoscaled

[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   0%/20%    2         5         2          20s

[root@k8s-master1 test]# kubectl get po
NAME                               READY   STATUS    RESTARTS   AGE
deploy-vito-web-6779457c4c-6z5np   1/1     Running   0          5m59s
deploy-vito-web-6779457c4c-xpjjk   1/1     Running   0          5m59s
  • 测试
    • 编写循环测试脚本提升内存与 cpu 负载
    • while true; do wget -q -O- http://10.101.103:80 > /dev/null ; done
1
2
3
[zc@k8s-node1 ~]$ while true; do wget -q -O- http://10.106.0.34:80 > /dev/null ; done

[zc@k8s-node2 ~]$ while true; do wget -q -O- http://10.106.0.34:80 > /dev/null ; done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 查看 pods 资源使用情况
[root@k8s-master1 test]# kubectl top pods
NAME                               CPU(cores)   MEMORY(bytes)   
deploy-vito-web-6779457c4c-6z5np   25m          1Mi             
deploy-vito-web-6779457c4c-xpjjk   24m          1Mi             

[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   245%/20%   2         5         2          6m31s
[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   265%/20%   2         5         4          6m55s
[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   157%/20%   2         5         5          7m12s

[root@k8s-master1 test]# kubectl get po
NAME                               READY   STATUS    RESTARTS   AGE
deploy-vito-web-6779457c4c-6z5np   1/1     Running   0          10m
deploy-vito-web-6779457c4c-jqgg9   1/1     Running   0          53s
deploy-vito-web-6779457c4c-xcx6h   1/1     Running   0          38s
deploy-vito-web-6779457c4c-xpjjk   1/1     Running   0          10m
deploy-vito-web-6779457c4c-zgcb9   1/1     Running   0          53s
  • 关闭循环脚本,cpu 负载降低,自动缩容
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@k8s-master1 test]# kubectl top po
NAME                               CPU(cores)   MEMORY(bytes)   
deploy-vito-web-6779457c4c-6z5np   0m           1Mi             
deploy-vito-web-6779457c4c-jqgg9   0m           1Mi             
deploy-vito-web-6779457c4c-xcx6h   0m           1Mi             
deploy-vito-web-6779457c4c-xpjjk   0m           1Mi             
deploy-vito-web-6779457c4c-zgcb9   0m           1Mi             
[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   76%/20%   2         5         5          11m
[root@k8s-master1 test]# kubectl get hpa
NAME              REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
deploy-vito-web   Deployment/deploy-vito-web   0%/20%    2         5         2          16m
[root@k8s-master1 test]# kubectl get po
NAME                               READY   STATUS    RESTARTS   AGE
deploy-vito-web-6779457c4c-jqgg9   1/1     Running   0          10m
deploy-vito-web-6779457c4c-xpjjk   1/1     Running   0          20m

自定义 metrics

  • 控制管理器开启 –horizontal-pod-autoscaler-use-rest-clients(用来收集自定义指标)
  • 在自己的 app 中提供获取自定义指标的 API
  • 在 API Server Aggregator 中注册自定义的 metrics API


App ready for offline use.