HPA
-
通过观察 Pod 的
cpu
、内存
使用率或自定义 metrics 指标
进行自动的扩容或缩容 pod 的数量。 -
通常用于 Deployment,不适用于无法扩/缩容的对象,如 DaemonSet
-
控制管理器每隔30s(可以通过
–horizontal-pod-autoscaler-sync-period
修改)查询 metrics 的资源使用情况
开启指标服务(便于观察)
- 下载 metrics-server 组件配置文件
1
2
[root@k8s-master1 ~]# cd /opt/kubernetes/
[root@k8s-master1 kubernetes]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server-components.yaml
- 修改镜像地址为国内的地址
1
2
3
4
5
6
7
8
9
[root@k8s-master1 kubernetes]# grep image metrics-server-components.yaml
image: registry.k8s.io/metrics-server/metrics-server:v0.6.4
imagePullPolicy: IfNotPresent
[root@k8s-master1 kubernetes]# sed -i 's/registry.k8s.io\/metrics-server/registry.cn-hangzhou.aliyuncs.com\/google_containers/g' metrics-server-components.yaml
[root@k8s-master1 kubernetes]# grep image metrics-server-components.yaml
image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.6.4
imagePullPolicy: IfNotPresent
- 修改容器的 tls 配置,不验证 tls
- 在 containers 的 args 参数中增加
--kubelet-insecure-tls
参数
- 在 containers 的 args 参数中增加
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@k8s-master1 kubernetes]# vim metrics-server-components.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
- 安装组件
1
2
3
4
5
6
7
8
9
10
11
12
13
[root@k8s-master1 kubernetes]# kubectl apply -f metrics-server-components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
[root@k8s-master1 kubernetes]# kubectl get po --all-namespaces | grep metrics
kube-system metrics-server-b8b756d46-qj7m2 1/1 Running 0 42s
CPU、内存指标监控
- 实现 cpu 或内存的监控,首先有个前提条件是该对象必须配置了
resources.requests.cpu
或resources.requests.memory
才可以,可以配置当 cpu/memory 达到上述配置的百分比后进行扩容或缩容 - 准备好一个有资源限制的 Deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[root@k8s-master1 ~]# cd test
[root@k8s-master1 test]# vim deploy-vito-web.yaml
---
apiVersion: v1
kind: Service
metadata:
name: svc-vito-web # Service 的名字
spec:
selector: # 将服务流量路由到具有与此选择器匹配的 Pod
app: deploy-vito-nginx
type: NodePort
ports:
- port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-vito-web
spec:
replicas: 2 # 期望副本数
selector:
matchLabels:
app: deploy-vito-nginx
template:
metadata:
labels:
app: deploy-vito-nginx
spec:
containers:
- image: nginx:1.7.9
imagePullPolicy: IfNotPresent
name: deploy-vito-web-c-nginx
resources:
requests: # 最少需要多少资源
cpu: 10m # 限制 cpu 最少使用 0.01 个核心
memory: 128Mi # 限制内存最少使用 128 兆
limits: # 最多可以用多少资源
cpu: 200m # 限制 cpu 最多使用 0.2 个核心
memory: 256Mi # 限制内存最多使用 256 兆
[root@k8s-master1 test]# kubectl create -f deploy-vito-web.yaml
service/svc-vito-web created
deployment.apps/deploy-vito-web created
[root@k8s-master1 test]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
deploy-vito-web 2/2 2 2 36s
[root@k8s-master1 test]# kubectl get rs
NAME DESIRED CURRENT READY AGE
deploy-vito-web-5f74dfcf97 2 2 2 39s
[root@k8s-master1 test]# kubectl get po
NAME READY STATUS RESTARTS AGE
deploy-vito-web-5f74dfcf97-46lpj 1/1 Running 0 46s
deploy-vito-web-5f74dfcf97-6crbg 1/1 Running 0 46s
[root@k8s-master1 test]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d18h
svc-vito-web NodePort 10.106.0.34 <none> 80:30686/TCP 52s
- 执行命令
kubectl autoscale deploy deploy-vito-web --cpu-percent=20 --min=2 --max=5
- 通过
kubectl get hpa
可以获取 HPA 信息
- 通过
1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master1 test]# kubectl autoscale deploy deploy-vito-web --cpu-percent=20 --min=2 --max=5
horizontalpodautoscaler.autoscaling/deploy-vito-web autoscaled
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 0%/20% 2 5 2 20s
[root@k8s-master1 test]# kubectl get po
NAME READY STATUS RESTARTS AGE
deploy-vito-web-6779457c4c-6z5np 1/1 Running 0 5m59s
deploy-vito-web-6779457c4c-xpjjk 1/1 Running 0 5m59s
- 测试
- 编写循环测试脚本提升内存与 cpu 负载
while true; do wget -q -O- http://10.101.103:80 > /dev/null ; done
1
2
3
[zc@k8s-node1 ~]$ while true; do wget -q -O- http://10.106.0.34:80 > /dev/null ; done
[zc@k8s-node2 ~]$ while true; do wget -q -O- http://10.106.0.34:80 > /dev/null ; done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 查看 pods 资源使用情况
[root@k8s-master1 test]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
deploy-vito-web-6779457c4c-6z5np 25m 1Mi
deploy-vito-web-6779457c4c-xpjjk 24m 1Mi
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 245%/20% 2 5 2 6m31s
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 265%/20% 2 5 4 6m55s
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 157%/20% 2 5 5 7m12s
[root@k8s-master1 test]# kubectl get po
NAME READY STATUS RESTARTS AGE
deploy-vito-web-6779457c4c-6z5np 1/1 Running 0 10m
deploy-vito-web-6779457c4c-jqgg9 1/1 Running 0 53s
deploy-vito-web-6779457c4c-xcx6h 1/1 Running 0 38s
deploy-vito-web-6779457c4c-xpjjk 1/1 Running 0 10m
deploy-vito-web-6779457c4c-zgcb9 1/1 Running 0 53s
- 关闭循环脚本,cpu 负载降低,自动缩容
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@k8s-master1 test]# kubectl top po
NAME CPU(cores) MEMORY(bytes)
deploy-vito-web-6779457c4c-6z5np 0m 1Mi
deploy-vito-web-6779457c4c-jqgg9 0m 1Mi
deploy-vito-web-6779457c4c-xcx6h 0m 1Mi
deploy-vito-web-6779457c4c-xpjjk 0m 1Mi
deploy-vito-web-6779457c4c-zgcb9 0m 1Mi
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 76%/20% 2 5 5 11m
[root@k8s-master1 test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
deploy-vito-web Deployment/deploy-vito-web 0%/20% 2 5 2 16m
[root@k8s-master1 test]# kubectl get po
NAME READY STATUS RESTARTS AGE
deploy-vito-web-6779457c4c-jqgg9 1/1 Running 0 10m
deploy-vito-web-6779457c4c-xpjjk 1/1 Running 0 20m
自定义 metrics
- 控制管理器开启
–horizontal-pod-autoscaler-use-rest-clients
(用来收集自定义指标) - 在自己的 app 中提供获取自定义指标的 API
- 在 API Server Aggregator 中注册自定义的 metrics API