簡(jiǎn)介：應(yīng)用級(jí)擴(kuò)縮容是相對(duì)于運(yùn)維級(jí)而言的。像監(jiān)控CPU/內(nèi)存的利用率就屬于應(yīng)用無(wú)關(guān)的純運(yùn)維指標(biāo)，針對(duì)這種指標(biāo)進(jìn)行擴(kuò)縮容的HPA配置就是運(yùn)維級(jí)擴(kuò)縮容。而像請(qǐng)求數(shù)量、請(qǐng)求延遲、P99分布等指標(biāo)就屬于應(yīng)用相關(guān)的，或者叫業(yè)務(wù)感知的監(jiān)控指標(biāo)。 本篇將介紹3種應(yīng)用級(jí)監(jiān)控指標(biāo)在HPA中的配置，以實(shí)現(xiàn)應(yīng)用級(jí)自動(dòng)擴(kuò)縮容。

應(yīng)用級(jí)擴(kuò)縮容是相對(duì)于運(yùn)維級(jí)而言的。像監(jiān)控CPU/內(nèi)存的利用率就屬于應(yīng)用無(wú)關(guān)的純運(yùn)維指標(biāo)，針對(duì)這種指標(biāo)進(jìn)行擴(kuò)縮容的HPA配置就是運(yùn)維級(jí)擴(kuò)縮容。而像請(qǐng)求數(shù)量、請(qǐng)求延遲、P99分布等指標(biāo)就屬于應(yīng)用相關(guān)的，或者叫業(yè)務(wù)感知的監(jiān)控指標(biāo)。

本篇將介紹3種應(yīng)用級(jí)監(jiān)控指標(biāo)在HPA中的配置，以實(shí)現(xiàn)應(yīng)用級(jí)自動(dòng)擴(kuò)縮容。

Setup HPA

1 部署metrics-adapter

執(zhí)行如下命令部署kube-metrics-adapter(完整腳本參見(jiàn)：demo_hpa.sh)。：

helm --kubeconfig "$USER_CONFIG" -n kube-system install asm-custom-metrics \$KUBE_METRICS_ADAPTER_SRC/deploy/charts/kube-metrics-adapter \--set prometheus.url=http://prometheus.istio-system.svc:9090

執(zhí)行如下命令驗(yàn)證部署情況：

#驗(yàn)證POD kubectl --kubeconfig "$USER_CONFIG" get po -n kube-system | grep metrics-adapterasm-custom-metrics-kube-metrics-adapter-6fb4949988-ht8pv 1/1 Running 0 30s#驗(yàn)證CRD kubectl --kubeconfig "$USER_CONFIG" api-versions | grep "autoscaling/v2beta"autoscaling/v2beta1 autoscaling/v2beta2#驗(yàn)證CRD kubectl --kubeconfig "$USER_CONFIG" get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .{"kind": "APIResourceList","apiVersion": "v1","groupVersion": "external.metrics.k8s.io/v1beta1","resources": [] }

2 部署loadtester

執(zhí)行如下命令部署flagger loadtester：

kubectl --kubeconfig "$USER_CONFIG" apply -f $FLAAGER_SRC/kustomize/tester/deployment.yaml -n test kubectl --kubeconfig "$USER_CONFIG" apply -f $FLAAGER_SRC/kustomize/tester/service.yaml -n test

3 部署HPA

3.1 根據(jù)應(yīng)用請(qǐng)求數(shù)量擴(kuò)縮容

首先我們創(chuàng)建一個(gè)感知應(yīng)用請(qǐng)求數(shù)量(istio_requests_total)的HorizontalPodAutoscaler配置：

apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata:name: podinfo-totalnamespace: testannotations:metric-config.external.prometheus-query.prometheus/processed-requests-per-second: |sum(rate(istio_requests_total{destination_workload_namespace="test",reporter="destination"}[1m])) spec:maxReplicas: 5minReplicas: 1scaleTargetRef:apiVersion: apps/v1kind: Deploymentname: podinfometrics:- type: Externalexternal:metric:name: prometheus-queryselector:matchLabels:query-name: processed-requests-per-secondtarget:type: AverageValueaverageValue: "10"

執(zhí)行如下命令部署這個(gè)HPA配置：

kubectl --kubeconfig "$USER_CONFIG" apply -f resources_hpa/requests_total_hpa.yaml

執(zhí)行如下命令校驗(yàn)：

kubectl --kubeconfig "$USER_CONFIG" get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

結(jié)果如下：

{"kind": "APIResourceList","apiVersion": "v1","groupVersion": "external.metrics.k8s.io/v1beta1","resources": [{"name": "prometheus-query","singularName": "","namespaced": true,"kind": "ExternalMetricValueList","verbs": ["get"]}] }

類(lèi)似地，我們可以使用其他維度的應(yīng)用級(jí)監(jiān)控指標(biāo)配置HPA。舉例如下，不再冗述。

3.2 根據(jù)平均延遲擴(kuò)縮容

apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata:name: podinfo-latency-avgnamespace: testannotations:metric-config.external.prometheus-query.prometheus/latency-average: |sum(rate(istio_request_duration_milliseconds_sum{destination_workload_namespace="test",reporter="destination"}[1m]))/sum(rate(istio_request_duration_milliseconds_count{destination_workload_namespace="test",reporter="destination"}[1m])) spec:maxReplicas: 5minReplicas: 1scaleTargetRef:apiVersion: apps/v1kind: Deploymentname: podinfometrics:- type: Externalexternal:metric:name: prometheus-queryselector:matchLabels:query-name: latency-averagetarget:type: AverageValueaverageValue: "0.005"

3.3 根據(jù)P95分布擴(kuò)縮容

apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata:name: podinfo-p95namespace: testannotations:metric-config.external.prometheus-query.prometheus/p95-latency: |histogram_quantile(0.95,sum(irate(istio_request_duration_milliseconds_bucket{destination_workload_namespace="test",destination_canonical_service="podinfo"}[5m]))by (le)) spec:maxReplicas: 5minReplicas: 1scaleTargetRef:apiVersion: apps/v1kind: Deploymentname: podinfometrics:- type: Externalexternal:metric:name: prometheus-queryselector:matchLabels:query-name: p95-latencytarget:type: AverageValueaverageValue: "4"

驗(yàn)證HPA

1 生成負(fù)載

執(zhí)行如下命令產(chǎn)生實(shí)驗(yàn)流量，以驗(yàn)證HPA配置自動(dòng)擴(kuò)容生效。

alias k="kubectl --kubeconfig $USER_CONFIG" loadtester=$(k -n test get pod -l "app=flagger-loadtester" -o jsonpath='{.items..metadata.name}') k -n test exec -it ${loadtester} -c loadtester -- hey -z 5m -c 2 -q 10 http://podinfo:9898

這里運(yùn)行了一個(gè)持續(xù)5分鐘、QPS=10、并發(fā)數(shù)為2的請(qǐng)求。

hey命令詳細(xì)參考如下：

Usage: hey [options...] <url>Options:-n Number of requests to run. Default is 200.-c Number of workers to run concurrently. Total number of requests cannotbe smaller than the concurrency level. Default is 50.-q Rate limit, in queries per second (QPS) per worker. Default is no rate limit.-z Duration of application to send requests. When duration is reached,application stops and exits. If duration is specified, n is ignored.Examples: -z 10s -z 3m.-o Output type. If none provided, a summary is printed."csv" is the only supported alternative. Dumps the responsemetrics in comma-separated values format.-m HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS.-H Custom HTTP header. You can specify as many as needed by repeating the flag.For example, -H "Accept: text/html" -H "Content-Type: application/xml" .-t Timeout for each request in seconds. Default is 20, use 0 for infinite.-A HTTP Accept header.-d HTTP request body.-D HTTP request body from file. For example, /home/user/file.txt or ./file.txt.-T Content-type, defaults to "text/html".-a Basic authentication, username:password.-x HTTP Proxy address as host:port.-h2 Enable HTTP/2.-host HTTP Host header.-disable-compression Disable compression.-disable-keepalive Disable keep-alive, prevents re-use of TCPconnections between different HTTP requests.-disable-redirects Disable following of HTTP redirects-cpus Number of used cpu cores.(default for current machine is 4 cores)

2 自動(dòng)擴(kuò)容

執(zhí)行如下命令觀察擴(kuò)容情況：

watch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-total

結(jié)果如下：

Every 2.0s: kubectl --kubeconfig /Users/han/shop_config/ack_zjk -n test get hpa/podinfo East6C16G: Tue Jan 26 18:01:30 2021NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 10056m/10 (avg) 1 5 2 4m45s

另外兩個(gè)HPA類(lèi)似，命令如下：

kubectl --kubeconfig $USER_CONFIG -n test get hpawatch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-latency-avg watch kubectl --kubeconfig $USER_CONFIG -n test get hpa/podinfo-p95

3 監(jiān)控指標(biāo)

同時(shí)，我們可以實(shí)時(shí)在Prometheus中查看相關(guān)的應(yīng)用級(jí)監(jiān)控指標(biāo)的實(shí)時(shí)數(shù)據(jù)。示意如下：

原文鏈接：https://developer.aliyun.com/article/783551?

版權(quán)聲明：本文內(nèi)容由阿里云實(shí)名注冊(cè)用戶(hù)自發(fā)貢獻(xiàn)，版權(quán)歸原作者所有，阿里云開(kāi)發(fā)者社區(qū)不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。具體規(guī)則請(qǐng)查看《阿里云開(kāi)發(fā)者社區(qū)用戶(hù)服務(wù)協(xié)議》和《阿里云開(kāi)發(fā)者社區(qū)知識(shí)產(chǎn)權(quán)保護(hù)指引》。如果您發(fā)現(xiàn)本社區(qū)中有涉嫌抄襲的內(nèi)容，填寫(xiě)侵權(quán)投訴表單進(jìn)行舉報(bào)，一經(jīng)查實(shí)，本社區(qū)將立刻刪除涉嫌侵權(quán)內(nèi)容。

總結(jié)

以上是生活随笔為你收集整理的Flagger on ASM——基于Mixerless Telemetry实现渐进式灰度发布系列 2 应用级扩缩容的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。