k8s安装Prometheus
注:必須要先搭建網(wǎng)頁管理容器;
k8s部署dashboard_kali_yao的博客-CSDN博客
1.Prometheus的概述
Prometheus是一個(gè)最初在SoundCloud上構(gòu)建的開源監(jiān)控系統(tǒng) 。它現(xiàn)在是一個(gè)獨(dú)立的開源項(xiàng)目,為了強(qiáng)調(diào)這一點(diǎn),并說明項(xiàng)目的治理結(jié)構(gòu),Prometheus 于2016年加入CNCF,作為繼Kubernetes之后的第二個(gè)托管項(xiàng)目
現(xiàn)在最常見的k8s容器管理系統(tǒng)中,通常會搭配Prometheus進(jìn)行監(jiān)控,可以把他看成google BorgMon監(jiān)控的開源版本
prometheus的特點(diǎn)
-自定義多維度模型
-非常高效的存儲,平均一個(gè)采樣數(shù)據(jù)占~3.5bytes左右
-在多緯度上靈活且強(qiáng)大的查詢語言(PromQ)
-不依賴分布式存儲,支持單主節(jié)點(diǎn)工作通過基于HTTP的pull方式采集時(shí)序數(shù)據(jù)可以通過push gateway進(jìn)行時(shí)序列數(shù)據(jù)推送(pushing)可以通過服務(wù)發(fā)現(xiàn)或靜態(tài)配置去或取要采集的目標(biāo)服務(wù)器多種可視化圖表及儀表盤
鏡像及資源文件
網(wǎng)址:https://github.com/coreos/kube-prometheus
下載鏡像導(dǎo)入私有倉庫
prom/node-exporter v1.0.0 quay.io/coreos/prometheus-config-reloader v0.35.1 quay.io/coreos/prometheus-operator v0.35.1 quay.io/coreos/kube-state-metrics v1.9.2 grafana/grafana 6.4.3 jimmidyson/configmap-reload v0.3.0 quay.io/prometheus/prometheus v2.11.0 quay.io/prometheus/alertmanager v0.18.0 quay.io/coreos/k8s-prometheus-adapter-amd64 v0.5.0 quay.io/coreos/kube-rbac-proxy v0.4.1下載資源文件
# 從官方的地址獲,這里是release-0.4分支 ~]# git clone https://github.com/prometheus-operator/kube-prometheus.git # 默認(rèn)下載下來的文件較多,建議把文件進(jìn)行歸類處理,將相關(guān)yaml文件移動(dòng)到對應(yīng)目錄下 ~]# cd kube-prometheus/manifests ~]# mkdir -p grafana grafana-json metrics-state node-exporter prom-adapter prom-server setup # 整理如下 ~]# tree ./ ./ |-- alertmanager | ? |-- alertmanager-alertmanager.yaml | ? |-- alertmanager-secret.yaml | ? |-- alertmanager-serviceAccount.yaml | ? |-- alertmanager-serviceMonitor.yaml | ? `-- alertmanager-service.yaml |-- grafana | ? |-- grafana-dashboardDatasources.yaml | ? |-- grafana-dashboardDefinitions.yaml | ? |-- grafana-dashboardSources.yaml | ? |-- grafana-deployment.yaml | ? |-- grafana-serviceAccount.yaml | ? |-- grafana-serviceMonitor.yaml | ? `-- grafana-service.yaml |-- grafana-json | ? |-- kubernetes-for-prometheus-dashboard-cn-v20201010_rev3.json | ? `-- node-exporter-dashboard_rev1.json |-- metrics-state | ? |-- kube-state-metrics-clusterRoleBinding.yaml | ? |-- kube-state-metrics-clusterRole.yaml | ? |-- kube-state-metrics-deployment.yaml | ? |-- kube-state-metrics-roleBinding.yaml | ? |-- kube-state-metrics-role.yaml | ? |-- kube-state-metrics-serviceAccount.yaml | ? |-- kube-state-metrics-serviceMonitor.yaml | ? `-- kube-state-metrics-service.yaml |-- node-exporter | ? |-- node-exporter-clusterRoleBinding.yaml | ? |-- node-exporter-clusterRole.yaml | ? |-- node-exporter-daemonset.yaml | ? |-- node-exporter-serviceAccount.yaml | ? |-- node-exporter-serviceMonitor.yaml | ? `-- node-exporter-service.yaml |-- prom-adapter | ? |-- prometheus-adapter-apiService.yaml | ? |-- prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml | ? |-- prometheus-adapter-clusterRoleBindingDelegator.yaml | ? |-- prometheus-adapter-clusterRoleBinding.yaml | ? |-- prometheus-adapter-clusterRoleServerResources.yaml | ? |-- prometheus-adapter-clusterRole.yaml | ? |-- prometheus-adapter-configMap.yaml | ? |-- prometheus-adapter-deployment.yaml | ? |-- prometheus-adapter-roleBindingAuthReader.yaml | ? |-- prometheus-adapter-serviceAccount.yaml | ? `-- prometheus-adapter-service.yaml |-- prom-server | ? |-- prometheus-clusterRoleBinding.yaml | ? |-- prometheus-clusterRole.yaml | ? |-- prometheus-operator-serviceMonitor.yaml | ? |-- prometheus-prometheus.yaml | ? |-- prometheus-roleBindingConfig.yaml | ? |-- prometheus-roleBindingSpecificNamespaces.yaml | ? |-- prometheus-roleConfig.yaml | ? |-- prometheus-roleSpecificNamespaces.yaml | ? |-- prometheus-rules.yaml | ? |-- prometheus-serviceAccount.yaml | ? |-- prometheus-serviceMonitorApiserver.yaml | ? |-- prometheus-serviceMonitorCoreDNS.yaml | ? |-- prometheus-serviceMonitorKubeControllerManager.yaml | ? |-- prometheus-serviceMonitorKubelet.yaml | ? |-- prometheus-serviceMonitorKubeScheduler.yaml | ? |-- prometheus-serviceMonitor.yaml | ? `-- prometheus-service.yaml `-- setup|-- 0namespace-namespace.yaml|-- prometheus-operator-0alertmanagerCustomResourceDefinition.yaml|-- prometheus-operator-0podmonitorCustomResourceDefinition.yaml|-- prometheus-operator-0prometheusCustomResourceDefinition.yaml|-- prometheus-operator-0prometheusruleCustomResourceDefinition.yaml|-- prometheus-operator-0servicemonitorCustomResourceDefinition.yaml|-- prometheus-operator-clusterRoleBinding.yaml|-- prometheus-operator-clusterRole.yaml|-- prometheus-operator-deployment.yaml|-- prometheus-operator-serviceAccount.yaml`-- prometheus-operator-service.yaml架構(gòu)圖
數(shù)據(jù)庫:prometheus;其他的都是數(shù)據(jù)采集插件; grafana讀取數(shù)據(jù)并制成圖表展示;Alertmanager告警插件
所有監(jiān)控流程架構(gòu)圖
?2.安裝Prometheus
?1)下載并導(dǎo)入鏡像至私有倉庫
~]# for i in "prom/node-exporter v1.0.0 >quay.io/coreos/prometheus-config-reloader v0.35.1 >quay.io/coreos/prometheus-operator v0.35.1 >quay.io/coreos/kube-state-metrics v1.9.2 >grafana/grafana 6.4.3 >jimmidyson/configmap-reload v0.3.0 >quay.io/prometheus/prometheus v2.11.0 >quay.io/prometheus/alertmanager v0.18.0 >quay.io/coreos/k8s-prometheus-adapter-amd64 v0.5.0 >quay.io/coreos/kube-rbac-proxy v0.4.1" do docker pull $i done ]# img="prom/node-exporter v1.0.0 >quay.io/coreos/prometheus-config-reloader v0.35.1 >quay.io/coreos/prometheus-operator v0.35.1 >quay.io/coreos/kube-state-metrics v1.9.2 >grafana/grafana 6.4.3 >jimmidyson/configmap-reload v0.3.0 >quay.io/prometheus/prometheus v2.11.0 >quay.io/prometheus/alertmanager v0.18.0 >quay.io/coreos/k8s-prometheus-adapter-amd64 v0.5.0 >quay.io/coreos/kube-rbac-proxy v0.4.1" ~]# while read _f _v;do docker tag ${_f}:${_v} 172.17.0.98:5000/${_f##*/}:${_v}docker push 172.17.0.98:5000/${_f##*/}:${_v}docker rmi ${_f}:${_v} done <<<"${img}" # 測試查看 ~]# curl http://172.17.0.98:5000/v2/_catalog {"repositories":["alertmanager","configmap-reload","coredns","dashboard","etcd","flannel","grafana","k8s-prometheus-adapter-amd64","kube-apiserver","kube-controller-manager","kube-proxy","kube-rbac-proxy","kube-scheduler","kube-state-metrics","metrics-scraper","metrics-server","myos","nginx-ingress-controller","node-exporter","pause","prometheus","prometheus-config-reloader","prometheus-operator"]}2)先安裝核心數(shù)據(jù)庫的名稱空間
注:安裝核心數(shù)據(jù)庫之前要先安裝基礎(chǔ)環(huán)境(如名稱空間monitoring)
什么是Prometheus Operator
Prometheus Operator的本職就是一組用戶自定義的CRD資源以及Controller的實(shí)現(xiàn),Prometheus Operator負(fù)責(zé)監(jiān)聽這些自定義資源的變化,并且根據(jù)這些資源的定義自動(dòng)化的完成如Prometheus Server自身以及配置的自動(dòng)化管理工作。以下是Prometheus Operator的架構(gòu)圖
?
為什么用Prometheus Operator
由于Prometheus本身沒有提供管理配置的AP接口(尤其是管理監(jiān)控目標(biāo)和管理警報(bào)規(guī)則),也沒有提供好用的多實(shí)例管理手段,因此這一塊往往要自己寫一些代碼或腳本。為了簡化這類應(yīng)用程序的管理復(fù)雜度,CoreOS率先引入了Operator的概念,并且首先推出了針對在Kubernetes下運(yùn)行和管理Etcd的Etcd Operator。并隨后推出了Prometheus Operator
prometheus-operator官方地址:https://github.com/prometheus-operator/prometheus-operator kube-prometheus官方地址:https://github.com/prometheus-operator/kube-prometheus
兩個(gè)項(xiàng)目的關(guān)系:前者只包含了Prometheus Operator,后者既包含了Operator,又包含了Prometheus相關(guān)組件的部署及常用的Prometheus自定義監(jiān)控,具體包含下面的組件
## 先創(chuàng)建基礎(chǔ)環(huán)境 # 基礎(chǔ)環(huán)境運(yùn)用的鏡像(上面有導(dǎo)入) ~]# curl http://172.17.0.98:5000/v2/configmap-reload/tags/list {"name":"configmap-reload","tags":["v0.3.0"]} ~]# curl http://172.17.0.98:5000/v2/prometheus-config-reloader/tags/list {"name":"prometheus-config-reloader","tags":["v0.35.1"]} ~]# curl http://172.17.0.98:5000/v2/prometheus-operator/tags/list # 書寫資源文件 # 資源文件較多創(chuàng)建一個(gè)目錄存放 ~]# cd setup # 需要的文件 ~]# ls setup 0namespace-namespace.yaml prometheus-operator-0alertmanagerCustomResourceDefinition.yaml prometheus-operator-0podmonitorCustomResourceDefinition.yaml prometheus-operator-0prometheusCustomResourceDefinition.yaml prometheus-operator-0prometheusruleCustomResourceDefinition.yaml prometheus-operator-0servicemonitorCustomResourceDefinition.yaml prometheus-operator-clusterRoleBinding.yaml prometheus-operator-clusterRole.yaml prometheus-operator-deployment.yaml prometheus-operator-serviceAccount.yaml prometheus-operator-service.yaml ? # 只需要修改指定鏡像倉庫就可以了(190,274行) ~]# vim prometheus-operator-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/name: prometheus-operatorapp.kubernetes.io/version: v0.35.1name: prometheus-operatornamespace: monitoring spec:replicas: 1selector:matchLabels:app.kubernetes.io/component: controllerapp.kubernetes.io/name: prometheus-operatortemplate:metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/name: prometheus-operatorapp.kubernetes.io/version: v0.35.1spec:containers:- args:- --kubelet-service=kube-system/kubelet- --logtostderr=true- --config-reloader-image=172.17.0.98:5000/configmap-reload:v0.3.0 ?#(指定到本地參庫)- --prometheus-config-reloader=172.17.0.98:5000/prometheus-config-reloader:v0.35.1 ?#(指定到本地參庫)image: 172.17.0.98:5000/prometheus-operator:v0.35.1 ?#(指定到本地參庫)name: prometheus-operatorports:- containerPort: 8080name: httpresources:limits:cpu: 200mmemory: 200Mirequests:cpu: 100mmemory: 100MisecurityContext:allowPrivilegeEscalation: falsenodeSelector:beta.kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 65534serviceAccountName: prometheus-operator安裝
# 安裝并查看(k8s默認(rèn)會自動(dòng)順序安裝,只要指定目錄) ~]# kubectl apply -f setup/ ~]# kubectl get namespaces NAME ? ? ? ? ? ? ? ? ? STATUS ? AGE default ? ? ? ? ? ? ? Active ? 5d14h ingress-nginx ? ? ? ? Active ? 3d17h kube-node-lease ? ? ? Active ? 5d14h kube-public ? ? ? ? ? Active ? 5d14h kube-system ? ? ? ? ? Active ? 5d14h kubernetes-dashboard ? Active ? 130m monitoring ? ? ? ? ? ? Active ? 2m49s ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE prometheus-operator-75b4b59b74-72qhg ? 1/1 ? ? Running ? 0 ? ? ? ? 47s3)安裝數(shù)據(jù)庫Prometheus server
#對數(shù)據(jù)的監(jiān)控的獲取,存儲以及查詢 #Prometheus server用到的鏡像(上面有導(dǎo)入) ~]# curl http://172.17.0.98:5000/v2/prometheus/tags/list {"name":"prometheus","tags":["v2.11.0"]} ? ## 準(zhǔn)備資源文件 ~]# ls prom-server prometheus-clusterRoleBinding.yaml prometheus-clusterRole.yaml prometheus-operator-serviceMonitor.yaml prometheus-prometheus.yaml prometheus-roleBindingConfig.yaml prometheus-roleBindingSpecificNamespaces.yaml prometheus-roleConfig.yaml prometheus-roleSpecificNamespaces.yaml prometheus-rules.yaml prometheus-serviceAccount.yaml prometheus-serviceMonitorApiserver.yaml prometheus-serviceMonitorCoreDNS.yaml prometheus-serviceMonitorKubeControllerManager.yaml prometheus-serviceMonitorKubelet.yaml prometheus-serviceMonitorKubeScheduler.yaml prometheus-serviceMonitor.yaml prometheus-service.yaml # 只需要注意該文件的版本于鏡像即可(這里把鏡像的版本分開寫了) ~]# vim prom-server/prometheus-prometheus.yaml 14: ?baseImage: 172.17.0.98:5000/prometheus 34: ?version: v2.11.0 ~]# vim prometheus-prometheus.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata:labels:prometheus: k8sname: k8snamespace: monitoring spec:alerting:alertmanagers:- name: alertmanager-mainnamespace: monitoringport: webbaseImage: 172.17.0.98:5000/prometheus ?# 指定鏡像倉庫nodeSelector:kubernetes.io/os: linuxpodMonitorNamespaceSelector: {}podMonitorSelector: {}replicas: 2resources:requests:memory: 400MiruleSelector:matchLabels:prometheus: k8srole: alert-rulessecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.11.0 ? ?# 指定鏡像版本 # 安裝(以目錄開頭會自動(dòng)安裝) ~]# kubectl apply -f prom-server/ ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE prometheus-k8s-0 ? ? ? ? ? ? ? ? ? ? ? 3/3 ? ? Running ? 1 ? ? ? ? 45s prometheus-k8s-1 ? ? ? ? ? ? ? ? ? ? ? 3/3 ? ? Running ? 1 ? ? ? ? 45s3.安裝數(shù)據(jù)收集插件
注:以下3個(gè)插件主要收集(cpu,磁盤使用率,容器的狀態(tài)信息)
1)prom-adapter安裝
adapter
-獲取APIServer的資源指標(biāo)提供給Prom Server
adapter用到的鏡像(上面有傳)
~]# curl http://172.17.0.98:5000/v2/k8s-prometheus-adapter-amd64/tags/list {"name":"k8s-prometheus-adapter-amd64","tags":["v0.5.0"]} # 準(zhǔn)備鏡像(上面已經(jīng)上傳到私有倉庫了) ~]# curl http://172.17.0.98:5000/v2/k8s-prometheus-adapter-amd64/tags/list {"name":"k8s-prometheus-adapter-amd64","tags":["v0.5.0"]} # 準(zhǔn)備文件 ~]# ls prom-adapter/ prometheus-adapter-apiService.yaml prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml prometheus-adapter-clusterRoleBindingDelegator.yaml prometheus-adapter-clusterRoleBinding.yaml prometheus-adapter-clusterRoleServerResources.yaml prometheus-adapter-clusterRole.yaml prometheus-adapter-configMap.yaml prometheus-adapter-deployment.yaml prometheus-adapter-roleBindingAuthReader.yaml prometheus-adapter-serviceAccount.yaml prometheus-adapter-service.yaml # 只需要主要28行指定倉庫 ~]# vim prom-adapter/prometheus-adapter-deployment.yaml 28: ? ? ?image: 172.17.0.98:5000/k8s-prometheus-adapter-amd64:v0.5.0 ~]# prometheus-adapter-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata:name: prometheus-adapternamespace: monitoring spec:replicas: 1selector:matchLabels:name: prometheus-adapterstrategy:rollingUpdate:maxSurge: 1maxUnavailable: 0template:metadata:labels:name: prometheus-adapterspec:containers:- args:- --cert-dir=/var/run/serving-cert- --config=/etc/adapter/config.yaml- --logtostderr=true- --metrics-relist-interval=1m- --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/- --secure-port=6443image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0 ? # 指定鏡像name: prometheus-adapterports:- containerPort: 6443volumeMounts:- mountPath: /tmpname: tmpfsreadOnly: false- mountPath: /var/run/serving-certname: volume-serving-certreadOnly: false- mountPath: /etc/adaptername: configreadOnly: falsenodeSelector:kubernetes.io/os: linuxserviceAccountName: prometheus-adaptervolumes:- emptyDir: {}name: tmpfs- emptyDir: {}name: volume-serving-cert- configMap:name: adapter-configname: config # 部署于查看容器 ~]# kubectl apply -f prom-adapter ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE prometheus-adapter-856854f9f6-knqtq ? ?1/1 ? ? Running ? 0 ? ? ? ? 6s2)metrics-state安裝
-獲取各種資源的最新狀態(tài)(pod,deploy)
-metrics-state用到的鏡像
~]# curl http://172.17.0.98:5000/v2/kube-state-metrics/tags/list {"name":"kube-state-metrics","tags":["v1.9.2"]} ~]# curl http://172.17.0.98:5000/v2/kube-rbac-proxy/tags/list {"name":"kube-rbac-proxy","tags":["v0.4.1"]} 文件]# ls metrics-state/ kube-state-metrics-clusterRoleBinding.yaml kube-state-metrics-clusterRole.yaml kube-state-metrics-deployment.yaml kube-state-metrics-roleBinding.yaml kube-state-metrics-role.yaml kube-state-metrics-serviceAccount.yaml kube-state-metrics-serviceMonitor.yaml kube-state-metrics-service.yaml # 先查看鏡像是否上傳成功 ~]# curl http://172.17.0.98:5000/v2/kube-state-metrics/tags/list {"name":"kube-state-metrics","tags":["v1.9.2"]} ~]# curl http://172.17.0.98:5000/v2/kube-rbac-proxy/tags/list {"name":"kube-rbac-proxy","tags":["v0.4.1"]} # 只需要注意鏡像就可以了 ~]# vim metrics-state/kube-state-metrics-deployment.yaml 24: ? ? ? ?image: 172.17.0.98:5000/kube-rbac-proxy:v0.4.1 ?# 指定私有倉庫 41: ? ? ? ?image: 172.17.0.98:5000/kube-rbac-proxy:v0.4.1 # 指定私有倉庫 58: ? ? ? ?image: 172.17.0.98:5000/kube-state-metrics:v1.9.2 ?# 指定私有倉庫 ? ? ~]# cat metrics-state/kube-state-metrics-apiVersion: apps/v1 kind: Deployment metadata:labels:app: kube-state-metricsname: kube-state-metricsnamespace: monitoring spec:replicas: 1selector:matchLabels:app: kube-state-metricstemplate:metadata:labels:app: kube-state-metricsspec:containers:- args:- --logtostderr- --secure-listen-address=:8443- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256- --upstream=http://127.0.0.1:8081/image: quay.io/coreos/kube-rbac-proxy:v0.4.1 ?# 修該鏡像name: kube-rbac-proxy-mainports:- containerPort: 8443name: https-mainresources:limits:cpu: 20mmemory: 40Mirequests:cpu: 10mmemory: 20Mi- args:- --logtostderr- --secure-listen-address=:9443- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256- --upstream=http://127.0.0.1:8082/image: quay.io/coreos/kube-rbac-proxy:v0.4.1 ? # 修改鏡像倉庫name: kube-rbac-proxy-selfports:- containerPort: 9443name: https-selfresources:limits:cpu: 20mmemory: 40Mirequests:cpu: 10mmemory: 20Mi- args:- --host=127.0.0.1- --port=8081- --telemetry-host=127.0.0.1- --telemetry-port=8082image: quay.io/coreos/kube-state-metrics:v1.9.2 ? # 指定鏡像name: kube-state-metricsresources:limits:cpu: 100mmemory: 150Mirequests:cpu: 100mmemory: 150MinodeSelector:kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 65534serviceAccountName: kube-state-metrics # 創(chuàng)建資源與查看 ~]# kubectl apply -f metrics-state/ ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE kube-state-metrics-5894f64799-krvn6 ? ?3/3 ? ? Running ? 0 ? ? ? ? 4s3)node-exporter安裝
-采集node節(jié)點(diǎn)的數(shù)據(jù)提供給Prom Server
-node-exporter用到的鏡像
~]# curl http://172.17.0.98:5000/v2/node-exporter/tags/list {"name":"node-exporter","tags":["v1.0.0"]} ~]# curl http://172.17.0.98:5000/v2/kube-rbac-proxy/tags/list {"name":"kube-rbac-proxy","tags":["v0.4.1"]} 文件]# ls node-exporter/ node-exporter-clusterRoleBinding.yaml node-exporter-serviceAccount.yaml node-exporter-clusterRole.yaml ? ? ? ? node-exporter-serviceMonitor.yaml node-exporter-daemonset.yaml ? ? ? ? ? node-exporter-service.yaml # 鏡像準(zhǔn)備 ~]# curl http://172.17.0.98:5000/v2/node-exporter/tags/list {"name":"node-exporter","tags":["v1.0.0"]} ~]# curl http://172.17.0.98:5000/v2/kube-rbac-proxy/tags/list {"name":"kube-rbac-proxy","tags":["v0.4.1"]} # 注意修改鏡像 ~]# vim node-exporter/node-exporter-daemonset.yaml 27: ? ? ? ?image: 172.17.0.98:5000/node-exporter:v1.0.0 57: ? ? ? ?image: 172.17.0.98:5000/kube-rbac-proxy:v0.4.1 ? ~]# vim node-exporter/node-exporter- apiVersion: apps/v1 kind: DaemonSet metadata:labels:app: node-exportername: node-exporternamespace: monitoring spec:selector:matchLabels:app: node-exportertemplate:metadata:labels:app: node-exporterspec:containers:- args:- --web.listen-address=127.0.0.1:9100- --path.procfs=/host/proc- --path.sysfs=/host/sys- --path.rootfs=/host/root- --no-collector.wifi- --no-collector.hwmon- --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)- --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$image: quay.io/prometheus/node-exporter:v1.0.0 ?# 指定鏡像name: node-exporterresources:limits:cpu: 250mmemory: 180Mirequests:cpu: 102mmemory: 180MivolumeMounts:- mountPath: /host/procname: procreadOnly: false- mountPath: /host/sysname: sysreadOnly: false- mountPath: /host/rootmountPropagation: HostToContainername: rootreadOnly: true- args:- --logtostderr- --secure-listen-address=[$(IP)]:9100- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256- --upstream=http://127.0.0.1:9100/env:- name: IPvalueFrom:fieldRef:fieldPath: status.podIPimage: quay.io/coreos/kube-rbac-proxy:v0.4.1 ? # 指定鏡像name: kube-rbac-proxyports:- containerPort: 9100hostPort: 9100name: httpsresources:limits:cpu: 20mmemory: 40Mirequests:cpu: 10mmemory: 20MihostNetwork: truehostPID: truenodeSelector:kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 65534serviceAccountName: node-exportertolerations:- operator: Existsvolumes:- hostPath:path: /procname: proc- hostPath:path: /sysname: sys- hostPath:path: /name: root # 添加與查看 ~]# kubectl apply -f node-exporter/ ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE node-exporter-7h4l9 ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 7s node-exporter-7vxmx ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 7s node-exporter-mr6lw ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 7s node-exporter-zg2j8 ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 7s4.安裝告警插件
1)alertmanager安裝
-Prometheus體系中的告警處理中心
-alertmanager用到的鏡像
~]# curl http://172.17.0.98:5000/v2/alertmanager/tags/list {"name":"alertmanager","tags":["v0.18.0"]} 文件~]# ls alertmanager/ alertmanager-alertmanager.yaml ? alertmanager-serviceMonitor.yaml alertmanager-secret.yaml ? ? ? ? alertmanager-service.yaml alertmanager-serviceAccount.yaml # 查看鏡像 ~]# curl http://172.17.0.98:5000/v2/alertmanager/tags/list {"name":"alertmanager","tags":["v0.18.0"]} # 只需要修改指定鏡像 ~]# vim alertmanager/alertmanager-alertmanager.yaml 09: baseImage: 172.17.0.98:5000/alertmanager 18: version: v0.18.0 ? ~]# vim alertmanager-alertmanager.yaml apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata:labels:alertmanager: mainname: mainnamespace: monitoring spec:baseImage: quay.io/prometheus/alertmanagernodeSelector:kubernetes.io/os: linuxreplicas: 3securityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: alertmanager-mainversion: v0.18.0 # 創(chuàng)建資源與查看 ~]# kubectl apply -f alertmanager/ ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE alertmanager-main-0 ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 16s alertmanager-main-1 ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 16s alertmanager-main-2 ? ? ? ? ? ? ? ? ? ?2/2 ? ? Running ? 0 ? ? ? ? 16s5.按裝展示插件
grafana安裝
-支持多種圖形和Dashboard的展示
-grafana用到的鏡像
~]# curl http://172.17.0.98:5000/v2/grafana/tags/list {"name":"grafana","tags":["6.4.3"]} ? ? 文件~]# ls grafana grafana-dashboardDatasources.yaml grafana-serviceAccount.yaml grafana-dashboardDefinitions.yaml grafana-serviceMonitor.yaml grafana-dashboardSources.yaml ? ? grafana-service.yaml grafana-deployment.yaml # 查看鏡像 ~]# curl http://172.17.0.98:5000/v2/grafana/tags/list {"name":"grafana","tags":["6.4.3"]} # 只需要修改鏡像 ~]# vim grafana/grafana-deployment.yaml 19: ? ? - image: 172.17.0.98:5000/grafana:6.4.3 ~]# vim grafana/grafana-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata:labels:app: grafananame: grafananamespace: monitoring spec:replicas: 1selector:matchLabels:app: grafanatemplate:metadata:labels:app: grafanaspec:containers:- image: grafana/grafana:6.4.3 ? # 指定鏡像name: grafanaports:- containerPort: 3000name: httpreadinessProbe:httpGet:path: /api/healthport: httpresources:limits:cpu: 200mmemory: 200Mirequests:cpu: 100mmemory: 100MivolumeMounts:- mountPath: /var/lib/grafananame: grafana-storagereadOnly: false- mountPath: /etc/grafana/provisioning/datasourcesname: grafana-datasourcesreadOnly: false- mountPath: /etc/grafana/provisioning/dashboardsname: grafana-dashboardsreadOnly: false- mountPath: /grafana-dashboard-definitions/0/apiservername: grafana-dashboard-apiserverreadOnly: false- mountPath: /grafana-dashboard-definitions/0/cluster-totalname: grafana-dashboard-cluster-totalreadOnly: false- mountPath: /grafana-dashboard-definitions/0/controller-managername: grafana-dashboard-controller-managerreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-clustername: grafana-dashboard-k8s-resources-clusterreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-namespacename: grafana-dashboard-k8s-resources-namespacereadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-nodename: grafana-dashboard-k8s-resources-nodereadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-podname: grafana-dashboard-k8s-resources-podreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloadname: grafana-dashboard-k8s-resources-workloadreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloads-namespacename: grafana-dashboard-k8s-resources-workloads-namespacereadOnly: false- mountPath: /grafana-dashboard-definitions/0/kubeletname: grafana-dashboard-kubeletreadOnly: false- mountPath: /grafana-dashboard-definitions/0/namespace-by-podname: grafana-dashboard-namespace-by-podreadOnly: false- mountPath: /grafana-dashboard-definitions/0/namespace-by-workloadname: grafana-dashboard-namespace-by-workloadreadOnly: false- mountPath: /grafana-dashboard-definitions/0/node-cluster-rsrc-usename: grafana-dashboard-node-cluster-rsrc-usereadOnly: false- mountPath: /grafana-dashboard-definitions/0/node-rsrc-usename: grafana-dashboard-node-rsrc-usereadOnly: false- mountPath: /grafana-dashboard-definitions/0/nodesname: grafana-dashboard-nodesreadOnly: false- mountPath: /grafana-dashboard-definitions/0/persistentvolumesusagename: grafana-dashboard-persistentvolumesusagereadOnly: false- mountPath: /grafana-dashboard-definitions/0/pod-totalname: grafana-dashboard-pod-totalreadOnly: false- mountPath: /grafana-dashboard-definitions/0/podsname: grafana-dashboard-podsreadOnly: false- mountPath: /grafana-dashboard-definitions/0/prometheus-remote-writename: grafana-dashboard-prometheus-remote-writereadOnly: false- mountPath: /grafana-dashboard-definitions/0/prometheusname: grafana-dashboard-prometheusreadOnly: false- mountPath: /grafana-dashboard-definitions/0/proxyname: grafana-dashboard-proxyreadOnly: false- mountPath: /grafana-dashboard-definitions/0/schedulername: grafana-dashboard-schedulerreadOnly: false- mountPath: /grafana-dashboard-definitions/0/statefulsetname: grafana-dashboard-statefulsetreadOnly: false- mountPath: /grafana-dashboard-definitions/0/workload-totalname: grafana-dashboard-workload-totalreadOnly: falsenodeSelector:beta.kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 65534serviceAccountName: grafanavolumes:- emptyDir: {}name: grafana-storage- name: grafana-datasourcessecret:secretName: grafana-datasources- configMap:name: grafana-dashboardsname: grafana-dashboards- configMap:name: grafana-dashboard-apiservername: grafana-dashboard-apiserver- configMap:name: grafana-dashboard-cluster-totalname: grafana-dashboard-cluster-total- configMap:name: grafana-dashboard-controller-managername: grafana-dashboard-controller-manager- configMap:name: grafana-dashboard-k8s-resources-clustername: grafana-dashboard-k8s-resources-cluster- configMap:name: grafana-dashboard-k8s-resources-namespacename: grafana-dashboard-k8s-resources-namespace- configMap:name: grafana-dashboard-k8s-resources-nodename: grafana-dashboard-k8s-resources-node- configMap:name: grafana-dashboard-k8s-resources-podname: grafana-dashboard-k8s-resources-pod- configMap:name: grafana-dashboard-k8s-resources-workloadname: grafana-dashboard-k8s-resources-workload- configMap:name: grafana-dashboard-k8s-resources-workloads-namespacename: grafana-dashboard-k8s-resources-workloads-namespace- configMap:name: grafana-dashboard-kubeletname: grafana-dashboard-kubelet- configMap:name: grafana-dashboard-namespace-by-podname: grafana-dashboard-namespace-by-pod- configMap:name: grafana-dashboard-namespace-by-workloadname: grafana-dashboard-namespace-by-workload- configMap:name: grafana-dashboard-node-cluster-rsrc-usename: grafana-dashboard-node-cluster-rsrc-use- configMap:name: grafana-dashboard-node-rsrc-usename: grafana-dashboard-node-rsrc-use- configMap:name: grafana-dashboard-nodesname: grafana-dashboard-nodes- configMap:name: grafana-dashboard-persistentvolumesusagename: grafana-dashboard-persistentvolumesusage- configMap:name: grafana-dashboard-pod-totalname: grafana-dashboard-pod-total- configMap:name: grafana-dashboard-podsname: grafana-dashboard-pods- configMap:name: grafana-dashboard-prometheus-remote-writename: grafana-dashboard-prometheus-remote-write- configMap:name: grafana-dashboard-prometheusname: grafana-dashboard-prometheus- configMap:name: grafana-dashboard-proxyname: grafana-dashboard-proxy- configMap:name: grafana-dashboard-schedulername: grafana-dashboard-scheduler- configMap:name: grafana-dashboard-statefulsetname: grafana-dashboard-statefulset- configMap:name: grafana-dashboard-workload-totalname: grafana-dashboard-workload-total # 安裝與查看 ~]# kubectl apply -f grafana/ ~]# kubectl -n monitoring get pod NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE grafana-647d948b69-d2hv9 ? ? ? ? ? ? ? 1/1 ? ? Running ? 0 ? ? ? ? 19s6.發(fā)布服務(wù)
grafana默認(rèn)的服務(wù)使用Cluster IP
-使用nodePort發(fā)布服務(wù)
# 使用etid在線直接改 ~]# kubctl etid svc grafana -o yaml ? # 直接修改資源文件然后更新 ~]# cp grafana/grafana-service.yaml ./ ~]# vim grafana-service.yaml apiVersion: v1 kind: Service metadata:labels:app: grafananame: grafananamespace: monitoring spec:type: NodePort ? ? ? ? ?# 新添加NodePortports:- name: httpport: 3000nodePort: 30000 ? ? ? # 新添加映射端口targetPort: httpselector:app: grafana ~]# kubectl apply -f grafana-service.yaml ~]# kubectl -n monitoring get serviceNAME ? ? ? ? ? ? ? ? ? TYPE ? ? ? CLUSTER-IP ? ? ? EXTERNAL-IP ? PORT(S)grafana ? ? ? ? ? ? ? ? NodePort ? ?10.254.79.49 ? ? <none> ? ? ? ?3000:30000/TCP # 服務(wù)發(fā)布以后可以通過云彈性公網(wǎng)IP直接訪問即可grafana 第一次默認(rèn)登錄的用戶名/密碼(admin/admin),登錄之后會強(qiáng)制修改密碼
7.圖形操作
1)綁定數(shù)據(jù)庫
# 先查看數(shù)據(jù)庫服務(wù),因?yàn)槿绻胮od的話只要重啟就會找不到 ~]# kubectl get service -n monitoring ....... prometheus-k8s ClusterIP 10.254.192.100 <none> ? ?9090/TCP 3h4m ~]# curl http://prometheus-k8s:90902) 收集傳輸數(shù)據(jù)地址
名字隨意(這里是prometheus),但是后續(xù)的圖表導(dǎo)入要填寫這個(gè)
URL填寫內(nèi)部DNS名稱如上(kubectl get service -n monitoring)
端口默認(rèn):9090
注:這里必須填寫該服務(wù)域名
?
?prometheus的內(nèi)部訪問是順序
用戶訪問開放端口30000,然后映射到內(nèi)部grafana的3000;grafana容器再問prometheus(在容器內(nèi)部就可以用域名訪問了)
?默認(rèn)有展示頁面
3)導(dǎo)入下載好的頁面展示圖
官網(wǎng)下載地址:儀表板|格拉法納實(shí)驗(yàn)室 (grafana.com)
注:導(dǎo)入的文件就是如下任意一個(gè)
~]# ls grafana-json kubernetes-for-prometheus-dashboard-cn-v20201010_rev3.json node-exporter-dashboard_rev1.json點(diǎn)擊這個(gè)位置,輸入模板ID
導(dǎo)入文件,數(shù)據(jù)源就是剛剛定義的prometheus1
?4)調(diào)整展示時(shí)間范圍
默認(rèn)十二小時(shí)(調(diào)成1小時(shí))
總結(jié)
以上是生活随笔為你收集整理的k8s安装Prometheus的全部內(nèi)容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: kuberbetes的架构与kubern
- 下一篇: HPA集群控制器