某次cloudera-scm-agnet系統啟動不了

前幾天添加磁盤的時候系統以外重啟了，那個時候并沒stop cloudera-scm-server和cloudera-scm-agent兩個進程，導致了重新啟動的時候cloudera-scm-server可以啟動而cloudera-scm-agent無法啟動。
使用命令: systemctl status cloudera-scm-agent得到的反饋

● cloudera-scm-agent.service - LSB: Cloudera SCM AgentLoaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)Active: failed (Result: exit-code) since Thu 2019-08-22 22:52:37 CST; 10h agoDocs: man:systemd-sysv-generator(8)Process: 16911 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)Process: 18991 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=1/FAILURE)Aug 22 22:52:37 test cloudera-scm-agent[18991]: install: cannot create directory ‘/var/run’: File exists Aug 22 22:52:37 test su[19009]: (to root) root on none Aug 22 22:52:37 test su[19009]: pam_systemd(su:session): Failed to connect to system bus: No such file or directory Aug 22 22:52:37 test su[19009]: pam_unix(su:session): session opened for user root by (uid=0) Aug 22 22:52:37 test su[19009]: pam_unix(su:session): session closed for user root Aug 22 22:52:37 test cloudera-scm-agent[18991]: Starting cloudera-scm-agent: [FAILED] Aug 22 22:52:37 test systemd[1]: cloudera-scm-agent.service: control process exited, code=exited status=1 Aug 22 22:52:37 test systemd[1]: Failed to start LSB: Cloudera SCM Agent. Aug 22 22:52:37 test systemd[1]: Unit cloudera-scm-agent.service entered failed state. Aug 22 22:52:37 test systemd[1]: cloudera-scm-agent.service failed.

查看日志

一開始看著這一堆問題的時候完全不知道發生了什么事情，決定去看日志
使用命令:tail -1000f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

Traceback (most recent call last):File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _sendself._port)File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__self.conn.connect()File "/usr/lib64/python2.7/httplib.py", line 824, in connectself.timeout, self.source_address)File "/usr/lib64/python2.7/socket.py", line 571, in create_connectionraise err error: [Errno 111] Connection refused

tm還是沒有看出些什么來，然后又看另外一個日志
使用命令:tail -100f /var/log/cloudera-scm-agent/cloudera-scm-agent.out

[root@test cloudera-scm-agent]# tail -10000f cloudera-scm-agent.out [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent INFO SCM Agent Version: 5.14.0 [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent INFO SCM Agent Version: 5.14.0 [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent Unable to create the pidfile. [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent Unable to create the pidfile. [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent INFO SCM Agent Version: 5.14.0 [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent WARNING Expected mode 0751 for /var/run/cloudera-scm-agent but was 0755 [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent

現在有一些線索了，這里有這么一句Unable to create the pidfile，然后去百度了一下大部分的博客都是說在/run目錄下沒有cloudera-scm-agent這個目錄，只要創建一下就有了，然而我這里卻是有的。
完了，現在線索好像又斷掉了。

事情還是不要想的太復雜

再次執行systemctl status cloudera-scm-agent這個命令，卻在這里找到一個新線索

Process: 18991 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=1/FAILURE)Aug 22 22:52:37 test cloudera-scm-agent[18991]: install: cannot create directory ‘/var/run’: File exists Aug 22 22:52:37 test su[19009]: (to root) root on none Aug 22 22:52:37 test su[19009]: pam_systemd(su:session): Failed to connect to system bus: No such file or directory

這里說cannot create directory ‘/var/run’: File exists，那我就去/var下看了看，這個地方有一個軟連接由:
/var/run -> /run的，然后又百度了一下/run這個目錄是干嘛的，就一個臨時目錄，重啟后會刪掉的。
好吧，這時果斷把軟連接刪除掉。然后cloudera-scm-agent竟然啟動成功了，看來問題就是出在這里。

事情還沒結束

進入到cm后，在啟動HDFS時又出問題了，namenode啟動不了。查看日志上顯示:

Failed to start namenode. java.io.IOException: NameNode is not formatted.

好吧，我手動到集群里面格式化一下namenode吧

接著，我啟動Hive和Spark的過程中都發生錯誤了，顯示/user目錄的讀寫權限變了，我需要重寫變更，接著就是Spark的History Server啟動不了，查看日志是:

File does not exist: hdfs://xxx:8020/user/spark/spark2ApplicationHistory

那么沒有就手動創建一下吧，終于所有事情都解決掉了。
哎…心累

總結

以上是生活随笔為你收集整理的大数据开发之CDH篇----cloudera-scm-agent启动不了后的一堆事的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。