log4net直切ElasticSearch,小步快跑首选
很多小步快跑的公司,開發人員可能就3-4個,面對巨大業務壓力,日連夜的趕著上線,快速試錯,自然就沒時間搭建一些基礎設施,比如說logCenter,但初期項目不穩定,bug又多,每次都跑到生產去找日志,確實也不大方便,用elk或者用hadoop做日志中心,雖然都是沒問題的,但基于成本和人手還是怎么簡化怎么來,
本篇就來說說直接使用log4net的ElasticSearchAppender擴展直接將log寫入到es中。
一:從nuget下載
nuget下來的都是開箱可用,如果看源碼的話可以在github上找一下log4net.ElasticSearch項目。https://github.com/jptoto/log4net.ElasticSearch
1. App.config配置
nuget包下來之后,就可以配置config文件了,其實還是蠻簡單的,大家可以根據自己的項目合理的配置里面的各項參數,為了方便大家理解,我在每個配置項上加了詳細的注釋,大家可以仔細看看。
<?xml?version="1.0"?encoding="utf-8"??> <configuration><configSections><p?name="log4net"?type="log4net.Config.Log4NetConfigurationSectionHandler,?log4net"?/></configSections><log4net><!--?日志的處理類:log4net.ElasticSearch.ElasticSearchAppender --><appender?name="ElasticSearchAppender"?type="log4net.ElasticSearch.ElasticSearchAppender,?log4net.ElasticSearch"><!-- ES地址? rolling=true:表示每天一個index(datamipcrm_log_2018.05.31)??--><connectionString?value="Scheme=http;Server=192.168.23.145;Index=datamiprm_log;Port=9200;rolling=true"/><!--?有損配置:?如果配置的buffer滿了還沒來的及刷新到es中,那么新來的log將會被丟棄。?--><lossy?value="false"?/><evaluator?type="log4net.Core.LevelEvaluator"><!--表示?小于ERROR級別的日志會進入buffer數組,大于等于這個級別的,直接提交給es。通常情況下,ERROR級別的錯誤,我們直接塞到ES中,這樣更有利于我們發現問題。DEBUG,INFO WARN ERROR --><threshold?value="ERROR"?/></evaluator><!--?buffer池的閾值50,一旦滿了就會觸發flush?到?es的動作(bulk?api)?--><bufferSize?value="50"?/></appender><root><!--?指定所有的loglevel(DEBUG,INFO,WARN,ERROR)級別都是用?ElasticSearchAppender?處理?--><level?value="ALL"/><appender-ref?ref="ElasticSearchAppender"?/></root></log4net><startup><supportedRuntime?version="v4.0"?sku=".NETFramework,Version=v4.6.1"?/></startup> </configuration>上面有幾點要特別注意一下:
- bufferSize 
這個大家理解成緩存區,方便批量提交到es中,否則的話,過于頻繁的和es進行交互,對帶寬,對application,es都是比較大的壓力。
有時候我們有這樣的需求,我希望ERROR,Fault這種級別的錯誤不要走buffer,直接提交給es,這樣更容易讓初創團隊發現問題,找到問題,恭喜你,上面這個配置就是解決這個事的。
2. 在AssemblyInfo中追加如下配置
//?可以指定所有值,也可以使用以下所示的?"*"?預置版本號和修訂號 //?方法是按如下所示使用“*”:?: //?[assembly:?AssemblyVersion("1.0.*")] [assembly:?AssemblyVersion("1.0.0.0")] [assembly:?AssemblyFileVersion("1.0.0.0")] [assembly:?log4net.Config.XmlConfigurator(Watch?=?true)]3. 然后就可以寫一段測試代碼啦。
namespace?ConsoleApp1 {class?Program{private?static?readonly?ILog?_log?=?LogManager.GetLogger(typeof(Program));static?void?Main(string[]?args){????????????for?(int?i?=?0;?i?<?1000;?i++){try{var?m?=?"1";var?result?=?100?/?Convert.ToInt32(m);_log.Info("我要開始記錄日志啦");}catch?(Exception?ex){_log.Error("調用失敗"?+?i,?ex);//_log.Info("調用失敗"?+?i,?ex);//Console.WriteLine(i);}}Console.Read();}} }然后通過elasticsearch-head 插件進行查看,各種數據就都在es中了。
二:elasticsearch-header 插件
關于這個插件的下載,因為google官網被屏蔽,大家如果有VPN的話,可以自行在chrome商店搜索,當然也可以直接下載我的zip包,elasticsearch-head。在瀏覽器中加載插件的時候選擇0.1.3.0 文件夾即可
完了之后點擊右上邊的 “放大鏡“ 按鈕就可以看到你想看到的UI了。
三:下載elasticsearch
這是一個基于luncene的分布式搜索框架,用起來還是挺順手的,你可以下載5.6.4版本。
[elsearch@localhost?myapp]$?ls elasticsearch-5.6.4.tar.gz??一般來說,安裝的過程中你可能會遇到3個坑。
1. ?不能用root賬號登錄
[root@localhost?bin]#?./elasticsearch [2018-05-31T04:01:59,402][WARN?][o.e.b.ElasticsearchUncaughtExceptionHandler]?[]?uncaught?exception?in?thread?[main] org.elasticsearch.bootstrap.StartupException:?java.lang.RuntimeException:?can?not?run?elasticsearch?as?rootat?org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.cli.Command.main(Command.java:90)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)?~[elasticsearch-5.6.4.jar:5.6.4] Caused?by:?java.lang.RuntimeException:?can?not?run?elasticsearch?as?rootat?org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)?~[elasticsearch-5.6.4.jar:5.6.4]at?org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)?~[elasticsearch-5.6.4.jar:5.6.4]...?6?more這個簡單,增加一個elasearch用戶就可以了。
groupadd?elsearch???????????????????#新建elsearch組 useradd?elsearch?-g?elsearch?-p?elasticsearch??#新建一個elsearch用戶 chown?-R?elsearch:elsearch??./elasticsearch????#指定elasticsearch所屬elsearch組2. ?內存不足的問題。
[elsearch@localhost?bin]$?./elasticsearch Java?HotSpot(TM)?64-Bit?Server?VM?warning:?INFO:?os::commit_memory(0x00000000a9990000,?1449590784,?0)?failed;?error='Cannot?allocate?memory'?(errno=12) # #?There?is?insufficient?memory?for?the?Java?Runtime?Environment?to?continue. #?Native?memory?allocation?(mmap)?failed?to?map?1449590784?bytes?for?committing?reserved?memory. #?An?error?report?file?with?more?information?is?saved?as: #?/usr/myapp/elasticsearch/bin/hs_err_pid33341這是ES默認分配的堆內存是2g,如果出現這個問題,一般來說是你的虛擬機擁有的內存小于2g,只需要在 jvm.options 中將2g修改1g就可以了。
[root@localhost?config]#?ls elasticsearch.yml??jvm.options??log4j2.properties??scripts#?Xms?represents?the?initial?size?of?total?heap?space #?Xmx?represents?the?maximum?size?of?total?heap?space-Xms1g -Xmx1g3. max file descriptors 太少 和 max virtual memory 虛擬內存太低。
[elsearch@localhost?bin]$?./elasticsearch [2018-05-30T20:44:56,484][INFO?][o.e.n.Node???????????????]?[]?initializing?... [2018-05-30T20:44:56,632][INFO?][o.e.e.NodeEnvironment????]?[f9t2Sfl]?using?[1]?data?paths,?mounts?[[/?(rootfs)]],?net?usable_space?[14.4gb],?net?total_space?[22.1gb],?spins??[unknown],?types?[rootfs] [2018-05-30T20:44:56,632][INFO?][o.e.e.NodeEnvironment????]?[f9t2Sfl]?heap?size?[989.8mb],?compressed?ordinary?object?pointers?[true] [2018-05-30T20:44:56,634][INFO?][o.e.n.Node???????????????]?node?name?[f9t2Sfl]?derived?from?node?ID?[f9t2SfljReiND4XeMLUbyA];?set?[node.name]?to?override [2018-05-30T20:44:56,634][INFO?][o.e.n.Node???????????????]?version[5.6.4],?pid[33546],?build[8bbedf5/2017-10-31T18:55:38.105Z],?OS[Linux/3.10.0-327.el7.x86_64/amd64],?JVM[Oracle?Corporation/Java?HotSpot(TM)?64-Bit?Server?VM/1.8.0_144/25.144-b01] [2018-05-30T20:44:56,634][INFO?][o.e.n.Node???????????????]?JVM?arguments?[-Xms1g,?-Xmx1g,?-XX:+UseConcMarkSweepGC,?-XX:CMSInitiatingOccupancyFraction=75,?-XX:+UseCMSInitiatingOccupancyOnly,?-XX:+AlwaysPreTouch,?-Xss1m,?-Djava.awt.headless=true,?-Dfile.encoding=UTF-8,?-Djna.nosys=true,?-Djdk.io.permissionsUseCanonicalPath=true,?-Dio.netty.noUnsafe=true,?-Dio.netty.noKeySetOptimization=true,?-Dio.netty.recycler.maxCapacityPerThread=0,?-Dlog4j.shutdownHookEnabled=false,?-Dlog4j2.disable.jmx=true,?-Dlog4j.skipJansi=true,?-XX:+HeapDumpOnOutOfMemoryError,?-Des.path.home=/usr/myapp/elasticsearch] [2018-05-30T20:44:57,530][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[aggs-matrix-stats] [2018-05-30T20:44:57,530][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[ingest-common] [2018-05-30T20:44:57,530][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[lang-expression] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[lang-groovy] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[lang-mustache] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[lang-painless] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[parent-join] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[percolator] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[reindex] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[transport-netty3] [2018-05-30T20:44:57,531][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?loaded?module?[transport-netty4] [2018-05-30T20:44:57,532][INFO?][o.e.p.PluginsService?????]?[f9t2Sfl]?no?plugins?loaded [2018-05-30T20:44:59,469][INFO?][o.e.d.DiscoveryModule????]?[f9t2Sfl]?using?discovery?type?[zen] [2018-05-30T20:45:00,107][INFO?][o.e.n.Node???????????????]?initialized [2018-05-30T20:45:00,107][INFO?][o.e.n.Node???????????????]?[f9t2Sfl]?starting?... [2018-05-30T20:45:00,339][INFO?][o.e.t.TransportService???]?[f9t2Sfl]?publish_address?{192.168.23.143:9300},?bound_addresses?{[::]:9300} [2018-05-30T20:45:00,356][INFO?][o.e.b.BootstrapChecks????]?[f9t2Sfl]?bound?or?publishing?to?a?non-loopback?or?non-link-local?address,?enforcing?bootstrap?checks ERROR:?[2]?bootstrap?checks?failed [1]:?max?file?descriptors?[4096]?for?elasticsearch?process?is?too?low,?increase?to?at?least?[65536] [2]:?max?virtual?memory?areas?vm.max_map_count?[65530]?is?too?low,?increase?to?at?least?[262144] [2018-05-30T20:45:00,365][INFO?][o.e.n.Node???????????????]?[f9t2Sfl]?stopping?... [2018-05-30T20:45:00,444][INFO?][o.e.n.Node???????????????]?[f9t2Sfl]?stopped [2018-05-30T20:45:00,444][INFO?][o.e.n.Node???????????????]?[f9t2Sfl]?closing?... [2018-05-30T20:45:00,455][INFO?][o.e.n.Node???????????????]?[f9t2Sfl]?closed這個max file descriptors 的問題,我只需要修改 vim /etc/security/limits.conf 文件增加句柄值即可。
*?hard?nofile?65536 *?soft?nofile?65536虛擬內存的 max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] ?只需要在sysctl.conf 中修改max即可。
[root@localhost?config]#?vim?/etc/sysctl.confvm.max_map_count=655360這些坑解決的話,es就可以正常啟動了,最后記得在elasticsearch.yml 中將host設為0.0.0.0 讓遠程機器可以訪問。
[elsearch@localhost?config]$?vim?elasticsearch.ymlnetwork.host:?0.0.0.0最后執行 ./elasticsearch -d ?讓es在后臺執行。
[elsearch@localhost?bin]$?./elasticsearch?-d[elsearch@localhost?bin]$?netstat?-tlnp (Not?all?processes?could?be?identified,?non-owned?process?infowill?not?be?shown,?you?would?have?to?be?root?to?see?it?all.) Active?Internet?connections?(only?servers) Proto?Recv-Q?Send-Q?Local?Address???????????Foreign?Address?????????State???????PID/Program?name???? tcp????????0??????0?192.168.122.1:53????????0.0.0.0:*???????????????LISTEN??????-??????????????????? tcp????????0??????0?0.0.0.0:22??????????????0.0.0.0:*???????????????LISTEN??????-??????????????????? tcp????????0??????0?127.0.0.1:631???????????0.0.0.0:*???????????????LISTEN??????-??????????????????? tcp????????0??????0?127.0.0.1:25????????????0.0.0.0:*???????????????LISTEN??????-??????????????????? tcp6???????0??????0?:::9200?????????????????:::*????????????????????LISTEN??????17523/java?????????? tcp6???????0??????0?:::9300?????????????????:::*????????????????????LISTEN??????17523/java?????????? tcp6???????0??????0?:::22???????????????????:::*????????????????????LISTEN??????-??????????????????? tcp6???????0??????0?::1:631?????????????????:::*????????????????????LISTEN??????-??????????????????? tcp6???????0??????0?::1:25??????????????????:::*????????????????????LISTEN??????-??????????????????? [elsearch@localhost?bin]$好了,本篇就說這么多,如果你要更精細化的查詢,可以再搭建一個配套版本的kibana即可。
總結
以上是生活随笔為你收集整理的log4net直切ElasticSearch,小步快跑首选的全部內容,希望文章能夠幫你解決所遇到的問題。
 
                            
                        - 上一篇: 好的自我介绍,面试成功一大半
- 下一篇: 壹佰文章总结| 关于ASP.NETCor
