ï»??xml version="1.0" encoding="utf-8" standalone="yes"?>这里只有精品在线,91涩漫在线观看,1769视频在线播放免费观看http://www.aygfsteel.com/jjshcc/category/54849.htmlzh-cnFri, 12 Jun 2015 05:15:31 GMTFri, 12 Jun 2015 05:15:31 GMT60安装logstash+kibana+elasticsearch+redis搭徏集中式日志分析åã^å? http://www.aygfsteel.com/jjshcc/archive/2015/06/12/425647.htmlEric_jiangEric_jiangFri, 12 Jun 2015 03:30:00 GMThttp://www.aygfsteel.com/jjshcc/archive/2015/06/12/425647.htmlhttp://www.aygfsteel.com/jjshcc/comments/425647.htmlhttp://www.aygfsteel.com/jjshcc/archive/2015/06/12/425647.html#Feedback0http://www.aygfsteel.com/jjshcc/comments/commentRss/425647.htmlhttp://www.aygfsteel.com/jjshcc/services/trackbacks/425647.html本文是参考logstash官方文档实践的笔讎ͼŒæ­å¾çŽ¯å¢ƒå’Œæ‰€éœ€¾l„äšg如下åQ?/p>
  • Redhat 5.7 64bit / CentOS 5.x
  • JDK 1.6.0_45
  • logstash 1.3.2 (内带kibana)
  • elasticsearch 0.90.10
  • redis 2.8.4

搭徏的集中式日志分析òq›_°‹¹ç¨‹å¦‚下åQ?nbsp;

安装logstash+kibana+elasticsearch+redis搭徏日志分析òq›_° - 傲风 - 0ä¸?构筑世界åQŒç¨‹åºå‘˜åˆ›é€ æ—¶ä»? src=

 

elasticsearch

1、下载elasticsearch�/p>

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz 

2、解压后åQŒè¿›å…¥bin目录。执行如下命令,让elasticsearch以前台方式启动:(x¨¬)

./elasticsearch -f 
[2014-01-16 16:21:31,825][INFO ][node                     ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z] [2014-01-16 16:21:31,826][INFO ][node                     ] [Saint Elmo] initializing ... [2014-01-16 16:21:31,836][INFO ][plugins                  ] [Saint Elmo] loaded [], sites [] [2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] initialized [2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] starting ... [2014-01-16 16:21:35,578][INFO ][transport                ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]} 

Redis

1、其安装方式可以参考我的另一½‹‡æ–‡ç«?a style="color: #4183c4; line-height: 26px; text-decoration: none; box-sizing: border-box;">Redis¾~–译安装ã€?/p>

2、进入其bin目录åQŒæ‰§è¡Œå¦‚下命令,使之在控制台输出debug信息åQ?/p>

./redis-server --loglevel verbose 
[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379 [32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use 

logstash日志生成器(shipperåQ?/h2>

1、新å»ÞZ¸€ä¸ªé…¾|®æ–‡ä»Óž¼š(x¨¬)shipper.confåQŒå…¶å†…容如下åQ?/p>

input {     stdin {         type => "example"     } }  output {     stdout {         codec => rubydebug     }     redis {         host => "127.0.0.1"         port => 6379         data_type => "list"         key => "logstash"     } } 

2、启动shipper。执行如下命令:(x¨¬)

java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf  

¾lˆç«¯½H—口ž®†å‡ºçŽ°å¦‚ä¸‹æ½CÞZ¿¡æ¯ï¼š(x¨¬)

Using milestone 2 output plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} 

然后在终端窗口直接按回èžRåQŒå°†å‡ºçŽ°å¦‚ä¸‹ä¿¡æ¯åQ?/p>

{        "message" => "",       "@version" => "1",     "@timestamp" => "2014-01-16T08:15:19.400Z",           "type" => "example",           "host" => "redhat" } 

˜q™ä¸ªjson信息ž®†å‘送给redisåQ?同时redis的终端窗口将出现¾cÖM¼¼ä¸‹é¢çš„æ½CÞZ¿¡æ¯ï¼š(x¨¬)

[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640 [32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT. [32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use 

logstash日志索引器(indexeråQ?/h2>

1、新å»ÞZ¸€ä¸ªé…¾|®æ–‡ä»Óž¼š(x¨¬)indexer.confåQŒå…¶å†…容如下åQ?/p>

input {   redis {     host => "127.0.0.1"     # these settings should match the output of the agent     data_type => "list"     key => "logstash"      # We use the 'json' codec here because we expect to read     # json events from redis.     codec => json   } }  output {   stdout { debug => true debug_format => "json"}    elasticsearch {     host => "127.0.0.1"   } } 

2、启动日志烦引器。执行如下命令:(x¨¬)

java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf  

¾lˆç«¯½H—口ž®†å‡ºçŽ°å¦‚ä¸‹æ½CÞZ¿¡æ¯ï¼š(x¨¬)

Using milestone 2 input plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} You are using a deprecated config setting "debug_format" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future.  If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"debug_format", :plugin=>, :level=>:warn} 

索引器从Redis接收åˆîC¿¡æ¯ï¼Œåœ¨ç»ˆç«¯çª—口会(x¨¬)昄¡¤º¾cÖM¼¼å¦‚下的信息:(x¨¬)

{"message":"","@version":"1","@timestamp":"2014-01-16T17:10:03.831+08:00","type":"example","host":"redhat"}{"message":"","@version":"1","@timestamp":"2014-01-16T17:13:20.545+08:00","type":"example","host":"redhat"}{ 

logstash WEB界面åQˆkibanaåQ?/h2>

1、启动kibana。执行如下命令:(x¨¬)

java -jar logstash-1.3.2-flatjar.jar web 

2、打开‹¹è§ˆå™¨ï¼ˆ™åÀL”¯æŒHTML5åQ‰ï¼Œè¾“入地址åQšhttp://127.0.0.1:9292/index.html#/dashboard/file/logstash.json。界面效果如下:(x¨¬) 

安装logstash+kibana+elasticsearch+redis搭徏日志分析òq›_° - 傲风 - 0ä¸?构筑世界åQŒç¨‹åºå‘˜åˆ›é€ æ—¶ä»? src=
 

参考资�/span>



]]>fluentd¾l“合kibana、elasticsearch实时搜烦分析hadoop集群日志http://www.aygfsteel.com/jjshcc/archive/2015/06/12/425645.htmlEric_jiangEric_jiangFri, 12 Jun 2015 03:05:00 GMThttp://www.aygfsteel.com/jjshcc/archive/2015/06/12/425645.htmlhttp://www.aygfsteel.com/jjshcc/comments/425645.htmlhttp://www.aygfsteel.com/jjshcc/archive/2015/06/12/425645.html#Feedback0http://www.aygfsteel.com/jjshcc/comments/commentRss/425645.htmlhttp://www.aygfsteel.com/jjshcc/services/trackbacks/425645.htmlFluentd是一个开源收集事件和日志¾pȝ»ŸåQŒå®ƒç›®å‰æä¾›150+扩展插äšg让你存储大数据用于日志搜索,数据分析和存储ã€?/p>

官方地址http://fluentd.org/  插äšg地址http://fluentd.org/plugin/

Kibana æ˜¯ä¸€ä¸ªäØ“(f¨´) ElasticSearch 提供日志分析çš?Web ui工具åQŒå¯ä½¿ç”¨å®ƒå¯¹æ—¥å¿—˜q›è¡Œé«˜æ•ˆçš„æœç´¢ã€å¯è§†åŒ–、分析等各种操作。官方地址http://www.elasticsearch.org/overview/kibana/

elasticsearch æ˜¯å¼€æºçš„åQˆApache2协议åQ‰ï¼Œåˆ†å¸ƒå¼çš„åQŒRESTful的,构徏在Apache Lucene之上的的搜烦引擎.

官方地址http://www.elasticsearch.org/overview/    中文地址 http://es-cn.medcl.net/


具体的工作流½E‹å°±æ˜¯åˆ©ç”¨fluentd 监控òq¶è¿‡æ»¤hadoop集群的系¾lŸæ—¥å¿—,ž®†è¿‡æ»¤åŽçš„æ—¥å¿—内容发¾l™å…¨æ–‡æœç´¢æœåŠ¡ElasticSearch, 然后用ElasticSearch¾l“合Kibana ˜q›è¡Œè‡ªå®šä¹‰æœç´¢web™åµé¢å±•示.

下面开始说部çÖvæ–ÒŽ(gu¨©)³•和过½E‹ã€‚以下安装步骤在centos 5 64位测试通过

一ã€? elasticsearch安装部çÖv

elasticsearch 官方提供了几¿Uå®‰è£…包åQŒé€‚用于windowsçš„zip压羃包,适用于unix/linuxçš„tar.gz压羃包,适用于centos¾pȝ»Ÿçš„rpm包和ubuntuçš„deb包。大家可以自己选择安装使用ã€?br />

因䨓(f¨´)elasticsearch 需要java环境˜qè¡ŒåQŒé¦–先需要安装jdk,安装步骤ž®Þqœç•¥äº†ã€?/span>

使用.tar.gz压羃包安装部¾|²çš„话,先下载压¾~©åŒ…

# wget  https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.tar.gz

# tar zxvf elasticsearch-0.90.5.tar.gz

////////////如果是单机部¾|?/p>

# cd elasticsearch-0.90.5

# elasticsearch-0.90.5/bin/elasticsearch -f

ž®±å¯ä»¥å¯åŠ¨æœç´¢æœåŠ¡äº†åQŒæŸ¥çœ‹ç«¯å?200是否打开åQŒå¦‚果打开说明启动正常ã€?/p>

////////////////如果是部¾|²é›†¾Ÿ¤çš„话,需要进行配¾|?/p>

例如åœ?92.168.0.1   192.168.0.2 两台服务器部¾|ÔŒ¼Œä¸¤å°æœåŠ¡å™¨éƒ½å®‰è£…å¥½j(lu¨°)dkåQŒä¸‹è½½elasticsearch 解压¾~©ï¼Œç„¶åŽ¾~–辑配置文äšg

//////////////////////192.168.0.1   服务器编辑文ä»?/p>

vi elasticsearch-0.90.5/config/elasticsearch.yml

删除cluster.name 前面注释åQŒä¿®æ”šw›†¾Ÿ¤å¿U?nbsp;

cluster.name: es_cluster

删除node.name前注é‡?åQŒä¿®æ”¹èŠ‚ç‚¹å¿UŽÍ¼Œä¸ä¿®æ”¹çš„话,¾pȝ»Ÿå¯åŠ¨åŽä¼š(x¨¬)生成随即node名称ã€?/p>

node.name: "elastic_inst1"

node.master: true    è®„¡½®è¯¥èŠ‚ç‚¹äØ“(f¨´)主节ç‚?br />


/////////////////////////192.168.0.2  ¾~–辑文äšg

vi elasticsearch-0.90.5/config/elasticsearch.yml

删除cluster.name 前面注释åQŒä¿®æ”šw›†¾Ÿ¤å¿U?nbsp;

cluster.name: es_cluster

删除node.name前注é‡?åQŒä¿®æ”¹èŠ‚ç‚¹å¿UŽÍ¼Œä¸ä¿®æ”¹çš„话,¾pȝ»Ÿå¯åŠ¨åŽä¼š(x¨¬)生成随即node名称ã€?/p>

node.name: "elastic_inst2"

node.master: false    è®„¡½®è¯¥èŠ‚ç‚¹äØ“(f¨´)主节ç‚?/p>


分别启动两台服务器的服务后,åœ?92.168.0.2的日志中ä¼?x¨¬)看å?/p>

 [elastic_inst2] detected_master [elastic_inst1]     日志信息。说明集¾Ÿ¤è¿žæŽ¥æˆåŠŸã€?br />


二、安装部¾|²fluentd

在需要监控分析的hadoop集群节点中安装f(xi¨¦)luentdåQŒå®‰è£…步骤很½Ž€å?/span>


curl -L http://toolbelt.treasure-data.com/sh/install-redhat.sh | sh

安装完成后,¾~–辑配置文äšg

# vim /etc/td-agent/td-agent.conf

  1. <source>  
  2.   type tail  #### tail方式采集日志  
  3.   path /var/log/hadoop/mapred/hadoop-mapred-tasktracker-node-128-70.log   ### hadoop日志路径  
  4.   pos_file /var/log/td-agent/task-access.log.pos  
  5.   tag task.mapred  
  6.   format /^(?<message>.+(WARN|ERROR).+)$/   #### æ”‰™›†error æˆ–者warn æ—¥å¿—ã€?nbsp; 
  7. </source>  
  8.   
  9. <match task.**>  
  10.   host 192.168.0.1  #####  <span style="font-family:Arial,Helvetica,sans-serif">elasticsearch æœåŠ¡å™¨åœ°å€</span>  
  11.   type elasticsearch  
  12.   logstash_format true  
  13.   flush_interval 5s  
  14.   include_tag_key true  
  15.   tag_key mapred  
  16. </match>  

启动fluentd 服务

# service td-agent start


三、安装部¾|²kibana 3

kibana 3 �span style="font-family: Helvetica, arial, freesans, clean, sans-serif;">使用html 和javascript 开发的web ui前端工具�/span>

下蝲 wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip

解压¾~?unzip kibana-latest.zip

安装apache    yum -y install httpd

cp -r  kibana-latest /var/www/html

因䨓(f¨´)我将kibana3 安装在和elasticsearch同一台服务器中,所以不用修攚w…¾|®æ–‡ä»?/span>

启动apache  service httpd start

打开‹¹è§ˆå™?http://ip/kibana ž®±å¯ä»¥çœ‹åˆ°kibana 界面

初次使用kibana 需要自己定义模�/span>






]]>
Ö÷Õ¾Ö©Öë³ØÄ£°å£º ÌÆº£ÏØ| ÈêÖÝÊÐ| ¾¸ÖÝ| ºÚºÓÊÐ| µÂ²ýÏØ| ¶¼²ýÏØ| ·ö·çÏØ| µÂÁî¹þÊÐ| ¿µÀÖÏØ| ÂêÇúÏØ| °ºÈÊÏØ| ÎâÇÅÏØ| ͨÐíÏØ| ÄþÁêÏØ| Ñ·¿ËÏØ| ¶î¶û¹ÅÄÉÊÐ| »ëÔ´ÏØ| ±£¶¨ÊÐ| ½¨Æ½ÏØ| ÁøºÓÏØ| ¸ßÐÛÏØ| Î×É½ÏØ| Ê®ÑßÊÐ| ËçÖÐÏØ| ÍòÈÙÏØ| ñçÑôÊÐ| ÎäÇ¿ÏØ| Öн­ÏØ| ÒÁÄþÏØ| ³²ºþÊÐ| Öз½ÏØ| ÃÖ¶ÉÏØ| ÅÍʯÊÐ| Îå»ªÏØ| ÁúÀïÏØ| ÄþÇ¿ÏØ| Ѱµé| Ë«ÁÉÊÐ| ÊÙÑôÏØ| ³Â°Í¶û»¢Æì| ÌÚ³åÏØ|