æå¾çš„集ä¸å¼æ—¥å¿—分æžòq›_°‹¹ç¨‹å¦‚下åQ?nbsp;
1ã€ä¸‹è½½elasticsearchã€?/p>
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz
2ã€è§£åŽ‹åŽåQŒè¿›å…¥bin目录。执行如下命令,让elasticsearch以å‰å°æ–¹å¼å¯åŠ¨ï¼š(x¨¬)
./elasticsearch -f
[2014-01-16 16:21:31,825][INFO ][node ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z] [2014-01-16 16:21:31,826][INFO ][node ] [Saint Elmo] initializing ... [2014-01-16 16:21:31,836][INFO ][plugins ] [Saint Elmo] loaded [], sites [] [2014-01-16 16:21:35,425][INFO ][node ] [Saint Elmo] initialized [2014-01-16 16:21:35,425][INFO ][node ] [Saint Elmo] starting ... [2014-01-16 16:21:35,578][INFO ][transport ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]}
1ã€å…¶å®‰è£…æ–¹å¼å¯ä»¥å‚考我的å¦ä¸€½‹‡æ–‡ç«?a style="color: #4183c4; line-height: 26px; text-decoration: none; box-sizing: border-box;">Redis¾~–译安装ã€?/p>
2ã€è¿›å…¥å…¶bin目录åQŒæ‰§è¡Œå¦‚下命令,使之在控制å°è¾“出debugä¿¡æ¯åQ?/p>
./redis-server --loglevel verbose
[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379 [32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use
1ã€æ–°å»ÞZ¸€ä¸ªé…¾|®æ–‡ä»Óž¼š(x¨¬)shipper.conf
åQŒå…¶å†…容如下åQ?/p>
input { stdin { type => "example" } } output { stdout { codec => rubydebug } redis { host => "127.0.0.1" port => 6379 data_type => "list" key => "logstash" } }
2ã€å¯åЍshipper。执行如下命令:(x¨¬)
java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf
¾lˆç«¯½H—壞®†å‡ºçް如䏋æ½CÞZ¿¡æ¯ï¼š(x¨¬)
Using milestone 2 output plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}
ç„¶åŽåœ¨ç»ˆç«¯çª—å£ç›´æŽ¥æŒ‰å›žèžRåQŒå°†å‡ºçŽ°å¦‚ä¸‹ä¿¡æ¯åQ?/p>
{ "message" => "", "@version" => "1", "@timestamp" => "2014-01-16T08:15:19.400Z", "type" => "example", "host" => "redhat" }
˜q™ä¸ªjson信毞®†å‘é€ç»™redisåQ?åŒæ—¶redis的终端窗å£å°†å‡ºçް¾cÖM¼¼ä¸‹é¢çš„æ½CÞZ¿¡æ¯ï¼š(x¨¬)
[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640 [32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT. [32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use
1ã€æ–°å»ÞZ¸€ä¸ªé…¾|®æ–‡ä»Óž¼š(x¨¬)indexer.conf
åQŒå…¶å†…容如下åQ?/p>
input { redis { host => "127.0.0.1" # these settings should match the output of the agent data_type => "list" key => "logstash" # We use the 'json' codec here because we expect to read # json events from redis. codec => json } } output { stdout { debug => true debug_format => "json"} elasticsearch { host => "127.0.0.1" } }
2ã€å¯åŠ¨æ—¥å¿—çƒ¦å¼•å™¨ã€‚æ‰§è¡Œå¦‚ä¸‹å‘½ä»¤ï¼š(x¨¬)
java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf
¾lˆç«¯½H—壞®†å‡ºçް如䏋æ½CÞZ¿¡æ¯ï¼š(x¨¬)
Using milestone 2 input plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} You are using a deprecated config setting "debug_format" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"debug_format", :plugin=>, :level=>:warn}
索引器从Redis接收åˆîC¿¡æ¯ï¼Œåœ¨ç»ˆç«¯çª—å£ä¼š(x¨¬)昄¡¤º¾cÖM¼¼å¦‚下的信æ¯ï¼š(x¨¬)
{"message":"","@version":"1","@timestamp":"2014-01-16T17:10:03.831+08:00","type":"example","host":"redhat"}{"message":"","@version":"1","@timestamp":"2014-01-16T17:13:20.545+08:00","type":"example","host":"redhat"}{
1ã€å¯åЍkibana。执行如下命令:(x¨¬)
java -jar logstash-1.3.2-flatjar.jar web
2ã€æ‰“å¼€‹¹è§ˆå™¨ï¼ˆ™åÀL”¯æŒHTML5
åQ‰ï¼Œè¾“入地å€åQšhttp://127.0.0.1:9292/index.html#/dashboard/file/logstash.jsonã€‚ç•Œé¢æ•ˆæžœå¦‚下:(x¨¬)
å‚考资æ–?/span>
官方地å€http://fluentd.org/ æ’äšg地å€http://fluentd.org/plugin/
Kibana æ˜¯ä¸€ä¸ªäØ“(f¨´) ElasticSearch æä¾›æ—¥å¿—分æžçš?Web ui工具åQŒå¯ä½¿ç”¨å®ƒå¯¹æ—¥å¿—˜q›è¡Œé«˜æ•ˆçš„æœç´¢ã€å¯è§†åŒ–ã€åˆ†æžç‰å„ç§æ“作。官方地å€http://www.elasticsearch.org/overview/kibana/
elasticsearch 是开æºçš„åQˆApache2åè®®åQ‰ï¼Œåˆ†å¸ƒå¼çš„åQŒRESTful的,构å¾åœ¨Apache Lucene之上的的æœçƒ¦å¼•擎.
官方地å€http://www.elasticsearch.org/overview/ 䏿–‡åœ°å€ http://es-cn.medcl.net/
具体的工作浽E‹å°±æ˜¯åˆ©ç”¨fluentd 监控òq¶è¿‡æ»¤hadoop集群的系¾lŸæ—¥å¿—,ž®†è¿‡æ»¤åŽçš„æ—¥å¿—内容呾l™å…¨æ–‡æœç´¢æœåŠ¡ElasticSearch, ç„¶åŽç”¨ElasticSearch¾l“åˆKibana ˜q›è¡Œè‡ªå®šä¹‰æœç´¢web™åµé¢å±•示.
下é¢å¼€å§‹è¯´éƒ¨çÖvæ–ÒŽ(gu¨©)³•和过½E‹ã€‚以下安装æ¥éª¤åœ¨centos 5 64使µ‹è¯•通过
一ã€? elasticsearch安装部çÖv
elasticsearch 官方æä¾›äº†å‡ ¿U安装包åQŒé€‚用于windowsçš„zip压羃包,适用于unix/linuxçš„tar.gz压羃包,适用于centos¾pÈ»Ÿçš„rpm包和ubuntuçš„deb包。大家å¯ä»¥è‡ªå·±é€‰æ‹©å®‰è£…使用ã€?br />
å› äØ“(f¨´)elasticsearch 需è¦java环境˜q行åQŒé¦–先需è¦å®‰è£…jdk,安装æ¥éª¤ž®Þqœç•¥äº†ã€?/span>
使用.tar.gz压羃包安装部¾|²çš„è¯ï¼Œå…ˆä¸‹è½½åŽ‹¾~©åŒ…
# wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.tar.gz
# tar zxvf elasticsearch-0.90.5.tar.gz
////////////å¦‚æžœæ˜¯å•æœºéƒ¨¾|?/p>
# cd elasticsearch-0.90.5
# elasticsearch-0.90.5/bin/elasticsearch -f
ž®±å¯ä»¥å¯åЍæœç´¢æœåŠ¡äº†åQŒæŸ¥çœ‹ç«¯å?200æ˜¯å¦æ‰“å¼€åQŒå¦‚果打开说明å¯åЍæ£å¸¸ã€?/p>
////////////////如果是部¾|²é›†¾Ÿ¤çš„è¯ï¼Œéœ€è¦è¿›è¡Œé…¾|?/p>
例如åœ?92.168.0.1 192.168.0.2 䏤尿œåŠ¡å™¨éƒ¨¾|ÔŒ¼Œä¸¤å°æœåŠ¡å™¨éƒ½å®‰è£…å¥½j(lu¨°)dkåQŒä¸‹è½½elasticsearch 解压¾~©ï¼Œç„¶å޾~–辑é…置文äšg
//////////////////////192.168.0.1 æœåŠ¡å™¨ç¼–è¾‘æ–‡ä»?/p>
vi elasticsearch-0.90.5/config/elasticsearch.yml
åˆ é™¤cluster.name å‰é¢æ³¨é‡ŠåQŒä¿®æ”šw›†¾Ÿ¤å¿U?nbsp;
cluster.name: es_cluster
åˆ é™¤node.name剿³¨é‡?åQŒä¿®æ”¹èŠ‚ç‚¹å¿UŽÍ¼Œä¸ä¿®æ”¹çš„è¯ï¼Œ¾pÈ»Ÿå¯åЍåŽä¼š(x¨¬)生æˆéšå³nodeåç§°ã€?/p>
node.name: "elastic_inst1"
node.master: true 讄¡½®è¯¥èŠ‚ç‚¹äØ“(f¨´)主节ç‚?br />
/////////////////////////192.168.0.2 ¾~–辑文äšg
vi elasticsearch-0.90.5/config/elasticsearch.yml
åˆ é™¤cluster.name å‰é¢æ³¨é‡ŠåQŒä¿®æ”šw›†¾Ÿ¤å¿U?nbsp;
cluster.name: es_cluster
åˆ é™¤node.name剿³¨é‡?åQŒä¿®æ”¹èŠ‚ç‚¹å¿UŽÍ¼Œä¸ä¿®æ”¹çš„è¯ï¼Œ¾pÈ»Ÿå¯åЍåŽä¼š(x¨¬)生æˆéšå³nodeåç§°ã€?/p>
node.name: "elastic_inst2"
node.master: false 讄¡½®è¯¥èŠ‚ç‚¹äØ“(f¨´)主节ç‚?/p>
分别å¯åЍ䏤尿œåŠ¡å™¨çš„æœåŠ¡åŽï¼Œåœ?92.168.0.2的日志ä¸ä¼?x¨¬)看å?/p>
[elastic_inst2] detected_master [elastic_inst1] 日志信æ¯ã€‚说明集¾Ÿ¤è¿žæŽ¥æˆåŠŸã€?br />
二ã€å®‰è£…部¾|²fluentd
在需è¦ç›‘控分æžçš„hadoop集群节点ä¸å®‰è£…f(xi¨¦)luentdåQŒå®‰è£…æ¥éª¤å¾ˆ½Ž€å?/span>
curl -L http://toolbelt.treasure-data.com/sh/install-redhat.sh | sh安装完æˆåŽï¼Œ¾~–辑é…置文äšg
# vim /etc/td-agent/td-agent.conf
# service td-agent start
三ã€å®‰è£…部¾|²kibana 3
kibana 3 æ˜?span style="font-family: Helvetica, arial, freesans, clean, sans-serif;">使用html å’Œjavascript å¼€å‘çš„web uiå‰ç«¯å·¥å…·ã€?/span>
ä¸‹è² wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip
解压¾~?unzip kibana-latest.zip
安装apache yum -y install httpd
cp -r kibana-latest /var/www/html
å› äØ“(f¨´)我将kibana3 安装在和elasticsearchåŒä¸€å°æœåС噍ä¸ï¼Œæ‰€ä»¥ä¸ç”¨ä¿®æ”šw…¾|®æ–‡ä»?/span>
å¯åЍapache service httpd start
打开‹¹è§ˆå™?http://ip/kibana ž®±å¯ä»¥çœ‹åˆ°kibana 界é¢
åˆæ¬¡ä½¿ç”¨kibana 需è¦è‡ªå·±å®šä¹‰æ¨¡å?/span>