]]>
]]>
logstash 是一个应用程序日志、事件的传输、处理、管理和搜烦(ch)的^台?br />你可以用它来l一对应用程序日志进行收集管理，提供 Web 接口用于查询和统计?br />
]]>
]]>
]]>
查询Q?br /> http://www.elasticsearch.org/blog/your-data-your-search/
JAVA APIQ注意端口是9300Q不?200
http://stackoverflow.com/questions/16670219/why-cant-i-connect-to-elasticsearch-through-java-api
http://www.elasticsearch.org/guide/reference/java-api/client/
书籍
http://fuxiaopang.gitbooks.io/learnelasticsearch/getting_started/README.html
]]>
Kafka部v与代码实?br /> http://shift-alt-ctrl.iteye.com/blog/1930791
Flume-ng+Kafka+storm的学?fn)笔?br />http://blog.csdn.net/zxcvg/article/details/18600335
Storm应用pd?#8212;—集成Kafka(0.8版的KAFKA)
http://blog.csdn.net/xeseo/article/details/18615761
]]>
zkServer.sh startstorm nimbus &storm supervisor &storm ui &storm jar /opt/hadoop/loganalyst/storm-dependend/data/teststorm-1.0.jar teststorm.TopologyMain /opt/hadoop/loganalyst/storm-dependend/data/words.txtstorm kill {toponame}storm active {toponame}storm deactive {toponame}storm list]]>
http://www.cnblogs.com/kemaswill/archive/2012/10/24/2737833.html
http://abentotoro.blog.sohu.com/197023262.html
http://www.cnblogs.com/panfeng412/archive/2012/11/30/how-to-install-and-deploy-storm-cluster.html
使用 Twitter Storm 处理实时的大数据
http://www.ibm.com/developerworks/cn/opensource/os-twitterstorm/
Storm数据?hu)模型的分析及(qing)讨?br />http://www.cnblogs.com/panfeng412/archive/2012/07/29/storm-stream-model-analysis-and-discussion.html
http://www.cnblogs.com/panfeng412/tag/Storm/
storm-kafka
https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka
使用Storm实现实时大数据分析！
http://www.csdn.net/article/2012-12-24/2813117-storm-realtime-big-data-analysis
storm-deploy-aws
https://github.com/nathanmarz/storm-deploy/wiki
!!!知乎|站上的Twitter Storm
http://www.zhihu.com/topic/19673110
storm-elastic-search
https://github.com/hmsonline/storm-elastic-search
storm-examples
https://github.com/stormprocessor/storm-examples
kafka-aws
https://github.com/nathanmarz/kafka-deploy
Next Gen Real-time Streaming with Storm-Kafka Integration
http://blog.infochimps.com/2012/10/30/next-gen-real-time-streaming-storm-kafka-integration/
flume+kafka+storm+mysql 数据?hu)?
http://blog.csdn.net/baiyangfu/article/details/8096088
http://blog.csdn.net/baiyangfu/article/category/1244640
Kafka学习(fn)W记
http://blog.csdn.net/baiyangfu/article/details/8096084
STORM+KAFKA
https://github.com/buildlackey/cep
STORM+KETTLE
https://github.com/buildlackey/kettle-storm
]]>
通过O(1)的磁盘数据结构提供消息的持久化，q种l构对于即CTB的消息存储也能够保持长时间的E_性能?br />高吞吐量Q即使是非常普通的gkafka也可以支持每U数十万的消息?br />支持通过kafka服务器和消费机集来分区消息?br />支持Hadoopq行数据加蝲?br />
设计侧重高吞吐量Q用于好友动态，相关性统计，排行l计Q访问频率控Ӟ批处理等pȝ。大部分的消息中间g能够处理实时性要求高的消?数据Q但是对于队列中大量未处理的消息/数据在持久性方面比较弱?br />
kakfa的consumer使用拉的方式工作?br />
安装kafka
下蝲Qhttp://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
> tar xzf kafka-.tgz
> cd kafka-
> ./sbt update
> ./sbt package
启动zkserver:
bin/zookeeper-server-start.sh config/zookeeper.properties
启动server:
bin/kafka-server-start.sh config/server.properties
是q么单?br />
使用kafka
import java.util.Arrays;
import java.util.List;
import java.util.Properties;
import kafka.javaapi.producer.SyncProducer;
import kafka.javaapi.message.ByteBufferMessageSet;
import kafka.message.Message;
import kafka.producer.SyncProducerConfig;
Properties props = new Properties();
props.put(“zk.connect”, “127.0.0.1:2181”);
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
Send a single message
// The message is sent to a randomly selected partition registered in ZK
ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message");
producer.send(data);
producer.close();
import java.util.List;
import java.util.Properties;
import kafka.javaapi.producer.SyncProducer;
import kafka.javaapi.message.ByteBufferMessageSet;
import kafka.message.Message;
import kafka.producer.SyncProducerConfig;
Properties props = new Properties();
props.put(“zk.connect”, “127.0.0.1:2181”);
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
Send a single message
// The message is sent to a randomly selected partition registered in ZK
ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message");
producer.send(data);
producer.close();
q样是一个标准的producer?br />
consumer的代?br />
// specify some consumer properties
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("zk.connectiontimeout.ms", "1000000");
props.put("groupid", "test_group");
// Create the connection to the cluster
ConsumerConfig consumerConfig = new ConsumerConfig(props);
ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
// create 4 partitions of the stream for topic “test”, to allow 4 threads to consume
Map<String, List<KafkaMessageStream<Message>>> topicMessageStreams =
consumerConnector.createMessageStreams(ImmutableMap.of("test", 4));
List<KafkaMessageStream<Message>> streams = topicMessageStreams.get("test");
// create list of 4 threads to consume from each of the partitions
ExecutorService executor = Executors.newFixedThreadPool(4);
// consume the messages in the threads
for(final KafkaMessageStream<Message> stream: streams) {
executor.submit(new Runnable() {
public void run() {
for(Message message: stream) {
// process message
}
}
});
}
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("zk.connectiontimeout.ms", "1000000");
props.put("groupid", "test_group");
// Create the connection to the cluster
ConsumerConfig consumerConfig = new ConsumerConfig(props);
ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
// create 4 partitions of the stream for topic “test”, to allow 4 threads to consume
Map<String, List<KafkaMessageStream<Message>>> topicMessageStreams =
consumerConnector.createMessageStreams(ImmutableMap.of("test", 4));
List<KafkaMessageStream<Message>> streams = topicMessageStreams.get("test");
// create list of 4 threads to consume from each of the partitions
ExecutorService executor = Executors.newFixedThreadPool(4);
// consume the messages in the threads
for(final KafkaMessageStream<Message> stream: streams) {
executor.submit(new Runnable() {
public void run() {
for(Message message: stream) {
// process message
}
}
});
}
]]>
apache kafka在数据处理中特别是日志和消息的处理上?x)有很多(gu)的表玎ͼq里写个索引Q关于kafka的文章暂时就更新到这里，最q利用空闲时间在对kafka做一些功能性增强，qjava化，虽然现在已经有很多这L(fng)版本Q但是根据实际需求来改变才是最适合的?/p>
首先当然推荐的是kafka的官|?nbsp;http://kafka.apache.org/
在官|最值得参考的文章是kafka designQ?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">http://kafka.apache.org/design.htmlQ我的文章也基本都是参照q里的说明，大家要特别重视这文章，里面有好多理念都特别好，推荐多读几遍?/p>
在OSC的翻译频道有kafka design全中文的译Q翻得挺好的Q推荐一下：(x)http://www.oschina.net/translate/kafka-design
kafka的wiki是很不错的学?fn)文档?x)https://cwiki.apache.org/confluence/display/KAFKA/Index
——————————————————————————————————
接下来就是我写的一pd文章Q文章都是@序渐q的方式带你?jin)解kafkaQ?/p>
关于kafka的基本知识，分布式的基础Q?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《分布式消息pȝKafka初步?/a>
kafka的分布式搭徏Qquick startQ?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《kafka分布式环境搭建?/a>
关于kafka的实现细节，q主要就是讲design的部分：(x)《细节上?/a>?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《细节下?/a>
关于kafka开发环境，scala环境的搭建：(x)《开发环境搭建?/a>
数据生者，producer的用法：(x)《producer的用法?/a>?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《producer使用注意?/a>
数据消费者，consumer的用法：(x)《consumer的用法?/a>
q有些零的Q关于通信D늚源码解读Q?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《net包源码解诅R?/a>?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《broker配置?/a>
——————————————————————————————————
扩展的阅读还有下面这些：(x)
我的好友写的关于kafka和jafka的相兛_客，特别好，我有很多问题?sh)都找他解决的，大神一般的存在Q?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">http://rockybean.github.com/ @rockybean
kafka的java化版本jafkaQ?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">https://github.com/adyliu/jafka
淘宝的metaQQ?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">https://github.com/killme2008/Metamorphosis
我最q在写的inforQQ刚开始写Q我也纯_Ҏ(gu)Z(jin)M源码Q不定期更新哈：(x)https://github.com/ielts0909/inforq
后面一阶段可能更新点儿关于cas的东西吧Q具体也没想好，最q一直出差，写代码的旉都很?/p>
--------------------------------------------------------------------------------
0.8版本的相x(chng)新如下：(x)
0.8更新内容介绍Q?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">《kafka0.8版本的一些更新?/a>
]]>