精品亚洲欧美一区,一区二区美女,成人性生交大片免费看96

Clojure DSL

徐红�? — Thu, 19 Jan 2012 07:38:00 GMT

     Storm 用Clojure DSL 来定�?spouts, bolts, and topologies�?Clojure的DSL讉K��的�Q何公开的Java API�Q�如果你是一个Clojure的用��P��你编写直接用Clojure �~�写Storm 的Topologies�Q�而不用接触Java�?Clojure的DSL定义在backtype.storm.clojure命名�I�间里�?br />       本页面概�q�C��Clojure的DSL的所有细节，包括�Q?br />        1.定义Topology�Q�拓扑结构）
        2.定义bolt
        3.定义spout
        4.在本地模式下或在集群模式下运行Topology
        5.��试Topology

定义 topologies

       要定义Topology�Q?需要��用Topology函数。Topology需要两个参敎ͼ�一个关�?#8220;Spou Specs”的映��和一个关�?#8220;Bolt Spec”的映��。每个Spout和Bolt指定�l��g到Topology上，如输入和�q�行拓扑�l�构的代码�?br />      让我们来看看在Storm启动��目的例子的拓扑定义
(topology
{"1" (spout-spec sentence-spout)
"2" (spout-spec (sentence-spout-parameterized
                   ["the cat jumped over the door"
                    "greetings from a faraway land"])
                   :p 2)}
{"3" (bolt-spec {"1" :shuffle "2" :shuffle}
                 split-sentence
                 :p 5)
"4" (bolt-spec {"3" ["word"]}
                 word-count
                 :p 6)})
映射 Spout 和Bolt Spces 都是从组件ID到Correponding Spec的映��。组件ID必须在映��间唯一。就像在Java中定义Topology一��P��在一个Topology里，在申明bolts的输入时�Q�组件ID��用到�?br />

spout-spec

spout-spec 作�ؓSpout实现的参数和可选的关键字参��C��?�?目前唯一的可选参数是�Q�P, �q�个用来定义Spout的�ƈ行度。如果你忽略 :p, spout��会作�ؓ单一��d��执行�?/p>

bolt-spec

bolt-spec作�ؓbolt的输入声明参数和可选的关键字参��C��?code> 。输入声明是数据��ID到数据流�l�的一个映��。数据流ID可以用以下两�U��Ş式中的一�U�：

[==component id== ==stream id==]: 在组件上订阅指定��?br />
==component id==: 在组件上订阅默认��?br />

数据��组可以是以下中的一�?/p>

:shuffle: 订阅shuffle�l?br />
字段名称的向�? like ["id" "name"]: 订阅指定字段上的字段�l?/li>
:global: 订阅一�?global grouping
:all: subscribes with an all grouping
:direct: subscribes with a direct grouping

可以参�?Concepts 获得更多关于��组的信�? �q�里有一个示例来展示不同的方法来声明输入:

{["2" "1"] :shuffle  "3" ["field1" "field2"]  ["4" "2"] :global} 
输入声明��d��订阅三种���。他在组�?#8220;2”上定义流“1”�Q�是Shuffle分组方式。在�l��g"3"上订阅默认的���，是Fileds分组方式�Q�分�l�标准是"Field1"�?Field2"。在�l��g4上定义流“2”�Q�是Global分组方式�Q?br />跟Spout-Spec 方式�c�M���Q�bolt-spec目前唯一支持的关键参数是:p,�q�个用来定义bolt的�ƈ行度�?br />

shell-bolt-spec

shell-bolt-spec是用在non-JVM语言环境下来实现bolts。他作�ؓ参数输入�Q�命令行�E�序去跑。the name of the file implementing the bolt, an output specification, and then the same keyword arguments that bolt-spec accepts.

以下�?shell-bolt-spec的一个示�?/code>:

  (shell-bolt-spec {"1" :shuffle "2" ["id"]}  "python"  "mybolt.py"  ["outfield1" "outfield2"]  :p 25) 
  
输出声明的语法是在下面的defbolt部分详细描述。有如何在Storm上��用multilang的更多细节，请参阅��用非JVM语言

 
 defbolt
 defbolt is used for defining bolts in Clojure. Bolts have the  constraint that they must be serializable, and this is why you can't just reify  IRichBolt to implement a bolt (closures aren't serializable).  defbolt works around this restriction and provides a nicer syntax  for defining bolts than just implementing a Java interface.
 At its fullest expressiveness, defbolt supports parameterized  bolts and maintaining state in a closure around the bolt implementation. It also  provides shortcuts for defining bolts that don't need this extra functionality.  The signature for defbolt looks like the following:
 (defbolt name output-declaration *option-map &  impl)
 Omitting the option map is equivalent to having an option map of  {:prepare false}.
 Simple bolts
 Let's start with the simplest form of defbolt. Here's an example  bolt that splits a tuple containing a sentence into a tuple for each word:
  (defbolt split-sentence ["word"] [tuple collector]  (let [words (.split (.getString tuple 0) " ")]  (doseq [w words]  (emit-bolt! collector [w] :anchor tuple))  (ack! collector tuple)  )) 
 Since the option map is omitted, this is a non-prepared bolt. The DSL simply  expects an implementation for the execute method of  IRichBolt. The implementation takes two parameters, the tuple and  the OutputCollector, and is followed by the body of the  execute function. The DSL automatically type-hints the parameters  for you so you don't need to worry about reflection if you use Java interop.
 This implementation binds split-sentence to an actual  IRichBolt object that you can use in topologies, like so:
  (bolt-spec {"1" :shuffle}  split-sentence  :p 5) 
 Parameterized bolts
 Many times you want to parameterize your bolts with other arguments. For  example, let's say you wanted to have a bolt that appends a suffix to every  input string it receives, and you want that suffix to be set at runtime. You do  this with defbolt by including a :params option in the  option map, like so:
  (defbolt suffix-appender ["word"] {:params [suffix]}  [tuple collector]  (emit-bolt! collector [(str (.getString tuple 0) suffix)] :anchor tuple)  ) 
 Unlike the previous example, suffix-appender will be bound to a  function that returns an IRichBolt rather than be an  IRichBolt object directly. This is caused by specifying  :params in its option map. So to use suffix-appender  in a topology, you would do something like:
  (bolt-spec {"1" :shuffle}  (suffix-appender "-suffix")  :p 10) 
 Prepared bolts
 To do more complex bolts, such as ones that do joins and streaming  aggregations, the bolt needs to store state. You can do this by creating a  prepared bolt which is specified by including {:prepare true} in  the option map. Consider, for example, this bolt that implements word  counting:
  (defbolt word-count ["word" "count"] {:prepare true}  [conf context collector]  (let [counts (atom {})]  (bolt  (execute [tuple]  (let [word (.getString tuple 0)]  (swap! counts (partial merge-with +) {word 1})  (emit-bolt! collector [word (@counts word)] :anchor tuple)  (ack! collector tuple)  ))))) 
 The implementation for a prepared bolt is a function that takes as input the  topology config, TopologyContext, and OutputCollector,  and returns an implementation of the IBolt interface. This design  allows you to have a closure around the implementation of execute  and cleanup. 
 In this example, the word counts are stored in the closure in a map called  counts. The bolt macro is used to create the  IBolt implementation. The bolt macro is a more concise  way to implement the interface than reifying, and it automatically type-hints  all of the method parameters. This bolt implements the execute method which  updates the count in the map and emits the new word count.
 Note that the execute method in prepared bolts only takes as  input the tuple since the OutputCollector is already in the closure  of the function (for simple bolts the collector is a second parameter to the  execute function).
 Prepared bolts can be parameterized just like simple bolts.
 Output declarations
 The Clojure DSL has a concise syntax for declaring the outputs of a bolt. The  most general way to declare the outputs is as a map from stream id a stream  spec. For example:
  {"1" ["field1" "field2"]  "2" (direct-stream ["f1" "f2" "f3"])  "3" ["f1"]} 
 The stream id is a string, while the stream spec is either a vector of fields  or a vector of fields wrapped by direct-stream. direct  stream marks the stream as a direct stream (See Concepts and  Direct  groupings for more details on direct streams).
 If the bolt only has one output stream, you can define the default stream of  the bolt by using a vector instead of a map for the output declaration. For  example:
  ["word" "count"] 
This declares the output of the bolt as the fields ["word" "count"]  on the default stream id.  Emitting, acking, and failing
 Rather than use the Java methods on OutputCollector directly,  the DSL provides a nicer set of functions for using  OutputCollector: emit-bolt!,  emit-direct-bolt!, ack!, and fail!.
 emit-bolt!: takes as parameters the  OutputCollector, the values to emit (a Clojure sequence), and  keyword arguments for :anchor and :stream.  :anchor can be a single tuple or a list of tuples, and  :stream is the id of the stream to emit to. Omitting the keyword  arguments emits an unanchored tuple to the default stream.
emit-direct-bolt!: takes as parameters the  OutputCollector, the task id to send the tuple to, the values to  emit, and keyword arguments for :anchor and :stream.  This function can only emit to streams declared as direct streams.
ack!: takes as parameters the OutputCollector and  the tuple to ack.
fail!: takes as parameters the OutputCollector and  the tuple to fail.
 See Guaranteeing  message processing for more info on acking and anchoring.
 defspout
 defspout is used for defining spouts in Clojure. Like bolts,  spouts must be serializable so you can't just reify IRichSpout to  do spout implementations in Clojure. defspout works around this  restriction and provides a nicer syntax for defining spouts than just  implementing a Java interface.
 The signature for defspout looks like the following:
 (defspout name output-declaration *option-map  & impl)
 If you leave out the option map, it defaults to {:prepare true}. The output  declaration for defspout has the same syntax as  defbolt.
 Here's an example defspout implementation from storm-starter:
  (defspout sentence-spout ["sentence"]  [conf context collector]  (let [sentences ["a little brown dog"  "the man petted the dog"  "four score and seven years ago"  "an apple a day keeps the doctor away"]]  (spout  (nextTuple []  (Thread/sleep 100)  (emit-spout! collector [(rand-nth sentences)])   )  (ack [id]  ;; You only need to define this method for reliable spouts  ;; (such as one that reads off of a queue like Kestrel)  ;; This is an unreliable spout, so it does nothing here  )))) 
 The implementation takes in as input the topology config,  TopologyContext, and SpoutOutputCollector. The  implementation returns an ISpout object. Here, the  nextTuple function emits a random sentence from  sentences. 
 This spout isn't reliable, so the ack and fail  methods will never be called. A reliable spout will add a message id when  emitting tuples, and then ack or fail will be called  when the tuple is completed or failed respectively. See Guaranteeing  message processing for more info on how reliability works within Storm.
 emit-spout! takes in as parameters the  SpoutOutputCollector and the new tuple to be emitted, and accepts  as keyword arguments :stream and :id.  :stream specifies the stream to emit to, and :id  specifies a message id for the tuple (used in the ack and  fail callbacks). Omitting these arguments emits an unanchored tuple  to the default output stream.
 There is also a emit-direct-spout! function that emits a tuple  to a direct stream and takes an additional argument as the second parameter of  the task id to send the tuple to.
 Spouts can be parameterized just like bolts, in which case the symbol is  bound to a function returning IRichSpout instead of the  IRichSpout itself. You can also declare an unprepared spout which  only defines the nextTuple method. Here is an example of an  unprepared spout that emits random sentences parameterized at runtime:
  (defspout sentence-spout-parameterized ["word"] {:params [sentences] :prepare false}  [collector]  (Thread/sleep 500)  (emit-spout! collector [(rand-nth sentences)])) 
 The following example illustrates how to use this spout in a  spout-spec:
  (spout-spec (sentence-spout-parameterized  ["the cat jumped over the door"  "greetings from a faraway land"])  :p 2) 
 Running topologies in local mode or on a cluster
 That's all there is to the Clojure DSL. To submit topologies in remote mode  or local mode, just use the StormSubmitter or  LocalCluster classes just like you would from Java.
 To create topology configs, it's easiest to use the  backtype.storm.config namespace which defines constants for all of  the possible configs. The constants are the same as the static constants in the  Config class, except with dashes instead of underscores. For  example, here's a topology config that sets the number of workers to 15 and  configures the topology in debug mode:
  {TOPOLOGY-DEBUG true  TOPOLOGY-WORKERS 15} 
 Testing topologies
 This  blog post and its follow-up  give a good overview of Storm's powerful built-in facilities for testing  topologies in Clojure.



徐红�? 2012-01-19 15:38 发表评论



徐红�? — Thu, 19 Jan 2012 06:29:00 GMT
  本文���译至Storm官方Wiki, �Ƣ迎转蝲�Q��{载请注明出处�Q?div>   http://www.aygfsteel.com/xuhongxing016/articles/368731.html
   初次���译�Q�英文好的同学，可以查看英文文档�Q?https://github.com/nathanmarz/storm/wiki/Serialization
    本文是介�l�关于Storm 0.6.0及以上版本的序列化系�l�，Storm 在之前版本��用了另外一套序列化�pȝ���?br />    Tuple可以包含��M���c�d��的对象�?/span>�׃��Storm是一个分布式�pȝ���Q�它需要知道�Q务之间传递对象时�Q�怎样序列化和反序列化对象�?br />    Storm使用Kryo来进行序列化�?Kryo是一个灵�z�d��快速的的序列化库，序列化对象较����?/span>默认情况下，Storm可以序列化的原始�c�d���Q�字�W�串�Q�字节数�l�的 ArrayList�Q�HashMap�Q�HashSet和Clojure的集合类型�?/span>如果你想在你的元�l���用另一�U�类型，你需要注册一个自定义序列化�?br />
动态类�?/strong>
Tuple中的字段没有�q�行�c�d��声明�?/span>你把对象攑ֈ�Fields里，Storm动�?/span>序列化对象�?br />在我们获得序列化接口之前�Q�让我们��q��旉���理解Storm 的Tuple ��Z��么是动态类型�?br />��d��静态类型的到Tuple 的Fileds�Q�将会给Storm的API增加大量的复杂性�?/span>例如�Q�Hadoop�Q�静态类型的键和��|��在用户��用时�Q�需要的很多注释�?Hadoop的API使用��h��比较�ȝ���Q?#8220;�c�d��安全”是不值得的�?/span>动态类型是���单容易��用�?br />很难用一�U�合理的静态方式来�l�计Storm的Tuples。假设一个Bolt订阅了多个流�Q�这些流的Tuple可能在Fields上有不同的类型，当一个Bolt在执行阶�D�|��受Tuple�Q�这些Tuple可能来自于�Q何流�Q?/span>�q�有可能使�Q何类型的�l�合�?/span>可能有一些反���魔法，你可以�ؓ每一个被Bolt订阅的Tuple���定义不同的�Ҏ���Q�但�?/span>
�Q�但���单，直接的方法是动态类型�?br />最后，使用动态类型的另一个原因是Storm可以以一�U�简单直接的方式来��用类��g��Clojure和JRuby的动态类型语�a�
自定义序列化
如前所�q�ͼ�Storm使用Kryo来进行序列化
�?/span>要实现自定义的序列化�Q�您需要注册新的序列化与Kryo�?/span>强烈�����您通过Kryo的主��阅�ȝ��解它是如何处理自定义序列�?br />
��d��自定义的序列化是在Topology的配�|�属�?#8220;topology.kryo.register”完成的�?/span>它需要一个注册的清单�Q�其中每个登记可以采取以下两�U��Ş式之一�Q?br />
1�?需要注册登记的�cȝ��名称�?/span>在这�U�情况下�Q�Storm�����用Kryo的FieldsSerializer来序列化�c�R�?/span>�q�可能是最佳的�c�，也可能不�?- 更多�l�节见Kryo文档�?br />2�?映射下类的名字来注册登记com.esotericsoftware.kryo.Serializer实施�?br />让我们来看看一个例子�?/span>
 topology.kryo.register:  
 - com.mycompany.CustomType1  
 - com.mycompany.CustomType2: com.mycompany.serializer.CustomType2Serializer 
 - com.mycompany.CustomType3 
com.mycompany.CustomType1 �? com.mycompany.CustomType3 使用  FieldsSerializer 来进行序列化
com.mycompany.CustomType2 使用 com.mycompany.serializer.CustomType2Serializer 来进行序列化
注册序列化器�?/span>
的帮助。配�|�类有一个名为registerSerialization一个方法，在注册时��d��到配�|�里
有更高阶的配�|�称为Config.TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS。如果设�|��ؓtrue�Q�Storm���忽略�Q何已注册的，但在classpath�?div>没有自己的代�?/span>
可用的序列化。否则，Storm会引发错误时�Q�它无法扑ֈ�一个序列化。如果你�q�行多个���集上，每个人都有不同的序列化的拓扑�l�构�Q�这是有用的�?br />但要声明所有在storm.yaml文�g拓扑中所有的序列化�?br />Java序列�?br />Storm如果遇到一个类型，它没有一个序列化注册�Q�它有可能��用Java序列化。如果对象不能被Java序列化程序序列，Storm会抛��Z��个错误�?br />
Java序列化是极其昂贵的，无论是在CPU成本以及序列化的对象的大���。强烈徏议您在Topology上生产环境之前注册定制序列�?Java序列化在处理Topology原型的时候就是这��L���?br />
可以通过配置文�g来关闭Java序列化功能，只需���Config.TOPOLOGY_FALL_BACK_ON_JAVA_SERIALIZATION配置讄���为false






徐红�? 2012-01-19 14:29 发表评论


Twitter Storm 分布式RPC
徐红�? — Thu, 19 Jan 2012 06:17:00 GMT
本文转蝲�?div> 
http://chenlx.blog.51cto.com/4096635/748348
本来准备自己���译一下，后来Google一下，发现�|�上已有�Q�遂转蝲�?br /> 

分布式RPC
 分布式RPC�Q�DRPC�Q�的真正目的是��用storm实时�q�行计算极端功能。Storm拓扑需要一个输入流作�ؓ函数参数�Q�以一个输出流的�Ş式发���每个函数调用的�l�果�?/div>  
 DRPC没有多少storm�Ҏ��，因�ؓ它是从storm的原始流�Q�spouts�Q�bolts�Q�拓扑来表达一个模式。DRPC没有单独打包�Q�但它如此有用，以至于和storm捆绑在一赗��?/div>  
 概述
 分布式RPC通过“DRPC  server”协调。DRPC服务器协调接收一个RPC��h���Q�发送请求到storm拓扑�Q�从storm拓扑接收�l�果�Q�发送结果回�{�待的客��L��。从一个客��L��的角度来看，一个分布式RPC调用���像是一个常规的RPC调用。例如，一个客��L��如何为带“http://twitter.com”参数�?#8220;reach”功能计算�l�果�?/div>DRPCClient client = new DRPCClient("drpc-host", 3772); 
String result = client.execute("reach", "http://twitter.com"); 
  分布式RPC工作���程如下�Q?/div> 
  客户端发送功能名�U�及功能所需参数�?/span>DRPC服务器去执行�?/span>图中的拓扑实��C��此功能，它��用DRPCSpout从DRPC服务器接收功能调用流。每个功能调用通过DRPC服务器��用唯一ID标记�Q�随后拓扑计���结果，在拓扑的最后，一个称之�ؓ“ReturnResults”的bolt�q�接到DRPC服务器，把结果交�l�这个功能调用（�Ҏ��功能调用ID�Q�，DRPC服务器根据ID扑ֈ��{�待中的客户端，为等待中的客��L��消除��d���Q��ƈ发送结果给客户端�?/p> 
LinearDRPCTopologyBuilder
 Storm有一个称之�ؓLinearDRPCTopologyBuilder的拓扑Builder几乎自动完成DRPC所需的所有相��x��骤。包括：
 1.讄���spout
 2.�q�回�l�果�l�DRPC服务�?/div> 3.为bolt提供对一�l�元�l�的有限聚合功能
 让我们看一个简单的例子。这是一个DRPC拓扑的实玎ͼ�在输入参数的���N���q�加“�Q?#8221;�q�返回：
public static class ExclaimBolt implements IBasicBolt { 
    public void prepare(Map conf, TopologyContext context) { 
    } 
 
    public void execute(Tuple tuple, BasicOutputCollector collector) { 
        String input = tuple.getString(1); 
        collector.emit(new Values(tuple.getValue(0), input + "!")); 
    } 
 
    public void cleanup() { 
    } 
 
    public void declareOutputFields(OutputFieldsDeclarer declarer) { 
        declarer.declare(new Fields("id", "result")); 
    } 
 
} 
 
public static void main(String[] args) throws Exception { 
    LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("exclamation"); 
    builder.addBolt(new ExclaimBolt(), 3); 
    // ... 
} 
 如你所见，代码非常���。当创徏LinearDRPCTopologyBuilder�Ӟ��你把�q�个拓扑的DRPC功能名称告诉storm。一个DRPC服务器可以协调许多功能，功能名称用于区别不同的功能，首先声明的bolt���接收一个输入的2-tuples�Q�第一个字�D�|����h��ID�Q�第二个字段是请求参数。LinearDRPCTopologyBuilder认�ؓ最后的bolt会发���一个输出流�Q�该输出���包含[id,  result]格式�?-tuples。最后，所有拓扑中间过�E���生的元组�Q�tuple�Q�都包含��h��id作�ؓ其第一个字�D�c�?/p> 
 在这个例子中�Q�ExclaimBolt只是���单地在元�l�的�W�二个字�D�尾部追�?#8220;�Q?#8221;字符。LinearDRPCTopologyBuilder处理其余的协调工作，包括�q�接DRPC服务器，发送最�l�结果�?/div>  
 本地模式DRPC
  DRPC可以�q�行在本地模式。这是如何在本地模式�q�行上述例子�Q?/div>LocalDRPC drpc = new LocalDRPC(); 
LocalCluster cluster = new LocalCluster(); 
 
cluster.submitTopology("drpc-demo", conf, builder.createLocalTopology(drpc)); 
 
System.out.println("Results for 'hello':" + drpc.execute("exclamation", "hello")); 
 
cluster.shutdown(); 
drpc.shutdown(); 
  首先你创��Z��个LocalDRPC对象。这个对象在�q�程内模拟一个DRPC服务器，���像在进�E�内模拟一个storm集群一栗���然后你创徏本地集群�Q�在本地模式�q�行�q�个拓扑。创建本地拓扑和�q�程拓扑�Q�LinearDRPCTopologyBuilder有不同的�Ҏ��。在本地模式�Q�LocalDRPC未绑定�Q何端口，拓扑也需要知道与哪个对象通讯�Q�这是�ؓ什么createLocaclTopology�Ҏ��需要接受LocalDRPC对象作�ؓ输入参数的原因�?/div>  
 载入拓扑后，你可以用LocalDRPC的execute�Ҏ��执行DRPC调用�?/div>  
 �q�程模式DRPC
  在实际的集群使用DRPC也很���单。有三个步骤�Q?/div> 1.      启动DRPC服务�?/div> 2.      配置DRPC服务器位�|?/div> 3.      提交DRPC拓扑到storm集群
 使用storm脚本启动DRPC服务器，和启动nimbus和ui一��P��
bin/storm drpc 
  接下来，配置你的storm集群�Q�让集群知道DRPC服务器的位置�Q�这样DRPCSpout���q��道从哪里��d��功能调用。可以通过修改storm.yaml配置文�g或拓扑配�|�完成配�|�DRPC服务器位�|�。修改storm.yaml配置文�g如下所�C�：
drpc.servers: 
  - "drpc1.foo.com" 
  - "drpc2.foo.com" 
  最后，使用StormSubmitter启动DRPC拓扑�Q�就像启动其它拓扑一栗���在�q�程模式�q�行上述例子�Q�代码如下所�C�：
StormSubmitter.submitTopology("exclamation-drpc", conf, builder.createRemoteTopology()); 
  createRemoteTopology�Ҏ��用于在storm集群创徏拓扑�?/div>  
 一个更完整的例�?/strong>
  �q�个exclaimation  DRPC例子只是一个用来说明DRPC概念的玩兗���让我们看一个更完整的例子，该例子是一个真正需要storm集群的�ƈ行计���的DRPC功能。我们将要看的例子是对twitter�|�站上的一个URL的接触用戯���行统计�?/div> 一个URL的接触用��h��是在twitter�|�站上接触一个URL的用��h���Q�你需要以�?步：
 1. 获取tweeted the URL的全部用�?/div> 2. 获取�q�些用户的全部追随�?/div> 3. 使追随者集合中的用户唯一
 4. �l�计唯一的用��h��
 一个单独的reach计算在计���期间涉及到数千数据库访问和数千万追随者记录。它是一个真正的耗时计算。正如你���要看到的，在storm上实现这个功能非常简单。在一台机器上�Q�reach计算��p��数分钟，在storm集群�Q�最难计���reach的URL也只需数秒�?/div> Storm-starter��目  �q�个拓扑�?个步骤的形式执行�Q?/div> 1. GetTweeters获取tweeted the URL的用戗���它转换一个[id, url]形式的输入流到[id,  tweeter]形式的输出流。每个url元组���映���到多个tweeter元组�?/p> 
2. GetFollowers获取�q�些tweeter的追随者。它转换一个[id, tweeter]形式的输入流到[id,  follower]形式的输出流。跨所有�Q务，当某�����随多个tweeter�Q�这些tweeter又tweeted相同的URL�Ӟ���q�可能会得到重复的追随者�?/p> 3. PartialUniquer按追随者ID对追随者数据流�q�行分组。同一的追随者去到同一的�Q务，因此每个PartialUniquer��d��都接收到独立的相互独立的�q�随者集合。PartialUniquer一旦收到请求ID用于它的所有追随者元�l�，它就发射�q�随者子集的唯一��L���?/div> 4. 最后，CountAggregator从每个PartialUniquer��d��接收计数�q�对它们求和�? 让我们来看看PartialUniquer�Q?/div>public static class PartialUniquer implements IRichBolt, FinishedCallback { 
    OutputCollector _collector; 
    Map> _sets = new HashMap>(); 
     
    public void prepare(Map conf, TopologyContext context, OutputCollector collector) { 
        _collector = collector; 
    } 
 
    public void execute(Tuple tuple) { 
        Object id = tuple.getValue(0); 
        Set curr = _sets.get(id); 
        if(curr==null) { 
            curr = new HashSet(); 
            _sets.put(id, curr); 
        } 
        curr.add(tuple.getString(1)); 
        _collector.ack(tuple); 
    } 
 
    public void cleanup() { 
    } 
 
    public void finishedId(Object id) { 
        Set curr = _sets.remove(id); 
        int count; 
        if(curr!=null) { 
            count = curr.size(); 
        } else { 
            count = 0; 
        } 
        _collector.emit(new Values(id, count)); 
    } 
 
    public void declareOutputFields(OutputFieldsDeclarer declarer) { 
        declarer.declare(new Fields("id", "partial-count")); 
    } 
} 
 当PartialUniquer在exectue�Ҏ��中接收一个follower元组�Ӟ��它用一个内部HashMap��d��它到与请求ID对应的集合�?/div> PartialUniquer也实��C��FinishedCallback接口�Q�它告诉LinearDRPCTopologyBuilder�Q�对于�Q意给定的��h��ID�Q�当它已收到所有指向它的元�l�时�Q�请通知它。这个回调是finishedId�Ҏ��。在�q�个回调中，PartialUniquer发射单一的元�l�，元组包含它的�q�随者子集的唯一��L���?nbsp;
 在底层，CoordinatedBolt用于������一个bolt何时收到该请求ID的所有元�l�。CoordinatedBolt使用direct  stream���理协调�?/p> 
其它的拓扑应该是不言自明。如你所见，reach计算的每一单步都是�q�行执行的，而且定义一个DRPC拓扑也非常简单�?/div>  Non-Linear DRPC拓扑
 LinearDRPCTopologyBuilder仅处�?#8220;�U�性的”DRPC拓扑�Q�计���以一�q�串步骤的�Ş式表达（像reach�Q�。不难想象某些功能将需要更复杂的拓扑结构，�q�些拓扑带有带分支和合�ƈbolt。目前，要做到这一点，你需要直接��用CoordinateBolt。务必在邮�g列表中谈谈你的非�U�性DRPC拓扑用例�Q�写下DRPC拓扑更普遍的抽象�l�构�?/div> LinearDRPCTopologyBuilder如何工作�Q?/strong>
 DRPCSpout发射[args, return-info]�Q�return-info是DRPC服务器的��L��和端口，�q�有DRPC服务器生成的ID�?/p> 
拓扑�l�成部分�Q?/div> DRPCSpout
PrepareRequest�Q�生成一个请求ID�Q�创��Z��个返回信息流�Q�一个参数流�Q?/li>
CoordinatedBolt包装器和直接分组
JoinResult�Q�同�q�回信息一赯���接结果）
ReturnResult�Q�连接DRPC服务器�ƈ�q�回�l�果�Q?/li>
LinearDRPCTopologyBuilder是一个构建在Storm原语之上的高层次抽象的好例子�?/li>
 高��
 同时�~�排处理多个��L��KeyedFairBolt
如何直接使用CoordinateBolt


徐红�? 2012-01-19 14:17 发表评论


徐红�? — Sat, 14 Jan 2012 09:08:00 GMT
Storm����?br />Storm是一个分布式的、容错的实时计算�pȝ���Q�可以方便地在一个计���机集群中编写与扩展复杂的实时计���。在���量领域里，Storm用于实时数据的处理，Hadoop用于�Ҏ��据的处理�Q�两者可以说是绝代双雄！Storm保证每个消息都会得到处理�Q�而且它很�?#8212;—在一个小集群中，每秒可以处理��C��百万计的消息�?br />Storm的优�?br />1. ���单的�~�程模型。类��g��MapReduce降低了�ƈ行批处理复杂性，Storm降低了进行实时处理的复杂性�?br />2. 服务�?一个服务框�?支持热部�|?��x��上线或下�U�App.
3. 可以使用各种�~�程语言。你可以在Storm之上使用各种�~�程语言。默认支持Clojure、Java、Ruby和Python。要增加对其他语�a�的支持，只需实现一个简单的Storm通信协议卛_���?br />4. 定w��性。Storm会管理工作进�E�和节点的故障�?br />5. 水��^扩展。计���是在多个线�E�、进�E�和服务器之间�ƈ行进行的�?br />6. 可靠的消息处理。Storm保证每个消息臛_��能得��C���ơ完整处理。�Q务失败时�Q�它会负责从消息源重试消息�?br />7. 快速。系�l�的设计保证了消息能得到快速的处理�Q���用ZeroMQ作�ؓ其底层消息队列�?br />8. 本地模式。Storm有一�?#8220;本地模式”�Q�可以在处理�q�程中完全模拟Storm集群。这让你可以快速进行开发和单元���试�?br />有优点，必定有缺炏V��不�q�相�Ҏ����_��我觉得这�?span style="font-family: 'Arial', 'sans-serif'; color: #333333; font-size: 9pt; mso-font-kerning: 0pt; mso-fareast-font-family: 宋体" xml:lang="EN-US" lang="EN-US">问题都不�?br />1. 目前的开源版本中只是单节�?/span>Nimbus�Q�我们可以在生��环境实现一个双nimbus的布局�Q��?/span>
2. Clojure是一个在JVM�q�_���q�行的动态函数式�~�程语言,优势在于���程计算�Q?/span>Storm的部分核心内容由Clojure�~�写�Q�虽然性能上提高不���但同时也提升了�l�护成本--学学Clojure也很不错�Q�只要你融入里面�?br />
Storm架构


Storm集群�׃��个主节点和多个工作节点组成，分布式的架构大多如此�Q�没什么好说的。主节点�q�行了一个名�?/span>“Nimbus”的守护进�E�，用于分配代码、布�|��Q务及故障������。每个工作节炚w���q�行了一个名�?/span>“Supervisor”的守护进�E�，用于监听工作�Q�开始�ƈ�l�止工作�q�程�?/span>Nimbus�?/span>Supervisor都能快速失败，而且是无状态的�Q�这样十分健壮，两者的协调当然是由Zookeeper来完成的�Q?/span>ZooKeeper用于���理集群中的不同�l��g�Q?/span>ZeroMQ是内部消息系�l�，JZMQ�?/span>ZeroMQMQ�?/span>Java Binding�?br />

徐红�? 2012-01-14 17:08 发表评论