posts - 110, comments - 101, trackbacks - 0, articles - 7
            BlogJava :: 首頁 :: 新隨筆 :: 聯(lián)系 :: 聚合  :: 管理

          ZooKeeper是一個分布式開源框架,提供了協(xié)調(diào)分布式應(yīng)用的基本服務(wù),它向外部應(yīng)用暴露一組通用服務(wù)——分布式同步(Distributed Synchronization)、命名服務(wù)(Naming Service)、集群維護(Group Maintenance)等,簡化分布式應(yīng)用協(xié)調(diào)及其管理的難度,提供高性能的分布式服務(wù)。ZooKeeper本身可以以Standalone模式安裝運行,不過它的長處在于通過分布式ZooKeeper集群(一個Leader,多個Follower),基于一定的策略來保證ZooKeeper集群的穩(wěn)定性和可用性,從而實現(xiàn)分布式應(yīng)用的可靠性。

          有關(guān)ZooKeeper的介紹,網(wǎng)上很多,也可以參考文章后面,我整理的一些相關(guān)鏈接。

          下面,我們簡單說明一下ZooKeeper的配置。


          ZooKeeper Standalone模式


          從Apache網(wǎng)站上(zookeeper.apache.org)下載ZooKeeper軟件包,我選擇了3.3.4版本的(zookeeper-3.3.4.tar.gz),在一臺Linux機器上安裝非常容易,只需要解壓縮后,簡單配置一下即可以啟動ZooKeeper服務(wù)器進程。

          將zookeeper-3.3.4/conf目錄下面的 zoo_sample.cfg修改為zoo.cfg,配置文件內(nèi)容如下所示:

          [plain] view plaincopy
          1. tickTime=2000  
          2. dataDir=/home/hadoop/storage/zookeeper  
          3. clientPort=2181  
          4. initLimit=5  
          5. syncLimit=2  
          上面各個配置參數(shù)的含義也非常簡單,引用如下所示:

          [plain] view plaincopy
          1. tickTime —— the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.  
          2. dataDir —— the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.  
          3. clientPort —— the port to listen for client connections  
          下面啟動ZooKeeper服務(wù)器進程:

          [plain] view plaincopy
          1. cd zookeeper-3.3.4/  
          2. bin/zkServer.sh start  
          通過jps命令可以查看ZooKeeper服務(wù)器進程,名稱為QuorumPeerMain。

          在客戶端連接ZooKeeper服務(wù)器,執(zhí)行如下命令:

          [plain] view plaincopy
          1. bin/zkCli.sh -server dynamic:2181  
          上面dynamic是我的主機名,如果在本機執(zhí)行,則執(zhí)行如下命令即可:
          [plain] view plaincopy
          1. bin/zkCli.sh  
          客戶端連接信息如下所示:

          接著,可以使用help查看Zookeeper客戶端可以使用的基本操作命令。



          ZooKeeper Distributed模式


          ZooKeeper分布式模式安裝(ZooKeeper集群)也比較容易,這里說明一下基本要點。

          首先要明確的是,ZooKeeper集群是一個獨立的分布式協(xié)調(diào)服務(wù)集群,“獨立”的含義就是說,如果想使用ZooKeeper實現(xiàn)分布式應(yīng)用的協(xié)調(diào)與管理,簡化協(xié)調(diào)與管理,任何分布式應(yīng)用都可以使用,這就要歸功于Zookeeper的數(shù)據(jù)模型(Data Model)和層次命名空間(Hierarchical Namespace)結(jié)構(gòu),詳細(xì)可以參考http://zookeeper.apache.org/doc/trunk/zookeeperOver.html。在設(shè)計你的分布式應(yīng)用協(xié)調(diào)服務(wù)時,首要的就是考慮如何組織層次命名空間。

          下面說明分布式模式的安裝配置,過程如下所示:

          第一步:主機名稱到IP地址映射配置

          ZooKeeper集群中具有兩個關(guān)鍵的角色:Leader和Follower。集群中所有的結(jié)點作為一個整體對分布式應(yīng)用提供服務(wù),集群中每個結(jié)點之間都互相連接,所以,在配置的ZooKeeper集群的時候,每一個結(jié)點的host到IP地址的映射都要配置上集群中其它結(jié)點的映射信息。

          例如,我的ZooKeeper集群中每個結(jié)點的配置,以slave-01為例,/etc/hosts內(nèi)容如下所示:

          [plain] view plaincopy
          1. 192.168.0.179   slave-01  
          2. 192.168.0.178   slave-02  
          3. 192.168.0.177   slave-03  

          ZooKeeper采用一種稱為Leader election的選舉算法。在整個集群運行過程中,只有一個Leader,其他的都是Follower,如果ZooKeeper集群在運行過程中Leader出了問題,系統(tǒng)會采用該算法重新選出一個Leader。因此,各個結(jié)點之間要能夠保證互相連接,必須配置上述映射。

          ZooKeeper集群啟動的時候,會首先選出一個Leader,在Leader election過程中,某一個滿足選舉算的結(jié)點就能成為Leader。整個集群的架構(gòu)可以參考http://zookeeper.apache.org/doc/trunk/zookeeperOver.html#sc_designGoals

          第二步:修改ZooKeeper配置文件

          在其中一臺機器(slave-01)上,解壓縮zookeeper-3.3.4.tar.gz,修改配置文件conf/zoo.cfg,內(nèi)容如下所示:

          [plain] view plaincopy
          1. tickTime=2000  
          2. dataDir=/home/hadoop/storage/zookeeper  
          3. clientPort=2181  
          4. initLimit=5  
          5. syncLimit=2  
          6. server.1=slave-01:2888:3888  
          7. server.2=slave-02:2888:3888  
          8. server.3=slave-03:2888:3888  
          上述配置內(nèi)容說明,可以參考http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

          第三步:遠程復(fù)制分發(fā)安裝文件

          上面已經(jīng)在一臺機器slave-01上配置完成ZooKeeper,現(xiàn)在可以將該配置好的安裝文件遠程拷貝到集群中的各個結(jié)點對應(yīng)的目錄下:

          [plain] view plaincopy
          1. cd /home/hadoop/installation/  
          2. scp -r zookeeper-3.3.4/ hadoop@slave-02:/home/hadoop/installation/  
          3. scp -r zookeeper-3.3.4/ hadoop@slave-03:/home/hadoop/installation/  
          第四步:設(shè)置myid

          在我們配置的dataDir指定的目錄下面,創(chuàng)建一個myid文件,里面內(nèi)容為一個數(shù)字,用來標(biāo)識當(dāng)前主機,conf/zoo.cfg文件中配置的server.X中X為什么數(shù)字,則myid文件中就輸入這個數(shù)字,例如:

          [plain] view plaincopy
          1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ echo "1" > /home/hadoop/storage/zookeeper/myid  
          2. hadoop@slave-02:~/installation/zookeeper-3.3.4$ echo "2" > /home/hadoop/storage/zookeeper/myid  
          3. hadoop@slave-03:~/installation/zookeeper-3.3.4$ echo "3" > /home/hadoop/storage/zookeeper/myid  
          按照上述進行配置即可。

          第五步:啟動ZooKeeper集群

          在ZooKeeper集群的每個結(jié)點上,執(zhí)行啟動ZooKeeper服務(wù)的腳本,如下所示:

          [plain] view plaincopy
          1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
          2. hadoop@slave-02:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
          3. hadoop@slave-03:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
          以結(jié)點slave-01為例,日志如下所示:
          [plain] view plaincopy
          1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ tail -500f zookeeper.out   
          2. 2012-01-08 06:51:19,117 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
          3. 2012-01-08 06:51:19,133 - INFO  [main:QuorumPeerConfig@310] - Defaulting to majority quorums  
          4. 2012-01-08 06:51:19,167 - INFO  [main:QuorumPeerMain@119] - Starting quorum peer  
          5. 2012-01-08 06:51:19,227 - INFO  [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181  
          6. 2012-01-08 06:51:19,277 - INFO  [main:QuorumPeer@819] - tickTime set to 2000  
          7. 2012-01-08 06:51:19,278 - INFO  [main:QuorumPeer@830] - minSessionTimeout set to -1  
          8. 2012-01-08 06:51:19,279 - INFO  [main:QuorumPeer@841] - maxSessionTimeout set to -1  
          9. 2012-01-08 06:51:19,281 - INFO  [main:QuorumPeer@856] - initLimit set to 5  
          10. 2012-01-08 06:51:19,347 - INFO  [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888  
          11. 2012-01-08 06:51:19,393 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING  
          12. 2012-01-08 06:51:19,396 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id =  1, Proposed zxid = 0  
          13. 2012-01-08 06:51:19,400 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)  
          14. 2012-01-08 06:51:19,416 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
          15. java.net.ConnectException: Connection refused  
          16.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          17.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          18.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          19.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          20.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
          21.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
          22.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
          23.         at java.lang.Thread.run(Thread.java:662)  
          24. 2012-01-08 06:51:19,420 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          25. java.net.ConnectException: Connection refused  
          26.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          27.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          28.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          29.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          30.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
          31.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
          32.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
          33.         at java.lang.Thread.run(Thread.java:662)  
          34. 2012-01-08 06:51:19,612 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
          35. java.net.ConnectException: Connection refused  
          36.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          37.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          38.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          39.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          40.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          41.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          42.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          43. 2012-01-08 06:51:19,615 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          44. java.net.ConnectException: Connection refused  
          45.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          46.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          47.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          48.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          49.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          50.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          51.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          52. 2012-01-08 06:51:19,616 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 400  
          53. 2012-01-08 06:51:20,019 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
          54. java.net.ConnectException: Connection refused  
          55.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          56.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          57.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          58.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          59.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          60.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          61.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          62. 2012-01-08 06:51:20,021 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          63. java.net.ConnectException: Connection refused  
          64.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          65.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          66.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          67.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          68.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          69.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          70.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          71. 2012-01-08 06:51:20,022 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 800  
          72. 2012-01-08 06:51:20,825 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
          73. java.net.ConnectException: Connection refused  
          74.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          75.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          76.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          77.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          78.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          79.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          80.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          81. 2012-01-08 06:51:20,827 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          82. java.net.ConnectException: Connection refused  
          83.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          84.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          85.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          86.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          87.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          88.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          89.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          90. 2012-01-08 06:51:20,828 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 1600  
          91. 2012-01-08 06:51:22,435 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
          92. java.net.ConnectException: Connection refused  
          93.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          94.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          95.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          96.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          97.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          98.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          99.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          100. 2012-01-08 06:51:22,439 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          101. java.net.ConnectException: Connection refused  
          102.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          103.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          104.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          105.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          106.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          107.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          108.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          109. 2012-01-08 06:51:22,441 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 3200  
          110. 2012-01-08 06:51:22,945 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)  
          111. 2012-01-08 06:51:22,946 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal  
          112. 2012-01-08 06:51:22,949 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
          113. java.net.ConnectException: Connection refused  
          114.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
          115.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
          116.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
          117.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          118.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
          119.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
          120.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
          121.         at java.lang.Thread.run(Thread.java:662)  
          122. 2012-01-08 06:51:22,951 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)  
          123. 2012-01-08 06:51:23,156 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING  
          124. 2012-01-08 06:51:23,170 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true  
          125. 2012-01-08 06:51:23,206 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT  
          126. 2012-01-08 06:51:23,207 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=slave-01  
          127. 2012-01-08 06:51:23,207 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_30  
          128. 2012-01-08 06:51:23,208 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.  
          129. 2012-01-08 06:51:23,208 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/home/hadoop/installation/jdk1.6.0_30/jre  
          130. 2012-01-08 06:51:23,209 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hadoop/installation/zookeeper-3.3.4/bin/../build/classes:/home/hadoop/installation/zookeeper-3.3.4/bin/../build/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../zookeeper-3.3.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/log4j-1.2.15.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/jline-0.9.94.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-lang-2.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-collections-3.2.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-cli-1.1.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-tasks-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-core-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../src/java/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../conf:/home/hadoop/installation/jdk1.6.0_30/lib/*.jar:/home/hadoop/installation/jdk1.6.0_30/jre/lib/*.jar  
          131. 2012-01-08 06:51:23,210 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386/client:/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386:/home/hadoop/installation/jdk1.6.0_30/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib  
          132. 2012-01-08 06:51:23,210 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp  
          133. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>  
          134. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux  
          135. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386  
          136. 2012-01-08 06:51:23,213 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-14-generic  
          137. 2012-01-08 06:51:23,213 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hadoop  
          138. 2012-01-08 06:51:23,214 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hadoop  
          139. 2012-01-08 06:51:23,214 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hadoop/installation/zookeeper-3.3.4  
          140. 2012-01-08 06:51:23,223 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@151] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hadoop/storage/zookeeper/version-2 snapdir /home/hadoop/storage/zookeeper/version-2  
          141. 2012-01-08 06:51:23,339 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader  
          142. 2012-01-08 06:51:23,358 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@325] - Setting leader epoch 1  
          143. 2012-01-08 06:51:23,358 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 0  
          144. 2012-01-08 06:51:25,511 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), FOLLOWING (my state)  
          145. 2012-01-08 06:51:42,584 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), FOLLOWING (my state)  
          我啟動的順序是slave-01>slave-02>slave-03,由于ZooKeeper集群啟動的時候,每個結(jié)點都試圖去連接集群中的其它結(jié)點,先啟動的肯定連不上后面還沒啟動的,所以上面日志前面部分的異常是可以忽略的。通過后面部分可以看到,集群在選出一個Leader后,最后穩(wěn)定了。

          其他結(jié)點可能也出現(xiàn)類似問題,屬于正常。

          第六步:安裝驗證

          可以通過ZooKeeper的腳本來查看啟動狀態(tài),包括集群中各個結(jié)點的角色(或是Leader,或是Follower),如下所示,是在ZooKeeper集群中的每個結(jié)點上查詢的結(jié)果:

          [plain] view plaincopy
          1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ bin/zkServer.sh status  
          2. JMX enabled by default  
          3. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
          4. Mode: follower  
          5.   
          6. hadoop@slave-02:~/installation/zookeeper-3.3.4$  bin/zkServer.sh status  
          7. JMX enabled by default  
          8. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
          9. Mode: leader  
          10.   
          11. hadoop@slave-03:~/installation/zookeeper-3.3.4$  bin/zkServer.sh status  
          12. JMX enabled by default  
          13. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
          14. Mode: follower  
          通過上面狀態(tài)查詢結(jié)果可見,slave-02是集群的Leader,其余的兩個結(jié)點是Follower。

          另外,可以通過客戶端腳本,連接到ZooKeeper集群上。對于客戶端來說,ZooKeeper是一個整體(ensemble),連接到ZooKeeper集群實際上感覺在獨享整個集群的服務(wù),所以,你可以在任何一個結(jié)點上建立到服務(wù)集群的連接,例如:

          [plain] view plaincopy
          1. hadoop@slave-03:~/installation/zookeeper-3.3.4$ bin/zkCli.sh -server slave-01:2181  
          2. Connecting to slave-01:2181  
          3. 2012-01-08 07:14:21,068 - INFO  [main:Environment@97] - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT  
          4. 2012-01-08 07:14:21,080 - INFO  [main:Environment@97] - Client environment:host.name=slave-03  
          5. 2012-01-08 07:14:21,085 - INFO  [main:Environment@97] - Client environment:java.version=1.6.0_30  
          6. 2012-01-08 07:14:21,089 - INFO  [main:Environment@97] - Client environment:java.vendor=Sun Microsystems Inc.  
          7. 2012-01-08 07:14:21,095 - INFO  [main:Environment@97] - Client environment:java.home=/home/hadoop/installation/jdk1.6.0_30/jre  
          8. 2012-01-08 07:14:21,104 - INFO  [main:Environment@97] - Client environment:java.class.path=/home/hadoop/installation/zookeeper-3.3.4/bin/../build/classes:/home/hadoop/installation/zookeeper-3.3.4/bin/../build/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../zookeeper-3.3.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/log4j-1.2.15.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/jline-0.9.94.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-lang-2.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-collections-3.2.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-cli-1.1.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-tasks-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-core-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../src/java/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../conf:/home/hadoop/installation/jdk1.6.0_30/lib/*.jar:/home/hadoop/installation/jdk1.6.0_30/jre/lib/*.jar  
          9. 2012-01-08 07:14:21,111 - INFO  [main:Environment@97] - Client environment:java.library.path=/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386/client:/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386:/home/hadoop/installation/jdk1.6.0_30/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib  
          10. 2012-01-08 07:14:21,116 - INFO  [main:Environment@97] - Client environment:java.io.tmpdir=/tmp  
          11. 2012-01-08 07:14:21,124 - INFO  [main:Environment@97] - Client environment:java.compiler=<NA>  
          12. 2012-01-08 07:14:21,169 - INFO  [main:Environment@97] - Client environment:os.name=Linux  
          13. 2012-01-08 07:14:21,175 - INFO  [main:Environment@97] - Client environment:os.arch=i386  
          14. 2012-01-08 07:14:21,177 - INFO  [main:Environment@97] - Client environment:os.version=3.0.0-14-generic  
          15. 2012-01-08 07:14:21,185 - INFO  [main:Environment@97] - Client environment:user.name=hadoop  
          16. 2012-01-08 07:14:21,188 - INFO  [main:Environment@97] - Client environment:user.home=/home/hadoop  
          17. 2012-01-08 07:14:21,190 - INFO  [main:Environment@97] - Client environment:user.dir=/home/hadoop/installation/zookeeper-3.3.4  
          18. 2012-01-08 07:14:21,197 - INFO  [main:ZooKeeper@379] - Initiating client connection, connectString=slave-01:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@bf32c  
          19. 2012-01-08 07:14:21,305 - INFO  [main-SendThread():ClientCnxn$SendThread@1061] - Opening socket connection to server slave-01/192.168.0.179:2181  
          20. Welcome to ZooKeeper!  
          21. 2012-01-08 07:14:21,376 - INFO  [main-SendThread(slave-01:2181):ClientCnxn$SendThread@950] - Socket connection established to slave-01/192.168.0.179:2181, initiating session  
          22. JLine support is enabled  
          23. [zk: slave-01:2181(CONNECTING) 0] 2012-01-08 07:14:21,872 - INFO  [main-SendThread(slave-01:2181):ClientCnxn$SendThread@739] - Session establishment complete on server slave-01/192.168.0.179:2181, sessionid = 0x134bdcd6b730000, negotiated timeout = 30000  
          24.   
          25. WATCHER::  
          26.   
          27. WatchedEvent state:SyncConnected type:None path:null  
          28.   
          29. [zk: slave-01:2181(CONNECTED) 0] ls /  
          30. [zookeeper]  
          當(dāng)前根路徑為/zookeeper。

          總結(jié)說明


          主機名與IP地址映射配置問題

          啟動ZooKeeper集群時,如果ZooKeeper集群中slave-01結(jié)點的日志出現(xiàn)如下錯誤:

          [plain] view plaincopy
          1. java.net.SocketTimeoutException  
          2.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:109)  
          3.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          4.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          5.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          6.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          7. 2012-01-08 06:37:46,026 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 6400  
          8. 2012-01-08 06:37:57,431 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/202.106.199.35:3888  
          9. java.net.SocketTimeoutException  
          10.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:109)  
          11.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
          12.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
          13.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
          14.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
          15. 2012-01-08 06:38:02,442 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/202.106.199.35:3888  

          很顯然,slave-01在啟動時連接集群中其他結(jié)點(slave-02、slave-03)時,主機名映射的IP與我們實際配置的不一致,所以集群中各個結(jié)點之間無法建立鏈路,整個ZooKeeper集群啟動是失敗的。

          上面錯誤日志中slave-02/202.106.199.35:3888實際應(yīng)該是slave-02/202.192.168.0.178:3888就對了,但是在進行域名解析的時候映射有問題,修改每個結(jié)點的/etc/hosts文件,將ZooKeeper集群中所有結(jié)點主機名到IP地址的映射配置上。






          評論

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄[未登錄]  回復(fù)  更多評論   

          2012-11-28 15:54 by Eric Liu
          寫的很好,頂博主

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2012-12-07 11:17 by tangtianfly
          太感謝了!!贊!

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2013-04-19 17:36 by 白松
          寫的很好,贊!!!!!!!

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2013-07-11 15:40 by セ智軍ミ
          寫的不錯,贊!!

          請問樓主在配置zookeeper集群時,leader和follower是怎么決定的?
          不是按服務(wù)器啟動順序,是隨機分配嗎?

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2014-11-27 01:29 by 龍斌
          我的ipy=映射是對的但還是出現(xiàn)這種異常
          [myid=1]/0:0:0:0:0:0:0:0:2181:Learner@233] - Unexpected exception, tries=0, connecting to slave4/192.168.20.13:2888
          java.net.NoRouteToHostException: 沒有到主機的路由
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          at java.net.Socket.connect(Socket.java:589)
          at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225)
          at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:71)
          at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
          2014-11-27 01:19:30,946 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Learner@233] - Unexpected exception, tries=1, connecting to slave4/192.168.20.13:2888
          java.net.NoRouteToHostException: 沒有到主機的路由
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2015-10-20 20:10 by 張大
          @龍斌
          這個問題,我也遇到了,你最后怎么解決的?

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄[未登錄]  回復(fù)  更多評論   

          2015-12-07 12:28 by edward
          我解決了 就是關(guān)閉防火墻就可以了。

          # re: zookeeper 集群安裝(單點與分布式成功安裝)摘錄  回復(fù)  更多評論   

          2015-12-17 14:29 by 學(xué)途
          Zookeeper+Spring跨機房容災(zāi)系統(tǒng)以及灰度發(fā)布課程

          課程學(xué)習(xí)地址:http://www.xuetuwuyou.com/course/20

          一、課程使用到的軟件及版本: centos系統(tǒng),zookeeper 3.4.6,Spring4

          二、課程目標(biāo):
          1、熟練開發(fā)各種分布式應(yīng)用
          2、掌控zookeeper java客戶端開發(fā)
          3、熟練運用curator的各種特性

          三、課程目錄

          第1節(jié)、zookeepeer集群簡介,跨機房容災(zāi)系統(tǒng)架構(gòu)設(shè)計
          第2節(jié)、zookeeper的java客戶端與spring整合(上)
          第3節(jié)、zookeeper的java客戶端與spring整合(下)
          第4節(jié)、zookeeper-3watcher使用(上)
          第5節(jié)、zookeeper-3watcher使用(下)
          第6節(jié)、Curator介紹與使用
          第7節(jié)、zookeeper跨機房容災(zāi)系統(tǒng)設(shè)計兩種方案詳細(xì)講解
          第8節(jié)、zookeeper跨機房容災(zāi)具體實現(xiàn)
          第9節(jié)、zookeeper跨機房容災(zāi)系統(tǒng)的調(diào)試,以及分布式鎖的簡單介紹
          第10節(jié)、zookeeper分布式鎖(上)
          第11節(jié)、zookeeper分布式鎖(下)
          第12節(jié)、zookeeper實現(xiàn)灰度指定服務(wù)器發(fā)布
          第13節(jié)、課程總結(jié)以及未來展望

          只有注冊用戶登錄后才能發(fā)表評論。


          網(wǎng)站導(dǎo)航:
          博客園   IT新聞   Chat2DB   C++博客   博問  
           
          主站蜘蛛池模板: 安图县| 仙居县| 台江县| 思茅市| 甘德县| 扶绥县| 永丰县| 美姑县| 高阳县| 突泉县| 霍山县| 上犹县| 宜昌市| 微山县| 蓬安县| 来宾市| 徐闻县| 扶绥县| 凤阳县| 微山县| 迁西县| 噶尔县| 齐河县| 沈阳市| 安国市| 汝城县| 通榆县| 历史| 堆龙德庆县| 中牟县| 延庆县| 云梦县| 临洮县| 平顺县| 邛崃市| 富宁县| 轮台县| 黎城县| 山东省| 威海市| 邹平县|