黄色在线观看网,日本中文字幕一区二区有码在线,日韩中文字幕第一页

libevent 学习�W�记

Tue, 05 Jan 2010 03:21:00 GMT

学习�Q?br /> http://blog.chinaunix.net/u1/59571/showart_2077664.html

个�h对上面文章做�?�W�记�׃�n出来和大家分享！

刘凯�?/a> 2010-01-05 11:21 发表评论

Fri, 30 Oct 2009 04:05:00 GMT

beanstalk 消息队列 ��结
协议说明和各状态�{换情�?br />

基本知识�?
1. 对于beanstalk 消息队列中每条数据都�?job
2. beanstalk service�?�Q�会�l�护 tubes[多个��道]
3. client端可以监�?使用�?tube
4. client端可以指�?use ��道[ client生成一个新的job时会把此job提交�?指定��道]
5. client端可以指�?watch ��道 [ client接收处理job时会�?指定��道得到待处理的job]

官方�C�意�?
put            reserve               delete
-----> [READY] ---------> [RESERVED] --------> *poof*

一般情�?
1. ��d��提交到service�?job ��理攑օ�内存�I�间�q��ؓ其标记状�?[READY]
2. client通过轮训竞争得到�ơ状�? job 改�ؓ [RESERVED]
   2.1 当在默认旉�� 120 �U�内没处理完 , job.stats.timeouts ��׃��大于 0
      同时其他轮训竞争client会拿到这个job�?注意�?每次timeouts�?在轮训的客户端就会得到次job�Q�状态都�?ready,timeouts>0 �?br /> 3. 随便其中一台client处理�?job.delete   , 其他 client 中的此job 都会    *poof*

deom - python beanstalkc �?job.stats 参�?
使用 easy_install beanstalkc
API 参�?: http://github.com/earl/beanstalkc/blob/master/TUTORIAL
刚生成的 beanstalk
{'buries': 0, 'releases': 0, 'tube': 'default', 'timeouts': 0, 'ttr': 120,
'age': 6, 'pri': 2147483648L, 'delay': 0, 'state': 'reserved', 'time-left': 114,
'kicks': 0, 'id': 2}

以timeout了的 beanstalk,�q�且在其他client轮训�?job
{'buries': 0, 'releases': 0, 'tube': 'default', 'timeouts': 1, 'ttr': 120,
'age': 417, 'pri': 2147483648L, 'delay': 0, 'state': 'reserved', 'time-left': 110,
'kicks': 0, 'id': 2}
{'buries': 0, 'releases': 0, 'tube': 'default', 'timeouts': 1, 'ttr': 120, 'age': 415,
'pri': 2147483648L, 'delay': 0, 'state': 'reserved', 'time-left': 4294967163L,
'kicks': 0, 'id': 2}

当没所有client �?job 都到�?�?状�?br /> {'buries': 0, 'releases': 0, 'tube': 'default', 'timeouts': 2, 'ttr': 120,
'age': 417, 'pri': 2147483648L, 'delay': 0, 'state': 'ready', 'time-left': 4294967161L,
'kicks': 0, 'id': 2}
{'buries': 0, 'releases': 0, 'tube': 'default', 'timeouts': 2, 'ttr': 120, 'age': 415,
'pri': 2147483648L, 'delay': 0, 'state': 'ready', 'time-left': 4294967163L,
'kicks': 0, 'id': 2}

其中 client1 job.delete
client1 job.stats *poof*
client2 job.stats *poof*

比较全的状态说�?- [官方文档]
http://github.com/kr/beanstalkd/blob/v1.1/doc/protocol.txt?raw=true

官方�C�意�?

先简单说明下�Q�完全自��q��解的�Q�欢�q�拍砖。本人E人太差~看官档费�Ԍ��谅解下）:
job.stats状�?= [READY] 待处�? [RESERVED] 正处�? [DELAYED]延迟状�?, [BURIED] 隐藏状�?br />
1. 延迟提交
py.client1.put>>> beanstalk.put('yes!', delay=10)
py.client3.reserve>>> job = beanstalk.reserve()
# �{�待 10 �U?br />
2. ��道��试
put-job到service�?可以指定 put的tube��道
�?

py.client1.put>>> beanstalk.use('foo')
py.client1.put>>> beanstalk.put('hey!')

py.client2.reserve>>> job = beanstalk.reserve()
# 一直拥塞，应�ؓ �?watch ��道 'default'

py.client3.reserve>>> beanstalk.watch('foo')
# beanstalk.ignore('bar') 攑ּ�监听 bar
py.client3.reserve>>> job = beanstalk.reserve()
py.client3.reserve>>> job.body #输出 'hey!'

3. 隐藏状�?现在�?client 1/2/3 �?use watch 的管道都调回 default
py.client2.reserve>>> job = beanstalk.reserve()
py.client3.reserve>>> job = beanstalk.reserve()
py.client1.put>>> beanstalk.put('隐藏状�?')
py.client2.reserve>>> job.bury() #2 轮训得到 �q�且修改 job 为隐藏状�?br /> # 120 �U�后 client3 没有轮训得到此job
py.client2.reserve>>> job.stats()
{'buries': 1, 'releases': 0, 'tube': 'default', 'timeouts': 0, 'ttr': 120,
'age': 188, 'pri': 2147483648L, 'delay': 0, 'state': 'buried',
'time-left': 4294967228L, 'kicks': 0, 'id': 11}
py.client2.reserve>>> beanstalk.kick( job.stats()['id'] ) #修改状态�ؓ reserved
# 立刻 client3 得到 job
py.client3.reserve>>> job.stats()
{'buries': 1, 'releases': 0, 'tube': 'default', 'timeouts': 0, 'ttr': 120, 'age': 313,
'pri': 2147483648L, 'delay': 0, 'state': 'reserved',
'time-left': 110, 'kicks': 1, 'id': 11}
# �q�时�?client2 / 3 同时 �?job 11 状�?'buries': 1,'timeouts': 0,'state': 'reserved'

4. peek �H�见
可以得到一�?stats - read �?job �Q�其�?client 可以 job = beanstalk.reserve()
后马�?job.stats 会变�?nbsp; [RESERVED]
py.client2.reserve>>> job = beanstalk.peek_ready()
取得 job �q�看 �?client �?处理�?br /> >>> job = beanstalk.peek(3)
>>> job.body
    'yes!'
>>> job.stats()['state']
    'ready'
�q�种形式�?job 不能 bury �{�修改状态，�?可以 delete

peek �pȝ��
peek_buried
peek_ready

刘凯�?/a> 2009-10-30 12:05 发表评论

消息队列 beanstalkd 介绍

Wed, 28 Oct 2009 11:21:00 GMT

首先好东�?
http://kr.github.com/beanstalkd/

其次真的是好东西支持 java , python ,perl,ruby,erlang 和我不知道的语言
官方的原文介�l�：

$ ./beanstalkd -d -l 10.0.1.5 -p 11300

This starts up beanstalkd as a daemon listening on address 10.0.1.5, port 11300.

Use It

Here’s an example in Ruby (see the client libraries to find your favorite language).

First, have one process put a job into the queue:

beanstalk = Beanstalk::Pool.new(['10.0.1.5:11300'])



beanstalk.put('hello')

Then start another process to take jobs out of the queue and run them:

beanstalk = Beanstalk::Pool.new(['10.0.1.5:11300'])



loop do



job = beanstalk.reserve



puts job.body # prints "hello"



job.delete



end

Thanks

Many thanks to memcached for providing inspiration for simple protocol design and for the structure of the documentation. Not to mention a fantastic piece of software!

刘凯�?/a> 2009-10-28 19:21 发表评论

hadoop streaming( hadoop + perl )��试

Fri, 25 Sep 2009 06:33:00 GMT

参�?
http://hadoop.apache.org/common/docs/r0.15.2/streaming.html

注意
目前 streaming �?linux pipe #也就�?cat |wc -l �q�样的管�?不支持，但不妨碍我们使用perl,python 行式命��o�Q�！
原话�?�Q?br /> Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
    Currently this does not work and gives an "java.io.IOException: Broken pipe" error.
    This is probably a bug that needs to be investigated.
但如果你是强烈的 linux shell pipe 发烧�?�Q?参考下�?br /> $> perl -e 'open( my $fh, "grep -v null tt |sed -n 1,5p |");while ( <$fh> ) {print;} '
     #不过我没��试通过 �Q�！

环境 �Q�hadoop-0.18.3
$> find . -type f -name "*streaming*.jar"
./contrib/streaming/hadoop-0.18.3-streaming.jar

��试数据�Q?br />

-bash-3.00$ head tt
null    false    3702    208100
6005100    false    70    13220
6005127    false    24    4640
6005160    false    25    4820
6005161    false    20    3620
6005164    false    14    1280
6005165    false    37    7080
6005168    false    104    20140
6005169    false    35    6680
6005240    false    169    32140
......

�q�行�Q?br />

c1=" perl -ne  'if(/.*\t(.*)/){\$sum+=\$1;}END{print \"\$sum\";}' "
# 注意 �q�里 $ 要写�?\$    " 写成 \"
echo $c1; # 打印输出 perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}'
hadoop jar hadoop-0.18.3-streaming.jar
   -input file:///data/hadoop/lky/jar/tt
   -mapper   "/bin/cat"
   -reducer "$c1"
   -output file:///tmp/lky/streamingx8

�l�果:
cat /tmp/lky/streamingx8/*
1166480

本地�q�行输出:
perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}' < tt
1166480

�l�果正确!!!!

命��o自带文档�Q?br />

-bash-3.00$ hadoop jar hadoop-0.18.3-streaming.jar -info
09/09/25 14:50:12 ERROR streaming.StreamJob: Missing required option -input
Usage: $HADOOP_HOME/bin/hadoop [--config dir] jar \
          $HADOOP_HOME/hadoop-streaming.jar [options]
Options:
  -input    <path>     DFS input file(s) for the Map step
  -output   <path>     DFS output directory for the Reduce step
  -mapper   <cmd|JavaClassName>      The streaming command to run
  -combiner <JavaClassName> Combiner has to be a Java class
  -reducer  <cmd|JavaClassName>      The streaming command to run
  -file     <file>     File/dir to be shipped in the Job jar file
  -dfs    <h:p>|local  Optional. Override DFS configuration
  -jt     <h:p>|local  Optional. Override JobTracker configuration
  -additionalconfspec specfile  Optional.
  -inputformat TextInputFormat(default)|SequenceFileAsTextInputFormat|JavaClassName Optional.
  -outputformat TextOutputFormat(default)|JavaClassName  Optional.
  -partitioner JavaClassName  Optional.
  -numReduceTasks <num>  Optional.
  -inputreader <spec>  Optional.
  -jobconf  <n>=<v>    Optional. Add or override a JobConf property
  -cmdenv   <n>=<v>    Optional. Pass env.var to streaming commands
  -mapdebug <path>  Optional. To run this script when a map task fails
  -reducedebug <path>  Optional. To run this script when a reduce task fails
  -cacheFile fileNameURI
  -cacheArchive fileNameURI
  -verbose

刘凯�?/a> 2009-09-25 14:33 发表评论

hadoop jython ( windows )

Fri, 04 Sep 2009 09:14:00 GMT

参�?�Q?hadoop window 搭徏 �?�׃��?py 的语法喜��?�Q�一直想把hadoop,�Ҏ��jython �?
�q�次 �?自己电脑�?nbsp; �l�于完成,下面介绍�q�程:

��试环境�Q?br /> 依然�?windows + cygwin
hadoop 0.18 # C:/cygwin/home/lky/tools/java/hadoop-0.18.3
jython 2.2.1 # C:/jython2.2.1

参�? PythonWordCount

启动 hadoop �q�到 hdoop_home �?br />

# 在云环境中创�?input 目录
$>bin/hadoop dfs -mkdir input
# �?�?hadoop �?NOTICE.txt 拯��?input 目录�?/strong>
$>bin/hadoop dfs -copyFromLocal c:/cygwin/home/lky/tools/java/hadoop-0.18.3/NOTICE.txt hdfs:///user/lky/input

$>cd src/examples/python

# 创徏 �?脚本 ( jy->jar->hd run ) 一步完�?
# 当然 �?linux 写个脚本比这好看呵呵�Q?br /> $>vim run.bat

"C:\Program Files\Java\jdk1.6.0_11\bin\java.exe"  -classpath "C:\jython2.2.1\jython.jar;%CLASSPATH%" org.python.util.jython C:\jython2.2.1\Tools\jythonc\jythonc.py   -p org.apache.hadoop.examples -d -j wc.jar -c %1

sh C:\cygwin\home\lky\tools\java\hadoop-0.18.3\bin\hadoop jar wc.jar  %2 %3 %4 %5 %6 %7 %8 %9

# 修改 jythonc 打包环境 �?+hadoop jar
$>vim C:\jython2.2.1\Tools\jythonc\jythonc.py

# Copyright (c) Corporation for National Research Initiatives
# Driver script for jythonc2.  See module main.py for details
import sys,os,glob

for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/jython2.2.1/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/lib/*.jar') :sys.path.append(fn)

import main
main.main()

import os
os._exit(0)

# �q�行
C:/cygwin/home/lky/tools/java/hadoop-0.18.3/src/examples/python>
run.bat WordCount.py hdfs:///user/lky/input file:///c:/cygwin/home/lky/tools/java/hadoop-0.18.3/tmp2

�l�果输出�Q?/strong>
cat c:/cygwin/home/lky/tools/java/hadoop-0.18.3/tmp2/part-00000
(http://www.apache.org/).       1
Apache 1
Foundation      1
Software        1
The     1
This    1
by      1
developed       1
includes        1
product 1
software        1

下面重头来了 �Q�（��z�的 jy hdoop 代码�Q?/strong>

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from org.apache.hadoop.fs import Path
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred import *

import sys
import getopt

class WordCountMap(Mapper, MapReduceBase):
    one = IntWritable(1)
    def map(self, key, value, output, reporter):
        for w in value.toString().split():
            output.collect(Text(w), self.one)

class Summer(Reducer, MapReduceBase):
    def reduce(self, key, values, output, reporter):
        sum = 0
        while values.hasNext():
            sum += values.next().get()
        output.collect(key, IntWritable(sum))

def printUsage(code):
    print "wordcount [-m ] [-r ]  "
    sys.exit(code)

def main(args):
    conf = JobConf(WordCountMap);
    conf.setJobName("wordcount");

    conf.setOutputKeyClass(Text);
    conf.setOutputValueClass(IntWritable);

    conf.setMapperClass(WordCountMap);
    conf.setCombinerClass(Summer);
    conf.setReducerClass(Summer);
    try:
        flags, other_args = getopt.getopt(args[1:], "m:r:")
    except getopt.GetoptError:
        printUsage(1)
    if len(other_args) != 2:
        printUsage(1)

    for f,v in flags:
        if f == "-m":
            conf.setNumMapTasks(int(v))
        elif f == "-r":
            conf.setNumReduceTasks(int(v))
    conf.setInputPath(Path(other_args[0]))
    conf.setOutputPath(Path(other_args[1]))
    JobClient.runJob(conf);

if __name__ == "__main__":
    main(sys.argv)

刘凯�?/a> 2009-09-04 17:14 发表评论

hadoop cascading demo

Wed, 22 Jul 2009 02:01:00 GMT

Java 代码�Q?br />
package com.xunjie.dmsp.olduser;

import java.util.Properties;

import cascading.flow.Flow;
import cascading.flow.FlowConnector;
import cascading.operation.regex.RegexSplitter;
import cascading.pipe.Each;
import cascading.pipe.Pipe;
import cascading.scheme.TextLine;
import cascading.tap.Hfs;
import cascading.tap.Tap;
import cascading.tuple.Fields;

/**
* test.txt�Q?nbsp;
* 1    a
* 2    b
* 3    c
*
* /data/hadoop/hadoop/bin/hadoop jar
*         dmsp_test_jar-1.0-SNAPSHOT-dependencies.jar
*             hdfs:/user/hadoop/test/lky/test.txt
*             file:///data/hadoop/test/lky/output
*/
public class Test2 {
    public static void main(String[] args) {

        //讑֮�输入文�g
        String sourcePath= args[0];
        //讄��输出文�g�?/span>
        String sinkPath = args[1];

        //定义��d��?/span>
        Fields inputfields = new Fields("num", "value");
        //定义分解正则�Q�默�?nbsp;\t
        RegexSplitter spliter = new RegexSplitter(inputfields);


        //��道定义
        Pipe p1 = new Pipe( "test" );
        //��道嵌套�Q?br />         //分解日志源文�Ӟ��输出�l�定字段
        p1 = new Each(p1,new Fields("line") ,spliter);


        //讑֮�输入和输�?nbsp;�Q��?nbsp;泛型Hfs
        Tap source = new Hfs( new TextLine(),  sourcePath );
        Tap sink = new Hfs( new TextLine() , sinkPath );



        //配置job
        Properties properties = new Properties();
        properties.setProperty("hadoop.job.ugi", "hadoop,hadoop");

        FlowConnector.setApplicationJarClass( properties, Main.class );
        FlowConnector flowConnector = new FlowConnector(properties);

        Flow importFlow = flowConnector.connect( "import flow", source,sink,p1);

        importFlow.start();
        importFlow.complete();


    }
}

刘凯�?/a> 2009-07-22 10:01 发表评论

hadoop window 搭徏

Wed, 08 Jul 2009 03:07:00 GMT

hadoop 原理参考：windows ssh 搭徏
2. 搭徏hadoop Cygwin下的Hadoop快速入�?伪分布式模式的查�~��?/a>
    �q�里�q�有 cygwin putty 控制台方法设�|?br /> 3. 开发环境和 demo hadoop-site.xml �?br />    在这个版本已�l�在conf下找不到了，我这��试修改�?hdfs-site.xml 。测试的时候竟焉��过了�?br />

xml version="1.0"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.default.namename>
    <value>hdfs://localhost:9000value>
  property>

  <property>
    <name>mapred.job.trackername>
    <value>localhost:9001value>
  property>

  <property>
    <name>dfs.data.dirname>


    <value>C:\\cygwin\\home\\lky\\tools\\java\\hadoop-0.18.3\\dfs\\value>
  property>

  <property>
    <name>hadoop.tmp.dirname>
    <value>/tmpvalue>
  property>

  <property>
    <name>dfs.replicationname>
    <value>1value>
  property>

<property>
  <name>fs.trash.intervalname>
  <value>60value>
  <description>Number of minutes between trash checkpoints.
  If zero, the trash feature is disabled.
  description>
property>

configuration>

刘凯�?/a> 2009-07-08 11:07 发表评论

Tue, 19 May 2009 09:37:00 GMT

svn 帮助文档�Q�大体上��览了下一些比较好的特�?有：
http://www.subversion.org.cn/svnbook/

分支与合�q?/a>
锁定
svnsync
版本库钩�?/a>
一个SVN Python状态爬�?/a>

注意使用用户 (sysmen) �Q?/strong>
#指定新版�?/strong>
vim ~.bashrc #��d��?br /> alias svn="/usr/local/subversion/ bin/svn"
alias svnadmin="/usr/local/subversion/bin/svnadmin"
alias svnserve="/usr/local/subversion/bin/svnserve"

#最��?svn 服务器搭�?/strong>
cd /home/sysman/src/svn/
svnadmin create test
svnserve -d -r /home/sysman/src/svn/test/

#最��单配�|?/strong>
#vim svnserve.conf
[general]
password-db = passwd
realm = test

anon-access = read
auth-access = write

#查看
cd /home/sysman/src/svn/testsrc
svn list svn://127.0.0.1/test

#�q�出
cd /home/sysman/src/svn/testsrc
svn checkout svn://127.0.0.1/test

#�q�入
cd test
echo "google - baidu " > baidu
svn add baidu svn://127.0.0.1/test
svn commit -m 'add ok'
svn list svn://127.0.0.1/test
# baidu
# tt

#详细参�?�Q?br /> http://www.subversion.org.cn/svnbook/
在文档中有关有意思的介绍 �Q�svnsync�Q�：
http://www.subversion.org.cn/svnbook/1.4/svn.ref.svnsync.html

其他说明 �Q?br />
1.帮助
svn help
�?
svn help add

2.一般操�?br /> 更新你的工作拯��
    svn update

做出修改
    svn add
    svn delete
    svn copy
    svn move

��验修�?br />     svn status
    svn diff

可能会取消一些修�?br />     svn revert

解决冲突�Q�合�q�别人的修改�Q?br />     svn update
    svn resolved

提交你的修改
    svn commit

刘凯�?/a> 2009-05-19 17:37 发表评论

soap (java,perl,要写代码 �q�不�q?100 char)

Sun, 01 Mar 2009 16:00:00 GMT

可能最��单的 soap 哦！
服务器�ؓ java tomcat
下蝲服务器：http://www.apache.org/dist/ws/
   axis/ 为java
   axis-c/    �?c

1.我这下蝲�?axis 1_4 版本 �Q�解�?/webapps/axis 攑ֈ� tomcat 的webapps
启动tomcat�Q�这��׃��修改��M��Q�如有需�?�?web.xml �{�配�|�了�Q?a style="font-size: 12px;" onclick="NEBlog.gPrevBlog.oBlogList._oUpdownMenu.openBlog('blog/static/307422712007552499407', event);return false;"> 参�?>>
    blog �?�q�有很多介绍 axis 大家可以看看日志 > 其下�?"学习"

直接写个文�g HelloWorld.jws 攑ֈ� %TOMCAT_HOME%"webapps"axis
public class HelloWorld {
     public String sayHello()   {
     return "HELLO soap!";
}
}

�{�不�?可以使用看看
http://127.0.0.1:8080/axis/HelloWorld.jws?method=sayHello
�l�果 �Q?br />
<soapenv:Envelope>
<soapenv:Body>
<sayHelloResponse soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<sayHelloReturn xsi:type="xsd:string">HELLO soap!sayHelloReturn>
sayHelloResponse>
soapenv:Body>
soapenv:Envelope>

客户�D?br /> perl cpan >> install SOAP::Lite

#!perl -w
use SOAP::Lite;
print SOAP::Lite
    -> proxy('http://127.0.0.1:8080/axis/HelloWorld.jws')
    -> uri('http://127.0.0.1:8080/axis/HelloWorld.jws')
    -> sayHello()
    -> result;

�l�果�Q?HELLO soap�Q?/span>

刘凯�?/a> 2009-03-02 00:00 发表评论

黄色在线观看网,日本中文字幕一区二区有码在线,日韩中文字幕第一页

libevent 学习�W�记

消息队列 beanstalkd 介绍

Use It

Thanks

hadoop streaming( hadoop + perl )���试

hadoop jython ( windows )

hadoop cascading demo

hadoop window 搭徏

soap (java,perl,要写代码 �q�不�q?100 char)

hadoop streaming( hadoop + perl )��试