定义Q?
灰羊?nbsp; Q无主见的用L体）
黑羊 Q?对自己需要什么有明确的认识，我们一般称Z家用戗?nbsp; Q?br />
1. 区分 灎ͼ无主见） ?群
2.
user session 兌 #?兌关系l护使用 用户的会话IDQ用户不同心情，起始在数据中应该是不同分类的）
user 推荐 # 而推荐出产品 q是 ?用户唯一~号有关
#在推荐中需要描q?nbsp; 用户的多角度 问题
3.
蛮力推荐Q?全数?Q描q初期清z后的数?Q?适合 产品兌
清洗后期的数据（包含用户多维度描qͼ 适合 用户兌
4.
专家跟随推荐
描述Q?br /> 用户分类 扑ֈ黑ľ?
扑ֈ 一灰l늾 ?一只黑l늾的关联关p?br /> ?一灰l늾 可以?黑ľ?动作
]]>
比如Q我希望原始日志数据q行某一字段的排?是不是很??
有h?nbsp; - 数据导入数据?load into table ... Q?select order by 。之
q有 - linux sort -n...
恩！很好Q下面我们对大小?1TB 的数据开始进行这个简单的操作 -- ȝ?Q！
关于挖掘 - TB U别的数量在我目前学习挖掘不到半q_遇到过3-4ơ之?br />
解决办法:
对于q个问题 - 我现在希望能有个 大的链表 - Q大到内存装不下Q?/strong>Q?br /> 链表中的struct l构?:
>> 排序属性文件归?br /> >> 排序属性整条数据在文g中的 起始位置 - l束位置
>> 在排序中的排?Q?链表l构,只记入比自己的 属性在此链表的位置 Q?br />
比如 :
1. 文g1内容 =>
说明:
完整数据描述 : 此数据在文g中的 起始位置Q当然是通过E序取得的，qؓ了方便我标出Q?/strong>
..c
. 0 - 22
..a. 23 - 55
..b. 56- 76
..d. 77 - 130
..f. 131 - 220
..e. 221 - 243
完整数据描述 : 此数据在文g中的 起始位置Q当然是通过E序取得的，qؓ了方便我标出Q?/strong>
..c
. 0 - 22..a
. 23 - 55..b
. 56- 76..d
. 77 - 130..f
. 131 - 220..e
. 221 - 2432. 数据l构预开I间 100 byte
3. 文g存储在描q?: # 链表排序我就不介l了Q数据结构的最基本技能，修改数据l构中的比自己小的指?
我这q出结?br /> { /tmp/文g1, 0-22 , 300 } #说明 c Q?在链表位|?0
{ /tmp/文g1, 23-55 , 200 } # a Q?100
{ /tmp/文g1, 56-76 , 0 } # b : 200
{ /tmp/文g1, 77-130 , 500 } # d : 300
{ /tmp/文g1, 131-220 , } # f : 400
{ /tmp/文g1, 221-243 , 400 } # e : 500
4. 倒叙输出 由小到到
假设预存最??nbsp; 200 链表位置
扑և 使用 open /tmp/文g1
q?seek 文g游标 定位 23-55 取出 ..a...
Ҏ 链表?200 ?seek 56 76 取出 ..b...
{等
当然 上面
数据l构你可以?双向链表Q?btree , U黑 , 斐L那契。。。（ 数据l构l于感觉有用了，不枉Ҏ考的软证啊！Q?br />
通过说明Q我q?l大家提供个 可能需要的 技术细?(py),不之处 Ƣ迎拍砖Q！
1. 二进制文?l构?写，修改
#指定修改 190 byte 处的 内容
import os
from struct import *
fd = os.open( "pack1.txt", os.O_RDWR|os.O_CREAT )
ss = pack('ii11s', 3, 4, 'google')
os.lseek(fs, len(ss)*10, 0)
os.write(fs,ss)
os.fsync(fs)
#os.close( fs )
import os
from struct import *
fd = os.open( "pack1.txt", os.O_RDWR|os.O_CREAT )
ss = pack('ii11s', 3, 4, 'google')
os.lseek(fs, len(ss)*10, 0)
os.write(fs,ss)
os.fsync(fs)
#os.close( fs )
2. seek 指定位置l构化读?br />
from struct import *
file_object = open('pack1.txt', 'rb')
def ts(si,ss=len(ss)):
file_object.seek(si*ss)
chunk = file_object.read(ss)
a,b,c=unpack('ii11s', chunk )
print a,b,c
ts(10)
#输出 3 4 google
1. 其他语言?使用
struct l构定义 ,?python ?使用 struct 包，q样序列出来的数据到文g中其他语a也可以?
参? http://www.pythonid.com/bbs/archiver/?tid-285.html
pack1.py
from struct import *
# i ?intQ?Q?nbsp; 11s 为预?11 位置 ?string
# 此数据类??19 byte ss = pack('ii11s', 1, 2, 'hello world')
f = open("pack1.txt", "wb")
f.write(ss)
f.close()
上面的代码往C的结构中写入数据Q结构包括两个整型和一个字W串?br /> pack1.c
#include <stdio.h>
#include <string.h>
struct AA
{
int a;
int b;
char c[64];
};
int main()
{
struct AA aa;
FILE *fp;
int size, readsize;
memset(&aa, 0, sizeof(struct AA));
fp = fopen("pack1.txt", "rb");
if (NULL == fp) {
printf("open file error!"n");
return 0;
}
readsize = sizeof(struct AA);
printf("readsize: %d"n", readsize);
size = fread(&aa, 1, readsize, fp);
printf("read: %d"n", size);
printf("a=%d, b=%d, c=%s"n", aa.a, aa.b, aa.c);
fclose(fp);
return 0;
}
l果输出:
C:"Documents and Settings"lky"桌面"dataStructure>a
readsize: 72
read: 57
a=1, b=2, c=hello word
from struct import *
# i ?intQ?Q?nbsp; 11s 为预?11 位置 ?string
# 此数据类??19 byte ss = pack('ii11s', 1, 2, 'hello world')
f = open("pack1.txt", "wb")
f.write(ss)
f.close()
上面的代码往C的结构中写入数据Q结构包括两个整型和一个字W串?br /> pack1.c
#include <stdio.h>
#include <string.h>
struct AA
{
int a;
int b;
char c[64];
};
int main()
{
struct AA aa;
FILE *fp;
int size, readsize;
memset(&aa, 0, sizeof(struct AA));
fp = fopen("pack1.txt", "rb");
if (NULL == fp) {
printf("open file error!"n");
return 0;
}
readsize = sizeof(struct AA);
printf("readsize: %d"n", readsize);
size = fread(&aa, 1, readsize, fp);
printf("read: %d"n", size);
printf("a=%d, b=%d, c=%s"n", aa.a, aa.b, aa.c);
fclose(fp);
return 0;
}
l果输出:
C:"Documents and Settings"lky"桌面"dataStructure>a
readsize: 72
read: 57
a=1, b=2, c=hello word
最后罗嗦下Q?/strong>
能用数据l构了，很多东西都可以根据自己逻辑定制 存储很方??不再?关系数据?, key 数据??mapreduce 的限?
参?
http://docs.python.org/library/struct.html#module-struct #官方struct ?说明
http://blog.csdn.net/JGood/archive/2009/06/22/4290158.aspx # 使用 struct 的前辈留下的
http://www.tutorialspoint.com/python/os_lseek.htm #一个小demo
Python天天味(17) - opend文g
]]>
数据挖掘所发现的知识最常见的有以下四类Q?
q? 义知识的发现Ҏ和实现技术有很多Q如数据立方体、面向属性的归约{。数据立方体q有其他一些别名，?#8220;多维数据?#8221;?#8220;实现视图”?#8220;OLAP"{。该 Ҏ的基本思想是实现某些常用的代h较高的聚集函数的计算Q诸如计数、求和、^均、最大值等Qƈ这些实现视囑ւ存在多维数据库中。既然很多聚集函数需l? 帔R复计，那么在多l数据立方体中存N先计好的结果将能保证快速响应，q可灉|地提供不同角度和不同抽象层次上的数据视图。另一U广义知识发现方? 是加拿大SimonFraser大学提出的面向属性的归约Ҏ。这U方法以cSQL语言表示数据挖掘查询Q收集数据库中的相关数据集，然后在相x据集? 应用一pd数据推广技术进行数据推q，包括属性删除、概忉|提升、属性阈值控制、计数及其他聚集函数传播{?/td>
数据分类q有l计、粗p集QRoughSetQ等Ҏ。线性回归和U性L别分析是典型的统计模型。ؓ降低决策树生成代PZq提Z一U区间分cd。最q也有h研究使用经|络Ҏ在数据库中进行分cd规则提取?/td>
? 前，旉序列预测Ҏ有经典的l计Ҏ、神l网l和机器学习{?968qBox和Jenkins提出了一套比较完善的旉序列建模理论和分析方法，q些 l典的数学方法通过建立随机模型Q如自回归模型、自回归滑动q_模型、求和自回归滑动q_模型和季节调整模型等Q进行时间序列的预测。由于大量的旉序列 是非q稳的，其特征参数和数据分布随着旉的推U而发生变化。因此，仅仅通过ҎD历史数据的训练Q徏立单一的神l网l预模型，q无法完成准的预测? 务。ؓ此，Z提出了基于统计学和基于精性的再训l方法，当发现现存预模型不再适用于当前数据时Q对模型重新训练Q获得新的权重参敎ͼ建立新的模型? 也有许多pȝ借助q行法的计优势进行时间序列预?
采用上述技术的某些专门的分析工具已l发展了大约十年的历Ԍ不过q些工具所面对的数据量通常较小。而现在这些技术已l被直接集成到许多大型的工业标准的数据仓库和联机分析pȝ中去了?
]]>
注意
目前 streaming ?linux pipe #也就?cat |wc -l q样的管?不支持，但不妨碍我们使用perl,python 行式命oQ！
原话?Q?br /> Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
Currently this does not work and gives an "java.io.IOException: Broken pipe" error.
This is probably a bug that needs to be investigated.
但如果你是强烈的 linux shell pipe 发烧?Q?参考下?br /> $> perl -e 'open( my $fh, "grep -v null tt |sed -n 1,5p |");while ( <$fh> ) {print;} '
#不过我没试通过 Q！
环境 Qhadoop-0.18.3
$> find . -type f -name "*streaming*.jar"
./contrib/streaming/hadoop-0.18.3-streaming.jar
试数据Q?br />
-bash-3.00$ head tt
null false 3702 208100
6005100 false 70 13220
6005127 false 24 4640
6005160 false 25 4820
6005161 false 20 3620
6005164 false 14 1280
6005165 false 37 7080
6005168 false 104 20140
6005169 false 35 6680
6005240 false 169 32140
......
null false 3702 208100
6005100 false 70 13220
6005127 false 24 4640
6005160 false 25 4820
6005161 false 20 3620
6005164 false 14 1280
6005165 false 37 7080
6005168 false 104 20140
6005169 false 35 6680
6005240 false 169 32140
......
q行Q?br />
c1=" perl -ne 'if(/.*\t(.*)/){\$sum+=\$1;}END{print \"\$sum\";}' "
# 注意 q里 $ 要写?\$ " 写成 \"
echo $c1; # 打印输出 perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}'
hadoop jar hadoop-0.18.3-streaming.jar
-input file:///data/hadoop/lky/jar/tt
-mapper "/bin/cat"
-reducer "$c1"
-output file:///tmp/lky/streamingx8
# 注意 q里 $ 要写?\$ " 写成 \"
echo $c1; # 打印输出 perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}'
hadoop jar hadoop-0.18.3-streaming.jar
-input file:///data/hadoop/lky/jar/tt
-mapper "/bin/cat"
-reducer "$c1"
-output file:///tmp/lky/streamingx8
l果:
cat /tmp/lky/streamingx8/*
1166480
本地q行输出:
perl -ne 'if(/.*"t(.*)/){$sum+=$1;}END{print $sum;}' < tt
1166480
l果正确!!!!
命o自带文档Q?br />
-bash-3.00$ hadoop jar hadoop-0.18.3-streaming.jar -info
09/09/25 14:50:12 ERROR streaming.StreamJob: Missing required option -input
Usage: $HADOOP_HOME/bin/hadoop [--config dir] jar \
$HADOOP_HOME/hadoop-streaming.jar [options]
Options:
-input <path> DFS input file(s) for the Map step
-output <path> DFS output directory for the Reduce step
-mapper <cmd|JavaClassName> The streaming command to run
-combiner <JavaClassName> Combiner has to be a Java class
-reducer <cmd|JavaClassName> The streaming command to run
-file <file> File/dir to be shipped in the Job jar file
-dfs <h:p>|local Optional. Override DFS configuration
-jt <h:p>|local Optional. Override JobTracker configuration
-additionalconfspec specfile Optional.
-inputformat TextInputFormat(default)|SequenceFileAsTextInputFormat|JavaClassName Optional.
-outputformat TextOutputFormat(default)|JavaClassName Optional.
-partitioner JavaClassName Optional.
-numReduceTasks <num> Optional.
-inputreader <spec> Optional.
-jobconf <n>=<v> Optional. Add or override a JobConf property
-cmdenv <n>=<v> Optional. Pass env.var to streaming commands
-mapdebug <path> Optional. To run this script when a map task fails
-reducedebug <path> Optional. To run this script when a reduce task fails
-cacheFile fileNameURI
-cacheArchive fileNameURI
-verbose
09/09/25 14:50:12 ERROR streaming.StreamJob: Missing required option -input
Usage: $HADOOP_HOME/bin/hadoop [--config dir] jar \
$HADOOP_HOME/hadoop-streaming.jar [options]
Options:
-input <path> DFS input file(s) for the Map step
-output <path> DFS output directory for the Reduce step
-mapper <cmd|JavaClassName> The streaming command to run
-combiner <JavaClassName> Combiner has to be a Java class
-reducer <cmd|JavaClassName> The streaming command to run
-file <file> File/dir to be shipped in the Job jar file
-dfs <h:p>|local Optional. Override DFS configuration
-jt <h:p>|local Optional. Override JobTracker configuration
-additionalconfspec specfile Optional.
-inputformat TextInputFormat(default)|SequenceFileAsTextInputFormat|JavaClassName Optional.
-outputformat TextOutputFormat(default)|JavaClassName Optional.
-partitioner JavaClassName Optional.
-numReduceTasks <num> Optional.
-inputreader <spec> Optional.
-jobconf <n>=<v> Optional. Add or override a JobConf property
-cmdenv <n>=<v> Optional. Pass env.var to streaming commands
-mapdebug <path> Optional. To run this script when a map task fails
-reducedebug <path> Optional. To run this script when a reduce task fails
-cacheFile fileNameURI
-cacheArchive fileNameURI
-verbose
]]>
如果在开发中 请?cascading groupby, q行 hadoop join,
本文只是为探讨弄?cascading 实现做准备?br />
当然 如果有有?hadoop join q?误pLQ大家交下 Q?br />
文g可能需要的一些参考：
hadoop jython ( windows )
jython Qjython ~译以及jar ?
量 linux shell
本文介绍 hadoop 可能使用到的 join 接口试 Q已l参考：
使用Hadoop实现Inner Join操作的方法【from淘宝?/strong>Qhttp://labs.chinamobile.com/groups/58_547
下面 试?Q我q大体上 ?hadoop join 的方式是q样理解?Q猜惻IQ?br /> 数据1 ; 数据2
job1.map( 数据1 ) =Q时文?Q?gt; 文g标示1+需要join?nbsp; 数据
job2.map( 数据2 ) =Q时文?Q?gt; 文g标示2+需要join?nbsp; 数据
临时文g mapred.join.expr 生成
job3.map ->
文g标示1+需要join?: 数据
文g标示2+需要join?: 数据
......
job3.Combiner - >
需要join?: 文g标示1+数据
需要join?: 文g标示2+数据
job3.Reducer->
需要join?: 使用 java-list > 生成
文g2-列x [ 数据,数据... ]
文g1-列x [ 数据,数据... ]
然后 你这 left join ,?inner join ?xxx join 逻辑 p己来?br />
l果集合
[root@localhost python]# cat /home/megajobs/del/jobs/tools/hadoop-0.18.3/data/090907/1
1
2
3
4
5
[root@localhost python]# cat /home/megajobs/del/jobs/tools/hadoop-0.18.3/data/090907/2
2
4
3
1
修改 ..../hadoop-0.18.3/src/examples/python/compile
#!/usr/bin/env bash
export HADOOP_HOME=/home/xx/del/jobs/tools/hadoop-0.18.3
export CASCADING_HOME=/home/xx/del/jobs/tools/cascading-1.0.16-hadoop-0.18.3
export JYTHON_HOME=/home/xx/del/jobs/tools/jython2.2.1
export CLASSPATH="$HADOOP_HOME/hadoop-0.18.3-core.jar"
# so that filenames w/ spaces are handled correctly in loops below
IFS=
# add libs to CLASSPATH
for f in $HADOOP_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $HADOOP_HOME/lib/jetty-ext/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $CASCADING_HOME/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $CASCADING_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $JYTHON_HOME/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
# restore ordinary behaviour
unset IFS
/home/xx/del/jobs/tools/jython2.2.1/jythonc -p org.apache.hadoop.examples -d -j $1.jar -c $1.py
/home/xx/del/jobs/tools/hadoop-0.18.3/bin/hadoop jar $1.jar $2 $3 $4 $5 $6 $7 $8 $9
export HADOOP_HOME=/home/xx/del/jobs/tools/hadoop-0.18.3
export CASCADING_HOME=/home/xx/del/jobs/tools/cascading-1.0.16-hadoop-0.18.3
export JYTHON_HOME=/home/xx/del/jobs/tools/jython2.2.1
export CLASSPATH="$HADOOP_HOME/hadoop-0.18.3-core.jar"
# so that filenames w/ spaces are handled correctly in loops below
IFS=
# add libs to CLASSPATH
for f in $HADOOP_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $HADOOP_HOME/lib/jetty-ext/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $CASCADING_HOME/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $CASCADING_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $JYTHON_HOME/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
# restore ordinary behaviour
unset IFS
/home/xx/del/jobs/tools/jython2.2.1/jythonc -p org.apache.hadoop.examples -d -j $1.jar -c $1.py
/home/xx/del/jobs/tools/hadoop-0.18.3/bin/hadoop jar $1.jar $2 $3 $4 $5 $6 $7 $8 $9
?数据 链接 :
from org.apache.hadoop.fs import Path
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred.lib import *
from org.apache.hadoop.mapred.join import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class tMap(Mapper, MapReduceBase):
def map(self, key, value, output, reporter):
output.collect( Text( str(key) ) , Text( value.toString() ))
def main(args):
conf = JobConf(tMap)
conf.setJobName("wordcount")
conf.setMapperClass( tMap )
FileInputFormat.setInputPaths(conf,[ Path(sp) for sp in args[1:-1]])
conf.setOutputKeyClass( Text )
conf.setOutputValueClass( Text )
conf.setOutputPath(Path(args[-1]))
JobClient.runJob(conf)
if __name__ == "__main__":main(sys.argv)
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred.lib import *
from org.apache.hadoop.mapred.join import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class tMap(Mapper, MapReduceBase):
def map(self, key, value, output, reporter):
output.collect( Text( str(key) ) , Text( value.toString() ))
def main(args):
conf = JobConf(tMap)
conf.setJobName("wordcount")
conf.setMapperClass( tMap )
FileInputFormat.setInputPaths(conf,[ Path(sp) for sp in args[1:-1]])
conf.setOutputKeyClass( Text )
conf.setOutputValueClass( Text )
conf.setOutputPath(Path(args[-1]))
JobClient.runJob(conf)
if __name__ == "__main__":main(sys.argv)
q行
./compile test file:///home/xx/del/jobs/tools/hadoop-0.18.3/data/090907/1 file:///home/xx/del/jobs/tools/hadoop-0.18.3/data/090907/2 file:///home/xx/del/jobs/tools/hadoop-0.18.3/tmp/wc78
l果:
[xx@localhost wc78]$ cat ../wc78/part-00000
0 1
0 2
2 4
2 2
4 3
4 3
6 1
6 4
8 5
单的数据 join :
from org.apache.hadoop.fs import Path
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred.lib import *
from org.apache.hadoop.mapred.join import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class tMap(Mapper, MapReduceBase):
def map(self, key, value, output, reporter):
output.collect( Text( str(key) ) , Text( value.toString() ))
def main(args):
conf = JobConf(tMap)
conf.setJobName("wordcount")
conf.setMapperClass( tMap )
conf.set("mapred.join.expr", CompositeInputFormat.compose("override",TextInputFormat, args[1:-1] ) )
conf.setOutputKeyClass( Text )
conf.setOutputValueClass( Text )
conf.setInputFormat(CompositeInputFormat)
conf.setOutputPath(Path(args[-1]))
JobClient.runJob(conf)
if __name__ == "__main__":main(sys.argv)
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred.lib import *
from org.apache.hadoop.mapred.join import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class tMap(Mapper, MapReduceBase):
def map(self, key, value, output, reporter):
output.collect( Text( str(key) ) , Text( value.toString() ))
def main(args):
conf = JobConf(tMap)
conf.setJobName("wordcount")
conf.setMapperClass( tMap )
conf.set("mapred.join.expr", CompositeInputFormat.compose("override",TextInputFormat, args[1:-1] ) )
conf.setOutputKeyClass( Text )
conf.setOutputValueClass( Text )
conf.setInputFormat(CompositeInputFormat)
conf.setOutputPath(Path(args[-1]))
JobClient.runJob(conf)
if __name__ == "__main__":main(sys.argv)
q行l果 ( ) :
./compile test file:///home/xx/del/jobs/tools/hadoop-0.18.3/data/090907/1 file:///home/xx/del/jobs/tools/hadoop-0.18.3/data/090907/2 file:///home/xx/del/jobs/tools/hadoop-0.18.3/tmp/wc79
[xx@localhost wc78]$ cat ../wc79/part-00000
0 2
2 4
4 3
6 1
8 5
]]>
q次 ?自己电脑?nbsp; l于 完成,下面介绍q程:
试环境Q?br /> 依然?windows + cygwin
hadoop 0.18 # C:/cygwin/home/lky/tools/java/hadoop-0.18.3
jython 2.2.1 # C:/jython2.2.1
参? PythonWordCount
启动 hadoop q到 hdoop_home ?br />
# 在云环境中创?input 目录
$>bin/hadoop dfs -mkdir input
# ??hadoop ?NOTICE.txt 拯?input 目录?/strong>
$>bin/hadoop dfs -copyFromLocal c:/cygwin/home/lky/tools/java/hadoop-0.18.3/NOTICE.txt hdfs:///user/lky/input
$>cd src/examples/python
# 创徏 ?脚本 ( jy->jar->hd run ) 一步完?
# 当然 ?linux 写个脚本比这 好看 呵呵Q?br /> $>vim run.bat
# 修改 jythonc 打包 环境 ?+hadoop jar
$>vim C:\jython2.2.1\Tools\jythonc\jythonc.py
# q行
C:/cygwin/home/lky/tools/java/hadoop-0.18.3/src/examples/python>
run.bat WordCount.py hdfs:///user/lky/input file:///c:/cygwin/home/lky/tools/java/hadoop-0.18.3/tmp2
$>bin/hadoop dfs -mkdir input
# ??hadoop ?NOTICE.txt 拯?input 目录?/strong>
$>bin/hadoop dfs -copyFromLocal c:/cygwin/home/lky/tools/java/hadoop-0.18.3/NOTICE.txt hdfs:///user/lky/input
$>cd src/examples/python
# 创徏 ?脚本 ( jy->jar->hd run ) 一步完?
# 当然 ?linux 写个脚本比这 好看 呵呵Q?br /> $>vim run.bat
"C:\Program Files\Java\jdk1.6.0_11\bin\java.exe" -classpath "C:\jython2.2.1\jython.jar;%CLASSPATH%" org.python.util.jython C:\jython2.2.1\Tools\jythonc\jythonc.py -p org.apache.hadoop.examples -d -j wc.jar -c %1
sh C:\cygwin\home\lky\tools\java\hadoop-0.18.3\bin\hadoop jar wc.jar %2 %3 %4 %5 %6 %7 %8 %9
sh C:\cygwin\home\lky\tools\java\hadoop-0.18.3\bin\hadoop jar wc.jar %2 %3 %4 %5 %6 %7 %8 %9
# 修改 jythonc 打包 环境 ?+hadoop jar
$>vim C:\jython2.2.1\Tools\jythonc\jythonc.py
# Copyright (c) Corporation for National Research Initiatives
# Driver script for jythonc2. See module main.py for details
import sys,os,glob
for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/jython2.2.1/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/lib/*.jar') :sys.path.append(fn)
import main
main.main()
import os
os._exit(0)
# Driver script for jythonc2. See module main.py for details
import sys,os,glob
for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/jython2.2.1/*.jar') :sys.path.append(fn)
for fn in glob.glob('c:/cygwin/home/lky/tools/java/hadoop-0.18.3/lib/*.jar') :sys.path.append(fn)
import main
main.main()
import os
os._exit(0)
# q行
C:/cygwin/home/lky/tools/java/hadoop-0.18.3/src/examples/python>
run.bat WordCount.py hdfs:///user/lky/input file:///c:/cygwin/home/lky/tools/java/hadoop-0.18.3/tmp2
l果输出Q?/strong>
cat c:/cygwin/home/lky/tools/java/hadoop-0.18.3/tmp2/part-00000
(http://www.apache.org/). 1
Apache 1
Foundation 1
Software 1
The 1
This 1
by 1
developed 1
includes 1
product 1
software 1
下面重头来了 Q（z的 jy hdoop 代码Q?/strong>
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from org.apache.hadoop.fs import Path
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class WordCountMap(Mapper, MapReduceBase):
one = IntWritable(1)
def map(self, key, value, output, reporter):
for w in value.toString().split():
output.collect(Text(w), self.one)
class Summer(Reducer, MapReduceBase):
def reduce(self, key, values, output, reporter):
sum = 0
while values.hasNext():
sum += values.next().get()
output.collect(key, IntWritable(sum))
def printUsage(code):
print "wordcount [-m <maps>] [-r <reduces>] <input> <output>"
sys.exit(code)
def main(args):
conf = JobConf(WordCountMap);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text);
conf.setOutputValueClass(IntWritable);
conf.setMapperClass(WordCountMap);
conf.setCombinerClass(Summer);
conf.setReducerClass(Summer);
try:
flags, other_args = getopt.getopt(args[1:], "m:r:")
except getopt.GetoptError:
printUsage(1)
if len(other_args) != 2:
printUsage(1)
for f,v in flags:
if f == "-m":
conf.setNumMapTasks(int(v))
elif f == "-r":
conf.setNumReduceTasks(int(v))
conf.setInputPath(Path(other_args[0]))
conf.setOutputPath(Path(other_args[1]))
JobClient.runJob(conf);
if __name__ == "__main__":
main(sys.argv)
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from org.apache.hadoop.fs import Path
from org.apache.hadoop.io import *
from org.apache.hadoop.mapred import *
import sys
import getopt
class WordCountMap(Mapper, MapReduceBase):
one = IntWritable(1)
def map(self, key, value, output, reporter):
for w in value.toString().split():
output.collect(Text(w), self.one)
class Summer(Reducer, MapReduceBase):
def reduce(self, key, values, output, reporter):
sum = 0
while values.hasNext():
sum += values.next().get()
output.collect(key, IntWritable(sum))
def printUsage(code):
print "wordcount [-m <maps>] [-r <reduces>] <input> <output>"
sys.exit(code)
def main(args):
conf = JobConf(WordCountMap);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text);
conf.setOutputValueClass(IntWritable);
conf.setMapperClass(WordCountMap);
conf.setCombinerClass(Summer);
conf.setReducerClass(Summer);
try:
flags, other_args = getopt.getopt(args[1:], "m:r:")
except getopt.GetoptError:
printUsage(1)
if len(other_args) != 2:
printUsage(1)
for f,v in flags:
if f == "-m":
conf.setNumMapTasks(int(v))
elif f == "-r":
conf.setNumReduceTasks(int(v))
conf.setInputPath(Path(other_args[0]))
conf.setOutputPath(Path(other_args[1]))
JobClient.runJob(conf);
if __name__ == "__main__":
main(sys.argv)
]]>