ç”׃ºŽå…¬å¸çš„业务需要，要在¾|‘络攉™›†¾|‘页之后对网™åµè¿›è¡Œç»“构化的解析，˜q™ä¸ª¾l“构化过½E‹å¸Œæœ›èƒ½å¤ŸåŸºäº?/span>HDFSòq¶ä¸”使用MR½Ž—法实现ã€?/span>

我虚拟了一个需求，针对http://hadoop.apache.org/common/releases.html ™åµé¢åQŒå‡è®‘Ö·²¾lä¸‹è½½äº†™åµé¢òq¶å…¥åº“。要求最¾lˆä½“现的数据æ˜?/span> “版本å?/span>+完整链接åQˆå³a标签全部内容åQ?#8221; çš„结构ã€?/span>

伪分布式环境搭徏在虚拟机上，操作¾pȝ»Ÿæ˜?/span>centos5.5åQ?/span>hadoop的版本是1.0.0.

插入书中的一些环境搭建的介绍

书中çš?/span>2.1节中介绍了每个进½E‹çš„作用ã€?/span>2.2节中çš?/span>ssh讄¡½®ä¹Ÿæ¯”较重要，否则好像会一直提½CÞZ½ è¾“入密码ã€?/span>2.3.2节介¾läº†ä¼ªåˆ†å¸ƒå¼çš„配¾|®æ–¹å¼ï¼Œå¯?/span>core-site.xmlåQ?/span>mapred-site.xmlåQ?/span>hdfs-site.xml˜q›è¡Œé…ç½®ä¹‹åŽåQŒéœ€è¦å¯¹namenode节点˜q›è¡Œæ ¼å¼åŒ–ã€?/span>

bin/hadoop namenode –format

hadoop的根目录ä¸?/span>/usr/local/hadoop-1.0.0åQŒç›´æŽ¥å¯åŠ?/span>start-all.sh

[root@localhost hadoop-1.0.0]# ./bin/start-all.sh

启动成功后ä‹Éç”?/span>jps命ä×o

jdkž®å·¥å…?/span>jps介绍 

jps(Java Virtual Machine Process Status Tool)æ˜?/span>JDK 1.5提供的一个显½Cºå½“前所æœ?/span>java˜q›ç¨‹pid的命令，½Ž€å•å®žç”¨ï¼Œéžå¸¸é€‚合åœ?/span>linux/unixòq›_°ä¸Šç®€å•å¯Ÿçœ‹å½“å‰?/span>java˜q›ç¨‹çš„一些简单情å†üc€?/span> jps存放åœ?/span>JAVA_HOME/bin/jps

[root@localhost hadoop-1.0.0]# jps

5694 SecondaryNameNode

5461 NameNode

5578 DataNode

6027 Jps

5784 JobTracker

5905 TaskTracker

˜q™å‡ ä¸ªè¿›½E‹æ˜¯éžå¸¸é‡è¦çš„。很多时候出现意外就是因为某™åÒŽœåŠ¡æœªå¯åŠ¨æˆ–异常。可以看åˆîC¸Šé¢çš„命ä×o上打印出日志位置。出现异常后可以在日志中查看详细的堆栈信息ã€?/span>

è‡Ïx­¤åQ?/span>hadoop已经启动åQŒçŽ¯å¢ƒå·²¾lå‡†å¤‡å°±¾lªã€?/span>

下面准备我们的测试数据，ž®†ç›®æ ‡é¡µé¢çš„html保存ä¸?/span>news.txtåQŒä¼ªåˆ†å¸ƒå¼ä¹ŸåŒæ ·æ”¯æŒhdfsåQŒæ‰€ä»¥æˆ‘们ä‹Éç”?/span> fs –put ž®?/span>news.txt存入hdfs中ã€?/span>

[root@localhost hadoop-1.0.0]# ./bin/hadoop fs -put /mnt/hgfs/shared/news.txt /user/root

[root@localhost hadoop-1.0.0]# ./bin/hadoop fs -lsr /userdrwxr-xr-x   - root supergroup          0 2012-04-01 11:22 /user/root

-rw-r--r--   1 root supergroup       3935 2012-04-01 11:22 /user/root/news.txt

实现的代码在eclipse中ä‹Éç”?/span>maven打包åQŒä¸Šä¼ è‡³è™šæ‹Ÿæœºã€?/span>

æ–‡äšgå?/span>com.suntang.analyse.hadoop-0.0.1.jarã€?br />使用hadoop的中的jar命ä×o调用该jaræ–‡äšgã€?br />

[root@localhost hadoop-1.0.0]# ./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar com.suntang.analyse.hadoop.AnalyseJob /user/root/news.txt output_root_test

12/04/01 14:40:04 INFO input.FileInputFormat: Total input paths to process : 1

12/04/01 14:40:05 INFO mapred.JobClient: Running job: job_201204011420_0001

12/04/01 14:40:06 INFO mapred.JobClient:  map 0% reduce 0%

12/04/01 14:40:19 INFO mapred.JobClient:  map 100% reduce 0%

12/04/01 14:40:31 INFO mapred.JobClient:  map 100% reduce 100%

12/04/01 14:40:37 INFO mapred.JobClient: Job complete: job_201204011420_0001

…

此处注意我犯的一个错误：

[root@localhost hadoop-1.0.0]# ./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar AnalyseJob -libjars hadoop-core-1.0.0.jar /user/root/news.txt output_root_test

Exception in thread "main" java.lang.ClassNotFoundException: AnalyseJob

提示找不到类åQŒå› ä¸ºæˆ‘忘了写完整类名，命ä×o应该改äØ“

./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar com.suntang.analyse.hadoop.AnalyseJob -libjars hadoop-core-1.0.0.jar /user/root/news.txt output_root_test å›_¯ã€?/span>

 

此处˜qè¡Œå¯èƒ½å‡ºçŽ°å¦å¤–一个错误。在命ä×o行中出现

12/04/01 14:01:38 INFO mapred.JobClient: Task Id : attempt_201204011356_0001_m_000001_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Creation of symlink from /mnt/hgfs/shared/hadoop-1.0.0/libexec/../logs/userlogs/job_201204011356_0001/attempt_201204011356_0001_m_000001_0 to ã€‚。ã€?/span>

ž®×ƒ¸æ‰“全了，重点在与

Creation of symlinkåQŒçœ‹è¯¦ç»†æ—¥å¿—ä¸?/span>hadoop-root-tasktracker-localhost.localdomain.log中提½C?/span>org.apache.hadoop.fs.FileUtil: Command 'ln -s ....': Operation not supportedåQŒå³ln操作不支持ã€?/span>google可知˜q™ä¸ªæ˜¯ç”±äº?/span>vm中的å…׃ín区域的问题，解决æ–ÒŽ³•ž®±æ˜¯ž®?/span>hadoop完全转移è‡?/span>linux目录中。本例中ä»?/span>/mnt/hgfs/shared/hadoop-1.0.0转移è‡?/span>/usr/local/hadoop-1.0.0ã€?/span>

执行完成后可åœ?/span>hdfs中查看结果，查看目录¾l“æž„ä¸?/span>

-rw-r--r--   1 root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_SUCCESS

drwxr-xr-x   - root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_logs

drwxr-xr-x   - root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_logs/history

-rw-r--r--   1 root supergroup      13634 2012-04-01 14:40 /user/root/output_root_test/_logs/history/job_201204011420_0001_1333262405103_root_ccAnalyseJob

-rw-r--r--   1 root supergroup      20478 2012-04-01 14:40 /user/root/output_root_test/_logs/history/job_201204011420_0001_conf.xml

-rw-r--r--   1 root supergroup       3580 2012-04-01 14:40 /user/root/output_root_test/part-r-00000

/user/root/output_root_test/part-r-00000即äؓ最¾lˆç»“果文件ã€?/span>

 =======================================================================

附加AnalyseJob代码

package com.suntang.analyse.hadoop;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.conf.Configured;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import org.apache.hadoop.util.Tool;

import org.apache.hadoop.util.ToolRunner;

public class AnalyseJob extends Configured implements Tool {

       public static class MapClass extends Mapper<LongWritable, Text, Text, Text> {

              @Override

              protected void map(LongWritable key, Text value, Context context)

                            throws IOException, InterruptedException {

                     // TODO Auto-generated method stub

                     // super.map(key, value, context);

                     if (value.toString().matches("<a[^>]*>.*?release.*?</a>"))

                            context.write(

                                          new Text(value.toString().substring(

                                                        value.toString().indexOf("release") + 8,

                                                        value.toString().indexOf("available") - 1)),

                                          value);

              }

       }

       public static class ReduceClass extends Reducer<Text, Text, Text, Text> {

              @Override

              protected void reduce(Text arg0, Iterable<Text> arg1, Context arg2)

                            throws IOException, InterruptedException {

                     // TODO Auto-generated method stub

                     // super.reduce(arg0, arg1, arg2);

                     arg2.write(arg0, arg1.iterator().next());

              }

       }

       public int run(String[] args) throws Exception {

              Configuration conf = getConf();

              Job job = new Job(conf, "myAnalyseJob");

              job.setJarByClass(getClass());

              Path in = new Path(args[0]);

              Path out = new Path(args[1]);

              FileInputFormat.setInputPaths(job, in);

              FileOutputFormat.setOutputPath(job, out);

              job.setMapperClass(MapClass.class);

              job.setReducerClass(ReduceClass.class);

              job.setInputFormatClass(TextInputFormat.class);

              job.setOutputFormatClass(TextOutputFormat.class);

              job.setOutputKeyClass(Text.class);

              job.setOutputValueClass(Text.class);

              System.exit(job.waitForCompletion(true) ? 0 : 1);

              return 0;

       }

       public static void main(String[] args) throws Exception {

              int res = ToolRunner.run(new Configuration(), new AnalyseJob(), args);

              System.exit(res);

       }

}