伊人成人在线视频,久久亚洲精品视频,日韩在线一区二区

HADOOP各种框架应用领域

paulwong — Sun, 04 Jan 2015 04:57:00 GMT

***** Data Analytics : Technology Area *****
1. Real Time Analytics : Apache Storm
2. In-memory Analytics : Apache Spark
3. Search Analytics : Apache Elastic search, SOLR
4. Log Analytics : Apache ELK Stack,ESK Stack(Elastic Search, Log
Stash, Spark Streaming, Kibana)
5. Batch Analytics : Apache MapReduce

***** NO SQL DB *****
1. MongoDB
2. Hbase
3. Cassandra

***** SOA *****
1. Oracle SOA
2. JBoss SOA
3. TiBco SOA
4. SOAP, RESTful Webservices

paulwong 2015-01-04 12:57 发表评论

�~�译HADOOP源码

paulwong — Mon, 15 Dec 2014 17:41:00 GMT

https://github.com/apache/hadoop/blob/trunk/BUILDING.txt

配置 eclipse �~�译、开�?Hadoop�Q�MapReduce�Q�源代码
http://blog.csdn.net/basicthinker/article/details/6174442

hadoop2.2.0源代码编�?br /> http://my.oschina.net/cloudcoder/blog/192224

Apache Hadoop 源代码编译环境搭�?br /> http://qq85609655.iteye.com/blog/1986991

Download code from https://codeload.github.com/apache/hadoop/zip/trunk, then unzip it, there is a folder hadoop-trunk.

wget https://codeload.github.com/apache/hadoop/zip/trunk
unzip trunk
Install native libraries

Ubuntu
sudo apt-get -y install maven build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev

Cent OS
yum -y install lzo-devel zlib-devel gcc autoconf automake libtool openssl-devel cmake
get protobuf zip from http://f.dataguru.cn/thread-459689-1-1.html
./configure
make
make check
make install
$vi /etc/profile
export PROTOC_HOME=/root/java/hadoop-source/protobuf-2.5.0
export PATH=$PATH:$PROTOC_HOME/src
cd to hadoop-trunk, run

mvn compile -Pnative
cd to hadoop-maven-plugins, run

mvn install
cd to hadoop-trunk

mvn install -DskipTests
Make sure still in hadoop-trunk folder, Build Eclipse project

mvn eclipse:eclipse -DskipTests
Import the maven project to Eclipse

paulwong 2014-12-16 01:41 发表评论

Simplehbase

paulwong — Tue, 15 Jul 2014 00:35:00 GMT

https://github.com/zhang-xzhi/simplehbase/
https://github.com/zhang-xzhi/simplehbase/wiki

## simplehbase��?
simplehbase是java和hbase之间的轻量��中间件�?
主要包含以下功能�?
* 数据�c�d��映射�Q�java�c�d��和hbase的bytes之间的数据�{换�?
* ��单操作封装：��装了hbase的put,get,scan�{�操作�ؓ��单的java操作方式�?
* hbase query��装�Q�封装了hbase的filter�Q�可以��用sql-like的方式操作hbase�?
* 动态query��装�Q�类��g��myibatis�Q�可以��用xml配置动态语句查询hbase�?
* insert,update支持: 建立在hbase的checkAndPut之上�?
* hbase多版本支持：提供接口可以对hbase多版本数据进行查�?映射�?
* hbase原生接口支持�?

### v0.9
新增

支持HTable如下使用方式�Q�对HTable可以定时flush�?
主要场景�Q?
扚w��写入�Q�但是flush可以配置为指定时间间隔进行�?
不降低批操作的吞吐，同时�Q�有一定的实时性保证�?

支持用户自定义htablePoolService�?
多个HTable可以使用同一个线�E�池�?

intelligentScanSize功能,可以�Ҏ��limit的��D��定scan的cachingsize大小�?

### v0.8
扚w��操作接口新增

public  void putObjectList(List> putRequestList);
public void deleteObjectList(List rowKeyList, Class type);
public  void putObjectListMV(List> putRequests,long timestamp)
public  void putObjectListMV(List> putRequests,Date timestamp)
public  void putObjectListMV(List> putRequestList)
public void deleteObjectMV(RowKey rowKey, Class type, long timeStamp)
public void deleteObjectMV(RowKey rowKey, Class type, Date timeStamp)
public void deleteObjectListMV(List rowKeyList, Class type,long timeStamp)
public void deleteObjectListMV(List rowKeyList, Class type,Date timeStamp)
public void deleteObjectListMV(List deleteRequestList,Class type);

Util新增�Q�前�~�查询使用�Q?

public static RowKey getEndRowKeyOfPrefix(RowKey prefixRowKey)

性能改进
把get的实��C��scan调回get�?

### v0.7新增功能�Q?
支持查询时主记录和关联的RowKey同时�q�回�?nbsp;

paulwong 2014-07-15 08:35 发表评论

安装CLOUDERA

paulwong — Fri, 23 May 2014 10:16:00 GMT

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html

http://www.cnblogs.com/xuesong/p/3604080.html

http://www.linuxidc.com/Linux/2013-12/94180.htm

卸蝲
http://www.cnblogs.com/shudonghe/articles/3133290.html

安装文�g�Q?br /> http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=4ZFrtT9ZQN

change to no password

sudo chmod +w /etc/sudoers
sudo vi /etc/sudoers
ufuser ALL=(ALL) NOPASSWD: ALL
sudo chmod -w /etc/sudoers
change disable

sudo vi /etc/selinux/config
SELINUX=disabled
sudo reboot
add to /etc/hosts

sudo vi /etc/hosts

10.0.0.4 ufhdp001.cloudapp.net ufhdp001
10.0.0.5 ufhdp002.cloudapp.net ufhdp002
download bin

wget http://archive.cloudera.com/cm4/installer/latest/cloudera-manager-installer.bin
run the bin

chmod 755 cloudera-manager-installer.bin
sudo ./cloudera-manager-installer.bin

paulwong 2014-05-23 18:16 发表评论

paulwong — Fri, 23 May 2014 04:15:00 GMT

开源大数据框架Apache Hadoop已经成了大数据处理的事实标准�Q�同时也几乎成了大数据的代名词，虽然�q�多��有些以偏概全�?/p>

�Ҏ��Gartner的估计，目前的Hadoop生态系�l�市��模在7700万美元左叻I��2016�q�_��该市��模将快速增长至8.13亿美元�?/p>

但是在Hadoop�q�个快速扩增的蓝�v中游泛_ƈ非易事，不仅开发大数据基础设施技术��品这件事很难�Q�销售�v来也很难�Q�具体到大数据基��设施工具�?Hadoop、NoSQL数据库和��处理系�l�则更是难上加难。客户需要大量培训和教育�Q�付费用户需要大量支持和及时跟进的��品开发工作。而跟企业�U�客��h�� 交道往往�q��创业公司团队的强��V��此外，大数据基��设施技术创业通常寚w��险投资规模也有较高要求�?/p>

��管困难重重�Q�Hadoop创业公司依然如雨后春�W�冒出，除了Cloudera、Datameer、DataStax和MapR�{�已�l�功成名��q�� Hadoop创业公司外，最�q�CIO杂志评出�?014�q�十大最值得��x��的Hadoop创业公司�Q�了解这些公司的产品和商业模式对企业大数据技术创业者和大数据应用用��h��说都非常有参考�h��|��

一�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Platfora

业务�Q�所提供的大数据分析解决�Ҏ��能够��Hadoop中的原始数据转换成可互动的，��Z��内存计算的商业智能服务�?/p>

��介：创立�?011�q�_��q�今已募�?500万美元�?/p>

入选理由：Platfora的目标是��化复杂难用的Hadoop�Q�推动Hadoop在企业市场的应用。Platfora的做法是��化数据采集和分析 ��程�Q�将Hadoop中的原始数据自动转化成可以互动的商业��服务�Q�无需ETL或者数据仓库�?参考阅读：Hadoop只是�I��h的ETL)

二�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Alpine Data Labs

业务�Q�提供基于Hadoop的数据分析��^�?/p>

��介：创立�?010�q�_��q�今累计融资2350万美元�?/p>

入选理由：复杂的高�U�分析和机器学习应用通常都需要脚本和代码开发高手实玎ͼ��q�进一�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">推高了数据科学家的技术门�?/a>。实际上大数据企业高��和IT�l�理都没旉��也没兴致学习�~�程技术，或者去了解复杂的Hadoop。Alpine Data通过SaaS服务的方式大�q�降低了预测分析的应用门槛�?/p>

三�?/strong>Altiscale

业务�Q�提供Hadoop��x��务（HaaS�Q?/p>

��介：创立�?012�q?月，�q�今融资1200万美元�?/p>

入选理由：大数据正在闹人才荒，而通过云计��提供Hadoop相关服务无疑是普及Hadoo的一条捷径，�Ҏ��TechNavio的估计，2016�q?HaaS市场规模��高�?90亿美元，是块大蛋�p�。但是HaaS市场的竞争已�l�日��激烈，包括亚马逊EMR、微软的Hadoop on Azure�Q�以及Rackspace的Hortonworks云服务等都是重量�U�玩�Ӟ��Altiscale�q�需要与Hortonworks�?Cloudera、Mortar Data、Qubole、Xpleny展开直接竞争�?/p>

四�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Trifacta

业务�Q�提供��^台帮助用户将复杂的原始数据�{化成�q�净的结构化格式供分析��用�?/p>

��介：创立�?012�q�_��q�今融资1630万美元�?/p>

入选理由：大数据技术��^台和分析工具之间存在一个巨大的瓉��Q�那��是数据分析专家需要花费大量精力和旉��转化数据�Q�而且业务数据分析师们往往也�ƈ�?具备独立完成数据转化工作的技术能力。�ؓ了解册��个问题Trifacta开发出�?#8220;预测互动”技术，��数据操作可视化�Q�而且Trifacta的机器学习算法还能同时观察用户和数据属性，预测用户意图�Q��ƈ自动�l�出��。Trifata的竞争对手是Paxata、Informatica和CirroHow�?/p>

五�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Splice Machine

业务�Q�提供面向大数据应用的，��Z��Hadoop的SQL兼容数据库�?/p>

��介：创立�?012�q�_��q�今融资1900万美元�?/p>

入选理由：新的数据技术��得传�l�关�p�d��数据库的一些流行功能如ACID合规、交易一致性和标准的SQL查询语言�{�得以在廉�h可扩展的Hadoop�?延箋。Splice Machine保留了NoSQL数据库所有的优点�Q�例如auto-sharding�Q�容错、可扩展性等�Q�同时又保留了SQL�?/p>

六�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">DataTorrent

业务�Q�提供基于Hadoop�q�_��的实时流处理�q�_��

��介：创立�?012�q�_��2013�q?月获�?00万美元A轮融资�?/p>

入选理由：大数据的未来是快数据�Q�而DataTorrent正是要解军_��数据的问题�?/p>

七�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Qubole

业务�Q�提供大数据DaaS服务�Q�基�?#8220;真正的自动扩展Hadoop集群”�?/p>

��介：创立�?011�q�_��累计融资700万美元�?/p>

入选理由：大数据�h才一��难求，对于大多��C��业来��_��像��用SaaS企业应用一样��用Hadoop是一个现实的选择�?/p>

八�?a style="margin: 0px; padding: 0px; color: #3e62a6; outline: 0px;">Continuuity

业务�Q�提供基于Hadoop的大数据应用托管�q�_��

��介：创立�?011�q�_��累计获得1250万美元融资，创始人兼CEO Todd Papaioannou曾是雅虎副总裁云架构负责�h�Q�去�q�夏天Todd��d��Continuuity后，联合创始人CTO Jonathan Gray接替担�QCEO一职�?/p>

入选理由：Continuuity的商业模式非常聪明也非常独特�Q�他们绕�q�非帔R��~�的Hadoop专家�Q�直接向Java开发者提供应用开发��^収ͼ��?旗舰产品Reactor是一个基于Hadoop的Java集成化数据和应用框架�Q�Continuuity��底层基��设施�q�行抽象处理�Q�通过��单的Java 和REST API提供底层基础设施服务�Q��ؓ用户大大��化了Hadoop基础设施的复杂性。Continuuity最新发布的服务——Loom是一个集��管理方案，�?�q�Loom创徏的集��可以��用�Q意硬件和软�g堆叠的模板，从单一的LAMP服务器和传统应用服务器如JBoss到包含数千个节点的大规模的Hadoop�?��。集��还可以部��v在多个云服务商的环境中（例如Rackspace、Joyent、Openstack�{�）而且�q�能使用常见的SCM工具�?/p>

九�?/strong>Xplenty

业务�Q�提供HaaS服务

��介：创立�?012�q�_��从Magma风险投资获得金额不详的融资�?/p>

入选理由：虽然Hadoop已经成了大数据的事实工业标准�Q�但是Hadoop的开发、部�|�和�l�护�Ҏ��术�h员的技能依然有着极高要求。Xplenty 的技术通过无需�~�写代码的Hadoop开发环境提供Hadoop处理服务�Q�企业无需投资软硬件和专业人才��p��快速��n受大数据技术�?/p>

十�?/strong>Nuevora

业务�Q�提供大数据分析应用

��介：创立�?011�q�_��累计获得300万早期投资�?/p>

入选理由：Nuevora的着眼点是大数据应用最早启动的两个领域�Q�营销和客��h��触。Nuevora的nBAAP�Q�大数据分析与应用）�q�_��的主要功能包括基于最��x��间预��算法的定制分析应用�Q�nBAAP��Z��三个关键大数据技术：Hadoop�Q�大数据处理�Q�、R�Q�预��分析）和Tableau�Q�数据可�?化）

paulwong 2014-05-23 12:15 发表评论

KMEANS PAGERANK ON HADOOP

paulwong — Wed, 07 May 2014 15:57:00 GMT
https://github.com/keokilee/kmeans-hadoop

https://github.com/rorlig/hadoop-pagerank-java

http://wuyanzan60688.blog.163.com/blog/static/12777616320131011426159/

http://codecloud.net/hadoop-k-means-591.html

import java.io.*;
import java.net.URI;
import java.util.Iterator;
import java.util.Random;
import java.util.Vector;

import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.GenericOptionsParser;

public class KMeans {
    static enum Counter { CENTERS, CHANGE, ITERATIONS }

    public static class Point implements WritableComparable {
        // Longs because this will store sum of many ints
        public LongWritable x;
        public LongWritable y;
        public IntWritable num; // For summation points

        public Point() {
            this.x = new LongWritable(0);
            this.y = new LongWritable(0);
            this.num = new IntWritable(0);
        }

        public Point(int x, int y) {
            this.x = new LongWritable(x);
            this.y = new LongWritable(y);
            this.num = new IntWritable(1);
        }

        public Point(IntWritable x, IntWritable y) {
            this.x = new LongWritable(x.get());
            this.y = new LongWritable(y.get());
            this.num = new IntWritable(1);
        }

        public void add(Point that) {
            x.set(x.get() + that.x.get());
            y.set(y.get() + that.y.get());
            num.set(num.get() + that.num.get());
        }

        public void norm() {
            x.set(x.get() / num.get());
            y.set(y.get() / num.get());
            num.set(1);
        }

        public void write(DataOutput out) throws IOException {
            x.write(out);
            y.write(out);
            num.write(out);
        }

        public void readFields(DataInput in) throws IOException {
            x.readFields(in);
            y.readFields(in);
            num.readFields(in);
        }

        public long distance(Point that) {
            long dx = that.x.get() - x.get();
            long dy = that.y.get() - y.get();

            return dx * dx + dy * dy;
        }

        public String toString() {
            String ret = x.toString() + '\t' + y.toString();
            if (num.get() != 1)
                ret += '\t' + num.toString();
            return ret;
        }

        public int compareTo(Point that) {
            int ret = x.compareTo(that.x);
            if (ret == 0)
                ret = y.compareTo(that.y);
            if (ret == 0)
                ret = num.compareTo(that.num);
            return ret;
        }
    }

    public static class Map
            extends MapReduceBase
            implements Mapper
    {
        private Vector centers;
        private IOException error;

        public void configure(JobConf conf) {
            try {
                Path paths[] = DistributedCache.getLocalCacheFiles(conf);
                if (paths.length != 1)
                    throw new IOException("Need exactly 1 centers file");

                FileSystem fs = FileSystem.getLocal(conf);
                SequenceFile.Reader in = new SequenceFile.Reader(fs, paths[0], conf);

                centers = new Vector();
                IntWritable x = new IntWritable();
                IntWritable y = new IntWritable();
                while(in.next(x, y))
                    centers.add(new Point(x, y));
                in.close();

                // Generate new points if we don't have enough.
                int k = conf.getInt("k", 0);
                Random rand = new Random();
                final int MAX = 1024*1024;
                for (int i = centers.size(); i < k; i++) {
                    x.set(rand.nextInt(MAX));
                    y.set(rand.nextInt(MAX));
                    centers.add(new Point(x, y));
                }
            } catch (IOException e) {
                error = e;
            }
        }

        public void map(Text xt, Text yt,
                OutputCollector output, Reporter reporter)
            throws IOException
        {
            if (error != null)
                throw error;

            int x = Integer.valueOf(xt.toString());
            int y = Integer.valueOf(yt.toString());
            Point p = new Point(x, y);
            Point center = null;
            long distance = Long.MAX_VALUE;

            for (Point c : centers) {
                long d = c.distance(p);
                if (d <= distance) {
                    distance = d;
                    center = c;
                }
            }

            output.collect(center, p);
        }
    }

    public static class Combine
            extends MapReduceBase
            implements Reducer
    {
        public void reduce(Point center, Iterator points,
                OutputCollector output, Reporter reporter)
            throws IOException
        {
            Point sum = new Point();
            while(points.hasNext()) {
                sum.add(points.next());
            }

            output.collect(center, sum);
        }
    }

    public static class Reduce
            extends MapReduceBase
            implements Reducer
    {
        public void reduce(Point center, Iterator points,
                OutputCollector output,
                Reporter reporter)
            throws IOException
        {
            Point sum = new Point();
            while (points.hasNext()) {
                sum.add(points.next());
            }
            sum.norm();

            IntWritable x = new IntWritable((int) sum.x.get());
            IntWritable y = new IntWritable((int) sum.y.get());

            output.collect(x, y);

            reporter.incrCounter(Counter.CHANGE, sum.distance(center));
            reporter.incrCounter(Counter.CENTERS, 1);
        }
    }

    public static void error(String msg) {
        System.err.println(msg);
        System.exit(1);
    }

    public static void initialCenters(
            int k, JobConf conf, FileSystem fs,
            Path in, Path out)
        throws IOException
    {
        BufferedReader input = new BufferedReader(
                new InputStreamReader(fs.open(in)));
        SequenceFile.Writer output = new SequenceFile.Writer(
                fs, conf, out, IntWritable.class, IntWritable.class);
        IntWritable x = new IntWritable();
        IntWritable y = new IntWritable();
        for (int i = 0; i < k; i++) {
            String line = input.readLine();
            if (line == null)
                error("Not enough points for number of means");

            String parts[] = line.split("\t");
            if (parts.length != 2)
                throw new IOException("Found a point without two parts");

            x.set(Integer.valueOf(parts[0]));
            y.set(Integer.valueOf(parts[1]));
            output.append(x, y);
        }
        output.close();
        input.close();
    }

    public static void main(String args[]) throws IOException {
        JobConf conf = new JobConf(KMeans.class);
        GenericOptionsParser opts = new GenericOptionsParser(conf, args);
        String paths[] = opts.getRemainingArgs();

        FileSystem fs = FileSystem.get(conf);

        if (paths.length < 3)
            error("Usage:\n"
                    + "\tKMeans \n"
                    + "\tKMeans   "
                 );

        Path outdir  = new Path(paths[0]);
        int k = Integer.valueOf(paths[1]);
        Path firstin = new Path(paths[2]);

        if (k < 1 || k > 20)
            error("Strange number of means: " + paths[1]);

        if (fs.exists(outdir)) {
            if (!fs.getFileStatus(outdir).isDir())
                error("Output directory \"" + outdir.toString()
                        + "\" exists and is not a directory.");
        } else {
            fs.mkdirs(outdir);
        }

        // Input: text file, each line "x\ty"
        conf.setInputFormat(KeyValueTextInputFormat.class);
        for (int i = 2; i < paths.length; i++)
            FileInputFormat.addInputPath(conf, new Path(paths[i]));

        conf.setInt("k", k);

        // Map: (x,y) -> (centroid, point)
        conf.setMapperClass(Map.class);
        conf.setMapOutputKeyClass(Point.class);
        conf.setMapOutputValueClass(Point.class);

        // Combine: (centroid, points) -> (centroid, weighted point)
        conf.setCombinerClass(Combine.class);

        // Reduce: (centroid, weighted points) -> (x, y) new centroid
        conf.setReducerClass(Reduce.class);
        conf.setOutputKeyClass(IntWritable.class);
        conf.setOutputValueClass(IntWritable.class);

        // Output
        conf.setOutputFormat(SequenceFileOutputFormat.class);

        // Chose initial centers
        Path centers = new Path(outdir, "initial.seq");
        initialCenters(k, conf, fs, firstin, centers);

        // Iterate
        long change  = Long.MAX_VALUE;
        URI cache[] = new URI[1];
        for (int iter = 1; iter <= 1000 && change > 100 * k; iter++) {
            Path jobdir = new Path(outdir, Integer.toString(iter));
            FileOutputFormat.setOutputPath(conf, jobdir);

            conf.setJobName("k-Means " + iter);
            conf.setJarByClass(KMeans.class);

            cache[0] = centers.toUri();
            DistributedCache.setCacheFiles( cache, conf );

            RunningJob result = JobClient.runJob(conf);
            System.out.println("Iteration: " + iter);

            change   = result.getCounters().getCounter(Counter.CHANGE);
            centers  = new Path(jobdir, "part-00000");
        }
    }
}

192.5.53.208

paulwong 2014-05-07 23:57 发表评论

Packt celebrates International Day Against DRM, May 6th 2014

paulwong — Tue, 06 May 2014 12:05:00 GMT

Packt celebrates International Day Against DRM, May 6^th 2014

According to the definition of DRM on Wikipedia, Digital Rights Management (DRM) is a class of technologies that are used by hardware manufacturers, publishers, copyright holders, and individuals with the intent to control the use of digital content and devices after sale.

However, Packt Publishing firmly believes that you should be able to read and interact with your content when you want, where you want, and how you want – to that end they have been advocates of DRM-free content since their very first eBook was published back in 2004.

To show their continuing support for Day Against DRM, Packt Publishing is offering all its DRM-free content at $10 for 24 hours only on May 6^th – that’s all 2000+ eBooks and Videos. Check it out at: http://bit.ly/1q6bpha.

paulwong 2014-05-06 20:05 发表评论

A book: Web Crawling and Data Mining with Apache Nutch

paulwong — Mon, 03 Feb 2014 05:14:00 GMT

Recently I am reading a book , http://www.packtpub.com/web-crawling-and-data-mining-with-apache-nutch/book, it is really a great book. And I get help in my project.

In my project I need to crawl the web content and do the data analyst. From the book I can know how to use and integrate Nutch and Solr frameworks to implement it.

If you have similiar case, recommand to read this book.

paulwong 2014-02-03 13:14 发表评论

【�{载】经典�O画讲解HDFS原理

paulwong — Sat, 26 Oct 2013 01:15:00 GMT
分布式文件系�l�比较出名的有HDFS  �?GFS�Q�其中HDFS比较��单一炏V��本文是一��描�q�非常简�z�易懂的漫画形式讲解HDFS的原理。比一般PPT要通俗易懂很多。不隑־�的学习资料�?br />

1、三个部�? 客户端、nameserver�Q�可理解��Z��控和文�g索引,�c�M��linux的inode�Q�、datanode�Q�存攑֮�际数据）

在这里，client的�Ş式我所了解的有两种�Q�通过hadoop提供的api所�~�写的程序可以和hdfs�q�行交互�Q�另外一�U�就是安装了hadoop的datanode其也可以通过命��o行与hdfs�pȝ��q�行交互�Q�如在datanode上上传则使用如下命��o行：bin/hadoop fs -put example1 user/chunk/

2、如何写数据�q�程

3、读取数据过�E?/span>

4、容错：�W�一部分�Q�故障类型及其检��方法（nodeserver 故障�Q�和�|�络故障�Q�和脏数据问题）

5、容错第二部分：��d��定w��

6、容错第三部分：dataNode 失效

7、备份规�?/span>

8、结束语

paulwong 2013-10-26 09:15 发表评论

Install Hadoop in the AWS cloud

paulwong — Sun, 08 Sep 2013 05:45:00 GMT

get the Whirr tar file

wget http://www.eu.apache.org/dist/whirr/stable/whirr-0.8.2.tar.gz

untar the Whirr tar file
tar -vxf whirr-0.8.2.tar.gz
create credentials file
mkdir ~/.whirr
cp conf/credentials.sample ~/.whirr/credentials
add the following content to credentials file
# Set cloud provider connection details
PROVIDER=aws-ec2
IDENTITY=
CREDENTIAL=
generate a rsa key pair
ssh-keygen -t rsa -P ''
create a hadoop.properties file and add the following content
whirr.cluster-name=whirrhadoopcluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,2 hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop.version=1.0.2
whirr.aws-ec2-spot-price=0.08
launch hadoop
bin/whirr launch-cluster --config hadoop.properties
launch proxy
cd ~/.whirr/whirrhadoopcluster/
./hadoop-proxy.sh
add a rule to iptables
0.0.0.0/0 50030
0.0.0.0/0 50070
check the web ui in the browser
http://:50030
add to /etc/profile
export HADOOP_CONF_DIR=~/.whirr/whirrhadoopcluster/
check if the hadoop works
hadoop fs -ls /

paulwong 2013-09-08 13:45 发表评论

Install hadoop+hbase+nutch+elasticsearch

paulwong — Fri, 30 Aug 2013 17:17:00 GMT
     摘要: This document is for Anyela Chavarro. Only these version of each framework work together Code highlighting produced by Actipro CodeHighlighter (freeware) http://www.CodeHighlighter.com/ -->H...  阅读全文

paulwong 2013-08-31 01:17 发表评论

Implementation for CombineFileInputFormat Hadoop 0.20.205

paulwong — Thu, 29 Aug 2013 08:08:00 GMT
�q�行MAPREDUCE JOB�Ӟ��如果输入的文件比较小而多�Ӟ��默认情况下会生成很多的MAP JOB�Q�即一个文件一个MAP JOB�Q�因此需要优化，使多个文件能合成一个MAP JOB的输入�?br />
具体的原理是下述三步:

1.�Ҏ��输入目录下的每个文�g,如果光��度超�q�mapred.max.split.size,以block为单位分成多个split(一个split是一个map的输�?,每个split的长度都大于mapred.max.split.size, 因�ؓ以block为单�? 因此也会大于blockSize, 此文件剩下的长度如果大于mapred.min.split.size.per.node, 则生成一个split, 否则先暂时保�?

2. 现在剩下的都是一些长度效短的��片,把每个rack下碎片合�q? 只要长度��过mapred.max.split.size��合�q�成一个split, 最后如果剩下的��片比mapred.min.split.size.per.rack�? ��合�q�成一个split, 否则暂时保留.

3. 把不同rack下的��片合�ƈ, 只要长度��过mapred.max.split.size��合�q�成一个split, 剩下的碎片无论长�? 合�ƈ成一个split.
举例: mapred.max.split.size=1000
mapred.min.split.size.per.node=300
mapred.min.split.size.per.rack=100
输入目录下五个文�?rack1下三个文�?长度�?050,1499,10, rack2下两个文�?长度�?010,80. 另外blockSize�?00.
�l�过�W�一�? 生成五个split: 1000,1000,1000,499,1000. 剩下的碎片�ؓrack1�?50,10; rack2�?0:80
�׃��两个rack下的��片和都不超�q?00, 所以经�q�第二步, split和碎片都没有变化.
�W�三�?合�ƈ四个��片成一个split, 长度�?50.

如果要减��map数量, 可以调大mapred.max.split.size, 否则调小卛_��.

其特�Ҏ��: 一个块臛_��作�ؓ一个map的输入，一个文件可能有多个块，一个文件可能因为块多分�l�做��Z��同map的输入，一个map可能处理多个块，可能处理多个文�g�?br />
注：CombineFileInputFormat是一个抽象类�Q�需要编写一个��承类�?br />

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.InputSplit;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.LineRecordReader;
import org.apache.hadoop.mapred.RecordReader;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.lib.CombineFileInputFormat;
import org.apache.hadoop.mapred.lib.CombineFileRecordReader;
import org.apache.hadoop.mapred.lib.CombineFileSplit;

@SuppressWarnings("deprecation")
public class CombinedInputFormat extends CombineFileInputFormat {

    @SuppressWarnings({ "unchecked", "rawtypes" })
    @Override
    public RecordReader getRecordReader(InputSplit split, JobConf conf, Reporter reporter) throws IOException {

        return new CombineFileRecordReader(conf, (CombineFileSplit) split, reporter, (Class) myCombineFileRecordReader.class);
    }

    public static class myCombineFileRecordReader implements RecordReader {
        private final LineRecordReader linerecord;

        public myCombineFileRecordReader(CombineFileSplit split, Configuration conf, Reporter reporter, Integer index) throws IOException {
            FileSplit filesplit = new FileSplit(split.getPath(index), split.getOffset(index), split.getLength(index), split.getLocations());
            linerecord = new LineRecordReader(conf, filesplit);
        }

        @Override
        public void close() throws IOException {
            linerecord.close();

        }

        @Override
        public LongWritable createKey() {
            // TODO Auto-generated method stub
            return linerecord.createKey();
        }

        @Override
        public Text createValue() {
            // TODO Auto-generated method stub
            return linerecord.createValue();
        }

        @Override
        public long getPos() throws IOException {
            // TODO Auto-generated method stub
            return linerecord.getPos();
        }

        @Override
        public float getProgress() throws IOException {
            // TODO Auto-generated method stub
            return linerecord.getProgress();
        }

        @Override
        public boolean next(LongWritable key, Text value) throws IOException {

            // TODO Auto-generated method stub
            return linerecord.next(key, value);
        }

    }
}

在运行时�q�样讄��Q?br />
if (argument != null) {
                conf.set("mapred.max.split.size", argument);
            } else {
                conf.set("mapred.max.split.size", "134217728"); // 128 MB
            }
//

            conf.setInputFormat(CombinedInputFormat.class);

paulwong 2013-08-29 16:08 发表评论

paulwong — Sun, 18 Aug 2013 10:27:00 GMT
!!!��Z��Hadoop的大数据�q�_��实施�?#8212;—整体架构设计
http://blog.csdn.net/jacktan/article/details/9200979

paulwong 2013-08-18 18:27 发表评论

How to install Hadoop cluster(2 node cluster) and Hbase on Vmware Workstation. It also includes installing Pig and Hive in the appendix

paulwong — Sat, 17 Aug 2013 14:23:00 GMT
By Tzu-Cheng Chuang 1-28-2011

Requires: Ubuntu10.04, hadoop0.20.2, zookeeper 3.3.2 HBase0.90.0
1. Download Ubuntu 10.04 desktop 32 bit from Ubuntu website.

2. Install Ubuntu 10.04 with username: hadoop, password: password, disk size: 20GB, memory: 2048MB, 1 processor, 2 cores

3. Install build-essential (for GNU C, C++ compiler)    $ sudo apt-get install build-essential

4. Install sun-jave-6-jdk
    (1) Add the Canonical Partner Repository to your apt repositories
    $ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
     (2) Update the source list
    $ sudo apt-get update
     (3) Install sun-java-6-jdk and make sure Sun’s java is the default jvm
    $ sudo apt-get install sun-java6-jdk
     (4) Set environment variable by modifying ~/.bashrc file, put the following two lines in the end of the file
    export JAVA_HOME=/usr/lib/jvm/java-6-sun
export PATH=$PATH:$JAVA_HOME/bin

5. Configure SSH server so that ssh to localhost doesn’t need a passphrase
    (1) Install openssh server
    $ sudo apt-get install openssh-server
     (2) Generate RSA pair key
    $ ssh-keygen –t ras –P ""
     (3) Enable SSH access to local machine
    $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

6. Disable IPv6 by      modifying /etc/sysctl.conf file, put the following two lines in the end of the file
#disable
ipv6 net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

7. Install hadoop
    (1) Download hadoop-0.20.2.tar.gz(stable release on 1/25/2011) from Apache hadoop website
    (2) Extract hadoop archive file to /usr/local/
    (3) Make symbolic link
    (4) Modify /usr/local/hadoop/conf/hadoop-env.sh
Change from # The java implementation to use. Required. # export JAVA_HOME=/usr/lib/j2sdk1.5-sun To # The java implementation to use. Required. export JAVA_HOME=/usr/lib/jvm/java-6-sun
     (5)Create /usr/local/hadoop-datastore folder
$ sudo mkdir /usr/local/hadoop-datastore
$ sudo chown hadoop:hadoop /usr/local/hadoop-datastore
$ sudo chmod 750 /usr/local/hadoop-datastore
     (6)Put the following code in /usr/local/hadoop/conf/core-site.xml
hadoop.tmp.dir/usr/local/hadoop/tmp/dir/hadoop-${user.name}A base for other temporary directories.fs.default.namehdfs://master:54310The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.
    (7) Put the following code in /usr/local/hadoop/conf/mapred-site.xml
mapred.job.trackermaster:54311The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.
     (8) Put the following code in /usr/local/hadoop/conf/hdfs-site.xml
dfs.replication1Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
     (9) Add hadoop to environment variable by modifying ~/.bashrc
export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$PATH

8. Restart Ubuntu Linux

9. Copy this virtual machine to another folder. At least we have 2 copies of Ubuntu linux

10. Modify /etc/hosts on both Linux Virtual Image machines, add in the following lines in the file. The IP address depends on each machine. We can use (ifconfig) to find out IP address.
# /etc/hosts (for master AND slave) 192.168.0.1 master 192.168.0.2 slave     Modify the following line, because it might cause Hbase to find out wrong ip.
192.168.0.1 ubuntu

11. Check hadoop user access on both machines.
The hadoop user on the master (aka hadoop@master) must be able to connect a) to its own user account on the master – i.e. ssh master in this context and not necessarily ssh localhost – and b) to the hadoop user account on the slave (aka hadoop@slave) via a password-less SSH login. On both machines, make sure each one can connect to master, slave without typing passwords.

12. Cluster configuration
    (1) Modify /usr/local/hadoop/conf/masters
         only on master machine    master
     (2) Modify /usr/local/hadoop/conf/slaves
          only on master machine    master slave
     (3) Change “localhost” to “master” in /usr/local/conf/hadoop/conf/core-site.xml and /usr/local/hadoop/conf/mapred-site.xml
        only on master machine
    (4) Change dfs.replication to “1” in /usr/local/conf/hadoop/conf/hdfs-site.xml
    only on master machine

13. Format the namenode only once and only on master machine
$ /usr/local/hadoop/bin/hadoop namenode –format

14. Later on, start the multi-node cluster by typing following code only on master. So far, please don’t start hadoop yet.
$ /usr/local/hadoop/bin/start-dfs.sh $ /usr/local/hadoop/bin/start-mapred.sh

15. Install zookeeper only on master node
    (1) download zookeeper-3.3.2.tar.gz from Apache hadoop website
    (2) Extract zookeeper-3.3.2.tar.gz    $ tar –xzf zookeeper-3-3.2.tar.gz
     (3) Move folder zookeeper-3.3.2 to /home/hadoop/ and create a symbloink link
    $ mv zookeeper-3.3.2 /home/hadoop/ ; ln –s /home/hadoop/zookeeper-3.3.2 /home/hadoop/zookeeper
     (4) copy conf/zoo_sample.cfg to conf/zoo.cfg
    $ cp conf/zoo_sample.cfg confg/zoo.cfg
     (5) Modify conf/zoo.cfg    dataDir=/home/hadoop/zookeeper/snapshot

16. Install Hbase on both master and slave nodes, configure it as fully-distributed
    (1) Download hbase-0.90.0.tar.gz from Apache hadoop website
    (2) Extract hbase-0.90.0.tar.gz    $ tar –xzf hbase-0.90.0.tar.gz
     (3) Move folder hbase-0.90.0 to /home/hadoop/ and create a symbloink link    $ mv hbase-0.90.0 /home/hadoop/ ; ln –s /home/hadoop/hbase-0.90.0 /home/hadoop/hbase
     (4) Edit /home/hadoop/hbase/conf/hbase-site.xml, put the following in between and hbase.rootdirhdfs://master:54310/hbase The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR hbase.cluster.distributedtrueThe mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) hbase.zookeeper.quorummasterComma separated list of servers in the ZooKeeper Quorum. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on.
     (5) modify environment variables in /home/hadoop/hbase/conf/hbase-env.sh
    export JAVA_HOME=/usr/lib/jvm/java-6-sun/
export HBASE_IDENT_STRING=$HOSTNAME
export HBASE_MANAGES_ZK=false
     (6)Overwrite /home/hadoop/hbase/conf/regionservers
on both machines    master slave
     (7)copy /usr/local/hadoop-0.20.2/haoop-0.20.2-core.jar to /home/hadoop/hbase/lib/ on both machines.
      This is very important to fix version difference issue. Pay attention to its ownership and mode(755).

17. Start zookeeper. It seems the zookeeper bundled with Hbase is not set up correctly.
$ /home/hadoop/zookeeper/bin/zkServer.sh start     (Optional)We can test if zookeeper is running correctly by typing     $ /home/hadoop/zookeeper/bin/zkCli.sh –server 127.0.0.1:2181

18. Start hadoop cluster
$ /usr/local/hadoop/bin/start-dfs.sh $ /usr/local/hadoop/bin/start-mapred.sh

19. Start Hbase
$ /home/hadoop/hbase/bin/start-hbase.sh

20. Use Hbase shell
$ /home/hadoop/hbase/bin/hbase shell     Check if hbase is running smoothly
    Open your browser, and type in the following.
    http://localhost:60010

21. Later on, stop the multi-node cluster by typing following code only on master
    (1) Stop Hbase    $ /home/hadoop/hbase/bin/stop-hbase.sh
     (2) Stop hadoop file system (HDFS)
$ /usr/local/hadoop/bin/stop-mapred.sh
$ /usr/local/hadoop/bin/stop-dfs.sh
     (3) Stop zookeeper
$ /home/hadoop/zookeeper/bin/zkServer.sh stop

Reference
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
http://wiki.apache.org/hadoop/Hbase/10Minutes
http://hbase.apache.org/book/quickstart.html
http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/

Author
Tzu-Cheng Chuang

Appendix- Install Pig and Hive
1. Install Pig 0.8.0 on this cluster
    (1) Download pig-0.8.0.tar.gz from Apache pig project website. Then extract the file and move it to /home/hadoop/
$ tar –xzf pig-0.8.0.tar.gz ; mv pig-0.8.0 /home/hadoop/
     (2) Make symbolink link under pig-0.8.0/conf/
$ ln -s /usr/local/hadoop/conf/core-site.xml /home/hadoop/pig-0.8.0/conf/core-site.xml
$ ln -s /usr/local/hadoop/conf/mapred-site.xml /home/hadoop/pig-0.8.0/conf/mapred-site.xml
$ ln -s /usr/local/hadoop/conf/hdfs-site.xml /home/hadoop/pig-0.8.0/conf/hdfs-site.xml
     3) Start pig in map-reduce mode: $ /home/hadoop/pig-0.8.0/bin/pig
     (4) Exit pig from grunt>    quit

2. Install Hive on this cluster
    (1) Download hive-0.6.0.tar.gz from Apache hive project website, and then extract the file and move it to /home/hadoop/    $ tar –xzf hive-0.6.0.tar.gz ; mv hive-0.6.0 ~/
     (2) Modify java heap size in hive-0.6.0/bin/ext/execHiveCmd.sh Change 4096 to 1024
    (3) Create /tmp and /user/hive/warehouse and set them chmod g+w in HDFS before a table can be created in Hive    $ hadoop fs –mkdir /tmp $ hadoop fs –mkdir /user/hive/warehouse $ hadoop fs –chmod g+w /tmp $ hadoop fs –chmod g+w /user/hive/warehouse
     (4) start Hive     $ /home/hadoop/hive-0.6.0/bin/hive

     3. (Optional)Load data by using Hive
    Create a file /home/hadoop/customer.txt    1, Kevin 2, David 3, Brian 4, Jane 5, Alice     After hive shell is started, type in    > CREATE TABLE IF NOT EXISTS customer(id INT, name STRING) > ROW FORMAT delimited fields terminated by ',' > STORED AS TEXTFILE; >LOAD DATA INPATH '/home/hadoop/customer.txt' OVERWRITE INTO TABLE customer; >SELECT customer.id, customer.name from customer;

http://chuangtc.info/ParallelComputing/SetUpHadoopClusterOnVmwareWorkstation.htm

paulwong 2013-08-17 22:23 发表评论

Kettle - HADOOP数据转换工具

paulwong — Thu, 01 Aug 2013 09:21:00 GMT
ETL�Q�Extract-Transform-Load的羃写，��x��据抽取、�{换、装载的�q�程�Q�，对于企业或行业应用来��_��我们�l�常会遇到各�U�数据的处理�Q��{换，�q�移�Q�所以了解�ƈ掌握一�U�etl工具的��用，必不可少�Q�这里我介绍一个我在工作中使用�?�q�左右的ETL工具Kettle,本着好东西不独��n的想法，跟大家分享碰撞交��一下！在��用中我感觉这个工��L��的很强大�Q�支持图形化的GUI设计界面�Q�然后可以以工作��的形式��{�Q�在做一些简单或复杂的数据抽取、质量检��、数据清�z�、数据�{换、数据过滤等斚w��有着比较�E�_��的表玎ͼ�其中最主要的我们通过熟练的应用它�Q�减��了非常多的研发工作量，提高了我们的工作效率�Q�不�q�对于我�q�个.net研发者来说唯一的遗憑ְ�是这个工��h��Java�~�写的�?br />
http://www.cnblogs.com/limengqiang/archive/2013/01/16/KettleApply1.html

paulwong 2013-08-01 17:21 发表评论

使用Sqoop实现HDFS与Mysql互�{

paulwong — Sat, 11 May 2013 13:27:00 GMT

��?br /> Sqoop是一个用来将Hadoop和关�p�d��数据库中的数据相互�{�Uȝ��工具�Q�可以将一个关�p�d��数据库（例如 �Q?MySQL ,Oracle ,Postgres�{�）中的数据导入到Hadoop的HDFS中，也可以将HDFS的数据导入到关系型数据库中�?br />
http://sqoop.apache.org/

环境
当调试过�E�出现IncompatibleClassChangeError一般都是版本兼定w��题�?br />
��Z��保证hadoop和sqoop版本的兼�Ҏ��，使用Cloudera�Q?br />
Cloudera��介：

Cloudera��Z��让Hadoop的配�|�标准化�Q�可以帮助企业安装，配置�Q�运行hadoop以达到大规模企业数据的处理和分析�?br />
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDHTarballs/3.25.2013/CDH4-Downloadable-Tarballs/CDH4-Downloadable-Tarballs.html

下蝲安装hadoop-0.20.2-cdh3u6�Q�sqoop-1.3.0-cdh3u6�?br />
安装
安装比较��单，直接解压卛_��

唯一需要做的就是将mysql的jdbc适配包mysql-connector-java-5.0.7-bin.jar copy�?SQOOP_HOME/lib下�?br />
配置好环境变量：/etc/profile

export SQOOP_HOME=/home/hadoop/sqoop-1.3.0-cdh3u6/

export PATH=$SQOOP_HOME/bin:$PATH

MYSQL转HDFS-�C�Z��
./sqoop import --connect jdbc:mysql://10.8.210.166:3306/recsys --username root --password root --table shop -m 1 --target-dir /user/recsys/input/shop/$today

HDFS转MYSQ-�C�Z��
./sqoop export --connect jdbc:mysql://10.8.210.166:3306/recsys --username root --password root --table shopassoc --fields-terminated-by ',' --export-dir /user/recsys/output/shop/$today

�C�Z��参数说明
(其他参数我未使用�Q�故不作解释�Q�未使用�Q�就没有发言权，详见命��ohelp)

参数�c�d��

参数�?br />
解释

公共

connect

Jdbc-url

公共

username

---

公共

password

---

公共

table

表名

Import

target-dir

制定输出hdfs目录�Q�默认输出到/user/$loginName/

export

fields-terminated-by

Hdfs文�g中的字段分割�W�，默认�?#8220;\t”

export

export-dir

hdfs文�g的�\�?img src ="http://www.aygfsteel.com/paulwong/aggbug/399153.html" width = "1" height = "1" />

paulwong 2013-05-11 21:27 发表评论

hadoop集群监控工具ambari安装

paulwong — Fri, 03 May 2013 05:55:00 GMT

　　Apache Ambari是对Hadoop�q�行监控、管理和生命周期��理的开源项目。它也是一个�ؓHortonworks数据�q�_��选择��理�l�徏的项目。Ambari向Hadoop MapReduce、HDFS�?HBase、Pig, Hive、HCatalog以及Zookeeper提供服务。最�q�准备装ambari�Q�在�|�上找了�怹��Q�没扑ֈ�比较�pȝ��的ambari安装�q�程�Q�于是，��根据官�|�进行了安装�Q�下面是我推荐的正确的较完善的安装方式，希望对大家有所帮助�?/span>
　　一、准备工�?/span>
　　1、系�l�：我的�pȝ��是CentOS6.2�Q�x86_64�Q�本�ơ集��采用两个节炏V��管理节点：192.168.10.121�Q�客��L��节点�Q?92.168.10.122
　　2、系�l�最好配�|�能上网�Q�这��h��便后面的操作�Q�否则需要配�|�yum仓库�Q�比较麻烦�?/span>
　　3、集��中ambari-serveer�Q�管理节点）到客��L��配置无密码登录�?/span>
　　4、集��同步时�?/span>
　　5、SELinux�Q�iptables都处于关闭状态�?/span>
　　6、ambari版本�Q?.2.0
　　二、安装步�?/span>
　　A、配�|�好集群环境
############  配置无密码登�?nbsp; #################
[root@ccloud121 ~]# ssh-keygen -t dsa
[root@ccloud121 ~]# cat /root/.ssh/id_dsa.pub >> /root/.ssh/authorized_keys
[root@ccloud121 ~]# scp /root/.ssh/id_dsa.pub 192.168.10.122:/root/
[root@ccloud121 ~]# ssh 192.168.10.122
[root@ccloud122 ~]# cat /root/.ssh/id_dsa.pub >> /root/.ssh/authorized_keys

#############  NTP 旉��同步  #################
[root@ccloud121 ~]# ntpdate time.windows.com
[root@ccloud121 ~]# ssh ccloud122 ntpdate time.windows.com

###########  SELinux & iptables 关闭   ###########
[root@ccloud121 ~]# setenforce 0
[root@ccloud121 ~]# ssh ccloud122 setenforce 0
[root@ccloud121 ~]# chkconfig iptables off
[root@ccloud121 ~]# service iptables stop
[root@ccloud121 ~]# ssh ccloud122 chkconfig iptables off
[root@ccloud121 ~]# ssh ccloud122 service iptables stop
　　B、管理节点上安装ambari-server
　　　　1、下载repo文�g　　　　　
[root@ccloud121 ~]# wget http://public-repo-1.hortonworks.com/AMBARI-1.x/repos/centos6/ambari.repo

[root@ccloud121 ~]# cp ambari.repo /etc/yum.repos.d
　　　　�q�样�Q�ambari-server的yum仓库��做好了�?/span>
　　　　2、安装epel仓库
[root@ccloud121 ~]# yum install epel-release   # 查看仓库列表�Q�应该有HDP�Q�EPEL [root@ccloud121 ~]# yum repolist
　　　　3、通过yum安装amabari bits�Q�这同时也会安装PostgreSQL
[root@ccloud121 ~]# yum install ambari-server
　　　　　�q�个步骤要等一会，它需要上�|�下载，大概39M左右�?/span>
　　　　4、运行ambari-server setup�Q�安装ambari-server�Q�它会自动安装配�|�PostgreSQL�Q�同时要求输入用户名和密码，如果按n�Q�它用默认的用户�?密码��|��ambari-server/bigdata。接着��开始下载安装JDK。安装完成后�Q�ambari-server��可以启动了�?/span>
　　三、集��启�?/span>
　　　　
　　　　1、直接接通过ambari-server start和amabari-server stop卛_��启动和关闭ambari-serveer�?/span>
　　　　2、启动成功后�Q�在��览器输入http://192.168.10.121:8080
　　　　界面如下图所�C�：
　　　　
��d��名和密码都是admin�?/span>
�q�样��可以登录到��理控制台�?/span>

paulwong 2013-05-03 13:55 发表评论

paulwong — Fri, 03 May 2013 01:05:00 GMT

下面��介�l�大数据领域支持Java的主��开源工�?/strong>�Q?/p>
1. HDFS
HDFS是Hadoop应用�E�序中主要的分布式储存系�l�， HDFS集群包含了一个NameNode�Q�主节点�Q�，�q�个节点负责��理所有文件系�l�的元数据及存储了真实数据的DataNode�Q�数据节点，可以有很多）。HDFS针对��量数据所设计�Q�所以相比传�l�文件系�l�在大批量小文�g上的优化�Q�HDFS优化的则是对��批量大型文件的讉K��和存储�?/p>
2. MapReduce
Hadoop MapReduce是一个��Y件框�Ӟ��用以��L��~�写处理��量�Q�TB�U�）数据的�ƈ行应用程序，以可靠和定w��的方式连�?span style="line-height: 1.45em;">大型集群�?/span>上万个节点（商用��g�Q��?/span>
3. HBase
Apache HBase是Hadoop数据库，一个分布式、可扩展的大数据存储。它提供了大数据集上随机和实时的�?写访问，�q��对了商用服务器集��上的大型表格做��Z��?#8212;—上百亿行�Q�上千万列。其核心是Google Bigtable论文的开源实玎ͼ�分布式列式存储。就像Bigtable利用GFS�Q�Google File System�Q�提供的分布式数据存储一��P��它是Apache Hadoop在HDFS基础上提供的一个类Bigatable�?/p>
4. Cassandra
Apache Cassandra是一个高性能、可�U�性扩展、高有效性数据库�Q�可以运行在商用��g或云基础设施上打造完��的��d��关键性数据��^台。在横跨数据中心的复制中�Q�Cassandra同类最佻I��为用��h��供更低的延时以及更可靠的��N��备䆾。通过log-structured update、反规范化和物化视图的强支持以及强大的内�|�缓存，Cassandra的数据模型提供了方便的二�U�烦引（column indexe�Q��?/p>
5. Hive
Apache Hive是Hadoop的一个数据仓库系�l�，促进了数据的�l�D��Q�将�l�构化的数据文�g映射��Z��张数据库表）、即席查询以及存储在Hadoop兼容�pȝ��中的大型数据集分析。Hive提供完整的SQL查询功能——HiveQL语言�Q�同时当使用�q�个语言表达一�?span style="line-height: 1.45em;">逻辑变得低效和繁�?/span>�Ӟ��HiveQL�q�允�怼��l�的Map/Reduce�E�序员��用自己定制的Mapper和Reducer�?/span>
6. Pig
Apache Pig是一个用于大型数据集分析的��^収ͼ�它包含了一个用于数据分析应用的高��语言以及评估�q�些应用的基��设施。Pig应用的闪光特性在于它们的�l�构�l�得起大量的�q�行�Q�也��是说让它们支撑起非常大的数据集。Pig的基��设施层包含了产生Map-Reduce��d��的编译器。Pig的语�a�层当前包含了一个原生语�a�——Pig Latin�Q�开发的初衷是易于编�E�和保证可扩展性�?/p>
7. Chukwa
Apache Chukwa是个开源的数据攉��pȝ��Q�用以监视大型分布系�l�。徏立于HDFS和Map/Reduce框架之上�Q��承了Hadoop的可扩展性和�E�_��性。Chukwa同样包含了一个灵�z�d��强大的工具包�Q�用以显�C�、监视和分析�l�果�Q�以保证数据的��用达到最��x��果�?/p>
8. Ambari
Apache Ambari是一个基于web的工��P��用于配置、管理和监视Apache Hadoop集群�Q�支持Hadoop HDFS,、Hadoop MapReduce、Hive、HCatalog,、HBase、ZooKeeper、Oozie、Pig和Sqoop。Ambari同样�q�提供了集群状况仪表盘，比如heatmaps和查看MapReduce、Pig、Hive应用�E�序的能力，以友好的用户界面对它们的性能�Ҏ��进行诊断�?/p>
9. ZooKeeper
Apache ZooKeeper是一个针对大型分布式�pȝ��的可靠协调系�l�，提供的功能包括：配置�l�护、命名服务、分布式同步、组服务�{�。ZooKeeper的目标就是封装好复杂易出错的关键服务�Q�将��单易用的接口和性能高效、功能稳定的�pȝ��提供�l�用戗��?/p>
10. Sqoop
Sqoop是一个用来将Hadoop和关�p�d��数据库中的数据相互�{�Uȝ��工具�Q�可以将一个关�p�d��数据库中数据导入Hadoop的HDFS中，也可以将HDFS中数据导入关�p�d��数据库中�?/p>
11. Oozie
Apache Oozie是一个可扩展、可靠及可扩充的工作��调度系�l�，用以��理Hadoop作业。Oozie Workflow作业是活动的Directed Acyclical Graphs�Q�DAGs�Q�。Oozie Coordinator作业是由周期性的Oozie Workflow作业触发�Q�周期一般决定于旉��Q�频率）和数据可用性。Oozie与余下的Hadoop堆栈�l�合使用�Q�开��即用的支持多种�c�d��Hadoop作业�Q�比如：Java map-reduce、Streaming map-reduce、Pig�?Hive、Sqoop和Distcp�Q�以及其它系�l�作业（比如Java�E�序和Shell脚本�Q��?/p>
12. Mahout
Apache Mahout是个可扩展的机器学习和数据挖掘库�Q�当前Mahout支持主要�?个用例：
推荐挖掘�Q�搜集用户动作�ƈ以此�l�用��h��荐可能喜�Ƣ的事物�?/span>
聚集�Q�收集文件�ƈ�q�行相关文�g分组�?/span>
分类�Q�从现有的分�c�L��档中学习�Q�寻找文档中的相似特征，�q��ؓ无标�{��文档�q�行正确的归�c�R�?/span>
频繁��w��挖掘�Q�将一�l�项分组�Q��ƈ识别哪些个别��会�l�常一起出现�?/span>
13. HCatalog
Apache HCatalog是Hadoop建立数据的映��表和存储管理服务，它包括：
提供一个共享模式和数据�c�d��机制�?/span>
提供一个抽象表�Q�这��L��户就不需要关注数据存储的方式和地址�?/span>
为类似Pig、MapReduce及Hive�q�些数据处理工具提供互操作性�?/span>

paulwong 2013-05-03 09:05 发表评论

paulwong — Tue, 30 Apr 2013 16:02:00 GMT
Centos集群服务器，公网ip
服务器地址
master�Q?mypetsbj.xicp.net:13283
slave1 �Q?mypetsbj.xicp.net:13282
slave2 �Q?mypetsbj.xicp.net:13286

http://mypetsbj.xicp.net:13296

http://mypetsbj.xicp.net:13304

http://mypetsbj.xicp.net:14113

http://mypetsbj.xicp.net:11103

服务器开机时�?br />08:00 �?23:59

opt/hadoop

用户�?密码
hadoop/wzp

paulwong 2013-05-01 00:02 发表评论

一个PIG脚本例子分析

paulwong — Sat, 13 Apr 2013 07:21:00 GMT
执行脚本�Q?br />
PIGGYBANK_PATH=$PIG_HOME/contrib/piggybank/java/piggybank.jar
INPUT=pig/input/test-pig-full.txt
OUTPUT=pig/output/test-pig-output-$(date  +%Y%m%d%H%M%S)
PIGSCRIPT=analyst_status_logs.pig

#analyst_500_404_month.pig
#analyst_500_404_day.pig
#analyst_404_percentage.pig
#analyst_500_percentage.pig
#analyst_unique_path.pig
#analyst_user_logs.pig
#analyst_status_logs.pig

pig -p PIGGYBANK_PATH=$PIGGYBANK_PATH -p INPUT=$INPUT -p OUTPUT=$OUTPUT $PIGSCRIPT

要分析的数据源，LOG 文�g
46.20.45.18 - - [25/Dec/2012:23:00:25 +0100] "GET / HTTP/1.0" 302 - "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 46.20.45.18 "" 11011AEC9542DB0983093A100E8733F8 0
46.20.45.18 - - [25/Dec/2012:23:00:25 +0100] "GET /sign-in.jspx HTTP/1.0" 200 3926 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 46.20.45.18 "" 11011AEC9542DB0983093A100E8733F8 0
69.59.28.19 - - [25/Dec/2012:23:01:25 +0100] "GET / HTTP/1.0" 302 - "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 69.59.28.19 "" 36D80DE7FE52A2D89A8F53A012307B0A 15

PIG脚本�Q?br />
--注册JAR包，因�ؓ要用到DateExtractor
register '$PIGGYBANK_PATH';

--声明一个短函数�?br />DEFINE DATE_EXTRACT_MM
org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM');

DEFINE DATE_EXTRACT_DD
org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM-dd');

-- pig/input/test-pig-full.txt
--把数据从变量所指的文�g加蝲到PIG中，�q�定义数据列名，此时的数据集为数�l?a,b,c)
raw_logs = load '$INPUT' USING org.apache.pig.piggybank.storage.MyRegExLoader('^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(\\S+) (\\S+) (HTTP[^"]+)" (\\S+) (\\S+) "([^"]*)" "([^"]*)" "(\\S+)" "(\\S+)" (\\S+) "(.*)" (\\S+) (\\S+)')
as (remoteAddr: chararray,
n2: chararray,
n3: chararray,
time: chararray,
method: chararray,
path:chararray,
protocol:chararray,
status: int,
bytes_string: chararray,
referrer: chararray,
browser: chararray,
n10:chararray,
remoteLogname: chararray,
remoteAddr12: chararray,
path2: chararray,
sessionid: chararray,
n15: chararray
);

--�q��o数据
filter_logs = FILTER raw_logs BY not (browser matches '.*pingdom.*');
--item_logs = FOREACH raw_logs GENERATE browser;

--percent 500 logs
--重定义数据项�Q�数据集只取2��status,month
reitem_percent_500_logs = FOREACH filter_logs GENERATE status,DATE_EXTRACT_MM(time) as month;
--分组数据集，此时的数据结构�ؓMAP(a{(aa,bb,cc),(dd,ee,ff)},b{(bb,cc,dd),(ff,gg,hh)})
group_month_percent_500_logs = GROUP reitem_percent_500_logs BY (month);
--重定义分�l�数据集数据��，�q�行分组�l�计�Q�此时要联合分组数据集和原数据集�l�计
final_month_500_logs = FOREACH group_month_percent_500_logs
{
    --对原数据集做count�Q�因为是在foreachj里做count的，即��是对原数据集�Q�也会自动会加month==group的条�?br />    --从这里可以看出对于group里的数据集，完全没用�?br />    --�q�时是以每一行�ؓ单位的，�l�计MAP中的KEY-a对应的数�l�在原数据集中的个数
    total = COUNT(reitem_percent_500_logs);
    --对原数据集做filter�Q�因为是在foreachj里做count的，即��是对原数据集�Q�也会自动会加month==group的条�?br />    --重新�q��o一下原数据集，得到status==500,month==group的数据集
    t = filter reitem_percent_500_logs by status== 500; --create a bag which contains only T values
    --重定义数据项�Q�取group�Q�统计结�?br />    generate flatten(group) as col1, 100*(double)COUNT(t)/(double)total;
}
STORE final_month_500_logs into '$OUTPUT' using PigStorage(',');

paulwong 2013-04-13 15:21 发表评论

paulwong — Wed, 10 Apr 2013 07:32:00 GMT
http://wiki.apache.org/pig/ParameterSubstitution

%pig -param input=/user/paul/sample.txt -param output=/user/paul/output/

PIG中获�?br />
records = LOAD $input;

paulwong 2013-04-10 15:32 发表评论

paulwong — Wed, 10 Apr 2013 06:13:00 GMT
http://stackoverflow.com/questions/15318785/pig-calculating-percentage-of-total-for-a-field

http://stackoverflow.com/questions/13476642/calculating-percentage-in-a-pig-query

paulwong 2013-04-10 14:13 发表评论

PIG��议

paulwong — Fri, 05 Apr 2013 13:33:00 GMT

什么是PIG
是一�U�设计语�a��Q�通过设计数据怎么��动�Q�然后由相应的引擎将此变成MAPREDUCE JOB去HADOOP中运行�?/div>
PIG与SQL
两者有相同之处�Q�执行一个或多个语句�Q�然后出来一些结果�?/div>
但不同的是，SQL要先把数据导到表中才能执行，SQL不关心中间如何做�Q�即发一个SQL语句�q�去�Q�就有结果出来�?/div>
PIG�Q�无��d��数据到表中，但要设计直到出结果的中间�q�程�Q�步骤如何等�{��?/div>

paulwong 2013-04-05 21:33 发表评论

PIG资源

paulwong — Fri, 05 Apr 2013 10:19:00 GMT
Hadoop Pig学习�W�记(一) 各种SQL在PIG中实�?br /> http://guoyunsky.iteye.com/blog/1317084

http://guoyunsky.iteye.com/category/196632

Hadoop学习�W�记(9) Pig��?br /> http://www.distream.org/?p=385

[hadoop�p�d��]Pig的安装和��单示�?br /> http://blog.csdn.net/inkfish/article/details/5205999

Hadoop and Pig for Large-Scale Web Log Analysis
http://www.devx.com/Java/Article/48063

Pig实战
http://www.cnblogs.com/xuqiang/archive/2011/06/06/2073601.html

[原创]Apache Pig中文教程�Q�进�Ӟ��
http://www.codelast.com/?p=4249

��Z��hadoop�q�_��的pig语言对apache日志�pȝ��的分�?br /> http://goodluck-wgw.iteye.com/blog/1107503

!!Pig语言
http://hi.baidu.com/cpuramdisk/item/a2980b78caacfa3d71442318

Embedding Pig In Java Programs
http://wiki.apache.org/pig/EmbeddedPig

一个pig事例(REGEX_EXTRACT_ALL, DBStorage�Q�结果存�q�数据库)
http://www.myexception.cn/database/1256233.html

Programming Pig
http://ofps.oreilly.com/titles/9781449302641/index.html

[原创]Apache Pig的一些基��概念及用法�ȝ��Q?�Q?br /> http://www.codelast.com/?p=3621

!PIG手册
http://pig.apache.org/docs/r0.11.1/func.html#built-in-functions

paulwong 2013-04-05 18:19 发表评论

paulwong — Sat, 16 Mar 2013 15:04:00 GMT
在新节点安装好hadoop

把namenode的有关配�|�文件复制到该节�?br />

修改masters和slaves文�g,增加该节�?br />

讄��ssh免密码进��节点

单独启动该节点上的datanode和tasktracker(hadoop-daemon.sh start datanode/tasktracker)

�q�行start-balancer.sh�q�行数据负蝲均衡

负蝲均衡:作用:当节点出现故�?或新增加节点�?数据块分布可能不均匀,负蝲均衡可以重新�q��各个datanode上数据块的分�?img src ="http://www.aygfsteel.com/paulwong/aggbug/396544.html" width = "1" height = "1" />

paulwong 2013-03-16 23:04 发表评论

HBASE��M��W�记-基础功能

paulwong — Wed, 06 Feb 2013 01:53:00 GMT

HBASE的SHELL命��o使用

HBASE的JAVA CLIENT的��?br />
新增和修改记录用PUT�?br />
PUT的执行流�E�：
首先会在内存中增加MEMSTORE�Q�如果这个表有N个COLOUMN FAMILY�Q�则会��生N个MEMSTORE�Q�记录中的值属于不同的COLOUMN FAMILY的，会保存到不同的MEMSTORE中。MEMSTORE中的��g��会马上FLUSH到文件中�Q�而是到MEMSTORE满的时候再FLUSH�Q�且FLUSH的时候不会写入已存在的HFILE中，而是新增一个HFILE��M��存。另外会写WRITE AHEAD LOG�Q�这是由于新增记录时不是马上写入HFILE的，如果中途出现DOWN机时�Q�则HBASE重启时会�Ҏ��q�个LOG来恢复数据�?br />
删除记录用DELETE�?br />
删除时�ƈ不会��在HFILE中的内容删除�Q�而是作一标记�Q�然后在查询的时候可以不取这些记录�?br />
��d��单条记录用GET�?br />
��d��的时候会��记录保存到CAHE中，同样如果�q�个表有N个COLOUMN FAMILY�Q�则会��生N个CAHE
�Q�记录中的值属于不同的COLOUMN FAMILY的，会保存到不同的CAHE中。这样下�ơ客��L��再取记录时会�l�合CAHE和MEMSTORE来返回数据�?br />
新增表用HADMIN�?br />
查询多条记录用SCAN和FILTER�?br />

HBASE的分布式计算

��Z��么会有分布式计算
前面的API是针对ONLINE的应用，卌��求低延时的，相当于OLTP。而针对大量数据时�q�些API��׃��适用了�?br />如要针对全表数据�q�行分析时用SCAN�Q�这样会��全表数据取回本圎ͼ�如果数据量在100G时会耗几个小�Ӟ��Z��节省旉��Q�引入多�U�程做法�Q�但要引入多�U�程�Ӟ��需遵从新算法：��全表数据分成N个段�Q�每�D는�一个线�E�处理，处理完后�Q�交�l�果合成�Q�然后进行分析�?br />
如果数据量在200G或以上时间就加倍了�Q�多�U�程的方式不能满��了�Q�因此引入多�q�程方式�Q�即��计��放在不同的物理��Z��处理�Q�这时就要考虑每个物理机DOWN机时的处理方式等情况了，HADOOP的MAPREDUCE则是�q�种分布式计��的框架了，对于应用者而言�Q�只��d��理分散和聚合的算法，其他的无��考虑�?br />
HBASE的MAPREDUCE
使用TABLEMAP和TABLEREDUCE�?br />
HBASE的部�|�架构和�l�成的组�?br />架构在HADOOP和ZOOPKEEPER之上�?br />
HBASE的查询记录和保存记录的流�E?br />说见前一�~�博文�?br />
HBASE作�ؓ数据来源地、保存地和共享数据源的处理方�?br />即相当于数据库中JOIN的算法：REDUCE SIDE JOIN、MAP SIDE JOIN�?br />

paulwong 2013-02-06 09:53 发表评论

监控HBASE

paulwong — Mon, 04 Feb 2013 07:08:00 GMT
@import url(http://www.aygfsteel.com/CuteSoft_Client/CuteEditor/Load.ashx?type=style&file=SyntaxHighlighter.css);@import url(/css/cuteeditor.css);
Hadoop/Hbase是开源版的google Bigtable, GFS, MapReduce的实玎ͼ�随着互联�|�的发展�Q�大数据的处理显得越发重要，Hadoop/Hbase的用武之��C��发�q�泛。�ؓ了更好的使用Hadoop/Hbase�pȝ��Q�需要有一套完善的监控�pȝ��Q�来了解�pȝ��q�行的实时状态，做到一切尽在掌握。Hadoop/Hbase有自己非常完善的metrics framework, 里面包种各种�l�度的系�l�指标的�l�计�Q�另外，�q�套metrics framework设计的也非常不错�Q�用户可以很方便地添加自定义的metrics。更为重要的一�Ҏ��metrics的展�C�方式，目前它支持三�U�方式：一�U�是落地到本地文�Ӟ��一�U�是report�l�Ganglia�pȝ��Q�另一�U�是通过JMX来展�C�。本文主要介�l�怎么把Hadoop/Hbase的metrics report�l�Ganglia�pȝ��Q�通过��览器来查看�?br />
介绍后面的内容之前有必要先简单介�l�一下Ganglia�pȝ��。Ganglia是一个开源的用于�pȝ��监控的系�l�，它由三部分组成：gmond, gmetad, webfrontend, 三部分是�q�样分工的：

gmond: 是一个守护进�E�，�q�行在每一个需要监��的节点上，攉��监测�l�计�Q�发送和接受在同一个组播或单播通道上的�l�计信息
gmetad: 是一个守护进�E�，定期��查gmond�Q�从那里拉取数据�Q��ƈ��他们的指标存储在RRD存储引擎�?br /> webfrontend: 安装在有gmetad�q�行的机器上�Q�以便读取RRD文�g�Q�用来做前台展示

��单�ȝ��它们三者的各自的功用，gmond攉��数据各个node上的metrics数据�Q�gmetad汇总gmond攉��到的数据�Q�webfrontend在前台展�C�gmetad汇�ȝ��数据。Ganglia�~�省是对�pȝ��的一些metric�q�行监控�Q�比如cpu/memory/net�{�。不�q�Hadoop/Hbase内部做了对Ganglia的支持，只需要简单的攚w��|�就可以��Hadoop/Hbase的metrics也接入到ganglia�pȝ��中进行监控�?br />
接下来介�l�如何把Hadoop/Hbase接入到Ganglia�pȝ��Q�这里的Hadoop/Hbase的版本号�?.94.2�Q�早期的版本可能会有一些不同，��h��意区别。Hbase本来是Hadoop下面的子��目�Q�因此所用的metrics framework原本是同一套Hadoop metrics�Q�但后面hadoop有了改进版本的metrics framework:metrics2(metrics version 2), Hadoop下面的项目都已经开始��用metrics2, 而Hbase成了Apache的顶�U�子��目�Q�和Hadoop成�ؓ�q��的项目后�Q�目前还没跟�q�metrics2�Q�它用的�q�是原始的metrics.因此�q�里需要把Hadoop和Hbase的metrics分开介绍�?br />
Hadoop接入Ganglia:

1. Hadoop metrics2对应的配�|�文件�ؓ�Q�hadoop-metrics2.properties
2. hadoop metrics2中引用了source和sink的概念，source是用来收集数据的, sink是用来把source攉��的数据consume的（包括落地文�g�Q�上报ganglia�Q�JMX�{�）
3. hadoop metrics2配置支持Ganglia:

#*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30
*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

*.sink.ganglia.period=10
*.sink.ganglia.supportsparse=true
*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40

#uncomment as your needs
namenode.sink.ganglia.servers=10.235.6.156:8649
#datanode.sink.ganglia.servers=10.235.6.156:8649
#jobtracker.sink.ganglia.servers=10.0.3.99:8649
#tasktracker.sink.ganglia.servers=10.0.3.99:8649
#maptask.sink.ganglia.servers=10.0.3.99:8649
#reducetask.sink.ganglia.servers=10.0.3.99:8649

�q�里需要注意的几点�Q?br />
(1) 因�ؓGanglia3.1�?.0不兼容，需要根据Ganglia的版本选择使用GangliaSink30或者GangliaSink31
(2) period配置上报周期�Q�单位是�U?s)
(3) namenode.sink.ganglia.servers指定Ganglia gmetad所在的host:port�Q�用来向其上报数�?br /> (4) 如果同一个物理机器上同时启动了多个hadoop�q�程(namenode/datanode, etc)�Q�根据需要把相应的进�E�的sink.ganglia.servers配置好即�?br /> Hbase接入Ganglia:

1. Hbase所用的hadoop metrics对应的配�|�文件是: hadoop-metrics.properties
2. hadoop metrics里核心是Context�Q�写文�g有写文�g的TimeStampingFileContext, 向Ganglia上报有GangliaContext/GangliaContext31
3. hadoop metrics配置支持Ganglia:

# Configuration of the "hbase" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
hbase.period=10
hbase.servers=10.235.6.156:8649

�q�里需要注意几点：

(1) 因�ؓGanglia3.1�?.0不兼容，所以如果是3.1以前的版本，需要用GangliaContext, 如果�?.1版的Ganglia�Q�需要用GangliaContext31
(2) period的单位是�U?s)�Q�通过period可以配置向Ganglia上报数据的周�?br /> (3) servers指定的是Ganglia gmetad所在的host:port�Q�把数据上报到指定的gmetad
(4) 对rpc和jvm相关的指标都可以�q�行�c�M��的配�|?/div>

paulwong 2013-02-04 15:08 发表评论

HBASE部��v要点

paulwong — Mon, 04 Feb 2013 04:10:00 GMT

REGIONS SERVER和TASK TRACKER SERVER不要在同一台机器上�Q�最好如果有MAPREDUCE JOB�q�行的话�Q�应该分开两个CLUSTER�Q�即两群不同的服务器上，�q�样MAPREDUCE 的线下负载不会媄响到SCANER�q�些�U�上负蝲�?/div>

如果主要是做MAPREDUCE JOB的话�Q�将REGIONS SERVER和TASK TRACKER SERVER攑֜�一��h��可以的�?/div>

原始集群模式

10个或以下节点�Q�无MAPREDUCE JOB�Q�主要用于低延迟的访问。每个节点上的配�|��ؓ�Q�CPU4-6CORE�Q�内�?4-32G�Q?个SATA��盘。Hadoop NameNode, JobTracker, HBase Master, 和ZooKeeper全都在同一个NODE上�?

��型集群模式�Q?0-20台服务器�Q?/span>

HBase Master攑֜�单独一台机器上, 以便于��用较低配�|�的机器。ZooKeeper也放在单独一台机器上�Q�NameNode和JobTracker攑֜�同一台机器上�?/div>

中型集群模式�Q?0-50台服务器�Q?/span>

�׃��无须再节省费用，可以��HBase Master和ZooKeeper攑֜�同一台机器上, ZooKeeper和HBase Master要三个实例。NameNode和JobTracker攑֜�同一台机器上�?/div>

大型集群模式�Q?gt;50台服务器�Q?/span>

和中型集��模式相��|��但ZooKeeper和HBase Master要五个实例。NameNode和Second NameNode要有��_��大的内存�?/div>

HADOOP MASTER节点

NameNode和Second NameNode服务器配�|�要求：�Q�小型）8CORE CPU�Q?6G内存�Q?G�|�卡和SATA ��盘�Q�中弄再增加�?6G内存�Q�大型则再增加多32G内存�?/div>

HBASE MASTER节点

服务器配�|�要求：4CORE CPU�Q?-16G内存�Q?G�|�卡�?个SATA ��盘�Q�一个用于操作系�l�，另一个用于HBASE MASTER LOGS�?/div>

HADOOP DATA NODES和HBASE REGION SERVER节点

DATA NODE和REGION SERVER应在同一台服务器上，且不应该和TASK TRACKER在一赗��服务器配置要求�Q?-12CORE CPU�Q?4-32G内存�Q?G�|�卡�?2*1TB SATA ��盘�Q�一个用于操作系�l�，另一个用于HBASE MASTER LOGS�?/div>

ZOOPKEEPERS节点

服务器配�|�和HBASE MASTER�怼��Q�也可以与HBASE MASTER攑֜�一��P��但就要多增加一个硬盘单独给ZOOPKEEPER使用�?/div>

安装各节�?/span>

JVM配置�Q?/div> -Xmx8g—讄��HEAP的最大值到8G�Q�不��讑ֈ�15 GB.
-Xms8g—讄��HEAP的最��值到8GS.
-Xmn128m—讄��新生代的值到128 MB�Q�默认值太��?br /> -XX:+UseParNewGC—讄��对于新生代的垃圾回收器类型，�q�种�c�d��是会停止JAVA�q�程�Q�然后再�q�行回收的，但由于新生代体积比较��，持箋旉��通常只有几毫�U�，因此可以接受�?br /> -XX:+UseConcMarkSweepGC—讄��老生代的垃圾回收�c�d��Q�如果用新生代的那个会不合适，即会��D��JAVA�q�程停止的时间太长，用这�U�不会停止JAVA�q�程�Q�而是在JAVA�q�程�q�行的同�Ӟ��q�行的进行回收�?br /> -XX:CMSInitiatingOccupancyFraction—讄��CMS回收器运行的频率�?br />

paulwong 2013-02-04 12:10 发表评论

HBASE��M��W�记

paulwong — Fri, 01 Feb 2013 05:55:00 GMT

GET、PUT是ONLINE的操作，MAPREDUCE是OFFLINE的操�?/div>

HDFS写流�E?/span>

客户端收到要保存文�g的请求后�Q�将文�g�?4M为单位拆成若�q�䆾BLOCK�Q��Ş成一个列表，即由几个BLOCK�l�成�Q�将�q�些信息告诉NAME NODE�Q�我要保存这个，NAME NODE��出一个列表，哪段BLOCK应该写到哪个DATA NODE�Q�客��L��第一个BLOCK传到�W�一个节点DATA NODE A�Q�通知其保存，同时让它通知DATA NODE D和DATA NODE B也保存一份，DATA NODE D收到信息后进行了保存�Q�同旉��知DATA NODE B保存一份，DATA NODE B保存完成后则通知客户端保存完成，客户端再��d��NAME NODE中取下一个BLOCK要保存的位置�Q�重复以上的动作�Q�直到所有的BLOCK都保存完成�?/div>

HDFS��L��E?/span>

客户端向NAME NODE��h��M��个文�Ӟ��NAME NODE�q�回�q�个文�g所构成的所有BLOCK的DATA NODE IP及BLOCK ID�Q�客��L��q�行的向各DATA NODE发出��h��Q�要取某个BLOCK ID的BLOCK�Q�DATA NODE发回所要的BLOCK�l�客��L��Q�客��L��攉��到所有的BLOCK后，整合成一个完整的文�g后，此流�E�结束�?br />

MAPREDUCE��程

输入数据 -- 非多�U�程了，而是多进�E�的挑选数据，卛_��输入数据分成多块�Q�每个进�E�处理一�?-- 分组 -- 多进�E�的汇集数据 -- 输出

HBASE表结�?/span>

HBASE中将一个大表数据分成不同的��表�Q�每个小表叫REGION�Q�存放REGION的服务器叫REGIONSERVER�Q�一个REGIONSERVER可以存放多个REGION。通常REGIONSERVER和DATA NODE是在同一服务器，以减��NETWORK IO�?/div>

-ROOT-表存放于MASTER SERVER上，记录了一共有多少个REGIONSERVER�Q�每个REGION SERVER上都有一�?META.表，上面记录了本REGION SERVER放有哪几个表的哪几个REGION。如果要知道某个表共有几个REGION�Q�就得去所有的REGION SERVER上查.META.表，�q�行汇��L��能得知�?/div>

客户端如果要查ROW009的信息，先去咨询ZOOPKEEPER�Q?ROOT-表在哪里�Q�然后问-ROOT-表，哪个.META.知道�q�个信息�Q�然后去�?META.表，哪个REGION有这个信息，然后去那个REGION问ROW009的信息，然后那个REGION�q�回此信息�?br />

HBASE MAPREDUCE

一个REGION一个MAP��d��Q�而�Q务里的map�Ҏ��执行多少�ơ，则由查询出来的记录有多少条，则执行多��次�?/div>
REDUCE��d��负责向REGION写数据，但写到哪个REGION则由那个KEY归属哪个REGION��，则写到哪个REGION�Q�有可能REDUCE��d��会和所有的REGION SERVER交互�?br />

在HBASE的MAPREDUCE JOB中��用JOIN

REDUCE-SIDE JOIN
利用现有的SHUTTLE分组机制�Q�在REDUCE阶段做JOIN�Q�但�׃��MAP阶段数据大，可能会有性能问题�?/div>
MAP-SIDE JOIN

��数据较��的一表读��C��公共文�g中，然后在MPA�Ҏ��中��@环另一表的数据�Q�再��要的数据从公共文�g中读取。这样可以减��SHUTTLE和SORT的时��_��同时也不需要REDUCE��d��?/div>

paulwong 2013-02-01 13:55 发表评论

Hadoop的几�U�Join�Ҏ��

paulwong — Thu, 31 Jan 2013 10:24:00 GMT
1) 在Reduce阶段�q�行Join,�q�样�q�算量比较小.(�q�个适合被Join的数据比较小的情况下.)
2) 压羃字段,�Ҏ��据预处理,�q��o不需要的字段.
3) 最后一步就是在Mapper阶段�q��o,�q�个��是Bloom Filter的用武之��C��.也就是需要详�l�说明的地方.

下面��拿一个我们大安��熟悉的场景来说明�q�个问题: 扑և�上个月动感地带的客户资费的��用情�?包括接入和拨�?

(�q�个只是我臆惛_��来的例子,�Ҏ��实际的DB数据存储�l�构,在这个场景下肯定有更好的解决�Ҏ��,大家不要太较真哦)

�q�个时候的两个个数据集都是比较大的,�q�两个数据集分别�?上个月的通话记录,动感地带的手机号码列�?

比较直接的处理方法有2�U?

1)�?Reduce 阶段,通过动感地带��L��来过�?

优点:�q�样需要处理的数据相对比较��?�q�个也是比较常用的方�?

�~�点:很多数据在Mapper阶段�׃��老��E子力气汇��M��,�q�通过�|�络Shuffle到Reduce节点,�l�果到这个阶�D늻��q��o�?

2)�?Mapper 阶段�?通过动感地带��L��来过滤数�?

优点:�q�样可以�q��o很多不是动感地带的数�?比如��州�?全球�?�q�些�q��o的数据就可以节省很多�|�络带宽�?

�~�点:��是动感地带的号码不是小数目,如果�q�样处理��需要把�q�个大块头复制到所有的Mapper节点,甚至是Distributed Cache.(Bloom Filter��是用来解决�q�个问题�?

Bloom Filter��是用来解决上面�Ҏ��2的缺点的.

�Ҏ��2的缺点就是大量的数据需要在多个节点复制.Bloom Filter通过多个Hash��法, 把这个号码列表压�~�到了一个Bitmap里面. 通过允许一定的错误率来换空�? �q�个和我们��^时经常提到的旉��和空间的互换�c�M��.详细情况可以参�?

http://blog.csdn.net/jiaomeng/article/details/1495500

但是�q�个��法也是有缺��L��,��是会把很多��州�?全球通之�cȝ��L��当成动感地带.但在�q�个场景�?�q�根本不是问�?因�ؓ�q�个��法只是�q��o一些号�?漏网之鱼会在Reduce阶段�q�行�_��匚w��旉��虑掉.

�q�个�Ҏ��改进之后基本上完全回避了�Ҏ��2的缺�?

1) 没有大量的动感地带号码发送到所有的Mapper节点.
2) 很多非动感地带号码在Mapper阶段��p��滤了(虽然不是100%),避免了网�l�带宽的开销及�g�?

�l�箋需要学习的地方:Bitmap的大��? Hash函数的多��? 以及存储的数据的多少. �q?个变量如何取值才能才能在存储�I�间与错误率之间取得一个��^�?

paulwong 2013-01-31 18:24 发表评论

伊人成人在线视频,久久亚洲精品视频,日韩在线一区二区

HADOOP各种框架应用领域

�~�译HADOOP源码

Simplehbase

安装CLOUDERA

KMEANS PAGERANK ON HADOOP

Packt celebrates International Day Against DRM, May 6th 2014

A book: Web Crawling and Data Mining with Apache Nutch

【�{载】经典�O画讲解HDFS原理

Install Hadoop in the AWS cloud

Install hadoop+hbase+nutch+elasticsearch

Implementation for CombineFileInputFormat Hadoop 0.20.205

How to install Hadoop cluster(2 node cluster) and Hbase on Vmware Workstation. It also includes installing Pig and Hive in the appendix

Kettle - HADOOP数据转换工具

使用Sqoop实现HDFS与Mysql互�{

hadoop集群监控工具ambari安装

一、准备工�?/span>

二、安装步�?/span>

三、集���启�?/span>

一个PIG脚本例子分析

PIG���议

PIG资源

HBASE��M���W�记-基础功能

监控HBASE

HBASE部��v要点

HBASE��M���W�记

Hadoop的几�U�Join�Ҏ��

　　一、准备工�?/span>

　　二、安装步�?/span>

　　三、集��启�?/span>

PIG��议

HBASE��M��W�记-基础功能

HBASE��M��W�记