我的家園

reduce exceed 100%
"Reduce Task Progress shows > 100% when the total size of map outputs (for a
single reducer) is high "
造成原因：
在reduce的merge過程中，check progress有誤差，導致status > 100%，在統計過程中就會出現以下錯誤：java.lang.ArrayIndexOutOfBoundsException: 3
        at org.apache.hadoop.mapred.StatusHttpServer$TaskGraphServlet.getReduceAvarageProgresses(StatusHttpServer.java:228)
        at org.apache.hadoop.mapred.StatusHttpServer$TaskGraphServlet.doGet(StatusHttpServer.java:159)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)

jira地址：

counters
3中counters：
1. built-in counters: Map input bytes, Map output records...
2. enum counters
   調用方式：
  enum Temperature {
    MISSING,
    MALFORMED
  }

reporter.incrCounter(Temperature.MISSING, 1)
   結果顯示：
09/04/20 06:33:36 INFO mapred.JobClient:   Air Temperature Recor
09/04/20 06:33:36 INFO mapred.JobClient:     Malformed=3
09/04/20 06:33:36 INFO mapred.JobClient:     Missing=66136856
3. dynamic countes:
   調用方式：
   reporter.incrCounter("TemperatureQuality", parser.getQuality(),1);
   
   結果顯示：
09/04/20 06:33:36 INFO mapred.JobClient:   TemperatureQuality
09/04/20 06:33:36 INFO mapred.JobClient:     2=1246032
09/04/20 06:33:36 INFO mapred.JobClient:     1=973422173
09/04/20 06:33:36 INFO mapred.JobClient:     0=1

7: Namenode in safe mode 
解決方法
bin/hadoop dfsadmin -safemode leave

8:java.net.NoRouteToHostException: No route to host
j解決方法：
sudo /etc/init.d/iptables stop

9：更改namenode后，在hive中運行select 依舊指向之前的namenode地址
這是因為：When youcreate a table, hive actually stores the location of the table (e.g.
hdfs://ip:port/user/root/...) in the SDS and DBS tables in the metastore . So when I bring up a new cluster the master has a new IP, but hive's metastore is still pointing to the locations within the old
cluster. I could modify the metastore to update with the new IP everytime I bring up a cluster. But the easier and simpler solution was to just use an elastic IP for the master
所以要將metastore中的之前出現的namenode地址全部更換為現有的namenode地址