永久免费观看精品视频,国产麻豆综合,黄色一区二区在线

Rosen — Tue, 12 Jul 2016 13:21:00 GMT

Hadoop周刊 �W?/span> 176 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>6�?/span>29�?/span>

Hadoop��C��本周在圣何塞召开�Q�所以很期待在下期周刊看到新��目的发布和�_�ֽ�演讲�Q�请向我们提供�Q何相关的�qȝ��片）。至于本期周刊，有大量关�?/span>Kafka Streams、从Amazon Kinesis�?/span>Google BigQuery传递流式数据�?/span>Google数据集搜索系�l�的文章�?/span>

技术新�?/span>

Shine介绍了他们如何��?/span>Amazon Lambda�?/span>Amazon Kinesis�Q�以及�ؓApache web服务器提供的Kinesis代理�Q�用于采日志�Q?/span>�Q�以及从EC2�U�d��数据�?/span>Google BigQuery的内宏V��本文提供了Lambda函数�Q?/span>javascript�~�写�Q�代码片�D�，规模和开销斚w��的信息，描述了如何通过gzip压羃数据从而优化传输开销�?/span>

https://blog.shinetech.com/2016/06/21/kinesis-lambda-bigquery/

Cloudera博客撰文介绍了如何通过Apache Spark�?/span>Apache Impala�Q�孵化中�Q��?/span>Hue�Ҏ��之队数据�q�行分析。本文主要聚焦在分析上，附带了些Spark代码以及Hue的功能演�C��?/span>

http://blog.cloudera.com/blog/2016/06/how-to-analyze-fantasy-sports-with-apache-spark-and-sql-part-2-data-exploration/

KDnuggets撰文介绍�?/span>13个和Apache Spark相关的主�?/span>API/��目/名词。包�?/span>RDD�?/span>DataFrame�?/span>Dataset、结构化��式计算�?/span>GraphX�?/span>Tungsten。每个条目都有一�D늫�节介�l�，��_��很好的了�?/span>Spark主要�Ҏ��了�?/span>

http://www.kdnuggets.com/2016/06/spark-key-terms-explained.html

本文来自Confluent博客�Q�介�l�了那些虽看��h��单却又不��单的Kafka Streams应用。例如用Kafka Streams�~�写�l�合用户点击��数据和用户位置数据的程序。后者存储在KTable中，KTable提供了类似带有数据库表主键的抽象�Q�主键的最新值通过API暴露�Q�。最后的�E�序倒是��?/span>——只有几行代码�?/span>

http://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events-using-kafka-streams

Cloudera博客撰文介绍�?/span>meinstadt.de构徏�?/span>Apache Flume�?/span>Apache Spark Streaming�?/span>Apache Impala�Q�孵化中�Q�上�?/span>HTTP��h��异常��系�l�。实��C��码放在了github上�?/span>

http://blog.cloudera.com/blog/2016/06/how-to-detect-and-report-web-traffic-anomalies-in-near-real-time/

AWS大数据博客有教程介绍了如何��?/span>Apache Spark�?/span>Apache Zeppelin�?/span>Amazon EMR集群处理Amazon Kinesis��数据。本文包含了一些通过Zeppelin notebook�q�行SQL产生的数据可视化范例�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx3K805CZ8WFBRP/Analyze-Realtime-Data-from-Amazon-Kinesis-Streams-Using-Zeppelin-and-Spark-Strea

Apache Kudu�Q�孵化中�Q�接�q?/span>1.0版发布了�Q�将全面支持高可用性。本文介�l�了�q�最后一块拼�?/span>“��d��?/span>”是如何实现的。晒了下JIRA上各�U�问题的跟进的情况，以及完成与剩余的��试�?/span>

http://kudu.apache.org/2016/06/24/multi-master-1-0-0.html

Google的所有数据��^台拥有超�q?/span>260亿的数据集，每天要添加和删除16亿的数据集�\径。�ؓ了跟�t�、查询、比较数据集�Q�他们研发了Google Dataset Search�Q?/span>GOODS�Q��?/span>GOODS跟踪�?/span>API暴露的元数据�Q�这些元数据被用于检索、监控等�?/span>

http://dl.acm.org/citation.cfm?id=2903730

其他新闻

SiliconAngle采访�?/span>Hortonworks CEO Rob Bearden。主题包括业界趋�ѝ�?/span>Hortonworks财务�?/span>Hortonworks的非Hadoop技术以及物联网�?/span>

http://siliconangle.com/blog/2016/06/24/hadoop-and-beyond-a-conversation-with-hortonworks-ceo-rob-bearden/

产品发布

Apache Sentry本周发布�?/span>1.7.0版，修复�?/span>bug�Q�增加了新特性和其他斚w��的提升。本�ơ发布把Hive授权框架升��C��W�二版�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201606.mbox/%3CCAPOmu3sDqdzu9ntDSvkMaDRQnVfHrkGV5qhyh-ZRiMmwgMMvBA@mail.gmail.com%3E

��Z��Apache Cassandra 3.0构徏�?/span>DataStax Enterprise 5.0�Q�增加了对图数据、分层存储�?/span>Cassandra多实例的支持。本�ơ发布也增加了诸如加密和��Z��角色讉K��控制的附加安全特性支持�?/span>

https://www.datastax.com/2016/06/introducing-datastax-enterprise-5-0

Driven�Q�大数据应用性能监控�pȝ��发布�?/span>2.2版。本�ơ发布的亮点是对Apache Spark的监控提供了支持�?/span>

http://www.driven.io/2016/06/driven-inc-delivering-hadoop-spark-performance-monitoring-announces-driven-2-2/

BlueData发布了他们�ؓAmazon Web Services提供�?/span>EPIC企业大数据既服务产品。本产品通过��单的点击��p��自动装蝲到基�?/span>Docker�?/span>Hadoop集群�?/span>

http://www.bluedata.com/blog/2016/06/big-data-as-a-service-on-prem-or-cloud-bdaas/

Apache Accumulo发布�?/span>1.7.2版。本�ơ发布修复了write-ahead日志处理方式�Q�优化了RFiles�Q�以及性能上的��提升�?/span>

https://accumulo.apache.org/release_notes/1.7.2.html

Apache ZooKeeper的顶�U?/span>SDK�Q?/span>Apache Curator发布�?/span>2.11.0�?/span>3.2.0版�?/span>

https://cwiki.apache.org/confluence/display/CURATOR/Releases#Releases-June23,2016,Releases2.11.0and3.2.0available

Apache Hive发布�?/span>2.1.0版。修复了大量bug和功能增强，包括�?/span>Hive�?/span>Live Longer�?/span>Prosper 改进和以�?/span>JDBC支持�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201606.mbox/%3C7194557D-CB5E-45B7-B905-82F27B7CB33F@apache.org%3E

�z�d��

中国

7�?/span>2�?/span> 上�vBigData Streaming�W�三�ơ见面会

Rosen 2016-07-12 21:21 发表评论

Rosen — Fri, 01 Jul 2016 07:44:00 GMT

Hadoop周刊 �W?/span> 175 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>6�?/span>19�?/span>

Hadoop��C��已过��M��周了�Q�我们已看到有多个��品（��目�Q�敲定了发布旉��。所以在技术新闻部分，有关�?/span>Hadoop Kerberos认证的内容另外还�?/span>Salsify应用Avro的文章。在产品发布部分�Q�包�?/span>Yandex新近开源的列式数据库在内的多个��目均有新版本发布�?/span>

技术新�?/span>

OpenCore博客撰文�C��了多�U?/span>Hadoop Kerberos认证协议调试工具。尤其示范了如何使用UserGropuInformation�?/span>“main()”�Ҏ��导出一些有用的调试信息�?/span>

http://www.opencore.com/blog/2016/5/user-name-handling-in-hadoop/

YARN�p�d��文章的第四部分，Cloduera博客介绍了如何配�|�公�q��度队列。尤其对资源�U�束讄��、队列安�|�策略和抢占�q�行了详解�?/span>

http://blog.cloudera.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/

Salsify��Z��Apache Kafka构徏了一个异步微服务架构�Q��ƈ采用Apache Avro�q�行数据序列化。该应用使用Ruby开发，他们创徏了多个新工具使得Avro能和Ruby语言很好的配合。本文介�l�了�q�些工具和它们的价��|��avro-builder用于定义记录、基�?/span>postgres的模式注册表�Q?/span>avromatic则从avro schema生成模型�?/span>

http://blog.salsify.com/engineering/adventures-in-avro

Apache Drill可以动态推断模式，�q�支持多模式(但相互兼�?/span>)数据。这�U�组合��得一些有��的用例得以实现�Q�例如跨多个不同模式�?/span>json文�g查询�?/span>MapR博客探究了这些特性�ƈ�q�行了示范�?/span>

https://www.mapr.com/blog/sql-query-mixed-schema-data-using-apache-drill

本教�E�展�C�Z��如何��?/span>Druid�?/span>Apache Kafka�l�合构徏��式分析和可视化�Q�借助Pivot�Q?/span>Druid�?/span>web UI�Q�应用�?/span>

http://www.confluent.io/blog/building-a-streaming-analytics-stack-with-apache-kafka-and-druid

Apache Beam�Q�孵化中�Q�博客撰文介�l�了他们在连�?/span>Apache Flink批处理集��方面的成果�?/span>Beam是一个开�?/span>SDK�Q�最初来自于Google�Q�用于暴露后端未知数据管�?/span>API�?/span>

http://beam.incubator.apache.org/blog/2016/06/13/flink-batch-runner-milestone.html

Cask Hydrator是一个通过UI界面采用拖拽方式构徏数据��道的工兗��本教程也演�C�Z��如何使用Hydrator把数据从MySQL导入�?/span>HDFS�?/span>

http://blog.cask.co/2016/06/bringing-relational-data-into-data-lakes/

Databricks撰文介绍了即��发布的Apache Spark 2.0中新�?/span>SQL子查询功能。有��的是，本文以手册�Ş式呈玎ͼ�最直截了当的展��C��代码和范例数据�?/span>

https://databricks.com/blog/2016/06/17/sql-subqueries-in-apache-spark-2-0.html

Apache Kudu�Q�孵化中�Q�博客撰写了在单集群节点使用Raft的文章，借此动态扩展到多主节点集群�?/span>

http://getkudu.io/2016/06/17/raft-consensus-single-node.html

其他新闻

本文指出Apache Spark�C�֌�如果不用心经营，可能会重走因��片化导�?/span>Apache Hadoop生态系�l��؜��q��老�\。�D例来��_��最新版本的CDH�?/span>HDP支持不同版本�?/span>Spark�?/span>

https://techcrunch.com/2016/06/12/spark-fragmentation-undermines-community/

New Stack撰写了一��关�?/span>Concord的文章，Concord是一个构建在Apache Mesos上新的流式处理框�Ӟ��公开��试状态）�?/span>Concord使用C++开发，支持动态拓扑（无需停机实现��道的增加和减少�Q��?/span>

http://thenewstack.io/concord-leverages-mesos-high-performance-stream-processing/

随着Databricks�C�֌�版的正式发布�Q?/span>Databricks发布了��?/span>Databricks�~�写Apache Spark应用�E�序�p�d��教程的第一��?/span>

https://databricks.com/blog/2016/06/15/an-introduction-to-writing-apache-spark-applications-on-databricks.html

Hadoop圣何塞峰会于几周前召开�Q�期间�D行了题�ؓ“大数据行业中的女�?/span>”专场午宴�?/span>Hortonworks博客�Ҏ��采访了午宴主持�hHortonworks CMO�Q?/span>Ingrid Burton�?/span>

http://hortonworks.com/blog/summer-hortonworks-part-2-wibd-assertive-innovative-take-risks/

产品发布

Apache SystemML�Q�孵化中�Q�最�q�发布了0.10.0版�?/span>SystemML是一个机器学习框�Ӟ��由多个项目在背后支撑�Q�包�?/span>Apache Spark�?/span>Apache Hadoop。本�ơ发布包括新�?/span>Spark Matrix Block�c�d��、支持深度学习、性能上的提升、新�?/span>KNN��法�{�等�?/span>

http://systemml.apache.org/0.10.0-incubating/release_notes.html

Apache Mahout�Q�另一个机器学习框架发布了0.12.2版。本�ơ发布向着集成Apache Zeppelin可视化和支持notebook的目标迈�q�了一步�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201606.mbox/%3CCAOtpBjgBAuQs5FiX5X_5A+Rd-A1fVz0R7SKttGe4cJuCLRiGww@mail.gmail.com%3E

Qubole宣布他们�?/span>HBase-as-a-Service已经�?/span>AWS上提供。它为长时运行集��提供了许多漂亮的特性。支�?/span>Hannibal和其它监控工��P��集成�?/span>Apache Zeppelin�Q��ƈ能通过节点引导�E�序�?/span>OpenTSDB�?/span>Apache Phoenix配置�?/span>

https://www.qubole.com/blog/product/quboles-hbase-as-a-service-is-generally-available-on-aws/

Altiscale发布�?/span>Altiscale Insight Cloud实时版。本�pȝ��?/span>Apache HBase�?/span>Spark Streaming支撑�?/span>

https://www.altiscale.com/blog/announcing-the-altiscale-insight-cloud-real-time-edition/

`hs2client`是一个�ؓApache Hive�?/span>Apache Impala�Q�孵化中�Q�提供的�?/span>C++库。除了支�?/span>C++�Q�这个库�q�绑定了python�Q�可以在pandas中把数据��d��DataFrame�?/span>

http://blog.cloudera.com/blog/2016/06/announcing-hs2client-a-fast-new-c-python-thrift-client-for-impala-and-hive/

MapR在其发行版中支持�?/span>Apache Spark 2.0开发者预览版�?/span>

https://www.mapr.com/blog/spark-20-now-developer-preview-mode-mapr-platform

Apache Beam发布了其0.1.0孵化版，是本��目加入Apache孵化器以来首�ơ发布�?/span>

http://beam.incubator.apache.org/beam/release/2016/06/15/first-release.html

Yandex开源了ClickHouse�Q�一个列式分析数据库。本�pȝ��为横向和�U�向扩展而生。支持复杂数据类型（例如数组�Q�和�q�似查询。该团队�q�发布了与其它数据库相比的基准测试结果�?/span>

https://clickhouse.yandex/

�z�d��

中国

Rosen 2016-07-01 15:44 发表评论

Rosen — Tue, 28 Jun 2016 09:39:00 GMT

Hadoop周刊 �W?/span> 174 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>6�?/span>12�?/span>

Spark��C��本周在旧金山召开�Q�正如所料，本期周刊有大量关�?/span>Apache Spark的新闅R��公告和版本发布。除Spark外，本期�q�有Kafka�?/span>Cask�?/span>Ambari斚w��的文章。在产品发布部分�Q�有一�q�来Apache Pig首次版本更新�Q�还一个�ؓ分布式系�l�设计的��z�新工具Runway�Q�最后是新版Apache Kudu�Q�孵化中�Q��?/span>

技术新�?/span>

Debezium是一个相对较新的��目�Q�用于数据库�?/span>Apache Kafka topic行��改变数据捕获。当面支�?/span>MySQL�?/span>Zookeeper�?/span>Kafka�Q�这是一��在Docker�?/span>Kubernetes容器上配�|?/span>Zookeeper, Kafka, MySQL的教�E��?/span>

http://debezium.io/blog/2016/05/31/Debezium-on-Kubernetes/

有些人对Apache Kafka��目宣布采用另一�U�流式处理引擎感到惊�Ӟ��q�就�?/span>Kafka Streams�?/span>Kafka Streams与其它系�l�存在显著的关键差异。本文很好的�C��了这些不同点——abstraction、部�|�模型、支持基于状态的计算�?/span>

https://softwaremill.com/kafka-streams-how-does-it-fit-stream-landscape/

每个使用MapReduce�?/span>Spark或类似系�l�的人都会陷入难以调试、数据特�?/span>bug�q�些问题中�?/span>BigDebug�?/span>UCLA�Q�加州大学洛杉矶分校�Q�的研究��目/论文�Q�旨在让开发�h员通过工具发现单机问题�Q�传入参数导致的崩溃�Q�跟�t�、断炏V��观察点、�g�q�报警等。该工具支持Apache Spark 1.2.1上�?/span>

https://blog.acolyer.org/2016/06/07/bigdebug-debugging-primitives-for-interactive-big-data-processing-in-spark/

Cask撰文介绍了在开�?/span>Cask Data Application Platform (CDAP)中运�?/span>Spark的文章。运行在CDAP�?/span>Spark�E�序通过讉K��Apache Tephra�Q�孵化中�Q�实现细�_�度事务支持。这��P��p��很容易利用快照隔��d��C��一个表复制到另一个表的一致性�?/span>CDAP中的Spark也能讉K��Cask Tracker�Q?/span>Cask Tracker提供数据血�~�信息（什么时候创建、��用等�Q�。根据应用的不同�Q?/span>CDAP工具�q�能发挥更大价倹{�?/span>

http://blog.cask.co/2016/06/cdap-spark-prototype-to-production/

IBM Hadoop Dev博客撰写了从cURL调用Ambari REST API的教�E�。还�C��了在vanilla和启用了kerberos的集��上建立会话�Q��ƈ为接下来的请求复用会话�?/span>

https://developer.ibm.com/hadoop/2016/06/07/ambari-rest-calls-for-kerberos-enabled-clusters/

Google云��^台博客撰文介�l�了如何调试�q�行�?/span>Google Dataflow上的Apache Beam�Q�孵化中�Q��Q务。�ؓ了调试性能瓉��Q?/span>Dataflow有一些有用的�l�计数据�?/span>UI来帮助��用者深入每一个步骤�?/span>

https://cloud.google.com/blog/big-data/2016/06/understanding-timing-in-cloud-dataflow-pipelines

其他新闻

Transaction Processing Performance Council(TPC)发布�?/span>TPCx-BB基准��试�Q�该基准��试为大数据�pȝ��设计。除了衡�?/span>SQL外，�q�可以对机器学习集群和分�c�问题进行测试�?/span>

http://www.datanami.com/2016/06/01/big-data-benchmark-gauges-hadoop-platforms/

伦敦Strata + Hadoop世界大会两周前已召开。演讲者的专题报告和��灯片已发布到会议�|�站上�?/span>

http://conferences.oreilly.com/strata/hadoop-big-data-eu/public/schedule/proceedings

Splice Machine�Q?/span>Hadoop上的RDBMS构徏者，宣布开源他们的软�g。当前，他们正在��L��贡献�?/span>/导师/豪杰来提升开源后的效果�?/span>Splice Machine有不��有��的�Ҏ��，例如ACID事务�Q�二�U�烦引，引用完整性�?/span>

http://www.splicemachine.com/were_going_open_source/

Altiscale博客�~�辑了许多关于客��h��务、情感分析、气候变化、智慧城市�?/span>bias�{�方面的大数据应用案例文章。还攉��了一些大数据怀疑论者的文章�?/span>

https://www.altiscale.com/blog/big-data-news-health-and-public-safety-sentiment-analysis-fixing-education-2/

Spark��C��本周在旧金山召开。会议组�l��?/span>Databricks概述了两天内的热点内容，链接了许多的演讲和专题报告�?/span>

https://databricks.com/blog/2016/06/08/another-record-setting-spark-summit.html

大数据即�?/span>�?/span>�Q�BDaaS�Q�公�?/span>Qubole�Q�撰文介�l�了他们的客户如何接受��?/span>Spark。接受速度之快——一半多的客��L��在开始用Spark�?/span>Qubole也支�?/span>Presto�Q�他们也看到了类似的增长�?/span>

https://www.qubole.com/blog/big-data/spark-usage/

Twitter�?/span>Apache孵化器提交了他们的复制日志服�?/span>DistributedLog�?/span>

https://wiki.apache.org/incubator/DistributedLogProposal

Big Data Day LA�?/span>6�?/span>9日在西洛杉矶学院召开。这�ơ活动是免费的（如果预先注册的话�Q�，演讲者来自于Confluent�?/span>Databricks�?/span>Yahoo�?/span>Netflix�{��?/span>

http://www.bigdatadayla.com/

产品发布

Apache Spark发布�?/span>Spark 2.0预览版。发布声明中说道API和功能都��未最�l�敲定�?/span>

https://spark.apache.org/news/spark-2.0.0-preview.html

JustOne构徏�q�开源了Kafka-to-PostgreSQL�q�接器。本文介�l�了该连接器的性能�Q�详�l�描�q�C��如何把消息�{换�ؓ行，�q�描�q�C��如何讑֮�配置�{��?/span>

http://www.confluent.io/blog/kafka-connect-sink-for-postgresql-from-justone-database

Salesforce开源了Runway�Q�这是一个徏模、仿真以及可视化分布式系�l�。在runway.system上有一个在�U�演�C�环境，演示�?/span>“too many bananas”模型�Q�电梯系�l�和Raft一致性系�l��?/span>

https://medium.com/salesforce-open-source/runway-intro-dc0d9578e248

Bloomberg最�q�开源了Presto Accumulo�Q�面�?/span>Apache Accumulo�?/span>Presto�q�接器。在声明中，链接�?/span>11��늚�论文�Q�比较了��Z��?/span>Presto查询和基�?/span>Accumulo Java API查询的基准测试结果�?/span>

http://www.bloomberg.com/company/announcements/open-source-at-bloomberg-reducing-application-development-time-via-presto-accumulo/

�?/span>�?/span>Azure发布了基�?/span>Apache Spark 1.6.1 �E�_��版的Azure HDInsight。本�ơ发布支持了面向Spark�?/span>Project Livy REST��d��服务支持�Q�集成了Azure数据湖存储（��Z��角色的访问控�Ӟ��Q�集成了IntelliJ�Q�支持了Jupyter�W�记本等�?/span>

https://azure.microsoft.com/en-us/blog/apache-spark-for-azure-hdinsight-now-generally-available/

LinkedIn开源了Photon ML�Q�他们的大规模回归分析库�?/span>Photon构徏�?/span>Spark之上�q�在LinkedIn�?/span>YARN上运行（�q�去��Z��MapReduce�Q�似乎因��提升性能才迁�U�）�?/span>

https://engineering.linkedin.com/blog/2016/06/open-sourcing-photon-ml

Hortonworks发布�?/span>Spark-HBase�q�接器的技术预览版。预览版原生支持Avro�Q�支持运行安全集��，原生支持Spark Datasource API�Q��ƈ优化了分��Z��剪，列修剪，谓词下推�?/span>

http://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/

Databricks发布�?/span>Apache Spark�q�_��的第一阶段安全�Ҏ��。本阶段寚w��?/span>ACL�?/span>SAML 2.0�q�行了支持，端对端的审计日志�?/span>

https://databricks.com/blog/2016/06/08/achieving-end-to-end-security-for-apache-spark-with-databricks.html

Apache ORC 1.1.0版发布了。本�ơ发布完成了从基�?/span>Apache Hive的代码到��Z��Java的代码迁�U�，修正�?/span>C++旉��戛_��理程序，增加�?/span>Hadoop MapReduce�q�接器�?/span>

http://orc.apache.org/news/2016/06/10/ORC-1.1.0/

Apache Kudu发布�?/span>0.9.0版。增加了UPSERT命��o�Q�新�?/span>Spark数据源不会依�?/span>MapReduce API�Q�提升了Tablet Server写性能�?/span>

http://getkudu.io/2016/06/10/apache-kudu-0-9-0-released.html

Google云服务��^台团队发布了支持Spark 2.0预览版的Google Cloud Dataproc�?/span>

https://cloud.google.com/blog/big-data/2016/06/google-cloud-dataproc-the-fast-easy-and-safe-way-to-try-spark-20-preview

Dory�Q?/span>Bruce的��承者）Kafka producer的守护进�E�，现在支持�?/span>UNIX domain sockets或本�?/span>TCP接收数据了�?/span>

http://mail-archives.apache.org/mod_mbox/kafka-users/201606.mbox/%3C1465683894.608424023@apps.rackspace.com%3E

Apache Pig 0.16.0版，一�q�来首次发布。坚定了�?/span>Tez的支持�?/span>

http://pig.apache.org/releases.html#8+June%2C+2016%3A+release+0.16.0+available

�z�d��

中国

Spark Meetup (上�v) – 周六, 6�?/span>18�?/span>

Rosen 2016-06-28 17:39 发表评论

Rosen — Mon, 20 Jun 2016 01:47:00 GMT

Hadoop周刊 �W?/span> 173 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>6�?/span>5�?/span>

本周�Q?/span>Spark�?/span>NiFi�?/span>Netflix Meson�?/span>Storm斚w��只有��量内容�?/span>Spark��C��本周在旧金山召开�Q�所以呢�Q�下周肯定有不少内容�?/span>

技术新�?/span>

Databricks博客介绍�?/span>Apache Spark 2.0的新�Ҏ�?/span>——跨语�a�支持存储和加载机器学习模型。模型通过��单的API被存储和加蝲�Q�模型的元数据与参数保存�?/span>JSON风格�Q�模型的数据保存�?/span>Parquet风格�?/span>

https://databricks.com/blog/2016/05/31/apache-spark-2-0-preview-machine-learning-model-persistence.html

Meson�?/span>Netflix用于执行机器学习工作��的框架。它�?/span>Apache Hive�?/span>Spark�?/span>Mesos�q�些大数据技术之间的�_�合剂。工作流使用DSL�q�行�~�写�Q?/span>Meson�q�提供了更加先进的流水线可视�?/span>UI�?/span>Netflix目前没开�?/span>Meson�Q�但他们有这斚w��的计划�?/span>

http://techblog.netflix.com/2016/05/meson_31.html

IBM Hadoop Dev博客��要介�l�和�C��?/span>HDFS归档存储能力�?/span>

https://developer.ibm.com/hadoop/2016/06/01/use-hdfs-archival-storage/

Apache Storm 1.0有了令�h惊讶的新�Ҏ��。本文关注了几个调试能力斚w��的增强：动态日志��别、统一日志搜烦�?/span>事�g抽样、集�?/span>jstack/heap dumps/java飞行记录器分�?/span>worker�?/span>

http://hortonworks.com/blog/whats-new-apache-storm-1-0-part-1-enhanced-debugging/

Cloudera博客撰文介绍了如何��?/span>Apache Spark来探索性分析存储在CSV文�g中的NBA历史�l�计数据。分析过�E��؜合��用了Scala�?/span>SQL�?/span>

http://blog.cloudera.com/blog/2016/06/how-to-analyze-fantasy-sports-using-apache-spark-and-sql/

Apache NiFi作�ؓ一�U�通用工具受到了很多的��x��。它�?/span>“��Z��程的处�?/span>”而生�Q�可能对很多人�ƈ不意味着什么，�?/span>NiFi支持标准�?/span>ETL�Q�流式处理等。许�?/span>NiFi例子都示范了如何�?/span>Twitter firehose把数据移动到HDFS中，但本文聚焦在NiFi另外的特性上——�C��了一些简单的�?/span>HTTP拉数据的�q�程�?/span>

http://hortonworks.com/blog/apache-nifi-not-scratch/

Amazon Redshift构徏�?/span>PostgreSQL引擎上，所以你可以利用PostgreSQL的扩展功能让Redshift集群�q�接PostgresSQL实例。这样一来，诸如跨数据库�q�接、将Redshift的结果�{换�ؓJSON、在Postgres中创�?/span>Redshift数据视图�?/span>

数据库之间复制数据等有趣的应用都能实现�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx1GQ6WLEWVJ1OX/JOIN-Amazon-Redshift-AND-Amazon-RDS-PostgreSQL-WITH-dblink

其他发布

FeatherCast发布了超�q?/span>100�?/span>ApacheCon北美��C��的相兛_��韟�?/span>

http://feathercast.apache.org/tag/apacheconna2016/

InfoWorld介绍�?/span>Heron�Q?/span>Twitter才开源的Apache Storm兼容��目。本文介�l�了两个��目在架构上的不同。主要指��Z��Heron��h��于几个月前（Storm已发布）�Q�就是说Storm在特性上�?/span>Heron更有优势�?/span>

http://www.infoworld.com/article/3078134/analytics/had-it-with-apache-storm-heron-swoops-to-the-rescue.html

Databricks�?/span>edX上开了一门新评��Q?/span>“Apache Spark入门”。课�E�从6�?/span>15日开始，一直持�l�两周�?/span>

launch-first-of-five-free-big-data-courses-on-apache-spark.html

产品发布

Amazon EMR发布�?/span>4.7.0版。本�ơ发布支持了Apache Tez�?/span>Apache Phoenix�Q��ƈ内置了新版本�?/span>Apache HBase�?/span>Apache Mahout�?/span>Presto。另外，AWS大数据博客还指导�?/span>Phoenix如何上手�?/span>

http://aws.amazon.com/blogs/aws/amazon-emr-4-7-0-apache-tez-phoenix-updates-to-existing-apps/

http://blogs.aws.amazon.com/bigdata/post/Tx2ZF1NDQYDJFGT/Supercharge-SQL-on-Your-Data-in-Apache-HBase-with-Apache-Phoenix

Apache Hive本周发布�?/span>2.0.1版。从二月发布2.0.0以来�Q�首�ơ小版本发布。本�ơ修复了60�?/span>bug�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201605.mbox/%3CD37344A3.77A64%25sershe@apache.org%3E

�z�d��

中国

�?/span>

Rosen 2016-06-20 09:47 发表评论

Rosen — Wed, 08 Jun 2016 16:11:00 GMT

Hadoop周刊 �W?/span> 172 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>5�?/span>22�?/span>

本周主要��x��式计算—— Twitter�?/span>Cloudera介绍了他们新的流式计��框�Ӟ��有文章介�l�了Apache Flink的流�?/span>SQL�Q?/span>DataTorrent介绍�?/span>Apache Apex定w��机制�Q�还�?/span>Concord�q�样新的��式计算框架�Q�另外还�?/span>Apache Kafka�?/span>0.10版。其他新��L��面，Apache孵化器有新动�?/span>——Apache TinkerPop�?/span>Apache Zeppelin孵化成�ؓ��目�Q?/span>Tephra�q�入孵化器。除了上�q�内容，Apache Spark�?/span>Apache HBase�?/span>Apache Drill�?/span>Apache Ambari�{�也有新文章�?/span>

技术新�?/span>

DataTorrent博客撰文介绍�?/span>Apache Apex在读写数据文件时的容错机制�?/span>Apex是专门处理流式数据的�Q�流式计��有一些微妙但重要的细节需要考虑。例如��?/span>HDFS输出�Ӟ��HDFS的租�U�机制会引发问题�?/span>

https://www.datatorrent.com/blog/fault-tolerant-file-processing/

Databricks博客介绍�?/span>Spark 2.0�?/span>Tungsten代码生成引擎带来的性能提升。博文�D例说明了�׃��虚拟函数的管理，更好地利�?/span>CPU寄存器和循环展开�Q�所以代码生成引擎能更快的生成代码。除�?/span>Databricks的博文外�Q?/span>Morning Paper�q�谈��C��上技术其实是受到VLDB论文的启发�?/span>

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html

https://blog.acolyer.org/2016/05/23/efficiently-compiling-efficient-query-plans-for-modern-hardware/

StreamScope是微软流式处理系�l�，�?/span>Morning Paper本周撰写的另一个流式计��文章。介�l�了该系�l�的特征——吞吐�?/span>/集群大小、编�E�模�?/span>(SQL)、时间模型、语义学/保证�Q�以及微软��品中的应用�?/span>

https://blog.acolyer.org/2016/05/24/streamscope-continuous-reliable-distributed-processing-of-big-data-streams/

Apache博客撰文介绍�?/span>HubSpot团队�?/span>Apache HBase�?/span>G1GC调优斚w��的经验。本文回��?/span>HubSpot如何��试和保障稳定性、如何保�?/span>99%的性能、如何羃短花在垃圑֛�收上的时间。该团队使用很多技巧，很好地决�l�了错综复杂�?/span>GC��法。本文最后，�q�一步步�C��?/span>HBase�?/span>G1GC调优�?/span>

https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase

LinkedIn撰文阐述了调�?/span>Kafka偏移量管理问题的诸多困难。本文聚焦了两个所�?/span>"offset rewind"事�g的症�Ӟ��如何在监控过�E�中��到�q�类事�g�Q�以及导致这两个事�g的根本原因（及解��x��案）�?/span>

https://engineering.linkedin.com/blog/2016/05/kafkaesque-days-at-linkedin--part-1

Databricks博客发布了��?/span>Apache Spark�q�行基因变异分析�p�d��文章的第三部分也是最后一��。本文从准备�Q�把文�g转换�?/span>Parquet�q�加载进Spark RRD�Q�到如何加蝲基因型数据再到运�?/span>kmeans聚类��法��Z��基因型特征预��地理种��?/span>

https://databricks.com/blog/2016/05/24/predicting-geographic-population-using-genome-variants-and-k-means.html

许多批处理大数据生态系�l�已从自定义API回到SQL上，所以如果流式处理框架也发生了同��L��变化�Q�一定很有趣。本文，Apache Flink团队介绍他们计划支持��式SQL�?/span>Flink已经有了Table API�Q�他们利�?/span>Apache Calcite提供了对SQL的支持。对�?/span>windowing�Q�他们计划用Calcite的流�?/span>SQL扩展。最初对SQL的支持将�?/span>1.1.0版中体现�Q�在1.2.0版加强�?/span>

http://flink.apache.org/news/2016/05/24/stream-sql.html

本文介绍�?/span>Apache Drill�?/span>XML插�g。尽��还没有�?/span>Drill集成在一��P��但它相当�Ҏ��被编译成jar和配�|�对XML的支持�?/span>

https://www.mapr.com/blog/how-use-xml-plugin-apache-drill

Hortonworks博客��略介�l�了Ambari监控度量�pȝ��的架构，最�q�加入了Grafana作�ؓ其前端��A表盘。该�pȝ��使用Apache Phoenix�?/span>Apache HBase作�ؓ存储支撑�Q�所以是可以横向扩展的�?/span>

http://hortonworks.com/blog/hood-ambari-metrics-grafana/

�q�篇教程介绍了怎样�?/span>Amazon EMR上��?/span>Spark SQL�?/span>Hue�?/span>Apache Zeppelin配合�q�行SQL查询存储�?/span>S3中跨制表�W�分割的数据。本文最后展�C�Z��如何�?/span>Spark�?/span>DynamoDB存储数据�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx2D93GZRHU3TES/Using-Spark-SQL-for-ETL

Heroku团队分��n了他们��用最新版Apache Kafka的体�?/span>——才引入的timestamp字段�Q?/span>8字节�Q�会��D��一些反直觉的性能变化�?/span>

https://engineering.heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-performance-in-distributed-systems/

其他新闻

O'Reilly数据播客�U��?/span>Spark 2.0中结构化��式计算斚w��的问题采访了来自Databricks�?/span>Michael Armbrust。网站上的一��文章选择引用了其中的话题—— Spark SQL、结构化��式计算的目标、端到端��道的保证、对在线处理�q�用Spark机器学习��法�?/span>

https://www.oreilly.com/ideas/structured-streaming-comes-to-apache-spark-2-0

本周两个大数据项目从Apache孵化器孵化完�?/span>——Apache TinkerPop�?/span>Apache Zeppelin�?/span>TinkerPop是图计算框架�Q?/span>Zeppelin是面向数据分析基�?/span>web�?/span>notebook�?/span>

https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces91

https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces92

Tephra�Q?/span>HBase的事务引擎进入了Apache孵化器�?/span>Tephra最初由Cask的团队创建，目前仅和Apache Phoenix�q�行了集成�?/span>

http://blog.cask.co/2016/05/tephra-a-transaction-engine-for-hbase-moves-to-apache-incubation/

TechRepublic撰文介绍�?/span>Concord.io�Q�一个由C++开发的��式处理框架。旨在填补高性能��式计算市场的空�~��?/span>

http://www.techrepublic.com/article/could-concord-topple-apache-spark-from-its-big-data-throne/

产品发布

Apache Avro本周发布�?/span>1.8.1版。修复了��过20�?/span>bug和一些其它进步�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201605.mbox/%3CCAO4re1nYMm79WQ2LUeODWjHmJ9EiYOF=mty6p2aiq-S_4R95iQ@mail.gmail.com%3E

Confluent发布了基�?/span>librdkafka开发的Kafka Python客户端�?/span>

https://pypi.python.org/pypi/confluent-kafka/0.9.1.1

伴随着新的Kafka ��式计算方式�Q?/span>Apache Kafka 0.10版发布了。新版本支持了机架感知和消息中的timestamp�Q�提升了SASL�?/span>Kafka Connect�{��?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201605.mbox/%3CCAPuboUuRyCRxDp5CLjv2yVM77SpYFF+HdnBeiiyeumYTJNpY4g@mail.gmail.com%3E

Confluent发布了基�?/span>Apache Kafka 0.10�?/span>Confluent Platform 3.0版。除�?/span>Kafka的核心特性，Confluent Platform�q�有一个商业组件�ؓKafka Connect提供配置工具和端到端��监控�?/span>

http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0

Apache Kylin�Q�大数据OLAP引擎�Q�发布了1.5.2版。作��Z��ơ补丁��的发布，1.5.2有不��新�Ҏ�?/span>/提升/bug修复�Q�包括支�?/span>CDH 5.7�?/span>MapR�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201605.mbox/%3CCA+LQBaTDxb4wVYVvtOC22gMbJ0p9cvhAWzEY_x2n1oNGvEDPSQ@mail.gmail.com%3E

Twitter开源了他们的流式处理系�l?/span>Heron�?/span>Heron�?/span>Twitter用于替换Apache Storm的��品，发力点在性能、调试以及开发�h员生产率�?/span>

https://blog.twitter.com/2016/open-sourcing-twitter-heron

Envelope是来自于Cloudera Labs的新��目�Q�它提供了基于配�|�文件的��式ETL处理�q�程。构建在Spark streaming之上�Q?/span>Envelope最�q�正在研发面�?/span>Kafka�?/span>Kudu的连接器�?/span>

http://blog.cloudera.com/blog/2016/05/new-in-cloudera-labs-envelope-for-apache-spark-streaming/

�z�d��

中国

Spark Meetup 4 (杭州) – 周日, 6�?/span>5�?/span>

http://www.meetup.com/Hangzhou-Apache-Spark-Meetup/events/231071384/

Rosen 2016-06-09 00:11 发表评论

Rosen — Wed, 08 Jun 2016 08:42:00 GMT

Hadoop周刊 �W?/span> 171 �?/span>

启明星辰�q�_��和大数据��M��l�编�?/span>

2016�q?/span>5�?/span>22�?/span>

本周�Q�包�?/span>LinkedIn新开源项目在内的几个��目都有版本发布。在技术新��d��其他新闻斚w��Q�多��文章回��了Apache: Big Data North America会议�Q�另外有一�l�跨��多个不同数据系�l�分析纽�U�出�U��R数据的系列文章�?/span>

技术新�?/span>

Databricks博客分析�?/span>Apache Spark中两�U��D��法。之一�Q?/span>“approxCountDistict”是用来评��C��同值的数量�Q�之二，“approxQuantile”用于生成��D��癑ֈ�比。本文介�l�了��法和可视化�_�ֺ�不同的残差�?/span>

https://databricks.com/blog/2016/05/19/approximate-algorithms-in-apache-spark-hyperloglog-and-quantiles.html

本教�E�描�q�C��如何使用Apache Hadoop HDFS�?/span>Apache Solr�?/span>Hue存储、烦引、查�?/span>DICOM格式的医学媄像。文章诏�I�了加蝲和获取数据的整个步骤�?/span>

http://blog.cloudera.com/blog/2016/05/how-to-process-and-index-medical-images-with-apache-hadoop-and-apache-solr/

MapR Streams是一�?/span>API兼容Apache Kafka的系�l�。本文在宏观上比较了MapR Streams�?/span>Kafka的异同。同旉��明了Kafka Streams怎样�?/span>MapR Streams扯上关系的�?/span>

https://www.mapr.com/blog/apache-kafka-and-mapr-streams-terms-techniques-and-new-designs

本文在我看来是最清晰介绍Paxos的文章之一�Q?/span>Paxos为分布式�pȝ��构徏了一致性协议。本文用�l�图计算机和分布式拍卖示范了�q�个协议�?/span>

http://ifeanyi.co/posts/understanding-consensus/

��Z��Apache: Big Data North America会议上的一��演讌Ӏ?/span>Datanami�H�探了即��发布的Apache Hadoop 3的新�Ҏ��。包括，shell脚本重写、�Q务集本地优化、内存大��自动�׾~�能力、支�?/span>HDFS erasure codings。本文着重在erasure codings上，文章密切��x��?/span>erasure codings在存储效率方面的提升�Q?/span>3x��盘消耗降低到1.5x�Q��?/span>

http://www.datanami.com/2016/05/18/hadoop-3-poised-boost-storage-capacity-resilience-erasure-coding/

�q�篇演讲来自�?/span>PyData柏林会议�Q�描�q�C��Apache Arrow�?/span>Feather文�g格式�Q�探�I�了数据在跨语言/框架互操作性的工作机制�?/span>

http://www.slideshare.net/wesm/python-data-ecosystem-thoughts-on-building-for-the-future

发布了两个来自于不同会议�?/span>Apache Kafka有关的演讲视频。第一个讨��Z��Kafka的安全特性，�W�二个探索了Kafka如何跨系�l�共享数据�?/span>

https://www.oreilly.com/learning/securing-apache-kafka

https://www.infoq.com/presentations/event-streams-kafka

�q�篇博客集成了数��利�?/span>Amazon Redshift�?/span>Google BigQuery�?/span>Postgres�?/span>Presto数据�pȝ��加蝲/查询�U�约出租车数据的文章。除了原始基准测试，�q�详�l�介�l�了如何处理故障、优化、比较替代方案（AWS�?/span>S3�?/span>HDFS比）�?/span>

http://tech.marksblogg.com/all-billion-nyc-taxi-rides-redshift.html

O'Reilly撰文介绍了通过Kafka�?/span>Flink�?/span>Elasticsearch�?/span>Kibana怎样实现kappa架构。文章概�q�C��lambda�?/span>kappa架构�Q�介�l�了主要的架构组�Ӟ��以及怎样讄��使用贝叶斯模型发现新奇事物�?/span>

http://www.oreilly.com/ideas/applying-the-kappa-architecture-in-the-telco-industry

其他新闻

本文列�D了最�q�在Apache: Big Data North America会议上提到的几个大数据生态系�l�项目。有不少是我们没�U�_��视线的内宏V�?/span>

http://www.datanami.com/2016/05/11/open-source-tour-de-force-apache-big-data-2016/

Pivotal博客有一��关于大数据和敏捷开发有��的文章。大数据�pȝ��往往停留在非敏捷的世界，例如在装载数据前需求要攉��C��Q�模型要定义好。本文认为，没有在云环境中经�q�长期验证的话，要对�q�种方式�q�行�U�束�Q�有限的能力和性能、竖井式数据�{�）�?/span>

https://blog.pivotal.io/big-data-pivotal/features/when-it-comes-to-big-data-cloud-and-agility-go-hand-in-hand

Databricks发布了他们记录的�|�络会议视频“Apache Spark MLlib: From Quick Start to Scikit-Learn”。除了视频内容，他们�q�在会议中解�{�了八个常见问题�?/span>

https://databricks.com/blog/2016/05/18/spark-mllib-from-quick-start-to-scikit-learn.html

Hortonworks博客回顾�?/span>Apache Storm的历双Ӏ?/span>2011�q�开源，2013�q�进�?/span>Apache孵化器，2014�q�成为顶�U�项目，今年初发布了1.0版。本文论�q�C��每个里程��的主要技术进步�?/span>

http://hortonworks.com/blog/brief-history-apache-storm/

HBaseCon本周在旧金山召开。这�ơ会议，Apple�?/span>Yahoo�?/span>Facebook都有演讲材料�?/span>

http://hbasecon.com

MapR发图庆祝了过��M��q�中Apache Drill取得的成�l�。一�q�中发布�?/span>7个版本，完成了多个里�E�碑�?/span>

https://www.mapr.com/blog/happy-anniversary-apache-drill-what-difference-year-makes

Datanami发布了在Apache: Big Data North America会议上，ASF�ȝ��Jim Jagielski�?/span>ODPi��目�ȝ��John Mertic的问�{�录�Q�如大家所料，主要话题�q�是ASF�?/span>ODPi的关�p�R�?/span>

http://www.datanami.com/2016/05/20/apache-foundation-keeps-eyes-wide-open-odpi/

产品发布

LinkedIn开源了Ambry�Q�他们的ObjectStore分布式系�l��?/span>Ambry代码已提交到github�Q�这��博文介�l�了Ambry的服务承诺，设计目标�Q�体�p�L��构和接口�?/span>

https://engineering.linkedin.com/blog/2016/05/introducing-and-open-sourcing-ambry---linkedins-new-distributed-

�?/span>apache HAWQ�Q�孵化中�Q�驱动的Pivotal HDB 本周发布�?/span>2.0版，HDB�?/span>Hadoop提供了分析数据库�?/span>

https://blog.pivotal.io/big-data-pivotal/products/fail-fast-and-ask-more-questions-of-your-data-with-hdb-2-0

Apache Mahout本周发布�?/span>0.12.1版，Mahout是一个机器学习和数据挖掘�pȝ��。本�ơ发布旨在推�q?/span>Flink�?/span>Mahout的集成�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201605.mbox/%3CCAOtpBjhshagyLN3Qnt0xRnc7YbnMVJjTS4piVXL7LiS2pQguXw@mail.gmail.com%3E

Apache Tajo发布�?/span>0.11.3版�?/span>Tajo�?/span>Hadoop的数据仓库。本�ơ发布修正了5�?/span>bug�?/span>

http://tajo.apache.org/releases/0.11.3/announcement.html

MongoDB�?/span>Apache Spark发布了新�?/span>MongoDB Connector。除了对�?/span>Spark�?/span>Hadoop InputFormat shim外，�?/span>Connector�q�有其他�Ҏ��。最后，�q�解释了MongoDB一些关键特性�?/span>

https://www.mongodb.com/blog/post/mongodb-connector-for-apache-spark-announcing-early-access-program-and-new-spark-training

http://rosslawley.co.uk/introducing-a-new=mongodb-spark-connector/

SyncSort发布�?/span>DMX-h v9�Q�支�?/span>Kafka以及新的��执行框架�?/span>

http://insidebigdata.com/2016/05/20/syncsorts-latest-innovations-simplify-integration-of-streaming-data-in-spark-kafka-and-hadoop-for-real-time-analytics/

�z�d��

中国

�?/span>

Rosen 2016-06-08 16:42 发表评论

Rosen — Sun, 15 May 2016 12:30:00 GMT

Hadoop周刊 �W?/span> 169 �?/span>

启明星辰�q�_��和大数据整体�l�编�?/span>

2016�q?/span>5�?/span>8�?/span>

本周内容短小�_��。主题覆�?/span>Apache Beam�?/span>MapR季度业�W、最�q�的Kafka��C��Q�以及来�?/span>Cloudera新开源的分布式单元测试框架�?/span>

技术新�?/span>

Elastic分析了宕��Z��件的�Ҏ��。错误配�|?/span>ZooKeeper内存讄��会引赯��度的GC�Q�这��从�Ҏ��上导�?/span>ZooKeeper集群丢失。文章介�l�了一些缓解策略，用来防止未来�c�M��问题的发生�?/span>

https://www.elastic.co/blog/elastic-cloud-outage-april-2016

Cask博客��明扼要的归纳了最�q?/span>Big Data Applications Meetup的花�i�。首先出场的�?/span>Pachyderm�Q�它��Z��Docker容器提供“数据Git”语义。第二个出场的是TubeMogul大数据��^収ͼ�TubeMogul构徏�?/span>Hadoop�?/span>Hive�?/span>Spark�?/span>Presto之上�?/span>

http://blog.cask.co/2016/05/pachyderm-and-tubemogul-share-their-big-data-application-platforms-and-experience/

Google�?/span>dataArtisans同时撰文介绍�?/span>Apache Beam�Q�前生是Google Dataflow SDK�Q��?/span>Google的文章解释了��Z��开源和开�?/span>Beam的动机，dataArtisans的文章介�l�他们对Beam模型的支持以及怎样考虑Flink�?/span>Beam API之间的关�p�R�?/span>

https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective

http://data-artisans.com/why-apache-beam/

IBM Hadoop dev博客有个关于安装Python�?/span>Scala和�ؓJupyter notebook嵌入R内核的操作说明。同�Ӟ��也说明了怎样�q�接Spark和通过SSL暴露notebook�?/span>

https://developer.ibm.com/hadoop/blog/2016/05/04/install-jupyter-notebook-spark/

本文介绍�?/span>Mongo Hadoop的连接函数是如何�H��vSpark�?/span>MongoDB的�?/span>

https://x.ai/using-the-mongo-hadoop-connector-as-a-translation-layer-to-spark/

Qubole博客撰文比较了用于大数据分析的流行编�E�语�a�—Python�?/span>R�?/span>Scala�?/span>

http://www.qubole.com/blog/big-data/programming-language/

其他新闻

MapR宣布本季度他们授权下单创�U�录的增长了99%�Q�以�?/span>146%的美元净增长率�?/span>

https://www.mapr.com/company/press-releases/mapr-achieves-another-record-quarter-99-software-subscription-license-growth

本文描述了最�q?/span>Google Cloud Dataflow�?/span>Apache Spark�?/span>Google Compute Engine上的基准��试表现�?/span>Dataflow胜过Spark2�Q?/span>5.7倍（一直以来，最好是在自��q��环境下评估工作负载，而不是一味的信�Q基准��试�Q�。本文还解释了一�U?/span>“��h��”�Q�通过它��每个使用大数据工��L��益�?/span>

http://www.datanami.com/2016/05/02/dataflow-tops-spark-benchmark-test/

Confluent博客回顾了最�q�召开�?/span>Kafka��C��Q�包括编�E�挑战预选赛�Q�主题演�Ԍ��分组会议�{�等�?/span>

http://www.confluent.io/blog/log-compaction-kafka-summit-edition-may-2016

��布斯介�l�了��国�q�通在�q�去5�q�间采用大数据技术的历程。本文中�Q�美国运通分享了一些技巧和学到的经验教训，例如采用新技术的困难�Q�得到组�l�高层的认同是多么的重要�Q�，以及雇䄦和留住工�E�师的挑战等�{��?/span>

http://www.forbes.com/sites/ciocentral/2016/04/27/inside-american-express-big-data-journey/

产品发布

Cask发布�?/span>Cask Data Application Platform (CDAP)3.4版本�?/span>新版本增加了Cask Tracker�Q�新的数据集�?/span>/审计/搜烦�pȝ��Q�升�U�了Cask Hydrator�?/span>UI�Q�增��Z��?/span>Spark的支持等�{��?/span>

http://blog.cask.co/2016/05/announcing-cdap-release-3-4-introducing-tracker-next-gen-hydrator-enhanced-spark-support-and-much-more/

Cloudera开源了“dist_tes”�Q��ƈ行执行单元测试的新工兗��通过该工��P��?/span>Hadoop�?/span>Kudu��目�q�行单元��试�Q�可以在数分钟而不是数��时内完成。该工具�l�定�?/span>C++�?/span>Java�Q��ƈ在网站上演示了这些特性�?/span>

http://blog.cloudera.com/blog/2016/05/quality-assurance-at-cloudera-distributed-unit-testing/

Google宣布Google BigQuery�?/span>Drive可集成在一��P��把输��Z��存到Google sheets�?/span>

http://techcrunch.com/2016/05/06/google-connects-bigquery-to-google-drive-and-sheets/

�z�d��

中国

�?/span>

Rosen 2016-05-15 20:30 发表评论

Rosen — Sat, 07 May 2016 15:37:00 GMT

Hadoop周刊 �W?/span> 168 �?/span>

启明星辰�q�_��和大数据整体�l�编�?/span>

2016�q?/span>5�?/span>1�?/span>

Kafka��C��本周在旧金山召开�Q�不容置疑本周期刊将有大量的Kafka内容。除此以外，�q�有大量关于Impala性能�?/span>Kudu�?/span>Druid斚w��的文章。在其他新闻部分�Q?/span>Apache Apex成�ؓ�?/span>Apache的顶�U�项目，Qubole开源了�?/span>StreamX��目�?/span>

技术新�?/span>

本文快速浏览了如何在可能或不可能创建新数据分区的情况下操作Spark RDD。尤�?/span>`mapValues`�?/span>`filter`会保存分��?/span>`map`却不会�?/span>

https://medium.com/@corentinanjuna/apache-spark-rdd-partitioning-preservation-2187a93bc33e

本文介绍了如何��?/span>Conda构徏独立�?/span>Python环境�Q�例�?/span>pandas插�g�Q�，以便做�ؓSpark job的一部分装蝲到集��节炏V��经�q�这��L��处理�Q�就能在没有python原生包被安装在主操作�pȝ��上的情况下运�?/span>PySpark job。这�U�方案同样适用�?/span>SparkR�?/span>

http://quasiben.github.io/blog/2016/4/15/conda-spark/

Datadog博客有三��监�?/span>Kafka的系列文章。第一��详�l�概括了broker�?/span>producer�?/span>consumers�?/span>ZooKeeper的关键度量指标。第二篇介绍了怎样�?/span>JConsole和其他工具上通过JMX查看指标�Q�第三篇介绍�?/span>Datadog集成斚w��的知识�?/span>

https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/

Salesforce撰文介绍�?/span>Kafka在他们组�l�内的成长史。最初，他们借助Kafka驱动了操作指标分析功能，渐渐地成��Z��个驱动众多系�l�的大��^台�?/span>Salesforce�q�用Kafka在多个数据中心运行，�q��?/span>MirrorMaker在集��间复制和聚合数据�?/span>

https://medium.com/salesforce-engineering/expanding-visibility-with-apache-kafka-e305b12c4aba#.5k7j921o3

Metamarkets博客有一��关于优化大规模分布式系�l�的有趣博文�?/span>Druid�Q�他们的分布式数据仓库，最�q�增加了一�U?/span>"先进先出"的查询模式，�q�在重型负蝲大集��间�q�行了测试。根据他们的假设�Q�推��Q何可能发生和攉��到有��的的指标�?/span>

https://metamarkets.com/2016/impact-on-query-speed-from-forced-processing-ordering-in-druid/

Google Cloud Big Data博客撰文介绍�?/span>BigQuery的内部存储格式，容器�Q�以及其它��得存储数据更有效率的优化措施�?/span>

https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format

Apache Kudu�Q�孵化中�Q�博客概�q�C��最�q��?/span>YCSB工具对系�l�性能分析和调优的�l�果�?/span>

http://getkudu.io/2016/04/26/ycsb.html

Impala 2.5无论�?/span>TPC基准��试�q�是其它斚w��均有显著的性能提升。提升项包括�q�行时过滤器�Q?/span>LLVM代码生成器对`SORT`�?/span>`DECIMAL`的支持，更快�?/span>metadata-only查询�Q�等�{��?/span>

http://blog.cloudera.com/blog/2016/04/apache-impala-incubating-in-cdh-5-7-4x-faster-for-bi-workloads-on-apache-hadoop/

本文介绍了，为支持高可用性，如何�?/span>Hive Metastore配置MariaDB的�?/span>

https://developer.ibm.com/hadoop/blog/2016/04/26/bigsql-ha-configure-ha-hive-metastore-db-using-mariadb10-1/

Altiscale博客撰文介绍了寻�?/span>NodeGroup相关bug的过�E�（跟进三月的文章）。如果你因没扑ֈ�Hadoop�Q�或其他分布式系�l�）�?/span>bug根结而气馁，不要�Ҏ��。本文告诉你�q�的��困难，甚至需要程序员在销�?/span>Hadoop服务的企业干�z�L��能搞定�?/span>

https://www.altiscale.com/blog/part-1-2-investigation-analysis-and-resolution-of-nodegroup-performance-issues-on-bare-metal-hardware-clusters/

Netflix现在�q�行了超�q?/span>4000�?/span>Kafka broker�Q�横�?/span>36个集��。在云中�q�行Kafka需要一些权衡，团队�q��了开销和数据丢失（日数据丢失小�?/span>0.01%�Q�。本文分享了团队�?/span>AWS中运�?/span>Kafka的经验，主要是一些典型问题，部��v�{�略�Q�小集群、隔��ȝ��zookeeper集群�Q�，集群�U�容错，支持AWS availability zones�Q?/span>Kafka UI可视化等�{��?/span>

http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html

Amazon大数据博客撰文介�l�了如何�?/span>Amazon EMR加密数据存放�?/span>S3中。这�U�集成方式同时支持客��L��和服务器端加密（借助�?/span>Amazon KMS�Q��?/span>

http://blogs.aws.amazon.com/bigdata/post/TxBQTAF 3X7VLEP/Process-Encrypted-Data-in-Amazon-EMR-with-Amazon-S3-and-AWS-KMS

TubeMogul介绍了他们大数据�q�_��的历�Ԍ��该��^台每月支撑万亿次数据分析��h��。该团队很早��p��?/span>Amazon EMR�Q�导入了Storm实时处理技术，最�l�把大数据服务落在了Qubole上�?/span>

https://www.tubemogul.com/engineering/the-big-data-lifecycle-at-tubemogul/

Caffe�Q�深度学习框�Ӟ��?/span>Spark�q�行了集�?/span>—CaffeOnSpark�?/span>MapR公司撰文介绍了如何在MapR YARN上运行，文章�q�包括了采用的性能优化手段�?/span>

https://www.mapr.com/blog/distributed-deep-learning-caffe-using-mapr-cluster

其他新闻

Apache Apex�Q�大数据��式处理和批处理�pȝ��Q�现在成��Z��Apache软�g基金会的��目�?/span>Apex��d��8月进入孵化器�?/span>

https://blogs.apache.org/foundation/entry/the_apache_ software_foundation_announces90

Heroku Kafka�Q�是一个分支于Heroku�?/span>Kafka��理服务。最�q�接�q�发�?/span>beta版�?/span>

https://blog.heroku.com/archives/2016/4/26/announcing-heroku-kafka-early-access

MapR博客上的一��文章强调�ؓ什么性别多样性是重要的，�q�提��C��大数据论坛中的女性，本文旨在鼓励��x��投�w�于�q�一领域�?/span>“大数据论坛中的女�?/span>”研讨会本周由MapR�l�织在圣何塞召开�?/span>

https://www.mapr.com/blog/case-women-big-data

产品发布

StreamX是一个来�?/span>Qubole的开源项目，它能�?/span>Kafka拯��数据�?/span>Amazon S3�q�样的目标存储中�?/span>Qubole�?/span>StreamX作�ؓ一�U�管理服务提供�?/span>

http://www.qubole.com/blog/big-data/streamx/

SnappyData是一个�ؓOLAP�?/span>OLTP查询��式数据的新�q�_��Q�和公司�Q��?/span>SnappyData�?/span>Apache Spark�?/span>GemFire的内存存储技术驱动�?/span>

http://www.infoworld.com/article/3062022/sql/apache-spark-powers-live-sql-analytics-in-snappydata.html

http://www.snappydata.io/

Apache Geode�Q�孵化中�Q�发布了1.0.0-incubating.M2版本�Q�它是一个分布式数据�q�_��Q�瞄准高性能和低延迟。新版本提供了广域网下的点对点连接等新特性�?/span>

http://mail-archives.apache.org/mod_mbox/incubator-geode-dev/201604.mbox/%3CCAFh%2B7k2eiK2TMGK sLqrY9CZDjxjYwiuTQ4QGUVC2s3geyJYwnA% 40mail.gmail.com%3E

Apache Knox发布�?/span>0.9.0版，它是Hadoop�?/span>REST API�|�关。新版本�?/span>Ranger�?/span>Ambari提供�?/span>UI界面支持�Q�以及一些其它的提升�?/span>bug修复�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201604.mbox/%3CCACRbFyjRF7zShb-NQ29d3FJ0hKZ57ts0Qfo31ffuNODpskwqPQ @mail.gmail.com%3E

�z�d��

中国

�?/span>

Rosen 2016-05-07 23:37 发表评论

Hadoop周刊—第 167 �?

Rosen — Tue, 03 May 2016 02:08:00 GMT

Hadoop周刊 �W?/span> 167 �?br />

启明星辰�q�_��和大数据整体�l�编�?/span>

2016�q?/span>4�?/span>25�?/span>

�Ƣ迎来到Hadoop周刊周一特别版。本周有大量来自Spark�?/span>Kafka�?/span>Beam�?/span>Kudu的技术新闅R��如果你正在��L��一些更前沿的技术，Apache Metron�Q�孵化中�Q�发布了它们�W�一个版本�?/span>Metron�Q�是一个构建在Hadoop上正在不断发展的通用安全�pȝ��?/span>

技术新�?/span>

本文介绍了如何在AWS上构建流式处理系�l�。包括了诸如Amazon Kinesis �?/span>AWS Lambda�?/span>Kineses S3 connector之类��单的搭配�Ҏ��Q�也介绍�?/span>AWS实现实时分析场景�q�样相对复杂点的�Ҏ��?/span>

http://cdn.oreillystatic.com/en/assets/1/event/144/Building%20a%20scalable%20architecture%20for%20processing%20streaming%20data%20on%20AWS%20Presentation.pdf

本文介绍了怎样使用Spark Testing Base�?/span>Spark Testing Base是一个用Scala�~�写�Q�通过Java调用�?/span>Spark��试框架。本文的样例代码展示了如何隔��L��试逻辑重构Spark代码�Q�同时还通过Java处理了一些臃肿的Scala API�?/span>

http://www.jesse-anderson.com/2016/04/unit-testing-spark-with-java/

Altiscale博客概述了在Spark环境下，构徏thin�?/span>uber jar包的优劣。示范了�?/span>Maven�?/span>SBT分别构徏两种包的情况�?/span>

https://www.altiscale.com/blog/spark-on-hadoop-thin-jars/

LinkedIn介绍了他们的Kafka生态系�l�，生态系�l�包含一个特�D�的Kafka producer�Q�一个�ؓ�?/span>Java客户端提供的REST API�Q�一�?/span>avro模式注册表，以及Gobblin�Q�装载数据到Hadoop的工��P��{�等�?/span>

https://engineering.linkedin.com/blog/2016/04/kafka-ecosystem-at-linkedin

�?/span>Spark Streaming教程介绍了怎样通过twitter4j API拉推文，��Z��标签�q��o�Q�对推文�q�行情感分析�?/span>

https://www.mapr.com/blog/spark-streaming-and-twitter-sentiment-analysis

Apache Kudu�Q�孵化中�Q�是Apache Impala�Q�孵化中�Q�的�l�佳伴��G�Q�因为它能高效地解决�q�泛的分析和有针�Ҏ��的查询。本文描�q�C��两者集成的技术细节，例如Kudu的设计如何保证高效地查询能力�Q�如何通过Impala�?/span>Kudu执行写／更新�Q�删除操作等�{��?/span>

http://blog.cloudera.com/blog/2016/04/how-to-use-impala-and-kudu-together-for-analytic-workloads/

MapR撰文介绍了��?/span>spark-sklearn扩展一个已存在�?/span>scikit-learn模型。文章介�l�了如何透过Airbnb数据集内部徏模，�q�介�l�了如何傍着spark-sklearn�q�行交叉验证�?/span>

https://www.mapr.com/blog/predicting-airbnb-listing-prices-scikit-learn-and-apache-spark

AWS大数据博客写了个如何�?/span>Amazon EMR中��?/span>HBase�?/span>Hive的教�E�。本教程介绍�?/span>HBase�Q�描�q�C��如何�?/span>S3中恢�?/span>HBase表，�C��?/span>Hive�?/span>HBase如何集成�{�等�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx3EGE8Z90LZ9WX/Combine-NoSQL-and-Massively-Parallel-Analytics-Using-Apache-HBase-and-Apache-Hiv

本文描述了�ؓ学生在大数据评��上提供实战经验的挑战。作者经历若�q�次的�P代和选择��g��有了一个好�Ҏ��— Altiscale�?/span>Hadoop-as-a-Service�?/span>

https://www.altiscale.com/blog/hadoop-as-a-service-in-the-classroom/

Cloudera博客的一��客做文章，作者比较了Parquet�?/span>Avro在跨两个数据集的不同处理方式�Q�一个数据集�H?/span>(3�?/span>)、一个数据集�?/span>(103�?/span>)�Q�。在�?/span>Spark�?/span>Spark SQL��试查询�Q�操作后�Q�作者发�?/span>Parquet�?/span>Avro在查询序列化数据斚w��有时表现很类��|��管在大多数情况下查�?/span>Parquet数据的时候更快点�Q�序列化数据更小�Q��?/span>

http://blog.cloudera.com/blog/2016/04/benchmarking-apache-parquet-the-allstate-experience/

本文介绍了如何在CDH�q�样的分布式环境中��?/span>SparkR�Q�尽��?/span>SparkR官方�q�没有支持这�U�方式。借助YARN�?/span>worker本地安装R语言包，job�E�加攚w��就能执行了�?/span>

http://www.nodalpoint.com/sparkr-in-cloudera-hadoop/

很多开源框枉��能执�?/span>MapReduce以及借助更高�U�的�~�程模型完成�c�M��的工作。纵观过去，它们依赖独立�q�行的框�Ӟ��例如MapReduce, Storm�Q�，但是最�q�的某些变化使得�q�一切充满了变数�?/span>Apache Beam�Q�孵化中�Q�更�q�一步地跨越了批处理、流式处理两�U�执行模式，内置更加复杂的计��模型�?/span>

http://www.datanami.com/2016/04/22/apache-beam-emerges-ambitious-goal-unify-big-data-development/

Apache博客发布�?/span>HBase�?/span>HDD�?/span>SSD以及RAMDISK上的写入性能��试比对�?/span>7��系列文章。通过�q�一分析�Q�作者发现�ƈ提议�?/span>HBase�?/span>HDFS上实��C��些未覆盖的功能�?/span>

https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_part

其他新闻

Tom White�Q?/span>“Hadoop权威指南”的作者撰文介�l�他是如何步�?/span>Apache Hadoop�D�堂的。他的早期�A献是�l�着Hadoop�?/span>Amazon Web Services集成展开�Q�而今AWS已成�?/span>Hadoop��目成功的重要部分�?/span>

http://vision.cloudera.com/how-i-got-into-hadoop/

Fluo�Q��ؓApache Accumulo准备的分布式处理引擎�Q�向Apache孵化器提交了孵化甌��?/span>

https://wiki.apache.org/incubator/FluoProposal

Apache Phoenix宣布��在HBaseCon后�D行会议，Apache Phoenix是一�?/span>SQL-on-HBase�pȝ��。该会议只有半天�Q�主题是介绍Phoenix内部情况和用例�?/span>

http://hortonworks.com/blog/announcing-first-annual-phoenixcon-apache-phoenix-user-conference/

产品发布

Apache Metron�Q�构��Z��Hadoop上的安全框架�Q�发布了0.1版�?/span>Hortonworks支撑其作为技术预览版�Q��ƈ撰写本文介绍了如何上手，如何贡献�Q�如何��?/span>Metron UI�{�等�?/span>

http://hortonworks.com/blog/apache-metron-tech-preview-1-come-get/

http://hortonworks.com/blog/apache-metron-use-case-finding-needle-haystack/

Apache NiFi本周发布�?/span>0.6.1版。这是修复了10多个bug后的修复版�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201604.mbox/%3CCALJK9a7yLnFeJ7Z=eU6mOB-DXvo8MHUr=_RshSjZcTbTcAHDZA@mail.gmail.com%3E

Apache Flink本周发布�?/span>1.0.2版。本�ơ发布包括了bug修复�Q?/span>RocksDB环境下的性能提升以及一些文档方面的�q�步�?/span>

http://flink.apache.org/news/2016/04/22/release-1.0.2.html

Amazon发布了新�?/span>Amazon EMR�Q�开始支�?/span>HBase 1.2�?/span>

https://aws.amazon.com/blogs/aws/amazon-emr-update-apache-hbase-1-2-is-now-available/

�z�d��

中国

�?/span>

Rosen 2016-05-03 10:08 发表评论

Rosen — Thu, 21 Apr 2016 07:07:00 GMT

Hadoop周刊 �W?/span> 166 �?/span>

2016�q?/span>4�?/span>17�?/span>

启明星辰——�q�_��和大数据整体�l�编�?nbsp;

Hortonworks在本�?/span>Hadoop�Ƨ洲��C��上有若干爆料�Q�诏�I�了本期整个内容。伴随着骄�h的新�Ҏ��，Apache Storm发布�?/span>1.0.0版。在技术新��L��面，有不��基�?/span>Kafka构徏大规模服务和分布式系�l�测试的文章。如果你错过�?/span>Hadoop��C��Q�那么不用担心，演讲视频已经攑ֈ�了网上�?/span>

技术新�?/span>

Smyte撰文介绍了他们基于事件数据流实时��垃��N��件和诈骗信息的基��设施。最初的事�g处理�pȝ��构徏�?/span>Kafka�?/span>Redis�?/span>Secor以及S3上，��Z��满��规模不断扩张和廉��L��要求�Q�他们把�pȝ��q�移到基于磁盘的�Ҏ��上，使用Redis协议�?/span>RocksDB交互�Q��?/span>Kafka�q�行复制�?/span>

https://medium.com/the-smyte-blog/counting-with-domain-specific-databases-73c660472da

本文�?/span>rsyslog�?/span>Kafka�?/span>AWS �?/span>ELK栈（ElasticSearch�?/span>Logstash�?/span>Kibana�Q�结合，处理诸如反压、规模以及维护方面的问题。本文覆盖了rsyslog集成Kafka以及schema斚w��的技巧，也介�l�了如何�q�行Kafka�?/span>Zookeeper以及AWS中大规模自动分组�?/span>

https://www.bashton.com/blog/2016/elk-on-ark/

Hortonworks撰文介绍�?/span>Apache Atlas以及Apache Range��要引入的数据管理特性。这些特性是�Q�分�c�访问控制、数据有效期�{�略、位�|�特性策略、禁止数据集�l�合、跨�l��g家族�Q�例如从Kafka�?/span>Storm再到Hive的数据跟�t�）�?/span>

http://hortonworks.com/blog/the-next-generation-of-hadoop-based-security-data-governance/

Apache HAWQ �Q�孵化中�Q�是一个基�?/span>Greenplum�?/span>HDFS上提供数据查询的SQL引擎。本文讨��Z��其典型设计以及新版本的诸多改�q�。包括它�?/span>Spark�?/span>MapReduce的区别，�q�有�?/span>Hadoop挑战�l�典MPP设计的内容，以及HAWQ的新设计怎样�l�合MPP和批处理技术进而��其两者兼��?/span>

https://blog.pivotal.io/big-data-pivotal/products/apache-hawq-next-step-in-massively-parallel-processing

Cloudera博客撰文介绍了对Hadoop分布式系�l�进行故障注入、组�|�的��试工具AgenTEST。它能注入网�l�故障（例如丢包�Q�，资源满蝲�Q�例�?/span>CPU�?/span>IO、磁盘空��_��{�等。当��试�|�络分区�Ӟ��可以评估环�Ş�l�网、桥接组�|�等�{��?/span>

http://blog.cloudera.com/blog/2016/04/quality-assurance-at-cloudera-fault-injection-and-elastic-partitioning/

Hortonworks博客展望了将包含新版�?/span>Spark�?/span>Zeppelin�?/span>HDP 2.4.2�?/span>Spark2.0预览版和Zeppelin新特性都��包含在内�?/span>

http://hortonworks.com/blog/apache-spark-apache-zeppelin-whats-coming-in-hdp-2-4-2/

Cask撰文介绍了在Hbase region compaction�q�样�|�见事�g发生的前后，他们是怎样通过长时间测试以评估分布式系�l�正��性的�?/span>

http://blog.cask.co/2016/04/long-running-tests-in-cdap/

本文介绍了如何结�?/span>SparkR与亚马�?/span>EMR�q�行地理�I�间分析的。通过SparkR�?/span>Hive集成�l��g�Q�可以立��d��?/span>S3上的数据映射Hive外部表。从�q�开始，数据��p��直接加蝲到内存中使用R语言分析�Q�很�Ҏ��实现高质量的数据可视化�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx1MECZ47VAV84F/Exploring-Geospatial-Intelligence-using-SparkR-on-Amazon-EMR

MapR�~�写了��?/span>Pig�?/span>Hive分析职业��球大联盟球队水�q�的教程�?/span>Pig用于数据初加工，Hive提供��Z��SQL的数据查询环境。借助Hive ODBC驱动�?/span>Hive服务器，使得微��YExcel也能用于获取和分析数据�?/span>

https://www.mapr.com/blog/using-hive-and-pig-baseball-statistics

SignalFX通过27节点�?/span>Kafka集群每天处理700多亿条消息。只有基于他们积累的大规�?/span>Kafka使用�l�验才能有如此高的量�Q�因此他们共享了不少调试Kafka的技巧，定位告警�Q�例如日志刷新�g�q�增加）�Q�以�?/span>Kafka横向扩展�?/span>

http://www.confluent.io/blog/how-we-monitor-and-run-kafka-at-scale-signalfx

dataArtisan's博客��Z��度量Flink在数据流效率、低延迟、正��性上的能力，专门写了�q�篇文章。�ؓ了证明效率，在高吞吐量的环境下运行了最新的Yahoo!��式基准��试�E�序。在正确性方面，文章�H�出�?/span>Flink事�g判别和处理事�Ӟ��星球大战电媄�q�表做类比）斚w��的优�ѝ��最后，文章描述�?/span>Flink未来版本��Z��内存的查询�Q务�?/span>

http://data-artisans.com/counting-in-streams-a-hierarchy-of-needs/

本教�E�介�l�了怎样�?/span>TCP Socket中的文本数据��{换�ؓSpark��式数据源�?/span>

https://medium.com/@anicolaspp/spark-custom-streaming-sources-e7d52da72e80

本文介绍了在构徏Hadoop的时候怎样防止AWS证书意外提交到补丁或git资源库。除Hadoop本��n外，本文�q�徏议��?/span>“git-secrets”工具防止意外提交讉K��/安全密钥。如果你用的�?/span>Hadoop S3�Q�还推荐了新补丁供评估�?/span>

http://steveloughran.blogspot.co.uk/2016/04/testing-against-s3-and-object-stores.html

Big Data & Brews采访�?/span>MapR�?/span>Ted Dunning�?/span>Jacques Nadeau�?/span>Apache Arrow也在本次采访范围内�?/span>

https://www.youtube.com/watch?v=l3mDDKjDjMk

https://www.youtube.com/watch?v=Xo9CO0a0VJI

其他新闻

DataEngConf最�q�在旧金山召开。本文�ȝ��?/span>Uber�?/span>Stripe�?/span>Microsoft�?/span>Instacart�?/span>Jawbone的发�a�内容。也介绍了会议主�?/span>“数据�U�学在现实世界中是一个��品和工程学科”�?/span>

https://medium.com/@eugmandel/software-engineering-invades-data-science-notes-from-dataengconf-4a3c066b081f#.g2h0duo44

Hortonworks在上周都柏林举行�?/span>Hadoop�Ƨ洲��C��上大攑ּ�彩�?/span>ZDNet报导了这些亮点，其中包括�?/span>Pivotal�Q�已转售�l?/span>HDP�Q�的扩展合作�Q�与Syncosrt的�{售协议，以及Atlas�?/span>Ranger�?/span>Zeppelin�?/span>Metron的技术预览。报��D��介绍�?/span>Hortonworks�?/span>Cloudera�?/span>MapR产品的不同之处�?/span>

http://www.zdnet.com/article/hortonworks-announces-new-alliances-and-releases-hadoop-comes-to-fork-in-road/

Flink 2016��C��在九月于�d国柏林�D行。讨��题征集将于六月末�l�束�?/span>

http://flink.apache.org/news/2016/04/14/flink-forward-announce.html

YouTube上发布了Hadoop都柏林峰会演讲视频。正如预期的那样�Q�这些演讲内�Ҏ��?/span>Hadoop生态系�l�的各个部分�?/span>

https://www.youtube.com/channel/UCAPa-K_rhylDZAUHVxqqsRA/videos?flow=list&live_view=500&view=0&sort=dd

产品发布

Metascope是一个配�?/span>Schedoscope�?/span>Hadoop集群中进行元数据��理的新工具。通过web界面�Q�利用数据沿袭它能洞察大量的数据。也提供��索、内嵌文档�?/span>REST API�{�等功能�?/span>

https://github.com/ottogroup/metascope

Apache HBase 1.2.1于本周发布，�?/span>1.2.0的基��上解决了27个问题。发布声明中重点介绍了四个高优先�U�的问题�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201604.mbox/%3CCAN5cbe7-T5uAYvGRbxw2dfvdbwe5s0nx3vKU8Nt2fzXbKPoQTg@mail.gmail.com%3E

Apache Mahout机器学习库发布了0.12.0版。该版本�?/span>“Samsara”数学环境开始支�?/span>Apache Flink了，�q�且是��^台无关的。发布声明中分��n了与Flink集成、已知问题、项目演�q�计划相关的内容�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201604.mbox/%3CCAOtpBjj5An876PStdn5kMeaF+up-B72WTmCk9j21EXdP=JOCUA@mail.gmail.com%3E

Apache Storm 1.0.0本周发布了。亮点包括性能提升�Q�普遍提�?/span>3倍以上）、新的分布式�~�存API�?/span>nimbus的高可用性、自动反压、动�?/span>worker性能分析�{�等�?/span>

http://storm.apache.org/2016/04/12/storm100-released.html

Apache Kudu�Q�孵化中�Q�本周发布了0.8.0版。本�ơ发布添加了Apache Flume sink、部分功能提升、修复了一�?/span>bug�?/span>

http://getkudu.io/releases/0.8.0/docs/release_notes.html

Cloudbreak本周发布�?/span>1.2版，它�ؓ云环境提�?/span>Hadoop集群Docker。新�Ҏ��包括支�?/span>OpenStack以及��定义服务器提供配�|�脚本�?/span>

http://hortonworks.com/blog/announcing-cloudbreak-1-2/

Cloudera发布�?/span>Cloudera Enterprise 5.4.10�Q�内�|�了Flume�?/span>Hadoop�?/span>HBase�?/span>Hive�?/span>Impala�{�组件�?/span>

http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Cloudera-Enterprise-5-4-10-Released/m-p/39790#U39790

Presto Accumulo是个新项目，�?/span>Accumulo��d��数据提供�?/span>Presto�q�接器�?/span>

https://github.com/bloomberg/presto-accumulo

�z�d��

中国

�?/span>

Rosen 2016-04-21 15:07 发表评论

Rosen — Thu, 14 Apr 2016 10:02:00 GMT

Hadoop周刊

�W?165 �?2016�q?�?0�?

启明星辰——�q�_��和大数据整体�l�编�?/strong>

本周�Q�包�?/span>LinkedIn �?/span>Airbnb新开源项目在内的��C��产品�q�行了重大版本发布。本期技术部分与��式处理有关——Spark�?/span>Flink�?/span>Kafka�{�等�Q�新闻部分是关于Spark Summit �?/span>HbaseCon的会议议�E��?/span>

技�?/span>

Zalando发表了他们是如何选择Apache Flink作�ؓ��式处理框架的文章。该文章阐述了对评�h标准�q�行验证后得出的�l�论�Q�阐明了选择Apache Flink的主�?/span>—在高吞吐量的情况下依然能保持低�g�q�，真正的流式处理，开发�h员支持�?/span>

https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/

Cloudera博客刊登了来�?/span>Wargaming.net的文章，通过本文可了解到他们如何通过Kafka�?/span>HBase�?/span>Drools�?/span>Spark构徏实时处理基础设施的。另外，在数据流�E�方面，他们介绍了如何对HBase的检索和序列化�?/span>HBase�?/span>Spark之间的数据本地化以及Spark计算斚w��的优化措施�?/span>

http://blog.cloudera.com/blog/2016/04/inside-wargamings-data-driven-real-time-rules-engine/

InfoQ发布了大规模��式处理—SMACK�Q?/span>Spark�?/span>Mesos�?/span>Akka�?/span>Cassandra以及 Kafka�Q�栈的介�l�视频。讨��Z��Z��?/span>SMACK栈在处理同样问题的时候比Lambda架构更简单�?/span>

http://www.infoq.com/presentations/stream-analytics-scalability

Confluent“日志压羃”�p�d��博文又有更新�Q�介�l�了Kafka��目三月份发生的事情。有不少令�h��x��的开发内容，包括机架感知�?/span>Kerberos支持、基于时间烦引方面的�q�展。以及不��你�Q�我也是�Q�没有时间持�l�关注的最新研发成果�?/span>

http://www.confluent.io/blog/log-compaction-highlights-in-the-kafka-and-stream-processing-community-april-2016

Apache Flink 1.0引入了新的复杂事件处理（CEP�Q�库。啰嗦几句，CEP提供了一�U�检��事件模式的�Ҏ��。本文借助传感器从数据中心服务器上攉��数据�Q�运用一�U�可能的异常��用例，诠释�?/span>Flink�?/span>CEP模式API �?/span>

http://flink.apache.org/news/2016/04/06/cep-monitoring.html

Genome Analysis Toolkit �Q?/span>GATK�Q�最�q�宣布，下一个版本（当前�?/span>alpha�Q�将支持Apache Spark。本文简要介�l�了工具��ƈ展示了怎样通过Spark来检��重�?/span>DNA片段的�?/span>

http://blog.cloudera.com/blog/2016/04/genome-analysis-toolkit-now-using-apache-spark-for-data-processing/

InfoWorld�l�D��?/span>Spark2.0关于�l�构化流式处理方面的计划。微批处理将依然延箋�Q�还有些新特性，例如无限数据帧（Infinite DataFrames�Q�、一��的重复查询支持�?/span>

http://www.infoworld.com/article/3052924/analytics/what-sparks-structured-streaming-really-means.html

AWS大数据博客发布了一��通过存储�?/span>AWS Key Management Service �Q?/span>KMS�Q�中的加密密钥加载数据到S3�?/span>Redshift的文章。除了描�q�所需步骤�Q�本文还介绍了如何在AWS S3中通过KMS密钥加密数据�?/span>

http://blogs.aws.amazon.com/bigdata/post/Tx2Q3ZBOZO9DHVQ/Encrypt-Your-Amazon-Redshift-Loads-with-Amazon-S3-and-AWS-KMS

Confluent博客介绍了如何��?/span>Kafka Connect �?/span> Kafka Streams �~�写非凡�?/span>“hello world”�E�序。更��切地说�Q�范例程序从IRC拉维基百�U�数据，�q�解析消息、进行多斚w��的统计计��。本文还用了若干�E�序展示了整个实现过�E��?/span>

http://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams

本文�?/span>Postgres �?/span> Cassandra转换��单的模式�Q?/span>schemas�Q�，�q�描�q�C��主要的差�?/span>—复制、数据类型（Cassandra不支�?/span>JSON�Q�、主键、最�l�以一致性�?/span>

http://neovintage.org/2016/04/07/data-modeling-in-cassandra-from-a-postgres-perspective/

新闻

ESG博客报导了最�q?/span>Strata+Hadoop World大会的情��c��ƈ有些重点��x��Q�例�?/span>Spark的良好势头、机器学习、云服务�?/span>

http://blog.esg-global.com/riding-high-at-stratahadoop-world

InformationWeek也报��g��Strata大会�Q�关注了MapR�?/span>Pivotal的关灯片、�h工智能等�?/span>

http://www.informationweek.com/big-data/ai-public-data-sets-real-time-strata-+-hadoop-keynote-sampling/d/d-id/1324943?

Spark Summit 2016议程敲定�Q�将�?/span>6�?/span>6-8日在旧金�׃�D行。会议将有两天展开五个方向的讨论�?/span>

https://databricks.com/blog/2016/04/04/agenda-announced-for-sparksummit-2016-in-san-francisco.html

��布斯采访了Cloudera CEO Tom Reilly�Q�他讨论了公司的机遇、竞争性市场、上市计划等�?/span>

http://www.forbes.com/sites/roberthof/2016/04/06/ceo-tom-reilly-makes-the-case-for-cloudera-and-its-ipo/

Datanami撰文��正在崛��L��Apache Kafka作�ؓ��式处理的支柱。文章还采访�?/span>Confluent联合创始人兼CTO Neha Narkhede�Q�坊间她表示最�q�将推出Kafka Connect �?/span> Kafka Streams�?/span>

http://www.datanami.com/2016/04/06/real-time-rise-apache-kafka/

HBaseCon��于5�?/span>24日在旧金山召开�Q�最�q�议�E�才正式宣布。在三个方向上，��有20个以上的议题要讨论�?/span>

http://blog.cloudera.com/blog/2016/04/hbasecon-2016-speaker-lineup-announced/

发布

Apache HBase 0.98.18 �?/span>1.1.4最�q�都发布了�?/span>1.1.4上有包括九个或正��性在内的若干修复�?/span>HBase 0.98.18��答�{�的仅解决了50个问题（bug、改善两个新�Ҏ��）�?/span>

http://mail-archives.apache.org/mod_mbox/hbase-user/201603.mbox/%3CCANZa%3DGu-mAxKEtfoRjctHcE0KD7z52oE010Fgsf6AMmW2tDZLA%40mail.gmail.com%3E
http://mail-archives.apache.org/mod_mbox/hbase-user/201603.mbox/%3CCA%2BRK%3D_CtZ1L07nS6Og2ekfVwet0qTE7jw-bmyD2pp5UPweUehQ%40mail.gmail.com%3E

Apache Lens发布�?/span>2.5.0-beta�Q�作为统一分析接口�Q�它已经支持Hadoop生态系�l�的执行引擎数据存储了。本�ơ发布解决了87��，主要�?/span>bug修复和实现新功能�?/span>

http://mail-archives.us.apache.org/mod_mbox/www-announce/201604.mbox/%3CCAL3kmZj60kpopRPpOVEs9o7oTg7YuaC_=c8zncBeMyUESrZsmQ@mail.gmail.com%3E

Airbnb 开源了 Caravel�Q�数据探索系�l�（数据可视化��^収ͼ��?/span>Caravel支持多种在商业��品上才能看到的特性，能够�q�接��C�Q意只要支�?/span>SQL方言的系�l�。尤其它支持面向Druid的实时分析�?/span>

https://medium.com/airbnb-engineering/caravel-airbnb-s-data-exploration-platform-15a72aa610e5

MapR 宣布支持Apache Drill 1.6作�ؓ他们的分布式�pȝ��。比较有亮点的发布有MapR-DB新存储插件、新SQL�H�口函数支持以及端对端安全。在�|�页介绍部分�Q�有些��?/span>MapR-DB API�?/span>�?/span>数据�q��?/span>�q?/span>Drill查询的例子�?/span>

https://www.mapr.com/blog/apache-drill-16-mapr-converged-platform-gearing-new-generation-stack-json-enabled-big-data

Apache Flink发布了修�?/span>bug后的1.0.x。这�ơ发布解决了23个问题，推荐所�?/span>1.0.0的用户升�U��?/span>

http://flink.apache.org/news/2016/04/06/release-1.0.1.html

Cloudera Enterprise 5.7发布附带�?/span>Spark�?/span>HBase�?/span>Impala�?/span>Kafka�{�组件版本的升��。本�ơ发布的亮点包括�?/span>Cloudera Labs 新鲜推荐�?/span>Hive-on-Spark�?/span>HBase-Spark�?/span>Impala性能重要提升�Q�支�?/span>SSD �?/span>HBase WAL�?/span>

http://blog.cloudera.com/blog/2016/04/cloudera-enterprise-5-7-is-released/

Apache Tajo�Q�构建在Hadoop上的数据仓库�pȝ��Q�发布了0.11.2版。新版本支持�?/span>Kerberos�Q�修复了ORC表对Hive的支持等�?/span>

http://tajo.apache.org/releases/0.11.2/announcement.html

LinkedIn 开源了 Dr. Elephant�Q�里面的工具能诊�?/span>Hadoop�?/span>Spark��d��的性能问题。基�?/span>metrics�?/span>YARN资源��理器收集已完成��d��数据�Q?/span>Dr. Elephant评估后生成诊断报表，内容包括数据错位�?/span>GC开销�{��?/span>LinkedIn宣称借助它能解决80%的问题�?/span>

https://engineering.linkedin.com/blog/2016/04/dr-elephant-open-source-self-serve-performance-tuning-hadoop-spark

�z�d��

中国

�?/span>

Rosen 2016-04-14 18:02 发表评论

Rosen — Fri, 09 Jul 2010 02:19:00 GMT
     摘要: 很多开发者对 hibernate 性能表示�|�疑�Q�下一�ơ技术革��C��是什么呢�Q�——对象数据库
�q�篇文章是开源面向对象数据库 db4o 之旅 �p�d��文章的第 4 部分�Q�介�l�面向对象数据库 db4o �?db4o Replication System(dRS) —�?db4o 复制�pȝ��Q��ƈ对其如何同步 Oracle 数据库进行分析�?nbsp; 阅读全文

Rosen 2010-07-09 10:19 发表评论

Rosen — Tue, 22 Jun 2010 07:27:00 GMT
     摘要: �q�是做个实际的SoftReference��试吧�?nbsp; 阅读全文

Rosen 2010-06-22 15:27 发表评论

Rosen — Sun, 13 Jun 2010 08:13:00 GMT
     摘要: 在��^时工作过�E�中�Q�有时会遇到OutOfMemoryError�Q�我们知道遇到Error一般表明程序存在着严重问题�Q�可能是��N��性的。所以找出是什么原因造成OutOfMemoryError非常重要。现在向大家引荐Eclipse Memory Analyzer tool(MAT)�Q�来化解我们遇到的难题�?nbsp; 阅读全文

Rosen 2010-06-13 16:13 发表评论

Rosen — Fri, 21 May 2010 12:59:00 GMT
     摘要: 在��^时工作过�E�中�Q�有时会遇到OutOfMemoryError�Q�我们知道遇到Error一般表明程序存在着严重问题�Q�可能是��N��性的。所以找出是什么原因造成OutOfMemoryError非常重要。现在向大家引荐Eclipse Memory Analyzer tool(MAT)�Q�来化解我们遇到的难题�?nbsp; 阅读全文

Rosen 2010-05-21 20:59 发表评论