In two weeks we’ll present a paper on the Dynamo technology at SOSP, the prestigious biannual
Operating Systems conference. Dynamo is internal technology developed at
Amazon to address the need for an incrementally scalable,
highly-available key-value storage system. The technology is designed to
give its users the ability to trade-off cost, consistency, durability
and performance, while maintaining high-availability.
Let me emphasize the internal technology part before it gets
misunderstood: Dynamo is not directly exposed externally as a web
service; however, Dynamo and similar Amazon technologies are
used to power parts of our Amazon Web Services, such as S3.
We submitted the technology for publication in SOSP because many of the
techniques used in Dynamo originate in the operating systems and
distributed systems research of the past years; DHTs, consistent
hashing, versioning, vector clocks, quorum, anti-entropy based recovery,
etc. As far as I know Dynamo is the first production system to use the
synthesis of all these techniques, and there are quite a few lessons
learned from doing so. The paper is mainly about these lessons.
We are extremely fortunate that the paper was selected for publication
in SOSP; only a very few true production systems have made it into the
conference and as such it is a recognition of the quality of the work
that went into building a real incrementally scalable storage system in
which the most important properties can be appropriately configured.
Dynamo is representative of a lot of the work that we are doing at
Amazon; we continuously develop cutting edge technologies using recent
research, and in many cases do the research ourselves. Much of the
engineering work at Amazon, whether it is in infrastructure, distributed
systems, workflow, rendering, search, digital, similarities, supply
chain, shipping or any of the other systems, is equally highly advanced.
The official reference for the paper is:
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan
Kakulapati, Avinash Lakshman, Alex Pilchin, Swami Sivasubramanian, Peter
Vosshall and Werner Vogels, “Dynamo: Amazon's Highly Available
Key-Value Store”, in the Proceedings of the 21st ACM Symposium on
Operating Systems Principles, Stevenson, WA, October 2007.
A pdf version is available here.
You can also read the full
online version.
The text of the paper is copyright of the ACM and as such the following
statement applies:
from http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html