SSTable的定ä¹?/h2>要解释这个术语的真正含义åQŒæœ€å¥½çš„æ–ÒŽ³•ž®±æ˜¯ä»Žå®ƒçš„出处找½{”案åQŒæ‰€ä»¥é‡æ–°ç¿»å¼€BigTable的论文。在˜q™ç¯‡è®ºæ–‡ä¸­ï¼Œæœ€åˆå¯¹SSTable是这么描˜q°çš„åQˆç¬¬ä¸‰é¡µæœ«å’Œ½W¬å››™åµåˆåQ‰ï¼š

SSTable

The Google SSTable file format is used internally to store Bigtable data. An SSTable provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. Operations are provided to look up the value associated with a specified key, and to iterate over all key/value pairs in a specified key range. Internally, each SSTable contains a sequence of blocks (typically each block is 64KB in size, but this is configurable). A block index (stored at the end of the SSTable) is used to locate blocks; the index is loaded into memory when the SSTable is opened. A lookup can be performed with a single disk seek: we first find the appropriate block by performing a binary search in the in-memory index, and then reading the appropriate block from disk. Optionally, an SSTable can be completely mapped into memory, which allows us to perform lookups and scans without touching disk.

½Ž€å•çš„非直译：
SSTable是Bigtable内部用于数据的文件格式，它的格式为文件本íw«å°±æ˜¯ä¸€ä¸ªæŽ’序的、不可变的、持久的Key/Value对MapåQŒå…¶ä¸­Keyå’Œvalue都可以是ä»ÀL„çš„byte字符丌Ӏ‚ä‹É用Key来查找ValueåQŒæˆ–通过¾l™å®šKey范围遍历所有的Key/Value寏V€‚每个SSTable包含一¾pÕdˆ—çš„BlockåQˆä¸€èˆ¬Block大小ä¸?4KBåQŒä½†æ˜¯å®ƒæ˜¯å¯é…ç½®çš„）åQŒåœ¨SSTable的末ž®¾æ˜¯Block索引åQŒç”¨äºŽå®šä½BlockåQŒè¿™äº›çƒ¦å¼•åœ¨SSTable打开时被加蝲到内存中åQŒåœ¨æŸ¥æ‰¾æ—‰™¦–先从内存中的索引二分查找扑ֈ°BlockåQŒç„¶åŽä¸€‹Æ¡ç£ç›˜å¯»é“即可读取到相应的Block。还有一¿Uæ–¹æ¡ˆæ˜¯ž®†è¿™ä¸ªSSTable加蝲到内存中åQŒä»Žè€Œåœ¨æŸ¥æ‰¾å’Œæ‰«æä¸­ä¸éœ€è¦è¯»å–磁盘ã€?/span>

˜q™ä¸ªè²Œä¼¼ž®±æ˜¯HFile½W¬ä¸€ä¸ªç‰ˆæœ¬çš„格式么，贴张图感受一下：

在HBase使用˜q‡ç¨‹ä¸­ï¼Œå¯¹è¿™ä¸ªç‰ˆæœ¬çš„HFile遇到以下一些问题（参è€?a >˜q™é‡ŒåQ‰ï¼š
1. 解析时内存ä‹É用量比较高ã€?br />2. Bloom Filterå’ŒBlock索引会变的很大，而媄响启动性能。具体的åQŒBloom Filter可以增长åˆ?00MB每个HFileåQŒè€ŒBlock索引可以增长åˆ?00MBåQŒå¦‚果一个HRegionServer中有20个HRegionåQŒåˆ™ä»–们分别能增长到2GBå’?GB的大ž®ã€‚HRegion需要在打开æ—Óž¼Œéœ€è¦åŠ è½½æ‰€æœ‰çš„Block索引到内存中åQŒå› è€Œåª„响启动性能åQ›è€Œåœ¨½W¬ä¸€‹Æ¡Requestæ—Óž¼Œéœ€è¦å°†æ•´ä¸ªBloom Filter加蝲到内存中åQŒå†å¼€å§‹æŸ¥æ‰¾ï¼Œå› è€ŒBloom Filter太大会媄响第一‹Æ¡è¯·æ±‚的延迟ã€?br />而HFile在版æœ?中对˜q™äº›é—®é¢˜åšäº†ä¸€äº›ä¼˜åŒ–，具体会在HFile解析时详¾l†è¯´æ˜Žã€?br />

SSTable作äؓ存储使用

¾l§ç®‹BigTable的论文往下走åQŒåœ¨5.3 Tablet Servingž®èŠ‚中这样写道（½W?™åµï¼‰åQ?br />

Tablet Serving

Updates are committed to a commit log that stores redo records. Of these updates, the recently committed ones are stored in memory in a sorted buffer called a memtable; the older updates are stored in a sequence of SSTables. To recover a tablet, a tablet server reads its metadata from the METADATA table. This metadata contains the list of SSTables that comprise a tablet and a set of a redo points, which are pointers into any commit logs that may contain data for the tablet. The server reads the indices of the SSTables into memory and reconstructs the memtable by applying all of the updates that have committed since the redo points.

When a write operation arrives at a tablet server, the server checks that it is well-formed, and that the sender is authorized to perform the mutation. Authorization is performed by reading the list of permitted writers from a Chubby file (which is almost always a hit in the Chubby client cache). A valid mutation is written to the commit log. Group commit is used to improve the throughput of lots of small mutations [13, 16]. After the write has been committed, its contents are inserted into the memtable.

When a read operation arrives at a tablet server, it is similarly checked for well-formedness and proper authorization. A valid read operation is executed on a merged view of the sequence of SSTables and the memtable. Since the SSTables and the memtable are lexicographically sorted data structures, the merged view can be formed efficiently.

Incoming read and write operations can continue while tablets are split and merged.

½W¬ä¸€ŒDµå’Œ½W¬ä¸‰ŒD늮€å•æ˜qŽÍ¼Œéžç¿»è¯‘：
在新数据写入æ—Óž¼Œ˜q™ä¸ªæ“ä½œé¦–先提交到日志中作äØ“redo¾Uªå½•åQŒæœ€˜q‘的数据存储在内存的排序¾~“å­˜memtable中；旧的数据存储在一¾pÕdˆ—çš„SSTable 中。在recover中，tablet server从METADATA表中è¯Õd–metadataåQŒmetadata包含了组成Tablet的所有SSTableåQˆçºªå½•äº†˜q™äº›SSTableçš„å…ƒ 数据信息åQŒå¦‚SSTable的位¾|®ã€StartKey、EndKey½{‰ï¼‰ä»¥åŠä¸€¾pÕdˆ—日志中的redo炏V€‚Tablet Serverè¯Õd–SSTable的烦引到内存åQŒåƈreplay˜q™äº›redo点之后的更新来重构memtableã€?br />在读æ—Óž¼Œå®Œæˆæ ¼å¼ã€æŽˆæƒç­‰‹‚€æŸ¥åŽåQŒè¯»ä¼šåŒæ—¶è¯»å–SSTable、memtableåQˆHBase中还包含了BlockCache中的数据åQ‰åƈ合åƈ他们的结果，ç”׃ºŽSSTableå’Œmemtable都是字典序排列，因而合òq¶æ“ä½œå¯ä»¥å¾ˆé«˜æ•ˆå®Œæˆã€?br />

SSTable在Compaction˜q‡ç¨‹ä¸­çš„使用

在BigTable论文5.4 Compactionž®èŠ‚中是˜q™æ ·è¯´çš„åQ?br />

Compaction

As write operations execute, the size of the memtable increases. When the memtable size reaches a threshold, the memtable is frozen, a new memtable is created, and the frozen memtable is converted to an SSTable and written to GFS. This minor compaction process has two goals: it shrinks the memory usage of the tablet server, and it reduces the amount of data that has to be read from the commit log during recovery if this server dies. Incoming read and write operations can continue while compactions occur.

Every minor compaction creates a new SSTable. If this behavior continued unchecked, read operations might need to merge updates from an arbitrary number of SSTables. Instead, we bound the number of such files by periodically executing a merging compaction in the background. A merging compaction reads the contents of a few SSTables and the memtable, and writes out a new SSTable. The input SSTables and memtable can be discarded as soon as the compaction has finished.

A merging compaction that rewrites all SSTables into exactly one SSTable is called a major compaction. SSTables produced by non-major compactions can contain special deletion entries that suppress deleted data in older SSTables that are still live. A major compaction, on the other hand, produces an SSTable that contains no deletion information or deleted data. Bigtable cycles through all of its tablets and regularly applies major compactions to them. These major compactions allow Bigtable to reclaim resources used by deleted data, and also allow it to ensure that deleted data disappears from the system in a timely fashion, which is important for services that store sensitive data.

随着memtable大小增加åˆîC¸€ä¸ªé˜€å€û|¼Œ˜q™ä¸ªmemtable会被å†ÖM½è€Œåˆ›å»ÞZ¸€ä¸ªæ–°çš„memtable以供使用åQŒè€Œæ—§çš„memtable会è{换成一个SSTable而写道GFS中，˜q™ä¸ª˜q‡ç¨‹å«åšminor compaction。这个minor compaction可以减少内存使用量，òq¶å¯ä»¥å‡ž®‘日志大ž®ï¼Œå› äؓ持久化后的数据可以从日志中删除ã€?/span>在minor compaction˜q‡ç¨‹ä¸­ï¼Œå¯ä»¥¾l§ç®‹å¤„理è¯Õd†™è¯äh±‚ã€?br />每次minor compaction会生成新的SSTableæ–‡äšgåQŒå¦‚æžœSSTableæ–‡äšg数量增加åQŒåˆ™ä¼šåª„响读的性能åQŒå› è€Œæ¯‹Æ¡è¯»éƒ½éœ€è¦è¯»å–所有SSTableæ–‡äšgåQŒç„¶åŽåˆòq¶ç»“果，因而对SSTableæ–‡äšg个数需要有上限åQŒåƈ且时不时的需要在后台做merging compactionåQŒè¿™ä¸ªmerging compactionè¯Õd–一些SSTableæ–‡äšgå’Œmemtable的内容，òq¶å°†ä»–们合åƈ写入一个新的SSTable中。当˜q™ä¸ª˜q‡ç¨‹å®ŒæˆåŽï¼Œ˜q™äº›æºSSTableå’Œmemtablež®±å¯ä»¥è¢«åˆ é™¤äº†ã€?br />如果一个merging compaction是合òq¶æ‰€æœ‰SSTableåˆîC¸€ä¸ªSSTableåQŒåˆ™˜q™ä¸ª˜q‡ç¨‹¿U°åšmajor compaction。一‹Æ¡major compaction会将mark成删除的信息、数据删除，而其他两‹Æ¡compaction则会保留˜q™äº›ä¿¡æ¯ã€æ•°æ®ï¼ˆmarkçš„åŞ式）。Bigtable会时不时的扫描所有的TabletåQŒåƈ对它们做major compaction。这个major compaction可以ž®†éœ€è¦åˆ é™¤çš„数据真正的删除从而节省空é—ß_¼Œòq¶ä¿æŒç³»¾lŸä¸€è‡´æ€§ã€?/span>

SSTableçš„localityå’ŒIn Memory

在Bigtable中，它的本地性是由Locality group来定义的åQŒå³å¤šä¸ªcolumn family可以¾l„合åˆîC¸€ä¸ªlocality group中，在同一个Tablet中，使用单独的SSTable存储˜q™äº›åœ¨åŒä¸€ä¸ªlocality groupçš„column family。HBase把这个模型简化了åQŒå³æ¯ä¸ªcolumn family在每个HRegion都ä‹É用单独的HFile存储åQŒHFile没有locality group的概念，或者一个column familyž®±æ˜¯ä¸€ä¸ªlocality groupã€?/span>

在Bigtable中，˜q˜å¯ä»¥æ”¯æŒåœ¨locality group¾U§åˆ«è®„¡½®æ˜¯å¦ž®†æ‰€æœ‰è¿™ä¸ªlocality group的数据加载到内存中，在HBase中通过column family定义时设¾|®ã€‚这个内存加载采用åšg时加载，主要应用于一些小的column familyåQŒåƈ且经常被用到的，从而提升读的性能åQŒå› è€Œè¿™æ ·å°±ä¸éœ€è¦å†ä»Žç£ç›˜ä¸­è¯Õd–了ã€?/span>

SSTable压羃

Bigtable的压¾~©æ˜¯åŸÞZºŽlocality group¾U§åˆ«åQ?br />

Compression

Clients can control whether or not the SSTables for a locality group are compressed, and if so, which compression format is used. The user-specified compression format is applied to each SSTable block (whose size is controllable via a locality group specific tuning parameter). Although we lose some space by compressing each block separately, we benefit in that small portions of an SSTable can be read without decompressing the entire file. Many clients use a two-pass custom compression scheme. The first pass uses Bentley and McIlroy’s scheme [6], which compresses long common strings across a large window. The second pass uses a fast compression algorithm that looks for repetitions in a small 16 KB window of the data. Both compression passes are very fast—they encode at 100–200 MB/s, and decode at 400–1000 MB/s on modern machines.

Bigtable的压¾~©ä»¥SSTable中的一个Block为单位，虽然每个Block为压¾~©å•ä½æŸå¤×ƒ¸€äº›ç©ºé—ß_¼Œä½†æ˜¯é‡‡ç”¨˜q™ç§æ–¹å¼åQŒæˆ‘们可以以Block为单位读取、解压、分析，而不是每‹Æ¡ä»¥ä¸€ä¸?#8220;å¤?#8221;çš„SSTable为单位读取、解压、分析ã€?/span>

SSTable的读¾~“å­˜

ä¸ÞZº†æå‡è¯Èš„性能åQŒBigtable采用两层¾~“存机制åQ?br />

Caching for read performance

To improve read performance, tablet servers use two levels of caching. The Scan Cache is a higher-level cache that caches the key-value pairs returned by the SSTable interface to the tablet server code. The Block Cache is a lower-level cache that caches SSTables blocks that were read from GFS. The Scan Cache is most useful for applications that tend to read the same data repeatedly. The Block Cache is useful for applications that tend to read data that is close to the data they recently read (e.g., sequential reads, or random reads of different columns in the same locality group within a hot row).

两层¾~“存分别是：
1. High LevelåQŒç¼“存从SSTableè¯Õd–çš„Key/Value寏V€‚提升那些們֐‘重复的读取相同的数据的操作（引用局部性原理）ã€?br />2. Low LevelåQŒBlockCacheåQŒç¼“å­˜SSTable中的Block。提升那些們֐‘于读取相˜q‘数据的操作ã€?br />

Bloom Filter

前文有提到Bigtable采用合åƈ读，即需要读取每个SSTable中的相关数据åQŒåƈ合åƈ成一个结果返回，然而每‹Æ¡è¯»éƒ½éœ€è¦è¯»å–所有SSTableåQŒè‡ªç„¶ä¼šè€—费性能åQŒå› è€Œå¼•å…¥äº†Bloom FilteråQŒå®ƒå¯ä»¥å¾ˆå¿«é€Ÿçš„扑ֈ°ä¸€ä¸ªRowKey不在某个SSTable中的事实åQˆæ³¨åQšå˜q‡æ¥åˆ™ä¸æˆç«‹åQ‰ã€?br />

Bloom Filter

As described in Section 5.3, a read operation has to read from all SSTables that make up the state of a tablet. If these SSTables are not in memory, we may end up doing many disk accesses. We reduce the number of accesses by allowing clients to specify that Bloom fil- ters [7] should be created for SSTables in a particu- lar locality group. A Bloom filter allows us to ask whether an SSTable might contain any data for a spec- ified row/column pair. For certain applications, a small amount of tablet server memory used for storing Bloom filters drastically reduces the number of disk seeks re- quired for read operations. Our use of Bloom filters also implies that most lookups for non-existent rows or columns do not need to touch disk.

SSTable设计成Immutable的好å¤?/h2>在SSTable定义中就有提到SSTable是一个Immutableçš„order mapåQŒè¿™ä¸ªImmutable的设计可以让¾pȝ»Ÿ½Ž€å•å¾ˆå¤šï¼š

Exploiting Immutability

Besides the SSTable caches, various other parts of the Bigtable system have been simplified by the fact that all of the SSTables that we generate are immutable. For example, we do not need any synchronization of accesses to the file system when reading from SSTables. As a result, concurrency control over rows can be implemented very efficiently. The only mutable data structure that is accessed by both reads and writes is the memtable. To reduce contention during reads of the memtable, we make each memtable row copy-on-write and allow reads and writes to proceed in parallel.

Since SSTables are immutable, the problem of permanently removing deleted data is transformed to garbage collecting obsolete SSTables. Each tablet’s SSTables are registered in the METADATA table. The master removes obsolete SSTables as a mark-and-sweep garbage collection [25] over the set of SSTables, where the METADATA table contains the set of roots.

Finally, the immutability of SSTables enables us to split tablets quickly. Instead of generating a new set of SSTables for each child tablet, we let the child tablets share the SSTables of the parent tablet.

关于Immutable的优ç‚ÒŽœ‰ä»¥ä¸‹å‡ ç‚¹åQ?/span>
1. 在读SSTable是不需要同步。读写同步只需要在memtable中处理，ä¸ÞZº†å‡å°‘memtable的读写竞争，Bigtablež®†memtableçš„row设计成copy-on-writeåQŒä»Žè€Œè¯»å†™å¯ä»¥åŒæ—¶è¿›è¡Œã€?/span>
2. æ°æ€¹…的移除数据è{变äØ“SSTableçš„Garbage Collect。每个Tablet中的SSTable在METADATA表中有注册，master使用mark-and-sweep½Ž—法ž®†SSTable在GC˜q‡ç¨‹ä¸­ç§»é™¤ã€?/span>
3. 可以让Tablet Split˜q‡ç¨‹å˜çš„高效åQŒæˆ‘们不需要äؓ每个子Tablet创徏新的SSTableåQŒè€Œæ˜¯å¯ä»¥å…׃ínçˆ?/span>Tabletçš„SSTableã€?/span>

DLevin 2015-09-25 01:35 发表评论

]]>