http://incubator.apache.org/kafka/design.html
1.
Why we built this
asd（activity stream data）數(shù)據(jù)是任何網(wǎng)站的一部分，反映網(wǎng)站使用情況，如：那些內(nèi)容被搜索、展示。通常，此部分?jǐn)?shù)據(jù)被以log方式記錄在文件，然后定期的整合和分析。od（operation data）是關(guān)于機器性能數(shù)據(jù)，和其它不同途徑整合的操作數(shù)據(jù)。
在近幾年，asd和od變成一個網(wǎng)站重要的一部分，更復(fù)雜的基礎(chǔ)設(shè)施是必須的。
數(shù)據(jù)特點：
a、大吞吐量的不變的ad，對實時計算是一個挑戰(zhàn)，會很容易超過10倍or100倍。
b、傳統(tǒng)的記錄log方式是respectable and scalable方式去支持離線處理，但是延遲太高。
Kafka is intended to be a single queuing platform that can support both offline and online use cases.
2.Major Design Elements
There is a small number of major design decisions that make Kafka different from most other messaging systems:
- Kafka is designed for persistent messages as the common case；消息持久
- Throughput rather than features are the primary design constraint；吞吐量是第一要求
- State about what has been consumed is maintained as part of the consumer not the server；狀態(tài)由客戶端維護(hù)
- Kafka is explicitly distributed. It is assumed that producers, brokers, and consumers are all spread over multiple machines；必須是分布式
3.Basics
Messages are the fundamental unit of communication；
Messages are published to a topic by a producer which means they are physically sent to a server acting as a broker，消息被生產(chǎn)者發(fā)布到一個topic，意味著物理的發(fā)送消息到broker；
多個consumer訂閱一個topic，則此topic的每個消息都會被分發(fā)到每個consumer；
kafka是分布式：producer、broker、consumer，均可以由集群的多臺機器組成，相互協(xié)作 a logic group；
屬于同一個consumer group的每一個consumer process，每個消息能準(zhǔn)確的由其中的一個process消費；
A more common case in our own usage is that we have multiple logical consumer groups, each consisting of a cluster of consuming machines that act as a logical whole.
kafka不管一個topic有多少個consumer，其消息僅會存儲一份。
4.
Message Persistence and Caching
4.1 Don't fear the filesystem !
kafka完全依賴文件系統(tǒng)去存儲和cache消息；
大家通常對磁盤的直覺是'很慢'，則使人們對持久化結(jié)構(gòu)，是否能提供有競爭力的性能表示懷疑；實際上，磁盤到底有多慢或多塊，完全取決于如何使用磁盤，
a properly designed disk structure can often be as fast as the network.
http://baike.baidu.com/view/969385.htm raid-5
http://www.china001.com/show_hdr.php?xname=PPDDMV0&dname=66IP341&xpos=172 磁盤種類
磁盤順序讀寫的性能非常高，
linear writes on a 6 7200rpm SATA RAID-5 array is about 300MB/sec；These linear reads and writes are the most predictable of all usage patterns, and hence the one detected and optimized best by the operating system using read-ahead and write-behind techniques。順序讀寫是最可預(yù)見的模式，因此操作系統(tǒng)通過read-head和write-behind技術(shù)去優(yōu)化。
現(xiàn)代操作系統(tǒng)，用mem作為disk的cache；Any modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and writes will go through this unified cache.
Jvm：a、對象的內(nèi)存開銷是非常大的，通常是數(shù)據(jù)存儲的2倍；b、當(dāng)heap數(shù)據(jù)增大時，gc代價越來越大；
As a result of these factors using the filesystem and relying on pagecache is superior to maintaining an in-memory cache or other structure。依賴文件系統(tǒng)和pagecache是優(yōu)于mem cahce或其它結(jié)構(gòu)的。
數(shù)據(jù)壓縮，Doing so will result in a cache of up to 28-30GB on a 32GB machine without GC penalties.
This suggests a design which is very simple: maintain as much as possible in-memory and flush to the filesystem only when necessary. 盡可能的維持在內(nèi)存中，僅當(dāng)必須時寫回到文件系統(tǒng).
當(dāng)數(shù)據(jù)被立即寫回到持久化的文件，而未調(diào)用flush，其意味著數(shù)據(jù)僅被寫入到os pagecahe，在后續(xù)某個時間由os flush。Then we add a configuration driven flush policy to allow the user of the system to control how often data is flushed to the physical disk (every N messages or every M seconds) to put a bound on the amount of data "at risk" in the event of a hard crash. 提供flush策略。
4.2 Constant Time Suffices
The persistent data structure used in messaging systems metadata is often a BTree. BTrees are the most versatile data structure available, and make it possible to support a wide variety of transactional and non-transactional semantics in the messaging system.
Disk seeks come at 10 ms a pop, and each disk can do only one seek at a time so parallelism is limited. Hence even a handful of disk seeks leads to very high overhead.
Furthermore BTrees require a very sophisticated page or row locking implementation to avoid locking the entire tree on each operation.
The implementation must pay a fairly high price for row-locking or else effectively serialize all reads.
持久化消息的元數(shù)據(jù)通常是BTree結(jié)構(gòu)，但磁盤結(jié)構(gòu)，其代價太大。原因：尋道、避免鎖整棵樹。
Intuitively a persistent queue could be built on simple reads and appends to files as is commonly the case with logging solutions.
持久化隊列可以構(gòu)建在讀和append to 文件。所以不支持BTree的一些語義，但其好處是：O(1)消耗，無鎖讀寫。
the performance is completely decoupled from the data size--one server can now take full advantage of a number of cheap, low-rotational speed 1+TB SATA drives.
Though they have poor seek performance, these drives often have comparable performance for large reads and writes at 1/3 the price and 3x the capacity.
4.3 Maximizing Efficiency
Furthermore we assume each message published is read at least once (and often multiple times), hence we optimize for consumption rather than production. 更進(jìn)一步，我們假設(shè)被發(fā)布的消息至少會讀一次，因此優(yōu)化consumer優(yōu)先于producer。
There are two common causes of inefficiency :
two many network requests, （
APIs are built around a "message set" abstraction，
This allows network requests to group messages together and amortize the overhead of the network roundtrip rather than sending a single message at a time.） 僅提供批量操作api，則每次網(wǎng)絡(luò)開銷是平分在一組消息，而不是單個消息。
and excessive byte copying.（
The message log maintained by the broker is itself just a directory of message sets that have been written to disk.
Maintaining this common format allows optimization of the most important operation : network transfer of persistent log chunks.）
To understand the impact of sendfile, it is important to understand the common data path for transfer of data from file to socket:
- The operating system reads data from the disk into pagecache in kernel space
- The application reads the data from kernel space into a user-space buffer
- The application writes the data back into kernel space into a socket buffer
- The operating system copies the data from the socket buffer to the NIC buffer where it is sent over the network
利用os提供的zero-copy，
only the final copy to the NIC buffer is needed.
4.4 End-to-end Batch Compression
In many cases the bottleneck is actually not CPU but network. This is particularly true for a data pipeline that needs to send messages across data centers.
Efficient compression requires compressing multiple messages together rather than compressing each message individually.
Ideally this would be possible in an end-to-end fashion — that is, data would be compressed prior to sending by the producer and remain compressed on the server, only being decompressed by the eventual consumers.
A batch of messages can be clumped together compressed and sent to the server in this form. This batch of messages will be delivered all to the same consumer and will remain in compressed form until it arrives there.
理解：kafka
producer api 提供批量壓縮，broker不對此批消息做任何操作，且以壓縮的方式，一起被發(fā)送到consumer。
4.5 Consumer state
Keeping track of what has been consumed is one of the key things a messaging system must provide.
State tracking requires updating a persistent entity and potentially causes random accesses.
Most messaging systems keep metadata about what messages have been consumed on the broker. That is, as a message is handed out to a consumer, the broker records that fact locally. 大部分消息系統(tǒng)，存儲是否被消費的元信息在broker。則是說，一個消息被分發(fā)到一個consumer，broker記錄。
問題：當(dāng)consumer消費失敗后，會導(dǎo)致消息丟失；改進(jìn)：每次consumer消費后，給broker ack，若broker在超時時間未收到ack，則重發(fā)此消息。
問題：1.當(dāng)消費成功，但未ack時，會導(dǎo)致消費2次 2.
now the broker must keep multiple states about every single message 3.當(dāng)broker是多臺機器時，則狀態(tài)之間需要同步
4.5.1 Message delivery semantics
So clearly there are multiple possible message delivery guarantees that could be provided : at most once 、at least once、exactly once。
This problem is heavily studied, and is a variation of the "transaction commit" problem. Algorithms that provide exactly once semantics exist, two- or three-phase commits and Paxos variants being examples, but they come with some drawbacks. They typically require multiple round trips and may have poor guarantees of liveness (they can halt indefinitely).
消費分發(fā)語義，是 ‘事務(wù)提交’ 問題的變種。算法提供 exactly onece 語義，兩階段 or 三階段提交，paxos 均是例子，但它們存在缺點。典型的問題是要求多次round trip，且
poor guarantees of liveness。
Kafka does two unusual things with respect to metadata.
First the stream is partitioned on the brokers into a set of distinct partitions.
Within a partition messages are stored in the order in which they arrive at the broker, and will be given out to consumers in that same order. This means that rather than store metadata for each message (marking it as consumed, say), we just need to store the "high water mark" for each combination of consumer, topic, and partition.
4.5.2 Consumer state
In Kafka, the consumers are responsible for maintaining state information (offset) on what has been consumed.
Typically, the Kafka consumer library writes their state data to zookeeper.
This solves a distributed consensus problem, by removing the distributed part!
There is a side benefit of this decision. A consumer can deliberately rewind back to an old offset and re-consume data.
4.5.3 Push vs. pull
A related question is whether consumers should pull data from brokers or brokers should push data to the subscriber.
There are pros and cons to both approaches.
However a push-based system has difficulty dealing with diverse consumers as the broker controls the rate at which data is transferred. push目標(biāo)是consumer能在最大速率去消費，可不幸的是，當(dāng)consume速率小于生產(chǎn)速率時，the consumer tends to be overwhelmed。
A pull-based system has the nicer property that the consumer simply falls behind and catches up when it can. This can be mitigated with some kind of backoff protocol by which the consumer can indicate it is overwhelmed, but getting the rate of transfer to fully utilize (but never over-utilize) the consumer is trickier than it seems. Previous attempts at building systems in this fashion led us to go with a more traditional pull model. 不存在push問題，且也保證充分利用consumer能力。
5. Distribution
Kafka is built to be run across a cluster of machines as the common case. There is no central "master" node. Brokers are peers to each other and can be added and removed at anytime without any manual configuration changes. Similarly, producers and consumers can be started dynamically at any time. Each broker registers some metadata (e.g., available topics) in Zookeeper. Producers and consumers can use Zookeeper to discover topics and to co-ordinate the production and consumption. The details of producers and consumers will be described below.
6. Producer
6.1 Automatic producer load balancing
Kafka supports client-side load balancing for message producers or use of a dedicated load balancer to balance TCP connections.
The advantage of using a level-4 load balancer is that each producer only needs a single TCP connection, and no connection to zookeeper is needed.
The disadvantage is that the balancing is done at the TCP connection level, and hence it may not be well balanced (if some producers produce many more messages then others, evenly dividing up the connections per broker may not result in evenly dividing up the messages per broker).
Client-side zookeeper-based load balancing solves some of these problems. It allows the producer to dynamically discover new brokers, and balance load on a per-request basis. It allows the producer to partition data according to some key instead of randomly.
The working of the zookeeper-based load balancing is described below. Zookeeper watchers are registered on the following events—
- a new broker comes up
- a broker goes down
- a new topic is registered
- a broker gets registered for an existing topic
Internally, the producer maintains an elastic pool of connections to the brokers, one per broker. This pool is kept updated to establish/maintain connections to all the live brokers, through the zookeeper watcher callbacks. When a producer request for a particular topic comes in, a broker partition is picked by the partitioner (see section on semantic partitioning). The available producer connection is used from the pool to send the data to the selected broker partition.
producer通過zk，管理與broker的連接。當(dāng)一個請求，根據(jù)partition rule 計算分區(qū)，從連接池選擇對應(yīng)的connection，發(fā)送數(shù)據(jù)。
6.2 Asynchronous send
Asynchronous non-blocking operations are fundamental to scaling messaging systems.
This allows buffering of produce requests in a in-memory queue and batch sends that are triggered by a time interval or a pre-configured batch size.
6.3 Semantic partitioning
The producer has the capability to be able to semantically map messages to the available kafka nodes and partitions. This allows partitioning the stream of messages with some semantic partition function based on some key in the message to spread them over broker machines.