前言
每一種該語言在某些極限情況下的表現(xiàn)一般都不太一樣，那么我常用的Java語言，在達(dá)到100萬個(gè)并發(fā)連接情況下，會(huì)怎么樣呢，有些好奇，更有些期盼。
這次使用經(jīng)常使用的順手的netty NIO框架（netty-3.6.5.Final），封裝的很好，接口很全面，就像它現(xiàn)在的域名 netty.io，專注于網(wǎng)絡(luò)IO。
整個(gè)過程沒有什么技術(shù)含量，淺顯分析過就更顯得有些枯燥無聊，準(zhǔn)備好，硬著頭皮吧。
測(cè)試服務(wù)器配置
運(yùn)行在VMWare Workstation 9中，64位Centos 6.2系統(tǒng)，分配14.9G內(nèi)存左右，4核。
已安裝有Java7版本：
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
在/etc/sysctl.conf中添加如下配置：
fs.file-max = 1048576
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_mem = 786432 2097152 3145728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
在/etc/security/limits.conf中添加如下配置：
* soft nofile 1048576
* hard nofile 1048576
測(cè)試端
測(cè)試端無論是配置還是程序和以前一樣，翻看前幾篇博客就可以看到client5.c的源碼，以及相關(guān)的配置信息等。
服務(wù)器程序
這次也是很簡單吶，沒有業(yè)務(wù)功能，客戶端HTTP請(qǐng)求，服務(wù)端輸出chunked編碼內(nèi)容。
入口HttpChunkedServer.java：
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
package com.test.server; |
|
|
|
import static org.jboss.netty.channel.Channels.pipeline; |
|
|
|
import java.net.InetSocketAddress; |
|
import java.util.concurrent.Executors; |
|
|
|
import org.jboss.netty.bootstrap.ServerBootstrap; |
|
import org.jboss.netty.channel.ChannelPipeline; |
|
import org.jboss.netty.channel.ChannelPipelineFactory; |
|
import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory; |
|
import org.jboss.netty.handler.codec.http.HttpChunkAggregator; |
|
import org.jboss.netty.handler.codec.http.HttpRequestDecoder; |
|
import org.jboss.netty.handler.codec.http.HttpResponseEncoder; |
|
import org.jboss.netty.handler.stream.ChunkedWriteHandler; |
|
|
|
public class HttpChunkedServer { |
|
private final int port; |
|
|
|
public HttpChunkedServer(int port) { |
|
this.port = port; |
|
} |
|
|
|
public void run() { |
|
// Configure the server. |
|
ServerBootstrap bootstrap = new ServerBootstrap( |
|
new NioServerSocketChannelFactory( |
|
Executors.newCachedThreadPool(), |
|
Executors.newCachedThreadPool())); |
|
|
|
// Set up the event pipeline factory. |
|
bootstrap.setPipelineFactory(new ChannelPipelineFactory() { |
|
public ChannelPipeline getPipeline() throws Exception { |
|
ChannelPipeline pipeline = pipeline(); |
|
|
|
pipeline.addLast("decoder", new HttpRequestDecoder()); |
|
pipeline.addLast("aggregator", new HttpChunkAggregator(65536)); |
|
pipeline.addLast("encoder", new HttpResponseEncoder()); |
|
pipeline.addLast("chunkedWriter", new ChunkedWriteHandler()); |
|
|
|
pipeline.addLast("handler", new HttpChunkedServerHandler()); |
|
return pipeline; |
|
} |
|
}); |
|
|
|
bootstrap.setOption("child.reuseAddress", true); |
|
bootstrap.setOption("child.tcpNoDelay", true); |
|
bootstrap.setOption("child.keepAlive", true); |
|
|
|
// Bind and start to accept incoming connections. |
|
bootstrap.bind(new InetSocketAddress(port)); |
|
} |
|
|
|
public static void main(String[] args) { |
|
int port; |
|
if (args.length > 0) { |
|
port = Integer.parseInt(args[0]); |
|
} else { |
|
port = 8080; |
|
} |
|
|
|
System.out.format("server start with port %d \n", port); |
|
new HttpChunkedServer(port).run(); |
|
} |
|
} |
唯一的自定義處理器HttpChunkedServerHandler.java:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
package com.test.server; |
|
|
|
import static org.jboss.netty.handler.codec.http.HttpHeaders.Names.CONTENT_TYPE; |
|
import static org.jboss.netty.handler.codec.http.HttpMethod.GET; |
|
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.BAD_REQUEST; |
|
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.METHOD_NOT_ALLOWED; |
|
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.OK; |
|
import static org.jboss.netty.handler.codec.http.HttpVersion.HTTP_1_1; |
|
|
|
import java.util.concurrent.atomic.AtomicInteger; |
|
|
|
import org.jboss.netty.buffer.ChannelBuffer; |
|
import org.jboss.netty.buffer.ChannelBuffers; |
|
import org.jboss.netty.channel.Channel; |
|
import org.jboss.netty.channel.ChannelFutureListener; |
|
import org.jboss.netty.channel.ChannelHandlerContext; |
|
import org.jboss.netty.channel.ChannelStateEvent; |
|
import org.jboss.netty.channel.ExceptionEvent; |
|
import org.jboss.netty.channel.MessageEvent; |
|
import org.jboss.netty.channel.SimpleChannelUpstreamHandler; |
|
import org.jboss.netty.handler.codec.frame.TooLongFrameException; |
|
import org.jboss.netty.handler.codec.http.DefaultHttpChunk; |
|
import org.jboss.netty.handler.codec.http.DefaultHttpResponse; |
|
import org.jboss.netty.handler.codec.http.HttpChunk; |
|
import org.jboss.netty.handler.codec.http.HttpHeaders; |
|
import org.jboss.netty.handler.codec.http.HttpRequest; |
|
import org.jboss.netty.handler.codec.http.HttpResponse; |
|
import org.jboss.netty.handler.codec.http.HttpResponseStatus; |
|
import org.jboss.netty.util.CharsetUtil; |
|
|
|
public class HttpChunkedServerHandler extends SimpleChannelUpstreamHandler { |
|
private static final AtomicInteger count = new AtomicInteger(0); |
|
|
|
private void increment() { |
|
System.out.format("online user %d\n", count.incrementAndGet()); |
|
} |
|
|
|
private void decrement() { |
|
if (count.get() <= 0) { |
|
System.out.format("~online user %d\n", 0); |
|
} else { |
|
System.out.format("~online user %d\n", count.decrementAndGet()); |
|
} |
|
} |
|
|
|
@Override |
|
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) |
|
throws Exception { |
|
HttpRequest request = (HttpRequest) e.getMessage(); |
|
if (request.getMethod() != GET) { |
|
sendError(ctx, METHOD_NOT_ALLOWED); |
|
return; |
|
} |
|
|
|
sendPrepare(ctx); |
|
increment(); |
|
} |
|
|
|
@Override |
|
public void channelDisconnected(ChannelHandlerContext ctx, |
|
ChannelStateEvent e) throws Exception { |
|
decrement(); |
|
super.channelDisconnected(ctx, e); |
|
} |
|
|
|
@Override |
|
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) |
|
throws Exception { |
|
Throwable cause = e.getCause(); |
|
if (cause instanceof TooLongFrameException) { |
|
sendError(ctx, BAD_REQUEST); |
|
return; |
|
} |
|
} |
|
|
|
private static void sendError(ChannelHandlerContext ctx, |
|
HttpResponseStatus status) { |
|
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, status); |
|
response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8"); |
|
response.setContent(ChannelBuffers.copiedBuffer( |
|
"Failure: " + status.toString() + "\r\n", CharsetUtil.UTF_8)); |
|
|
|
// Close the connection as soon as the error message is sent. |
|
ctx.getChannel().write(response) |
|
.addListener(ChannelFutureListener.CLOSE); |
|
} |
|
|
|
private void sendPrepare(ChannelHandlerContext ctx) { |
|
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK); |
|
response.setChunked(true); |
|
response.setHeader(HttpHeaders.Names.CONTENT_TYPE, |
|
"text/html; charset=UTF-8"); |
|
response.addHeader(HttpHeaders.Names.CONNECTION, |
|
HttpHeaders.Values.KEEP_ALIVE); |
|
response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING, |
|
HttpHeaders.Values.CHUNKED); |
|
|
|
Channel chan = ctx.getChannel(); |
|
chan.write(response); |
|
|
|
// 缓冲必须凑够256字节，浏览器端才能够正常接收 ... |
|
StringBuilder builder = new StringBuilder(); |
|
builder.append("<html><body><script>var _ = function (msg) { parent.s._(msg, document); };</script>"); |
|
int leftChars = 256 - builder.length(); |
|
for (int i = 0; i < leftChars; i++) { |
|
builder.append(" "); |
|
} |
|
|
|
writeStringChunk(chan, builder.toString()); |
|
} |
|
|
|
private void writeStringChunk(Channel channel, String data) { |
|
ChannelBuffer chunkContent = ChannelBuffers.dynamicBuffer(channel |
|
.getConfig().getBufferFactory()); |
|
chunkContent.writeBytes(data.getBytes()); |
|
HttpChunk chunk = new DefaultHttpChunk(chunkContent); |
|
|
|
channel.write(chunk); |
|
} |
|
} |
啟動(dòng)腳本start.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
set CLASSPATH=. |
|
nohup java -server -Xmx6G -Xms6G -Xmn600M -XX:PermSize=50M -XX:MaxPermSize=50M -Xss256K -XX:+DisableExplicitGC -XX:SurvivorRatio=1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:gc.log -Djava.ext.dirs=lib com.test.server.HttpChunkedServer 8000 >server.out 2>&1 & |
達(dá)到100萬并發(fā)連接時(shí)的一些信息
每次服務(wù)器端達(dá)到一百萬個(gè)并發(fā)持久連接之后，然后關(guān)掉測(cè)試端程序，斷開所有的連接，等到服務(wù)器端日志輸出在線用戶為0時(shí)，再次重復(fù)以上步驟。在這反反復(fù)復(fù)的情況下，觀察內(nèi)存等信息的一些情況。以某次斷開所有測(cè)試端為例后，當(dāng)前系統(tǒng)占用為（設(shè)置為list_free_1）:
total used free shared buffers cached
Mem: 15189 7736 7453 0 18 120
-/+ buffers/cache: 7597 7592
Swap: 4095 948 3147
通過top觀察，其進(jìn)程相關(guān)信息
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4925 root 20 0 8206m 4.3g 2776 S 0.3 28.8 50:18.66 java
在啟動(dòng)腳本start.sh中，我們?cè)O(shè)置堆內(nèi)存為6G。
ps aux|grep java命令獲得信息：
root 4925 38.0 28.8 8403444 4484764 ? Sl 15:26 50:18 java -server...HttpChunkedServer 8000
RSS占用內(nèi)存為4484764K/1024K=4379M
然后再次啟動(dòng)測(cè)試端，在服務(wù)器接收到online user 1023749時(shí)，ps aux|grep java內(nèi)容為：
root 4925 43.6 28.4 8403444 4422824 ? Sl 15:26 62:53 java -server...
查看當(dāng)前網(wǎng)絡(luò)信息統(tǒng)計(jì)
ss -s
Total: 1024050 (kernel 1024084)
TCP: 1023769 (estab 1023754, closed 2, orphaned 0, synrecv 0, timewait 0/0), ports 12
Transport Total IP IPv6
* 1024084 - -
RAW 0 0 0
UDP 7 6 1
TCP 1023767 12 1023755
INET 1023774 18 1023756
FRAG 0 0 0
通過top查看一下
top -p 4925
top - 17:51:30 up 3:02, 4 users, load average: 1.03, 1.80, 1.19
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.9%us, 2.6%sy, 0.0%ni, 52.9%id, 1.0%wa, 13.6%hi, 29.0%si, 0.0%st
Cpu1 : 1.4%us, 4.5%sy, 0.0%ni, 80.1%id, 1.9%wa, 0.0%hi, 12.0%si, 0.0%st
Cpu2 : 1.5%us, 4.4%sy, 0.0%ni, 80.5%id, 4.3%wa, 0.0%hi, 9.3%si, 0.0%st
Cpu3 : 1.9%us, 4.4%sy, 0.0%ni, 84.4%id, 3.2%wa, 0.0%hi, 6.2%si, 0.0%st
Mem: 15554336k total, 15268728k used, 285608k free, 3904k buffers
Swap: 4194296k total, 1082592k used, 3111704k free, 37968k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4925 root 20 0 8206m 4.2g 2220 S 3.3 28.4 62:53.66 java
四核都被占用了，每一個(gè)核心不太平均。這是在虛擬機(jī)中得到結(jié)果，可能真實(shí)服務(wù)器會(huì)更好一些。 因?yàn)椴皇荂PU密集型應(yīng)用，CPU不是問題，無須多加關(guān)注。
系統(tǒng)內(nèi)存狀況
free -m
total used free shared buffers cached
Mem: 15189 14926 263 0 5 56
-/+ buffers/cache: 14864 324
Swap: 4095 1057 3038
物理內(nèi)存已經(jīng)無法滿足要求了，占用了1057M虛擬內(nèi)存。
查看一下堆內(nèi)存情況
jmap -heap 4925
Attaching to process ID 4925, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.21-b01
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 6442450944 (6144.0MB)
NewSize = 629145600 (600.0MB)
MaxNewSize = 629145600 (600.0MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 1
PermSize = 52428800 (50.0MB)
MaxPermSize = 52428800 (50.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 419430400 (400.0MB)
used = 308798864 (294.49354553222656MB)
free = 110631536 (105.50645446777344MB)
73.62338638305664% used
Eden Space:
capacity = 209715200 (200.0MB)
used = 103375232 (98.5863037109375MB)
free = 106339968 (101.4136962890625MB)
49.29315185546875% used
From Space:
capacity = 209715200 (200.0MB)
used = 205423632 (195.90724182128906MB)
free = 4291568 (4.0927581787109375MB)
97.95362091064453% used
To Space:
capacity = 209715200 (200.0MB)
used = 0 (0.0MB)
free = 209715200 (200.0MB)
0.0% used
concurrent mark-sweep generation:
capacity = 5813305344 (5544.0MB)
used = 4213515472 (4018.321487426758MB)
free = 1599789872 (1525.6785125732422MB)
72.48054631000646% used
Perm Generation:
capacity = 52428800 (50.0MB)
used = 5505696 (5.250640869140625MB)
free = 46923104 (44.749359130859375MB)
10.50128173828125% used
1439 interned Strings occupying 110936 bytes.
老生代占用內(nèi)存為72%，較為合理，畢竟系統(tǒng)已經(jīng)處理100萬個(gè)連接。
再次斷開所有測(cè)試端，看看系統(tǒng)內(nèi)存（free -m）
total used free shared buffers cached
Mem: 15189 7723 7466 0 13 120
-/+ buffers/cache: 7589 7599
Swap: 4095 950 3145
記為list_free_2。
list_free_1和list_free_2兩次都釋放后的內(nèi)存比較結(jié)果，系統(tǒng)可用物理已經(jīng)內(nèi)存已經(jīng)降到7589M，先前可是7597M物理內(nèi)存。
總之，我們的JAVA測(cè)試程序在內(nèi)存占用方面已經(jīng)，最低需要7589 + 950 = 8.6G內(nèi)存為最低需求內(nèi)存吧。
GC日志
我們?cè)趩?dòng)腳本處設(shè)置的一大串參數(shù)，到底是否達(dá)到目標(biāo)，還得從gc日志處獲得具體效果，推薦使用GCViewer。
GC事件概覽：
其它:
總之：
- 只進(jìn)行了一次Full GC，代價(jià)太高，停頓了12秒。
- PartNew成為了停頓大戶，導(dǎo)致整個(gè)系統(tǒng)停頓了41秒之久，不可接受。
- 當(dāng)前JVM調(diào)優(yōu)喜憂參半，還得繼續(xù)努力等
小結(jié)
Java與與Erlang、C相比，比較麻煩的事情，需要在程序一開始就得準(zhǔn)備好它的堆棧到底需要多大空間，換個(gè)說法就是JVM啟動(dòng)參數(shù)設(shè)置堆內(nèi)存大小，設(shè)置合適的垃圾回收機(jī)制，若以后程序需要更多內(nèi)存，需停止程序，編輯啟動(dòng)參數(shù)，然后再次啟動(dòng)。總之一句話，就是麻煩。單單JVM的調(diào)優(yōu)，就得持續(xù)不斷的根據(jù)檢測(cè)、信息、日志等進(jìn)行適當(dāng)微調(diào)。
- JVM需要提前指定堆大小，相比Erlang/C，這可能是個(gè)麻煩
- GC(垃圾回收)，相對(duì)比麻煩，需要持續(xù)不斷的根據(jù)日志、JVM堆棧信息、運(yùn)行時(shí)情況進(jìn)行JVM參數(shù)微調(diào)
- 設(shè)置一個(gè)最大連接目標(biāo)，多次測(cè)試達(dá)到頂峰，然后釋放所有連接，反復(fù)觀察內(nèi)存占用，獲得一個(gè)較為合適的系統(tǒng)運(yùn)行內(nèi)存值
- Eclipse Memory Analyzer結(jié)合jmap導(dǎo)出堆棧DUMP文件，分析內(nèi)存泄漏，還是很方便的
- 想修改運(yùn)行時(shí)內(nèi)容，或者稱之為熱加載，默認(rèn)不可能
- 真實(shí)機(jī)器上會(huì)有更好的反映
吐槽一下：
JAVA OSGI，相對(duì)比Erlang來說，需要人轉(zhuǎn)換思路，不是那么原生的東西，總是有些別扭，社區(qū)或商業(yè)公司對(duì)此的修修補(bǔ)補(bǔ)，不過是實(shí)現(xiàn)一些面向?qū)ο笏痪邆涞臒峒虞d的企業(yè)特性。
測(cè)試源代碼，下載just_test。