å‰è¨€
å› äؓ能力有é™åQŒè¿˜æ˜¯æœ‰å¾ˆå¤šä¸œè¥¿åQˆSO_REUSEADDRå’ŒSO_REUSEPORT的区别ç‰åQ‰æ²¡æœ‰èƒ½å¤Ÿåœ¨ä¸€½‹‡æ–‡å—ä¸è¡¨è¾¾æ¸…楚åQŒä½œä¸ø™¡¥é—，也方便以åŽè‡ªå·±å›ž˜q‡å¤´æ¥å¤ä¹ ã€?/p>
SO_REUSADDR VS SO_REUSEPORT
两者ä¸æ˜¯ä¸€ç 事åQŒæ²¡æœ‰å¯æ¯”性。有时也会被其æžæ™•ï¼Œè‡ªå·±æ€È»“çš„ä¸å¥½ï¼ŒæŽ¨èStackOverflowçš?a >Socket options SO_REUSEADDR and SO_REUSEPORT, how do they differ?资料åQŒæ€È»“的很全é¢ã€?/p>
½Ž€å•æ¥è¯ß_¼š
- 讄¡½®äº†SO_REUSADDR的应用å¯ä»¥é¿å…TCP çš?TIME_WAIT 状æ€?æ—‰™—´˜q‡é•¿æ— 法å¤ç”¨ç«¯å£åQŒå°¤å…¶è¡¨çŽ°åœ¨åº”用½E‹åºå…³é—-é‡å¯äº¤æ›¿çš„瞬é—?
- SO_REUSEPORT更强大，隶属于åŒä¸€ä¸ªç”¨æˆøP¼ˆé˜²æ¢ç«¯å£åŠ«æŒåQ‰çš„多个˜q›ç¨‹/¾U¿ç¨‹å…׃ín一个端å£ï¼ŒåŒæ—¶åœ¨å†…æ ¸å±‚é¢æ›¿ä¸Šå±‚应用åšæ•°æ®åŒ…˜q›ç¨‹/¾U¿ç¨‹çš„处ç†å‡è¡?
若有困惑åQŒæŽ¨è两者都讄¡½®åQŒä¸ä¼šæœ‰å†²çªã€?/p>
Netty多线½E‹ä‹É用SO_REUSEPORT
上一½‹‡è®²åˆ°SO_REUSEPORTåQŒå¤šä¸ªç¨‹¾l‘定åŒä¸€ä¸ªç«¯å£ï¼Œå¯ä»¥æ ÒŽ®éœ€è¦æŽ§åˆ¶è¿›½E‹çš„æ•°é‡ã€‚这里讲讲基äº?code>Netty 4.0.25+Epoll navtie transport在å•ä¸ªè¿›½E‹å†…多个¾U¿ç¨‹¾l‘定åŒä¸€ä¸ªç«¯å£çš„情况åQŒä¹Ÿæ˜¯æ¯”较实用的ã€?/p>
TCPæœåŠ¡å™¨ï¼ŒåŒä¸€ä¸ªè¿›½E‹å¤š¾U¿ç¨‹¾l‘定åŒä¸€ä¸ªç«¯å?/h4>
˜q™æ˜¯ä¸€ä¸ªPING-PONG½Cø™Œƒåº”用åQ?/p>
public void run() throws Exception {
final EventLoopGroup bossGroup = new EpollEventLoopGroup();
final EventLoopGroup workerGroup = new EpollEventLoopGroup();
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(EpollServerSocketChannel. class)
.childHandler( new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(
new StringDecoder(CharsetUtil.UTF_8 ),
new StringEncoder(CharsetUtil.UTF_8 ),
new PingPongServerHandler());
}
}).option(ChannelOption. SO_REUSEADDR, true)
.option(EpollChannelOption. SO_REUSEPORT, true)
.childOption(ChannelOption. SO_KEEPALIVE, true);
int workerThreads = Runtime.getRuntime().availableProcessors();
ChannelFuture future;
for ( int i = 0; i < workerThreads; ++i) {
future = b.bind( port).await();
if (!future.isSuccess())
throw new Exception(String. format("fail to bind on port = %d.",
port), future.cause());
}
Runtime. getRuntime().addShutdownHook (new Thread(){
@Override
public void run(){
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
});
}
打æˆjar包，在CentOS 7下题qè¡ŒåQŒæ£€æŸ¥åŒä¸€ä¸ªç«¯å£æ‰€æ‰“开的文件å¥æŸ„ã€?/p>
# lsof -i:8000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 3515 root 42u IPv6 29040 0t0 TCP *:irdmi (LISTEN)
java 3515 root 43u IPv6 29087 0t0 TCP *:irdmi (LISTEN)
java 3515 root 44u IPv6 29088 0t0 TCP *:irdmi (LISTEN)
java 3515 root 45u IPv6 29089 0t0 TCP *:irdmi (LISTEN)
åŒä¸€˜q›ç¨‹åQŒä½†æ‰“开的文件å¥æŸ„是ä¸ä¸€æ ïLš„ã€?/p>
UDPæœåŠ¡å™¨ï¼Œå¤šä¸ª¾U¿ç¨‹¾l‘åŒä¸€ä¸ªç«¯å?/h4>/**
* UDPè°šè¯æœåŠ¡å™¨ï¼Œå•è¿›½E‹å¤š¾U¿ç¨‹¾l‘定åŒä¸€ç«¯å£½Cø™Œƒ
*/
public final class QuoteOfTheMomentServer {
private static final int PORT = Integer.parseInt(System. getProperty("port" ,
"9000" ));
public static void main(String[] args) throws Exception {
final EventLoopGroup group = new EpollEventLoopGroup();
Bootstrap b = new Bootstrap();
b.group(group).channel(EpollDatagramChannel. class)
.option(EpollChannelOption. SO_REUSEPORT, true )
.handler( new QuoteOfTheMomentServerHandler());
int workerThreads = Runtime.getRuntime().availableProcessors();
for (int i = 0; i < workerThreads; ++i) {
ChannelFuture future = b.bind( PORT).await();
if (!future.isSuccess())
throw new Exception(String.format ("Fail to bind on port = %d.",
PORT), future.cause());
}
Runtime. getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
group.shutdownGracefully();
}
});
}
}
}
@Sharable
class QuoteOfTheMomentServerHandler extends
SimpleChannelInboundHandler<DatagramPacket> {
private static final String[] quotes = {
"Where there is love there is life." ,
"First they ignore you, then they laugh at you, then they fight you, then you win.",
"Be the change you want to see in the world." ,
"The weak can never forgive. Forgiveness is the attribute of the strong.", };
private static String nextQuote() {
int quoteId = ThreadLocalRandom.current().nextInt( quotes .length );
return quotes [quoteId];
}
@Override
public void channelRead0(ChannelHandlerContext ctx, DatagramPacket packet)
throws Exception {
if ("QOTM?" .equals(packet.content().toString(CharsetUtil. UTF_8))) {
ctx.write( new DatagramPacket(Unpooled.copiedBuffer( "QOTM: "
+ nextQuote(), CharsetUtil. UTF_8), packet.sender()));
}
}
@Override
public void channelReadComplete(ChannelHandlerContext ctx) {
ctx.flush();
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
}
}
/**
* UDPè°šè¯æœåŠ¡å™¨ï¼Œå•è¿›½E‹å¤š¾U¿ç¨‹¾l‘定åŒä¸€ç«¯å£½Cø™Œƒ
*/
public final class QuoteOfTheMomentServer {
private static final int PORT = Integer.parseInt(System. getProperty("port" ,
"9000" ));
public static void main(String[] args) throws Exception {
final EventLoopGroup group = new EpollEventLoopGroup();
Bootstrap b = new Bootstrap();
b.group(group).channel(EpollDatagramChannel. class)
.option(EpollChannelOption. SO_REUSEPORT, true )
.handler( new QuoteOfTheMomentServerHandler());
int workerThreads = Runtime.getRuntime().availableProcessors();
for (int i = 0; i < workerThreads; ++i) {
ChannelFuture future = b.bind( PORT).await();
if (!future.isSuccess())
throw new Exception(String.format ("Fail to bind on port = %d.",
PORT), future.cause());
}
Runtime. getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
group.shutdownGracefully();
}
});
}
}
}
@Sharable
class QuoteOfTheMomentServerHandler extends
SimpleChannelInboundHandler<DatagramPacket> {
private static final String[] quotes = {
"Where there is love there is life." ,
"First they ignore you, then they laugh at you, then they fight you, then you win.",
"Be the change you want to see in the world." ,
"The weak can never forgive. Forgiveness is the attribute of the strong.", };
private static String nextQuote() {
int quoteId = ThreadLocalRandom.current().nextInt( quotes .length );
return quotes [quoteId];
}
@Override
public void channelRead0(ChannelHandlerContext ctx, DatagramPacket packet)
throws Exception {
if ("QOTM?" .equals(packet.content().toString(CharsetUtil. UTF_8))) {
ctx.write( new DatagramPacket(Unpooled.copiedBuffer( "QOTM: "
+ nextQuote(), CharsetUtil. UTF_8), packet.sender()));
}
}
@Override
public void channelReadComplete(ChannelHandlerContext ctx) {
ctx.flush();
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
}
}
åŒæ ·ä¹Ÿè¦‹‚€‹¹‹ä¸€ä¸‹ç«¯å£æ–‡ä»¶å¥æŸ„打开情况åQ?/p>
# lsof -i:9000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 3181 root 26u IPv6 27188 0t0 UDP *:cslistener
java 3181 root 27u IPv6 27217 0t0 UDP *:cslistener
java 3181 root 28u IPv6 27218 0t0 UDP *:cslistener
java 3181 root 29u IPv6 27219 0t0 UDP *:cslistener
ž®ç»“
以上为Netty+SO_REUSEPORT多线½E‹ç»‘定åŒä¸€ç«¯å£çš„一些情况，是äØ“è®°è²ã€?/p>
å‰è¨€
本篇用于记录å¦ä¹ SO_REUSEPORT的笔记和心得åQŒæœ«ž®¾è¿˜ä¼šæ供一个bindpž®å·¥å…·ä¹Ÿèƒ½äؓ已有的程åºäínå—这个新的特性ã€?/p>
当å‰Linux¾|‘络应用½E‹åºé—®é¢˜
˜q行在Linux¾pÈ»Ÿä¸Šç½‘¾lœåº”用程åºï¼Œä¸ÞZº†åˆ©ç”¨å¤šæ ¸çš„优势，一般ä‹É用以下比较典型的多进½E?多线½E‹æœåŠ¡å™¨æ¨¡åž‹åQ?/p>
- å•çº¿½E‹listen/acceptåQŒå¤šä¸ªå·¥ä½œçº¿½E‹æŽ¥æ”¶ä“Q务分å‘，虽CPU的工作负载ä¸å†æ˜¯é—®é¢˜åQŒä½†ä¼šå˜åœ¨ï¼š
- å•çº¿½E‹listeneråQŒåœ¨å¤„ç†é«˜é€ŸçŽ‡‹¹·é‡˜qžæŽ¥æ—Óž¼Œä¸€æ ·ä¼šæˆäؓ瓉™¢ˆ
- CPU¾~“å˜è¡Œä¸¢å¤±å¥—接嗾l“æž„(socket structure)现象严é‡
- 所有工作线½E‹éƒ½accept()在åŒä¸€ä¸ªæœåŠ¡å™¨å¥—接å—ä¸Šå‘¢ï¼Œä¸€æ ·å˜åœ¨é—®é¢˜ï¼š
- 多线½E‹è®¿é—®server socketé”竞争严é‡?/li>
- 高负载下åQŒçº¿½E‹ä¹‹é—´å¤„ç†ä¸å‡è¡¡åQŒæœ‰æ—‰™«˜è¾?:1ä¸å‡è¡¡æ¯”ä¾?/li>
- å¯ÆD‡´CPU¾~“å˜è¡ŒèŸ©è·?cache line bouncing)
- 在ç¹å¿™CPU上å˜åœ¨è¾ƒå¤§åšg˜q?/li>
上é¢æ¨¡åž‹è™½ç„¶å¯ä»¥åšåˆ°¾U¿ç¨‹å’ŒCPUæ ¸ç»‘å®šï¼Œä½†éƒ½ä¼šå˜åœ¨ï¼š
- å•ä¸€listener工作¾U¿ç¨‹åœ¨é«˜é€Ÿçš„˜qžæŽ¥æŽ¥å…¥å¤„ç†æ—¶ä¼šæˆäؓ瓉™¢ˆ
- ¾~“å˜è¡ŒèŸ©è·?/li>
- 很难åšåˆ°CPU之间的负载å‡è¡?/li>
- éšç€æ ¸æ•°çš„扩展，性能òq¶æ²¡æœ‰éšç€æå‡
比如HTTP CPS(Connection Per Second)åžåé‡åƈ没有éšç€CPUæ ¸æ•°å¢žåŠ å‘ˆçŽ°¾U¿æ€§å¢žé•¿ï¼š 
Linux kernel 3.9带æ¥äº†SO_REUSEPORTç‰ÒŽ€§ï¼Œå¯ä»¥è§£å†³ä»¥ä¸Šå¤§éƒ¨åˆ†é—®é¢˜ã€?/p>
SO_REUSEPORT解决了什么问�/h3>
linux man文档ä¸ä¸€ŒD‰|–‡å—æ˜q°å…¶ä½œç”¨åQ?/p>
The new socket option allows multiple sockets on the same host to bind to the same port, and is intended to improve the performance of multithreaded network server applications running on top of multicore systems.
SO_REUSEPORT支æŒå¤šä¸ª˜q›ç¨‹æˆ–者线½E‹ç»‘定到åŒä¸€ç«¯å£åQŒæ高æœåŠ¡å™¨½E‹åºçš„性能åQŒè§£å†³çš„问题åQ?/p>
- å…许多个套接å?bind()/listen() åŒä¸€ä¸ªTCP/UDP端å£
- æ¯ä¸€ä¸ªçº¿½E‹æ‹¥æœ‰è‡ªå·Þqš„æœåŠ¡å™¨å¥—接å—
- 在æœåŠ¡å™¨å¥—接å—上没有了é”çš„ç«žäº?/li>
- å†…æ ¸å±‚é¢å®žçŽ°è´Ÿè²å‡è¡¡
- 安全层é¢åQŒç›‘å¬åŒä¸€ä¸ªç«¯å£çš„套接å—åªèƒ½ä½äºŽåŒä¸€ä¸ªç”¨æˆ·ä¸‹é?/li>
å…¶æ ¸å¿ƒçš„å®žçŽ°ä¸»è¦æœ‰ä¸‰ç‚¹ï¼š
- 扩展 socket optionåQŒå¢žåŠ?SO_REUSEPORT 选项åQŒç”¨æ¥è®¾¾|?reuseportã€?/li>
- 修改 bind ¾pÈ»Ÿè°ƒç”¨å®žçŽ°åQŒä»¥ä¾¿æ”¯æŒå¯ä»¥ç»‘定到相åŒçš?IP 和端å?/li>
- 修改处ç†æ–°å¾˜qžæŽ¥çš„实玎ͼŒæŸ¥æ‰¾ listener 的时候，能够支æŒåœ¨ç›‘å¬ç›¸å?IP 和端å£çš„多个 sock 之间å‡è¡¡é€‰æ‹©ã€?/li>
代ç 分æžåQŒå¯ä»¥å‚考引用资æ–?[多个˜q›ç¨‹¾l‘定相åŒç«¯å£çš„实现分æž[Google Patch]]ã€?/p>
CPU之间òqŒ™¡¡å¤„ç†åQŒæ°´òqÏx‰©å±?/h4>
以å‰é€šè¿‡forkå½¢å¼åˆ›å¾å¤šä¸ªåè¿›½E‹ï¼ŒçŽ°åœ¨æœ‰äº†SO_REUSEPORTåQŒå¯ä»¥ä¸ç”¨é€šè¿‡forkçš„åÅžå¼ï¼Œè®©å¤š˜q›ç¨‹ç›‘å¬åŒä¸€ä¸ªç«¯å£ï¼Œå„个˜q›ç¨‹ä¸?code>accept socket fdä¸ä¸€æ øP¼Œæœ‰æ–°˜qžæŽ¥å»ºç«‹æ—Óž¼Œå†…æ ¸åªä¼šå”¤é†’一个进½E‹æ¥acceptåQŒåƈ且ä¿è¯å”¤é†’çš„å‡è¡¡æ€§ã€?/p>
模型½Ž€å•ï¼Œ¾l´æŠ¤æ–¹ä¾¿äº†ï¼Œ˜q›ç¨‹çš„管ç†å’Œåº”用逻辑解耦，˜q›ç¨‹çš„管ç†æ°´òqÏx‰©å±•æƒé™ä¸‹æ”„¡»™½E‹åºå‘?½Ž¡ç†å‘˜ï¼Œå¯ä»¥æ ÒŽ®å®žé™…˜q›è¡ŒæŽ§åˆ¶˜q›ç¨‹å¯åŠ¨/å…³é—åQŒå¢žåŠ 了ç‰|´»æ€§ã€?/p>
˜q™å¸¦æ¥äº†ä¸€ä¸ªè¾ƒä¸ºå¾®è§‚çš„æ°´åã^扩展æ€èµ\åQŒçº¿½E‹å¤šž®‘是å¦åˆé€‚，状æ€æ˜¯å¦å˜åœ¨å…±äº«ï¼Œé™ä½Žå•ä¸ª˜q›ç¨‹çš„资æºä¾èµ–ï¼Œé’ˆå¯¹æ— çŠ¶æ€çš„æœåŠ¡å™¨æž¶æž„最为适åˆäº†ã€?/p>
新特性测试或多个版本共å˜
å¯ä»¥å¾ˆæ–¹ä¾¿çš„‹¹‹è¯•æ–°ç‰¹æ€§ï¼ŒåŒä¸€ä¸ªç¨‹åºï¼Œä¸åŒç‰ˆæœ¬åŒæ—¶˜qè¡Œä¸ï¼Œæ ÒŽ®˜qè¡Œ¾l“果军_®šæ–°è€ç‰ˆæœ¬æ›´˜q与å¦ã€?/p>
针对对客æˆïL«¯è€Œè¨€åQŒè¡¨é¢ä¸Šæ„Ÿå—ä¸åˆ°å…¶å˜åŠ¨ï¼Œå› äØ“˜q™äº›å·¥ä½œå®Œå…¨åœ¨æœåŠ¡å™¨ç«¯è¿›è¡Œã€?/p>
æœåŠ¡å™¨æ— ¾~é‡å?切æ¢
æƒÏx³•æ˜¯ï¼Œæˆ‘们˜q代了一版本åQŒéœ€è¦éƒ¨¾|²åˆ°¾U¿ä¸ŠåQŒäؓ之å¯åŠ¨ä¸€ä¸ªæ–°çš„è¿›½E‹åŽåQŒç¨åŽå…³é—旧版本˜q›ç¨‹½E‹åºåQŒæœåŠ¡ä¸€ç›´åœ¨˜qè¡Œä¸ä¸é—´æ–åQŒéœ€è¦åã^衡过度。这ž®±åƒErlangè¯è¨€å±‚é¢æ‰€æ供的çƒæ›´æ–°ä¸€æ —÷€?/p>
æƒÏx³•ä¸é”™åQŒä½†æ˜¯å®žé™…æ“作è“væ¥ï¼Œž®×ƒ¸æ˜¯é‚£ä¹ˆåã^滑了åQŒè¿˜å¥½æœ‰ä¸€ä¸?a >hubtimeå¼€æºå·¥å…øP¼ŒåŽŸç†ä¸?code>SIGHUPä¿¡å·å¤„ç†å™?SO_REUSEPORT+LD_RELOADåQŒå¯ä»¥å¸®åŠ©æˆ‘们轻æ‘֚刎ͼŒæœ‰éœ€è¦çš„åŒå¦å¯ä»¥‹‚€å‡ø™¯•ç”¨ä¸€ä¸‹ã€?/p>
SO_REUSEPORT已知问题
SO_REUSEPORTæ ÒŽ®æ•°æ®åŒ…的四元¾l„{src ip, src port, dst ip, dst port}和当å‰ç»‘定åŒä¸€ä¸ªç«¯å£çš„æœåŠ¡å™¨å¥—接å—数釘q›è¡Œæ•°æ®åŒ…分å‘。若æœåŠ¡å™¨å¥—接å—æ•°é‡äº§ç”Ÿå˜åŒ–åQŒå†…æ 怼šæŠŠæœ¬è¯¥ä¸Šä¸€ä¸ªæœåŠ¡å™¨å¥—接å—所处ç†çš„客æˆïL«¯˜qžæŽ¥æ‰€å‘é€çš„æ•°æ®åŒ…（比如三次æ¡æ‰‹æœŸé—´çš„劘qžæŽ¥åQŒä»¥åŠå·²¾l完æˆæ¡æ‰‹ä½†åœ¨é˜Ÿåˆ—ä¸æŽ’队的连接）分å‘到其它的æœåŠ¡å™¨å¥—接å—上é¢åQŒå¯èƒ½ä¼šå¯ÆD‡´å®¢æˆ·ç«¯è¯·æ±‚失败，一般å¯ä»¥ä‹É用：
- 使用固定的æœåŠ¡å™¨å¥—接å—æ•°é‡ï¼Œä¸è¦åœ¨è´Ÿè½½ç¹å¿™æœŸé—´è½»æ˜“å˜åŒ?/li>
- å…许多个æœåŠ¡å™¨å¥—接å—å…׃ínTCPè¯äh±‚è¡?Tcp request table)
- ä¸ä‹É用四元组作äØ“Hashå€ÆD¿›è¡Œé€‰æ‹©æœ¬åœ°å¥—接å—处ç†ï¼ŒæŒ‘选隶属于åŒä¸€ä¸ªCPU的套接å—
与RFS/RPS/XPS-mqå作åQŒå¯ä»¥èŽ·å¾—进一æ¥çš„性能åQ?/p>
- æœåŠ¡å™¨çº¿½E‹ç»‘定到CPUs
- RPS分å‘TCP SYN包到对应CPUæ 怸Š
- TCP˜qžæŽ¥è¢«å·²¾l‘定到CPU上的¾U¿ç¨‹accept()
- XPS-mq(Transmit Packet Steering for multiqueue)åQŒä¼ 输队列和CPU¾l‘定åQŒå‘é€æ•°æ?/li>
- RFS/RPSä¿è¯åŒä¸€ä¸ªè¿žæŽ¥åŽ¾læ•°æ®åŒ…都会被分å‘到åŒä¸€ä¸ªCPUä¸?/li>
- ¾|‘å¡æŽ¥æ”¶é˜Ÿåˆ—已绾l‘定到CPUåQŒåˆ™RFS/RPSåˆ™æ— ™å»è®¾¾|?/li>
- 需è¦æ³¨æ„硬件支æŒä¸Žå?/li>
目的嘛，数æ®åŒ…的软硬ä¸æ–ã€æŽ¥æ”¶ã€å¤„ç†ç‰åœ¨ä¸€ä¸ªCPUæ 怸ŠåQŒåƈ行化处ç†åQŒå°½å¯èƒ½åšåˆ°èµ„æºåˆ©ç”¨æœ€å¤§åŒ–ã€?/p>
SO_REUSEPORTä¸æ˜¯ä¸€è´´ä¸‡èƒ½è†è?/h4>
虽然SO_REUSEPORT解决了多个进½E‹å…±åŒç»‘å®?监å¬åŒä¸€ç«¯å£çš„é—®é¢˜ï¼Œä½†æ ¹æ®æ–°‹¹ªæž—晓峰åŒå¦‹¹‹è¯•¾l“æžœæ¥çœ‹åQŒåœ¨å¤šæ ¸æ‰©å±•å±‚é¢ä¹Ÿæœªèƒ½å¤Ÿåšåˆ°ç†æƒ³çš„线性扩展：
å¯ä»¥å‚考Fastsocket在其基础之上的改˜q›ï¼Œé“¾æŽ¥åœ°å€ã€?/p>
支æŒSO_REUSEPORTçš„Tengine
æ·˜å®çš„Tengineå·²ç»æ”¯æŒäº†SO_REUSEPORTç‰ÒŽ€§ï¼Œåœ¨å…¶‹¹‹è¯•æŠ¥å‘Šä¸ï¼Œæœ‰ä¸€ä¸ªç®€å•æµ‹è¯•ï¼Œå¯ä»¥çœ‹å‡ºæ¥ç›¸å¯ÒŽ¯”SO_REUSEPORT所带æ¥çš„性能æå‡åQ?/p>
使用SO_REUSEPORT以åŽåQŒæœ€æ˜Žæ˜¾çš„效果是在压力下ä¸å®¹æ˜“出çŽîC¸¢è¯äh±‚的情况，CPUå‡è¡¡æ€§åã^½EŸë€?/p>
Java支æŒå¦ï¼Ÿ
JDK 1.6è¯è¨€å±‚é¢ä¸æ”¯æŒï¼Œè‡³äºŽä»¥åŽçš„版本，ç”׃ºŽæš‚时没有使用刎ͼŒä¸å¤šè¯´ã€?/p>
Netty 3/4版本默认都ä¸æ”¯æŒSO_REUSEPORTç‰ÒŽ€§ï¼Œä½†Netty 4.0.19以åŠä¹‹åŽç‰ˆæœ¬æ‰çœŸæ£æ供了JNIæ–¹å¼å•ç‹¬åŒ…装的epoll native transport版本åQˆåœ¨Linux¾pÈ»Ÿä¸‹è¿è¡Œï¼‰åQŒå¯ä»¥é…¾|®ç±»ä¼égºŽSO_REUSEPORT½{‰ï¼ˆJAVA NIIO没有æä¾›åQ‰é€‰é¡¹åQŒè¿™éƒ¨åˆ†æ˜¯åœ¨io.netty.channel.epoll.EpollChannelOptionä¸å®šä¹‰ï¼ˆåœ¨çº¿ä»£ç 部分åQ‰ã€?/p>
在linux环境下ä‹É用epoll native transportåQŒå¯ä»¥èŽ·å¾—å†…æ ¸å±‚é¢ç½‘¾lœå †æ ˆå¢žå¼ºçš„¾U¢åˆ©åQŒå¦‚何ä‹É用å¯å‚è€?a >Native transports文档ã€?/p>
使用epoll native transport倒也½Ž€å•ï¼Œ¾cÕd½E作替æ¢åQ?/p>
NioEventLoopGroup → EpollEventLoopGroup
NioEventLoop → EpollEventLoop
NioServerSocketChannel → EpollServerSocketChannel
NioSocketChannel → EpollSocketChannel
比如写一个PING-PONG应用æœåŠ¡å™¨ç¨‹åºï¼Œ¾cÖM¼¼ä»£ç åQ?/p>
public void run() throws Exception {
EventLoopGroup bossGroup = new EpollEventLoopGroup();
EventLoopGroup workerGroup = new EpollEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
ChannelFuture f = b
.group(bossGroup, workerGroup)
.channel(EpollServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)
throws Exception {
ch.pipeline().addLast(
new StringDecoder(CharsetUtil.UTF_8),
new StringEncoder(CharsetUtil.UTF_8),
new PingPongServerHandler());
}
}).option(ChannelOption.SO_REUSEADDR, true)
.option(EpollChannelOption.SO_REUSEPORT, true)
.childOption(ChannelOption.SO_KEEPALIVE, true).bind(port)
.sync();
f.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
è‹¥ä¸è¦è¿™ä¹ˆæŠ˜è…¾ï¼Œ˜q˜æƒ³è®©ä»¥å¾€Java/Netty应用½E‹åºåœ¨ä¸åšä“Q何改动的å‰æ下顺利在Linux kernel >= 3.9下åŒæ ·äínå—到SO_REUSEPORT带æ¥çš„好处，ä¸å¦¨ž®è¯•ä¸€ä¸?a >bindpåQŒæ›´ä¸ºç»‹¹Žï¼Œ˜q™ä¸€éƒ¨åˆ†ä¸‹é¢ä¼šè®²åˆ°ã€?/p>
bindpåQŒäؓ已有应用æ·ÕdŠ SO_REUSEPORTç‰ÒŽ€?/h3>
以å‰æ‰€å†?a >bindpž®ç¨‹åºï¼Œå¯ä»¥ä¸ºå·²æœ‰ç¨‹åºç»‘定指定的IP地å€å’Œç«¯å£ï¼Œä¸€æ–šw¢å¯ä»¥çœåŽ»¼‹¬ç¼–ç ，å¦ä¸€æ–šw¢ä¹ŸäØ“‹¹‹è¯•æ供了一些方ä¾Ñ€?/p>
å¦å¤–åQŒäؓ了让以å‰æ²¡æœ‰¼‹¬ç¼–ç ?code>SO_REUSEPORT的应用程åºå¯ä»¥åœ¨Linuxå†…æ ¸3.9以åŠä¹‹åŽLinux¾pÈ»Ÿä¸Šä¹Ÿèƒ½å¤Ÿå¾—åˆ°å†…æ ¸å¢žå¼ºæ”¯æŒåQŒç¨åšä¿®æ”¹ï¼Œæ·ÕdŠ 支æŒã€?/p>
但è¦æ±‚如下：
- Linuxå†…æ ¸(>= 3.9)支æŒSO_REUSEPORTç‰ÒŽ€?/li>
- 需è¦é…¾|?code>REUSE_PORT=1
ä¸æ»¡‘³ä»¥ä¸Šæ¡ä»Óž¼Œæ¤ç‰¹æ€§å°†æ— 法生效ã€?/p>
使用½Cø™ŒƒåQ?/p>
REUSE_PORT=1 BIND_PORT=9999 LD_PRELOAD=./libbindp.so java -server -jar pingpongserver.jar &
当然åQŒä½ å¯ä»¥æ ÒŽ®éœ€è¦è¿è¡Œå‘½ä»¤å¤š‹Æ¡ï¼Œå¤šä¸ª˜q›ç¨‹ç›‘å¬åŒä¸€ä¸ªç«¯å£ï¼Œå•æœº˜q›ç¨‹æ°´åã^扩展ã€?/p>
使用½Cø™Œƒ
使用python脚本快速构å»ÞZ¸€ä¸ªå°çš„示范原型，两个˜q›ç¨‹åQŒéƒ½ç›‘å¬åŒä¸€ä¸ªç«¯å?0000åQŒå®¢æˆïL«¯è¯äh±‚˜q”回ä¸åŒå†…容åQŒä»…供娱ä¹ã€?/p>
server_v1.pyåQŒç®€å•PING-PONGåQ?/p>
# -*- coding:UTF-8 -*-
import socket
import os
PORT = 10000
BUFSIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', PORT))
s.listen(1)
while True:
conn, addr = s.accept()
data = conn.recv(PORT)
conn.send('Connected to server[%s] from client[%s]\n' % (os.getpid(), addr))
conn.close()
s.close()
server_v2.pyåQŒè¾“出当å‰æ—¶é—ß_¼š
# -*- coding:UTF-8 -*-
import socket
import time
import os
PORT = 10000
BUFSIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', PORT))
s.listen(1)
while True:
conn, addr = s.accept()
data = conn.recv(PORT)
conn.send('server[%s] time %s\n' % (os.getpid(), time.ctime()))
conn.close()
s.close()
借助于bindp˜q行两个版本的程åºï¼š
REUSE_PORT=1 LD_PRELOAD=/opt/bindp/libindp.so python server_v1.py &
REUSE_PORT=1 LD_PRELOAD=/opt/bindp/libindp.so python server_v2.py &
模拟客户端请æ±?0‹Æ¡ï¼š
for i in {1..10};do echo "hello" | nc 127.0.0.1 10000;done
看看¾l“æžœå§ï¼š
Connected to server[3139] from client[('127.0.0.1', 48858)]
server[3140] time Thu Feb 12 16:39:12 2015
server[3140] time Thu Feb 12 16:39:12 2015
server[3140] time Thu Feb 12 16:39:12 2015
Connected to server[3139] from client[('127.0.0.1', 48862)]
server[3140] time Thu Feb 12 16:39:12 2015
Connected to server[3139] from client[('127.0.0.1', 48864)]
server[3140] time Thu Feb 12 16:39:12 2015
Connected to server[3139] from client[('127.0.0.1', 48866)]
Connected to server[3139] from client[('127.0.0.1', 48867)]
å¯ä»¥çœ‹å‡ºæ¥ï¼ŒCPU分é…很å‡è¡¡ï¼Œå„自分é…50%的请求é‡ã€?/p>
嗯，虽是ž®çŽ©å…øP¼Œæœ‰äº›æ„æ€?:))
bindpçš„ä‹É用方æ³?/h4>
更多使用说明åQŒè¯·å‚è€?a >READMEã€?/p>
å‚考资æ–?/h3>
- 《SO_REUSEPORT: Scaling Techniques for Servers with High Connection Rates》PPT
- huptime
- SO_REUSEPORT and accept(2) performance
- 多个˜q›ç¨‹¾l‘定相åŒç«¯å£çš„实现分æž[Google Patch]
nieyong 2015-02-12 16:50 å‘表评论]]>Fastsocketå¦ä¹ ½W”记之尾l“篇 http://www.aygfsteel.com/yongboy/archive/2015/02/05/422760.htmlnieyong nieyong Thu, 05 Feb 2015 07:21:00 GMT http://www.aygfsteel.com/yongboy/archive/2015/02/05/422760.html http://www.aygfsteel.com/yongboy/comments/422760.html http://www.aygfsteel.com/yongboy/archive/2015/02/05/422760.html#Feedback 3 http://www.aygfsteel.com/yongboy/comments/commentRss/422760.html http://www.aygfsteel.com/yongboy/services/trackbacks/422760.html å‰è¨€
å‰é¢å•°å•°å—¦å—¦çš„å‡ ½‹‡æ–‡å—，å„个斚w¢ä»‹ç»äº†FastsocketåQŒç›²äººæ‘¸è±¡ä¸€èˆ¬ï¼Œèƒ½åŠ›æœ‰é™åQŒè¿˜å¾—ç‘ô¾l深入å¦ä¹ ä¸æ˜¯ã€‚è¿™ä¸ï¼ŒåˆîCº†è¯¥å°¾l“收ž®„¡š„时候了ã€?/p>
¾~˜è“våQŒå†…æ ¸å·²¾læˆä¸ºç“¶é¢?/h3>
使用Linux作äØ“æœåŠ¡å™¨ï¼Œåœ¨è¯·æ±‚é‡å¾ˆå°çš„时候，是ä¸ç”¨æ‹…心其性能。但在æ“vé‡çš„æ•°æ®è¯äh±‚下，Linuxå†…æ ¸åœ¨TCP/IP¾|‘络处ç†æ–šw¢åQŒå·²¾læˆä¸ºç“¶é¢ˆã€‚比如新‹¹ªåœ¨æŸå°HAProxyæœåŠ¡å™¨ä¸Šå–æ ·åQ?0%çš„CPUæ—‰™—´è¢«å†…æ ¸å 用，应用½E‹åºåªèƒ½å¤Ÿåˆ†é…到较少的CPUæ—‰™’Ÿå‘¨æœŸçš„资æºã€?/p>
¾l过Haproxy¾pÈ»Ÿè¯¦å°½åˆ†æžåŽï¼Œå‘现大部分CPU资æºæ¶ˆè€—在kernel里，òq¶ä¸”åœ¨å¤šæ ¸åã^åîC¸‹åQŒkernel在网¾lœåè®®æ ˆå¤„ç†˜q‡ç¨‹ä¸å˜åœ¨ç€å¤§é‡åŒæ¥å¼€é”€ã€?/p>
åŒæ—¶åœ¨å¤šæ 怸Š˜q›è¡Œ‹¹‹è¯•åQŒHTTP CPS(Connection Per Second)åžåé‡åƈ没有éšç€CPUæ ¸æ•°å¢žåŠ å‘ˆçŽ°¾U¿æ€§å¢žé•¿ï¼š
å‰è¨€
å‰é¢å•°å•°å—¦å—¦çš„å‡ ½‹‡æ–‡å—，å„个斚w¢ä»‹ç»äº†FastsocketåQŒç›²äººæ‘¸è±¡ä¸€èˆ¬ï¼Œèƒ½åŠ›æœ‰é™åQŒè¿˜å¾—ç‘ô¾l深入å¦ä¹ ä¸æ˜¯ã€‚è¿™ä¸ï¼ŒåˆîCº†è¯¥å°¾l“收ž®„¡š„时候了ã€?/p>
¾~˜è“våQŒå†…æ ¸å·²¾læˆä¸ºç“¶é¢?/h3>
使用Linux作äØ“æœåŠ¡å™¨ï¼Œåœ¨è¯·æ±‚é‡å¾ˆå°çš„时候，是ä¸ç”¨æ‹…心其性能。但在æ“vé‡çš„æ•°æ®è¯äh±‚下，Linuxå†…æ ¸åœ¨TCP/IP¾|‘络处ç†æ–šw¢åQŒå·²¾læˆä¸ºç“¶é¢ˆã€‚比如新‹¹ªåœ¨æŸå°HAProxyæœåŠ¡å™¨ä¸Šå–æ ·åQ?0%çš„CPUæ—‰™—´è¢«å†…æ ¸å 用，应用½E‹åºåªèƒ½å¤Ÿåˆ†é…到较少的CPUæ—‰™’Ÿå‘¨æœŸçš„资æºã€?/p>
¾l过Haproxy¾pÈ»Ÿè¯¦å°½åˆ†æžåŽï¼Œå‘现大部分CPU资æºæ¶ˆè€—在kernel里，òq¶ä¸”åœ¨å¤šæ ¸åã^åîC¸‹åQŒkernel在网¾lœåè®®æ ˆå¤„ç†˜q‡ç¨‹ä¸å˜åœ¨ç€å¤§é‡åŒæ¥å¼€é”€ã€?/p>
åŒæ—¶åœ¨å¤šæ 怸Š˜q›è¡Œ‹¹‹è¯•åQŒHTTP CPS(Connection Per Second)åžåé‡åƈ没有éšç€CPUæ ¸æ•°å¢žåŠ å‘ˆçŽ°¾U¿æ€§å¢žé•¿ï¼š
å†…æ ¸3.9之å‰çš„Linux TCP调用
- kernel 3.9之å‰çš„tcp socket实现
- bind¾pÈ»Ÿè°ƒç”¨ä¼šå°†socketå’Œport˜q›è¡Œ¾l‘定åQŒåÆˆåŠ å…¥å…¨å±€tcp_hashinfoçš„bhash链表ä¸?
- 所有bind调用都会查询˜q™ä¸ªbhash链表åQŒå¦‚æžœport被å ç”¨ï¼Œå†…æ ¸ä¼šå¯¼è‡´bindå¤ÞpÓ|
- listen则是æ ÒŽ®ç”¨æˆ·è®„¡½®çš„队列大ž®é¢„å…ˆäØ“tcp˜qžæŽ¥åˆ†é…内嘽Iºé—´
- 一个应用在åŒä¸€ä¸ªport上åªèƒ½listen一‹Æ¡ï¼Œé‚£ä¹ˆä¹Ÿå°±åªæœ‰ä¸€ä¸ªé˜Ÿåˆ—æ¥ä¿å˜å·²ç»å»ºç«‹çš„è¿žæŽ?
- nginx在listen之åŽä¼šfork处多个workeråQŒæ¯ä¸ªworker会ç‘ô承listençš„socketåQŒæ¯ä¸ªworker会创å»ÞZ¸€ä¸ªepoll fdåQŒåƈž®†listen fdå’Œacceptçš„æ–°˜qžæŽ¥çš„fdåŠ å…¥epoll fd
- 但是一旦新的连接到æ¥ï¼Œå¤šä¸ªnginx workeråªèƒ½æŽ’队accept˜qžæŽ¥˜q›è¡Œå¤„ç†
- 对于大é‡çš„矘qžæŽ¥åQŒaccept昄¡„¶æˆäؓ了一个瓶é¢?
Linux¾|‘ç»œå †æ ˆæ‰€å˜åœ¨é—®é¢˜
-
TCP处ç†&å¤šæ ¸
- 一个完整的TCP˜qžæŽ¥åQŒä¸æ–å‘生在一个CPUæ 怸ŠåQŒä½†åº”用数æ®å¤„ç†å¯èƒ½ä¼šåœ¨å¦å¤–ä¸€ä¸ªæ ¸ä¸?
- ä¸åŒCPUæ ¸å¿ƒå¤„ç†åQŒå¸¦æ¥äº†é”竞争和CPU Cache MissåQˆæ‡L动ä¸òqŒ™¡¡åQ?
- 多个˜q›ç¨‹ç›‘å¬ä¸€ä¸ªTCP套接å—，å…׃ín一个listen queue队列
- 用于˜qžæŽ¥½Ž¡ç†å…¨å±€å“ˆå¸Œè¡¨æ ¼åQŒå˜åœ¨èµ„æºç«žäº?
- epoll IO模型多进½E‹å¯¹accept½{‰å¾…åQŒæƒŠ¾Ÿ¤çŽ°è±?br />
-
Linux VFSçš„åŒæ¥æŸè€—严é‡?/p>
- Socket被VFS½Ž¡ç†
- VFSå¯ÒŽ–‡ä»¶èŠ‚点Inode和目录Dentry有åŒæ¥éœ€æ±?
- SOCKETåªéœ€è¦åœ¨å†…å˜ä¸å˜åœ¨å³å¯ï¼Œéžä¸¥æ ¼æ„义上文äšg¾pÈ»ŸåQŒä¸éœ€è¦Inodeå’ŒDentry
- 代ç 层é¢ç•¥è¿‡ä¸å¿…™åÈš„常规é”，但åˆä¿æŒäº†èƒö够的兼容æ€?
Fastsocket所作改˜q?/h3> - TCPå•ä¸ª˜qžæŽ¥å®Œæ•´å¤„ç†åšåˆ°äº†CPU本地化，é¿å…了资æºç«žäº?
- ä¿æŒå®Œæ•´BSD socket API
CPU之间ä¸å…±äº«æ•°æ®ï¼Œòq¶è¡ŒåŒ–å„自独立处ç†TCP˜qžæŽ¥åQŒä¹Ÿæ˜¯å…¶é«˜æ•ˆçš„主è¦åŽŸå› 。其架构囑֯以看出其改进åQ?/p>
Fastsocketæž¶æž„å›‘Ö¯ä»¥å¾ˆæ¸…æ™°è¯´æ˜Žå…¶å¤§è‡´ç»“æž„ï¼Œå†…æ ¸æ€å’Œç”¨æˆ·æ€é€šè¿‡ioctlå‡½æ•°ä¼ è¾“ã€‚è®°å¾—netmap在é‡å†™ç½‘å¡é©±åŠ¨é‡Œé¢é€šè¿‡ioctl函数直接é€ä¼ 到用æˆäh€ä¸åQŒå…¶æ›´äؓ高效åQŒä½†æ²¡æœ‰å®Œæ•´çš„TCP/IP¾|‘ç»œå †æ ˆæ”¯æŒå˜›ã€?/p>
Fastsocket的TCP调用�/h4>
- 多个˜q›ç¨‹å¯ä»¥åŒæ—¶listen在åŒä¸€ä¸ªportä¸?
- 动æ€é“¾æŽ¥åº“libfsocket.so拦截socketã€bindã€listen½{‰ç³»¾lŸè°ƒç”¨åƈ˜q›å…¥˜q™ä¸ªé“¾æŽ¥åº“进行处ç?
- 对于listen¾pÈ»Ÿè°ƒç”¨åQŒfastsocket会记录下˜q™ä¸ªfdåQŒå½“应用通过epollž®†è¿™ä¸ªfdåŠ å…¥åˆ°epoll fdsetä¸æ—¶åQŒlibfsocket.so会通过ioctlä¸ø™¯¥˜q›ç¨‹clone listen fdå…Œ™”çš„socketã€sockã€fileçš„ç³»¾lŸèµ„æº?
- å†…æ ¸æ¨¡å—ž®†cloneçš„socketå†æ¬¡è°ƒç”¨bindå’Œlisten
- bind¾pÈ»Ÿè°ƒç”¨‹‚€‹¹‹åˆ°å¦å¤–一个进½E‹ç»‘定到已ç»è¢«ç»‘定的portæ—Óž¼Œä¼šè¿›è¡Œç›¸å…Ïx£€æŸ?
- 通过‹‚€æŸ¥sockž®†ä¼šè¢«è®°å½•åˆ°port相关è”的一个链表ä¸åQŒé€šè¿‡è¯¥é“¾è¡¨å¯ä»¥çŸ¥é“所有bindåŒä¸€ä¸ªportçš„sock
- 而sock是关è”到fd的，˜q›ç¨‹åˆ™æŒæœ‰fdåQŒé‚£ä¹ˆæ‰€æœ‰çš„资溞®±å·²¾lå…³è”到一èµ?
- æ–°çš„˜q›ç¨‹å†æ¬¡è°ƒç”¨listen¾pÈ»Ÿè°ƒç”¨çš„时候，fastsocketå†…æ ¸ä¼šå†‹Æ¡äؓ其关è”çš„sock分é…accept队列
- ¾l“果是多个进½E‹ä¹Ÿž®±æ‹¥æœ‰äº†å¤šä¸ªaccept队列åQŒå¯é¿å…cpu cache miss
- fastsocketæä¾›ž®†æ¯ä¸ªlistenå’Œacceptçš„è¿›½E‹ç»‘定到用户指定的CPUæ ?
- 如果用户未指定，fastsocketž®†ä¼šä¸ø™¯¥˜q›ç¨‹é»˜è®¤¾l‘定一个空闲的CPUæ ?
FastsocketçŸè¿žæŽ¥æ€§èƒ½
在新‹¹ªæµ‹è¯•ä¸åQŒåœ¨24æ ¸çš„å®‰è£…æœ‰Centos 6.5çš„æœåŠ¡å™¨ä¸Šï¼Œå€ŸåŠ©äºŽFastsocketåQŒNginxå’ŒHAProxyæ¯ç§’处熘qžæŽ¥æ•°æŒ‡æ ‡ï¼ˆconnection/secondåQ‰æ€§èƒ½å¾ˆæƒŠäººï¼Œåˆ†åˆ«å¢žåŠ 290%å’?20%。这也è¯æ˜Žäº†åQŒFastsocket带æ¥äº†TCP˜qžæŽ¥å¿«é€Ÿå¤„ç†çš„能力ã€?除æ¤ä¹‹å¤–åQŒå€ŸåŠ©äºŽç¡¬ä»¶ç‰¹æ€§ï¼š
- 借助于Intel‘…çñ”¾U¿ç¨‹åQŒå¯ä»¥èŽ·å¾—å¦å¤?0%的性能增长
- HAProxy代ç†æœåŠ¡å™¨å€ŸåŠ©äºŽç½‘å¡Flow-Directorç‰ÒŽ€§æ”¯æŒï¼Œåžåé‡å¯å¢žåŠ 15%
Fastsocket V1.0æ£å¼ç‰ˆä»Ž2014òq?月䆾开始已¾l在新浪生äñ”环境ä¸ä‹É用，用作代ç†æœåŠ¡å™¨ï¼Œå› æ¤å¤§å®¶å¯ä»¥è€ƒè™‘是å¦å¯ä»¥é‡‡ç”¨ã€‚é’ˆå¯?.0版本åQŒä»¥ä¸‹çŽ¯å¢ƒè¾ƒä¸ºæ”¶ç›Šï¼š
- æœåŠ¡å™¨è‡³ž®‘丞®‘于8个CPUæ ¸å¿ƒ
- çŸè¿žæŽ¥è¢«å¤§é‡ä½¿ç”¨
- CPU周期大部分消耗在¾|‘络软ä¸æ–和套接å—ç³»¾lŸè°ƒç”¨ä¸Š
- 应用½E‹åºä½¿ç”¨åŸÞZºŽepollçš„éžé˜Õd¡žIO
- 应用½E‹åºä½¿ç”¨å¤šä¸ª˜q›ç¨‹å•ç‹¬æŽ¥å—˜qžæŽ¥
多线½E‹å˜›åQŒå°±å¾—需è¦å‚考示范应用所æ供实践å»ø™®®äº†ã€?/p>
Nginx‹¹‹è¯•æœåŠ¡å™¨é…¾|?/h4> - nginx工作˜q›ç¨‹æ•°é‡è®„¡½®æˆCPUæ ¸æ•°ä¸?
- http keep-aliveç‰ÒŽ€§è¢«¼›ç”¨
- ‹¹‹è¯•ç«¯http_load从nginx获å–64å—节é™æ€æ–‡ä»Óž¼Œòq¶å‘é‡äØ“500*CPUæ ¸æ•°
- å¯ç”¨å†…嘾~“å˜é™æ€æ–‡ä»¶è®¿é—®ï¼Œç”¨äºŽæŽ’除¼‚盘影å“
- 务必¼›ç”¨accept_mutexåQˆå¤šæ ¸è®¿é—®accept产生é”竞争，å¦fastsocketå†…æ ¸æ¨¡å—为其去除了é”竞争åQ?
从下表测试图片ä¸åQŒå¯ä»¥çœ‹åˆŽÍ¼š
- Fastsocketåœ?4æ ¸æœåŠ¡å™¨è¾‘Öˆ°äº?75K Connection/SecondåQŒèŽ·å¾—了21å€çš„æå‡
- Centos 6.5在CPUæ ¸æ•°å¢žé•¿åˆ?2æ ¸æ—¶òq¶æ²¡æœ‰å‘ˆçŽ°çº¿æ€§å¢žé•¿åŠ¿å¤ß_¼Œå而在24æ ¸æ—¶ä¸‹é™åˆ?59k CPS
- Linux kernel 3.13åœ?4æ ¸æ—¶èŽ·å¾—äº†è¿‘ä¹Žä¸¤å€äºŽCentos 6.5çš„åžåé‡åQ?83K CPSåQŒä½†åœ?2æ ¸åŽå‘ˆçŽ°å‡ºæ‰©å±•æ€§ç“¶é¢?
HAProxyé‡è¦é…ç½®
- 工作˜q›ç¨‹æ•°é‡½{‰åŒäºŽCPUæ ¸æ•°ä¸?
- 需è¦å¯ç”¨RFD(Receive Flow Deliver)
- http keep-alive需è¦ç¦ç”?
- ‹¹‹è¯•ç«¯http_loadòq¶å‘é‡äØ“500*CPUæ ¸æ•°
- åŽç«¯æœåŠ¡å™¨å“应外å›?4个å—节的消æ¯
‹¹‹è¯•¾l“æžœä¸ï¼š
- fastsocket呈现å‡ÞZº†æƒŠäh的扩展性能
- 24æ ¸ï¼ŒLinux kernel 3.13æˆç‡Wä¸?39K CPS
- 24æ ¸ï¼ŒCentos 6.5借助FastsocketåQŒèŽ·å¾—了370K CPSçš„åžåé‡
实际部çÖv环境的戾l?/h4>
8æ ¸æœåŠ¡å™¨¾U¿ä¸ŠçŽ¯å¢ƒ˜qè¡Œäº?4ž®æ—¶çš„戾l©ï¼Œå›¾a展示了部¾|²fastsocket之å‰CPU利用率，图b为部¾|²äº†fastsocekt之åŽçš„CPU利用率ã€?Fastsocket带æ¥çš„收益：
- æ¯ä¸ªCPUæ ¸å¿ƒè´Ÿè²å‡è¡¡
- òq›_‡CPU利用率é™ä½?0%
- HAProxy处ç†èƒ½åŠ›å¢žé•¿85%
其实å§ï¼Œ˜q™ä¸€å—期待新‹¹ªå…¬å¸ƒæ›´å¤šçš„æ•°æ®ã€?/p>
长连接的支æŒæ£åœ¨å¼€å‘ä¸
长连接支æŒï¼Œ˜q˜æ˜¯éœ€è¦ç‰ä¸€½{‰çš„。但是è¦æ”¯æŒä»€ä¹ˆç±»åž‹é•¿˜qžæŽ¥åQŸç™¾ä¸‡çñ”别应用æœåŠ¡å™¨¾cÕdž‹åQŒè¿˜æ˜¯redisåQŒå¯èƒ½æ˜¯åŽè€…。虽然目å‰æ£åšï¼Œä½†ç›®å‰æ²¡æœ‰æ—¶é—´è¡¨åQŒä½†ç›®å‰æ‰€åšç‰¹æ€§æ€È»“如下åQ?/p>
- ¾|‘ç»œå †æ ˆçš„å®šåˆ?
- SKB-PoolåQŒæ¯ä¸€CPUæ ¸å¯¹åº”ä¸€ä¸ªé¢„åˆ†é…skb poolåQŒæ›¿æ¢å†…æ ¸ç¼“å†²åŒºkernel slab
- Percore skb pool
- åˆåƈskb头部和数æ?
- 本地Poolå’Œé‡å¤åó@环ä‹É用的PoolåQˆFlow-DirectoråQ?
- Fast-Epoll
- 多进½E‹ä¹‹é—´TCP˜qžæŽ¥å…׃ínå˜å¾—½E€ž®?
- 在file¾l“构体ä¸ä¿å˜Epoll entryåQŒç”¨ä»¥èŠ‚çœè°ƒç”¨epoll_ctlæ—¶çº¢é»‘æ ‘æŸ¥è¯¢çš„å¼€é”€
- SKB-PoolåQŒæ¯ä¸€CPUæ ¸å¯¹åº”ä¸€ä¸ªé¢„åˆ†é…skb poolåQŒæ›¿æ¢å†…æ ¸ç¼“å†²åŒºkernel slab
- 跨层的设�
- Direct-TCPåQŒæ•°æ®åŒ…隶属于已建立套接å—会直接跌™¿‡è·¯ç”±˜q‡ç¨‹
- 记录TCP套接å—的输入路由信æ¯åQˆRecord input route information in TCP socketåQ?
- 直接查找¾|‘络套接å—在˜q›å…¥¾|‘ç»œå †æ ˆä¹‹å‰åQˆLookup socket directly before network stackåQ?
- 从套接å—è¯Õd–输入路由信æ¯åQˆRead input route information from socketåQ?
- æ ‡è®°æ•°æ®åŒ…被路有˜q‡ï¼ˆMark the packet as routedåQ?
- Receive-CPU-Selection ¾cÖM¼¼äºŽRFSåQŒä½†æ›´è½»å·§ã€ç²¾å‡†ä¸Žå¿«é€?
- 把当å‰CPUæ ¸id¾~–ç 到套接å—ä¸ï¼ˆApplication marks current CPU id in the socketåQ?
- 直接查询套接å—在˜q›å…¥¾|‘ç»œå †æ ˆä¹‹å‰åQˆLookup socket directly before network stackåQ?
- è¯Õd–套接å—ä¸åŒ…å«çš„CPUæ ¸ï¼Œç„¶åŽå‘é€ç»™å®ƒï¼ˆRead CPU id from socket and deliver accordinglyåQ?
- RPS-Framework æ•°æ®åŒ…在˜q›å…¥¾|‘ç»œå †æ ˆä¹‹å‰åQŒè®©å¼€å‘è€…åœ¨å†…æ ¸æ¨¡å—之外定制数æ®åŒ…投递规则，扩充RPS功能
- Direct-TCPåQŒæ•°æ®åŒ…隶属于已建立套接å—会直接跌™¿‡è·¯ç”±˜q‡ç¨‹
Redis‹¹‹è¯•¾l“æžœ
‹¹‹è¯•çŽ¯å¢ƒ:
- CPU: Intel E5 2640 v2 (6 core) * 2
- NIC: Intel X520
Redisé…置选项:
- TCPæŒä¹…˜qžæŽ¥
- 8个Redis实例åQŒç»‘定ä¸åŒç«¯å?
- 使用åˆ?个CPUæ ¸å¿ƒåQŒåƈ且绑定CPUæ ?
‹¹‹è¯•¾l“æžœåQ?/p>
- ä»…å¼€å¯RSSåQ?0%çš„åžåé‡å¢žåŠ
- å¯ç”¨¾|‘å¡Flow-Directorç‰ÒŽ€§ï¼š45%åžåé‡å¢žåŠ?
但需è¦æ³¨æ„：
- ä»…äؓ实验‹¹‹è¯•é˜¶æ®µ
- 为V1.0补充åQŒNginxå’ŒHAProxyåŒæ ·ä¼šæ”¶ç›?
Fastsocket v1.1
V1.1版本è¦å¢žåŠ é•¿˜qžæŽ¥çš„支æŒï¼Œé‚£ä¹ˆ¾cÖM¼¼äºŽRedisçš„æœåŠ¡å™¨åº”用½E‹åºž®±å¾ˆå—ç›Šäº†ï¼Œå› äؓ没有具体的时间表åQŒåªèƒ½å¤Ÿæ…¢æ…¢½{‰å¾…了ã€?/p>
以åŽä¸€äº›ä¼˜åŒ–措æ–?/h3> - 在上下文切æ¢æ—Óž¼Œé¿å…拯‚´æ“作åQŒZero-Copy
- ä¸æ–机制完善åQŒå‡ž®‘ä¸æ–?
- 支æŒæ‰šw‡æ交åQŒé™ä½Žç³»¾lŸå‡½æ•°è°ƒç”?
- æ交到Linux kernelä¸Õdˆ†æ”¯ä¸ŠåŽ?
- HugeTLB/HugePage½{?
Fastsocketå’ŒmTCP½{‰ç®€å•å¯¹æ¯?/h3>
说是å¯ÒŽ¯”åQŒå…¶å®žæ˜¯æˆ‘从mTCP论文ä¸æ‘˜å–出æ¥ï¼Œå¢žåŠ 了Fastsocket一æ ，å¯ä»¥çœ‹å‡ºäºÞZ»¬ä¸€ç›´åŠªåŠ›çš„è„šæ¥ã€?/p>
| Types | Accept queue | Conn. Locality | Socket API | Event Handling | Packet I/O | Application Mod- ification | Kernel Modification |
| PSIO , DPDK , PF RING , netmap | No TCP stack | Batched | No interface for transport layer | No (NIC driver) | |||
| Linux-2.6 | Shared | None | BSD socket | Syscalls | Per packet | Transparent | No |
| Linux-3.9 | Per-core | None | BSD socket | Syscalls | Per packet | Add option SO REUSEPORT | No |
| Affinity-Accept | Per-core | Yes | BSD socket | Syscalls | Per packet | Transparent | Yes |
| MegaPipe | Per-core | Yes | lwsocket | Batched syscalls | Per packet | Event model to completion I/O | Yes |
| FlexSC,VOS | Shared | None | BSD socket | Batched syscalls | Per packet | Change to use new API | Yes |
| mTCP | Per-core | Yes | User-level socket | Batched function calls | Batched | Socket API to mTCP API | No (NIC driver) |
| Fastsocket | Per-core | Yes | BSD socket | Ioctl + kernel calls | Per packet | Transparent | No |
有一个大致的å°è±¡åQŒä¹Ÿæ–¹ä¾¿å¯ÒŽ¯”åQŒä½†˜q™åªèƒ½æ˜¯ä¸€ä¸ªæš‚时的摘è¦è€Œå·²åQŒäh¾cÕd¯¹æ€§èƒ½çš„æÍ求æ€ÀL˜¯æœç€æ›´å¥½çš„æ–¹å‘å‘展ç€ã€?/p>
部çÖvž®è¯•
怎么说呢åQŒFastsocket是äؓ大家耳熟能详æœåŠ¡å™¨ç¨‹åºNginxåQŒHAProxy½{‰è€Œå¼€å‘的。但若应用环境äؓ大é‡çš„矘qžæŽ¥åQŒåƈ且是ž®æ–‡ä»¶ç±»åž‹è¯·æ±‚，ä¸éœ€è¦å¼ºåˆ¶æ”¯æŒKeep-aliveç‰ÒŽ€§ï¼ˆçŸè¿žæŽ¥è¦çš„是快速请æ±?相应åQŒç„¶åŽå…³é—）åQŒé‚£ä¹ˆç®¡ç†å‘˜å¯ä»¥ž®è¯•ä¸€ä¸‹FastsocketåQŒè‡³äºŽéƒ¨¾|²ç–略，选择性部¾|²å‡ åîC½œä¸ºå®žéªŒçœ‹çœ‹ç»“æžœã€?/p>
ž®ç»“
本系列到æ¤ç®—是告一ŒDµè½å•¦ã€‚以åŽå‘¢åQŒè‡ªç„¶æ˜¯å¸Œæœ›Fastsocketž®½å¿«å‘布寚w•¿˜qžæŽ¥çš„支æŒï¼Œ˜q˜æœ‰æ›´é«˜æ€§èƒ½çš„æå‡å’¯ :))
资æºå¼•ç”¨
å‰è¨€
å‰é¢åˆ†æžFastsocket慢慢凑æˆäº†å‡ ½‹‡çƒ‚æ–‡å—åQŒè¦æŠŠä¸€ä»¶äº‹æƒ…åšæŒåšä¸‹æ¥åQŒæœ‰æ—¶å‘³åŒçˆµèœ¡ï¼Œä½†æ—¢ç„‰™€‰æ‹©äº†ï¼Œä¹Ÿå¾—¼‹¬ç€å¤´çš®åšä¸‹åŽ…R€‚é—²è¯å°‘è¯ß_¼Œæ–‡å½’æ£æ–‡ã€‚本文接自上½‹‡å†…æ ¸æ¨¡å—篇åQŒç‘ô¾l记录å¦ä¹ Fastsocketå†…æ ¸çš„ç¬”è®°å†…å®V€?/p>
Fastsocket建立在SO_REUSEPORT支æŒåŸºç¡€ä¸?/h3>
Linux kernel 3.9包å«TCP/UDP支æŒå¤šè¿›½E‹ã€å¤š¾U¿ç¨‹¾l‘定åŒä¸€ä¸ªIP和端å£çš„ç‰ÒŽ€§ï¼Œå?code>SO_REUSEPORTåQ›åœ¨å†…æ ¸å±‚é¢åŒæ—¶ä¹Ÿè®©¾U¿ç¨‹/˜q›ç¨‹ä¹‹é—´å„自独äínSOCKETåQŒé¿å…CPUæ 怹‹é—´ä»¥é”资æºäº‰å¤?code>accept queue的调用。在fastsocket/kernel/net/sock.h定义sock_common¾l“æž„æ—Óž¼Œå¯ä»¥çœ‹åˆ°å…¶èín影：
unsigned char skc_reuse:4;
unsigned char skc_reuseport:4;
在多个socket.hæ–‡äšgä¸ï¼ˆæ¯”如fastsocket/kernel/include/asm/socket.håQ‰ï¼Œå®šä¹‰äº†SO_REUSESORTçš„å˜é‡å€û|¼š
#define SO_REUSEPORT 15
在fastsocket/kernel/net/core/sock.cçš„sock_setsockoptå’Œsock_getsockopt函数ä¸ï¼Œéƒ½æœ‰SO_REUSEPORTçš„èín影：
sock_setsockopt函数ä¸ï¼š
case SO_REUSEADDR:
sk->sk_reuse = valbool;
break;
case SO_REUSEPORT:
sk->sk_reuseport = valbool;
break;
sock_getsockopt函数体ä¸åQ?/p>
case SO_REUSEADDR:
v.val = sk->sk_reuse;
break;
case SO_REUSEPORT:
v.val = sk->sk_reuseport;
break;
åœ?code>SO_REUSEPORTç‰ÒŽ€§æ”¯æŒä¹‹å‰çš„事äšg驱动驱动æœåŠ¡å™¨èµ„æºç«žäº‰ï¼š
之åŽå‘¢ï¼Œå¯ä»¥çœ‹åšæ˜¯åƈ行的了：
Fastsocket没有é‡å¤å‘明轮ååQŒåœ¨SO_REUSEPORT基础上进行进一æ¥çš„优化½{‰ã€?/p>
嗯，åŽé¢å‡†å¤‡å†™ä¸€ä¸ªåŠ¨æ€é“¾æŽ¥åº“ž®ç¨‹åºï¼Œæ‰“算让以å‰çš„没有¼‹¬ç¼–ç ?code>SO_REUSEPORT的程åºä¹Ÿèƒ½å¤Ÿåœ¨Linux kernel >= 3.9¾pÈ»Ÿä¸Šäínå—真æ£çš„端å£é‡ç”¨çš„æ–°ç‰ÒŽ€§çš„支æŒã€?/p>
Fastsocket架构�/h3>
下é¢æŒ‰ç…§å…¶æž¶æž„图所½Cºå†…æ ¸å±‚é¢ä»Žä¸Šåˆ°ä¸‹ä¸€ä¸€åˆ—出ã€?/p>
虚拟文äšg¾pÈ»ŸVFS的改˜q?/h3>
å› äØ“Linux Kernel VFSçš„åŒæ¥æŸè€—严é‡?/p>
- VFSå¯ÒŽ–‡ä»¶èŠ‚点Inode和目录Dentry有åŒæ¥éœ€æ±?
- 但SOCKETåªéœ€è¦åœ¨å†…å˜ä¸å˜åœ¨å³å¯ï¼Œéžä¸¥æ ¼æ„义上文äšg¾pÈ»ŸåQŒå…¶ä¸éœ€è¦èµ\径，ä¸éœ€è¦äØ“Inodeå’ŒDentryåŠ é”
- 代ç 层é¢ç•¥è¿‡ä¸å¿…™åÈš„常规é”，但åˆä¿æŒäº†èƒö够的兼容æ€?
æ交记录åQ?/p>
a209dfc vfs: dont chain pipe/anon/socket on superblock s_inodes list
4b93688 fs: improve scalability of pseudo filesystems
对VFS的改˜q›ï¼Œåœ¨æ‰€æå‡çš„性能ä¸å 有超˜q?0%的比例，效果éžå¸¸æ˜Žæ˜¾åQ?/p>
Local Listen Table
å¯¹äºŽå¤šæ ¸å¤šæŽ¥æ”‰™˜Ÿåˆ—æ¥è¯ß_¼Œlinux原生的åè®®æ ˆåªèƒ½listen在一个socket上é¢åQŒåƈ且所有完æˆä¸‰‹Æ¡æ¡æ‰‹è¿˜æ²¡æ¥å¾—åŠè¢«åº”用accept的套接å—都会攑օ¥å…‰™™„带的accept队列ä¸ï¼Œaccept¾pÈ»Ÿè°ƒç”¨å¿…须串行的从队列å–出åQŒå½“òq¶å‘é‡è¾ƒå¤§æ—¶å¤šæ ¸ç«žäº‰åQŒè¿™ž®†æˆä¸ºæ€§èƒ½ç“‰™¢ˆåQŒåª„å“å¾ç«‹è¿žæŽ¥å¤„ç†é€Ÿåº¦ã€?/p>
Local Listen TableåQŒfastsocket为æ¯ä¸€ä¸ªCPUæ ¸å…‹éš†ç›‘å¬å¥—接å—åQŒåƈä¿å˜åˆ°å…¶æœ¬åœ°è¡¨ä¸åQŒCPUæ 怹‹é—´ä¸ä¼šå˜åœ¨accept的竞争关¾p…R€‚下é¢äؓ引用æ述内容åQ?/p>
- æ¯ä¸ªcore有一个listen socket table。应用程åºå¾ç«‹è¿žæŽ¥çš„时候，执行˜q‡ç¨‹ä¼šè°ƒç”¨local_listen()函数åQŒæœ‰ä¸¤ä¸ªå‚æ•°åQŒä¸€ä¸ªæ˜¯socket FDåQŒä¸€ä¸ªæ˜¯core number. new socket从原始的listen socket(global)拯‚´åˆ°per-core local socket table. ˜q™äº›å¯¹äºŽåº”用½E‹åºæ¥è¯´éƒ½æ˜¯é€æ˜Žçš„，æä¾›¾l™åº”用程åºçš„socketFD是抽象过的，éšè—了底层的实现ã€?
- 当一个TCP SYN到达本机åQŒkernel首先去local listen tableä¸æ‰¾åŒšw…çš„listen socketåQŒå¦‚果找刎ͼŒž®±é€šè¿‡¾|‘å¡RSSä¼ é€’è¿™ä¸ªsocketåˆîC¸€ä¸ªcoreåQŒå¦åˆ™å°±åŽ»global listen tableä¸æ‰¾ã€?
- 定w”™æ–šw¢åQŒå½“˜q›ç¨‹å´©æºƒçš„è¯åQŒlocal listen socket会被关é—åQŒè¿›å…¥çš„˜qžæŽ¥ž®†ä¼šè¢«å¼•å¯¼åˆ°global Listen socketåQ?˜q™æ ·çš„è¯åQŒåˆ«çš„processå¯ä»¥å¤„熘q™äº›˜qžæŽ¥ã€‚由于local listen socketå’Œglobal listen socketå…׃ínFDåQŒæ‰€ä»¥kernelž®†ä¼šæŠŠæ–°çš„connet通知到相应的processã€?
- 如果应用½E‹åº˜q›ç¨‹ä½¿ç”¨accept()¾pÈ»Ÿè°ƒç”¨åQŒé‚£ä¹ˆå¤„ç†è¿‡½E‹æ˜¯é¦–先去global listen tableä¸æŸ¥æ‰‘Ö’Œæ“作åQˆå› 为是è¯ÀL“作，没有使用é”）åQŒå¦‚果没有找刎ͼŒé‚£ä¹ˆåŽ»coreçš„local tableä¸æŸ¥æ‰¾ã€‚如果找刎ͼŒž®Þp¿”回给应用½E‹åºã€‚由于listen的时候把socket¾l‘定åˆîCº†ä¸€ä¸ªcoreåQŒæ‰€ä»¥æŸ¥æ‰„¡š„时候也去这个coreçš„local tableä¸æŸ¥æ‰¾ã€?
- epoll兼容性，如果应用½E‹åºä½¿ç”¨epoll_ctl()¾pÈ»Ÿè°ƒç”¨åQŒæ¥æŠŠä¸€ä¸ªlisten socketæ·ÕdŠ 到Epoll setä¸ï¼Œé‚£ä¹ˆlocalçš„listen socketå’Œglobalçš„listen socket都被epoll监控。事件å‘生的时候，epoll_wait()¾pÈ»Ÿè°ƒç”¨ä¼šè¿”回listen socketåQŒaccept()¾pÈ»Ÿè°ƒç”¨ž®×ƒ¼šå¤„熘q™ä¸ªsocketã€‚è¿™æ ·å°±ä¿è¯äº†epoll实现的兼å®ÒŽ€§ã€?
使用‹¹ç¨‹å›¾æ¦‚括上é¢æ‰€˜qŽÍ¼š
Local Established Table
Linuxå†…æ ¸ä½¿ç”¨ä¸€ä¸ªå…¨å±€çš„hash表以åŠé”æ“作æ¥ç»´æŠ¤establised socketsåQˆè¢«ç”¨æ¥è·Ÿè¸ª˜qžæŽ¥çš„socketsåQ‰ã€‚Fastsocket æƒÏx³•æ˜¯æŠŠå…¨å±€table分散到per-Core tableåQŒå½“一个core需è¦è®¿é—®socket的时候，åªåœ¨éš¶å±žäºŽè‡ªå·Þqš„tableä¸æœç´¢ï¼Œå› æ¤ä¸éœ€è¦é”æ“纵åQŒä¹Ÿä¸å˜åœ¨èµ„æºç«žäº‰ã€‚ç”±fastsocket建立的socket本地local established tableä¸ï¼Œå…¶ä»–çš„regular socketsä¿å˜åœ¨globalçš„tableä¸ã€‚core首先去自å·Þqš„local tableä¸æŸ¥æ‰¾ï¼ˆä¸éœ€è¦é”åQ‰ï¼Œç„¶åŽåŽ»globalä¸æŸ¥æ‰¾ã€?/p>
Receive Flow Deliver
默认情况下，应用½E‹åºä¸ÕdŠ¨å‘包的时候，å‘出åŽÈš„包是通过æ£åœ¨æ‰§è¡Œæœ¬è¿›½E‹çš„那个CPU æ ¸ï¼ˆ¾pÈ»Ÿåˆ†é…的）æ¥å®Œæˆçš„åQ›è€ŒæŽ¥æ”¶æ•°æ®åŒ…çš„æ—¶CPU æ ¸æ˜¯ç”±å‰é¢æ到的RSS或RPSæ¥ä¼ é€’ã€‚è¿™æ ·ä¸€æ¥ï¼Œ˜qžæŽ¥å¯èƒ½ç”׃¸åŒçš„两个CPUæ ¸æ¥å®Œæˆã€‚连接应该在本地化处ç†ã€‚RFSå’ŒIntel¾|‘å¡çš„FlowDirectorå¯ä»¥ä»ŽèÊY件和¼‹¬äšg上缓解这¿U情况，但是ä¸å®Œå¤‡ã€?/p>
RFDåQˆReceive Flow DeliveråQ‰ä¸»è¦çš„æ€æƒ³æ˜¯CPUæ ¸æ•°ä¸ÕdŠ¨å‘è“v˜qžæŽ¥çš„时候å¯ä»¥æŠŠCPU coreçš„æ ‡è¯†å’Œ˜qžæŽ¥çš„source port¾~–ç åˆîC¸€èµ—÷€‚CPU coreså’Œportsçš„å…³¾pÈ”±ä¸€ä¸ªå…³¾p»é›†åˆæ¥å†›_®šã€coresåQŒports】， 对于一个portåQŒæœ‰å”¯ä¸€çš„一个core与之对应。当一个coreæ¥å¾ç«‹connection的时候，RFDéšæœºé€‰æ‹©ä¸€ä¸ªè·Ÿå½“å‰core匚w…çš„port。接收包的时候，RFD负责军_®š˜q™ä¸ªåŒ…应该让哪一个coreæ¥å¤„ç†ï¼Œå¦‚果当å‰coreä¸æ˜¯è¢«é€‰ä¸çš„cpu coreåQŒé‚£ä¹ˆå°±deliver到选ä¸çš„cpu coreã€?/p>
一般æ¥è¯ß_¼ŒRFD对代ç†ç¨‹åºæ”¶ç›Šæ¯”较大åQŒå•¾U¯çš„WEBæœåŠ¡å™¨å¯ä»¥é€‰æ‹©¼›ç”¨ã€?/p>
ž®ç»“
以上å‚考了大é‡çš„外部资料进行整ç†è€ŒæˆåQŒè¿›è€Œå¯ä»¥èŽ·å¾—一个较为整体的Fastsocketå†…æ ¸æž¶æž„å°è±¡ã€?/p>
Fastsocket的努力，在å•ä¸ªTCP˜qžæŽ¥çš„管ç†ä»Ž¾|‘å¡è§¦å‘的硬ä¸æ–ã€èÊYä¸æ–ã€ä¸‰‹Æ¡æ¡æ‰‹ã€æ•°æ®ä¼ 输ã€å››‹Æ¡æŒ¥æ‰‹ç‰å®Œæ•´çš„过½E‹åœ¨å®Œæ•´åœ¨ä¸€ä¸ªCPUæ 怸Š˜q›è¡Œå¤„ç†åQŒä»Žè€Œå®žçŽîCº†æ¯ä¸€ä¸ªCPUæ ¸å¿ƒTCP资æºæœ¬åœ°åŒ–，˜q™æ ·ä¸ºå¤šæ ¸æ°´òqÏx‰©å±•æ‰“好了基础åQŒå‡ž®‘全局资æºç«žäº‰åQŒåã^行化处熘qžæŽ¥åQŒåŒæ—‰™™ä½Žæ–‡ä»‰™”的副作用åQŒåšåˆîCº†æžäؓ高效的矘qžæŽ¥å¤„ç†æ–ÒŽ¡ˆåQŒä¸å¾—ä¸èµžå•Šã€?/p>
引用资料åQ?/h3>
- Fastsocket PPT
- FastSocket
nieyong 2015-02-04 14:22 å‘表评论]]>Fastsocketå¦ä¹ ½W”记之模å—篇 http://www.aygfsteel.com/yongboy/archive/2015/02/03/422694.htmlnieyong nieyong Tue, 03 Feb 2015 05:26:00 GMT http://www.aygfsteel.com/yongboy/archive/2015/02/03/422694.html http://www.aygfsteel.com/yongboy/comments/422694.html http://www.aygfsteel.com/yongboy/archive/2015/02/03/422694.html#Feedback 1 http://www.aygfsteel.com/yongboy/comments/commentRss/422694.html http://www.aygfsteel.com/yongboy/services/trackbacks/422694.html å‰è¨€
本篇å¦ä¹ Fastsocketå†…æ ¸æ¨¡å—fastsocket.soåQŒä½œä¸ºç”¨æˆäh€?code>libfsocket.soçš„å†…æ ¸æ€çš„支æŒåQŒå¤„ç?code>ioctlä¼ é€’åˆ°/dev/fastsocketçš„æ•°æ®ï¼Œéžå¸¸æ ¸å¿ƒå’ŒåŸº¼‹€ã€‚å—¯åQŒè¿˜æ˜¯å…ˆ¾˜»è¯‘åQŒéšåŽæŒŸå¸¦äº›ç‚¹è¯„˜q›æ¥ã€?/p>
模å—介ç»
Fastsocketå†…æ ¸æ¨¡å— (fastsocket.ko) æ供若干ç‰ÒŽ€§ï¼Œòq¶å„自具有开å¯å’Œå…³é—½{‰ä¸°å¯Œé€‰é¡¹å¯é…¾|®ã€?/p>
VFS 优化
CentOS 6.5带æ¥çš„å†…æ ”R”竞争处处å¯è§åQŒå¯¼è‡´æ— 论如何优化TCP/IP¾|‘ç»œå †æ ˆéƒ½ä¸èƒ½å¤Ÿå¸¦æ¥å¾ˆå¥½çš„性能扩展。比较严é‡é”竞争例ååQ?code>inode_lockå’?code>dcache_lockåQŒé’ˆå¯¹å¥—接å—æ–‡äšg¾pÈ»Ÿsockfs而言åQŒåƈä¸æ˜¯å¿…须。fastsocket通过在VFSåˆå§‹åŒ–结构时æä¾›fastpath快速èµ\径用以解å†Ïx¤™åšw—®é¢˜ï¼Œå·²ç»å‘代å·äؓ香è‰åQˆvanillaåQ‰çš„å†…æ ¸æ交了两处修改：
a209dfc vfs: dont chain pipe/anon/socket on superblock s_inodes list
4b93688 fs: improve scalability of pseudo filesystems
æ¤é¡¹ä¿®æ”¹æ²¡æœ‰æ供选项å¯ä¾›é…ç½®åQŒå› æ¤æ‰€æœ‰fastsocket创å¾çš„套接å—sockets都会强制¾lç”±fastpathä¼ è¾“ã€?/p>
å†…æ ¸æ¨¡å—å‚æ•°
enable_listen_spawn
fastsocket为æ¯ä¸ªCPU创å¾äº†ä¸€ä¸ªæœ¬åœ°socket监å¬è¡¨ï¼ˆlocal listen tableåQ‰ï¼Œåº”用½E‹åºå¯ä»¥å†›_®šåœ¨ä¸€ä¸ªç‰¹å®šCPUå†…æ ¸ä¸Šå¤„ç†æŸä¸ªæ–°çš„连接，具体ž®±æ˜¯é€šè¿‡æ‹¯‚´åŽŸå§‹ç›‘å¬å¥—接å—socketåQŒç„¶åŽæ’入到本地套接å—socket监å¬è¡¨ä¸ã€‚当新徘qžæŽ¥åœ¨æŸCPU处ç†æ—Óž¼Œ¾pÈ»Ÿå†…æ ¸ž®è¯•åŒšw…本地socket监å¬è¡¨ï¼ŒåŒšw…æˆåŠŸä¼šæ’入到本地accept队列ä¸ã€‚ç¨åŽï¼ŒCPU会从本地accept队列ä¸èŽ·å–进行处ç†ã€?/p>
˜q™ç§æ–¹å¼æ¯ä¸€ä¸ªç½‘¾lœèÊYä¸æ–都会有隶属于自己本地套接å—队列当新的˜qžæŽ¥˜q›æ¥æ—¶å¯ä»¥åŽ‹å…¥ï¼Œæ¯ä¸€ä¸ªè¿›½E‹ä»Žæœ¬åœ°é˜Ÿåˆ—ä¸å¼¹å‡ø™¿žæŽ¥è¿›è¡Œå¤„ç†ã€‚当˜q›ç¨‹å’ŒCPU˜q›è¡Œ¾l‘定åQŒä¸€æ—¦æœ‰¾|‘å¡æŽ¥å£å†›_®šæŠ•é€’到æŸä¸ªCPUå†…æ ¸ä¸Šï¼Œé‚£ä¹ˆåŒ…æ‹¬¼‹¬ä¸æ–ã€èÊYä¸æ–ã€ç³»¾lŸè°ƒç”¨ä»¥åŠç”¨æˆ¯‚¿›½E‹ï¼Œéƒ½ä¼šæœ‰è¿™ä¸ªCPU全程负责。好处就是客æˆïL«¯è¯äh±‚˜qžæŽ¥åœ¨æ²¡æœ‰é”的竞争环境下分散到å„个CPU上被动处ç†æœ¬åœ°è¿žæŽ¥ã€?/p>
本特性更适åˆä»¥ä¸‹æƒ…况åQ?/p>
- ž®½å¯èƒ½å¤šçš„网å¡Rx接收队列和CPUæ ¸æ•°
- 应用½E‹åºå·¥ä½œ˜q›ç¨‹è¢«é™æ€ç»‘定到æ¯ä¸€ä¸ªCPUä¸?
½W¬ä¸€¿U情况下åQŒRPSå¯ä»¥åœ¨ç½‘å¡æŽ¥æ”‰™˜Ÿåˆ—å°äºŽCPUæ ¸æ•°æ—¶è¢«ä½¿ç”¨ã€‚ç¬¬äºŒç§æ–ÒŽ¡ˆå¯ä»¥æ»¡èƒö两个斚w¢åQ?/p>
- 应用½E‹åºåœ¨å¯åŠ¨æ—¶è‡ªå·±¾l‘定工作˜q›ç¨‹å’ŒCPU亲和æ€?
- å…许fastsocket自动为工作进½E‹ç»‘定CPU亲和æ€?
å› æ¤åQ?code>enable_listen_spawnå…ähœ‰ä¸‰ä¸ªå€¼å¯ä¾›é…¾|®ï¼š
- enable_listen_spawn=0: å½Õdº•¼›æ¢
- enable_listen_spawn=1: å¯ç”¨åQŒä½†è¦æ±‚应用½E‹åºè‡ªå·±¾l‘定CPU
- enable_listen_spawn=2 (默认å€?: å¯ç”¨æ¤ç‰¹æ€§ï¼Œå…许fastsocket为æ¯ä¸€ä¸ªå·¥ä½œè¿›½E‹ç»‘定到CPUä¸?
enable_fast_epoll
一旦开å¯ï¼Œéœ€è¦äØ“æ–‡äšg¾l“æž„é¢å¤–æ·ÕdŠ 一å—段用以ä¿å˜æ–‡äšg与epitemçš„æ˜ ž®„å…³¾p»ï¼Œ˜q™æ ·å¯çœåŽÕdœ¨epoll_ctlæ–ÒŽ³•è¢«è°ƒç”¨æ—¶ä»Žepoll¾U¢é»‘æ ‘æŸ¥æ‰¾epitem的开销ã€?/p>
虽然æ¤é¡¹ä¼˜åŒ–有所修改epollè¯ä¹‰åQŒä½†å¸¦æ¥äº†å¥—接å—性能æå‡ã€‚å¼€å¯çš„å‰æ是一个套接å—åªå…è®¸æ·»åŠ åˆ°ä¸€ä¸ªepoll实例ä¸ï¼Œä½†ä¸åŒ…括监å¬å¥—接å—。默认å€égØ“trueå¯ä»¥é€‚用于ç»å¤§å¤šæ•°åº”用程åºï¼Œè‹¥ä½ 的程åºä¸æ»¡èƒöæ¡äšgž®±å¾—需è¦ç¦ç”¨äº†ã€?/p>
enable_fast_epoll 为布ž®”åž‹boolean选项:
- enable_fast_epoll=0: ¼›ç”¨fast-epoll
- enable_fast_epoll=1 (默认å€?: å¼€å¯fast-epoll
enable_receive_flow_deliver
RFDåQˆReceive Flow DeliveråQ‰ä¼šæŠŠäؓ新徘qžæŽ¥åˆ†é…çš„CPU IDž®è£…到其˜qžæŽ¥çš„端å£å·ä¸ï¼Œè€Œä¸æ˜¯éšæœºé€‰æ‹©æ–°åˆ›å»ºçš„ä¸ÕdŠ¨˜qžæŽ¥çš„æºç«¯å£˜q›è¡Œåˆ†é…到CPU上ã€?/p>
当应用从‹zÕdŠ¨˜qžæŽ¥æ”¶åˆ°æ•°æ®åŒ…RFD解ç æ—Óž¼Œä¼šä»Žç›®çš„地端å£ä¸Šè§£æžå‡ºå¯¹åº”çš„CPUå†…æ ¸IDåQŒç‘ô而è{å‘给对应的CPUå†…æ ¸ã€‚å†åŠ 上listen_spawnåQŒä¿è¯äº†ä¸€ä¸ªè¿žæŽ¥CPU处ç†çš„完全本地化ã€?/p>
enable_receive_flow是一个布ž®”型选项:
- enable_receive_flow=0 (默认å€?: ¼›ç”¨RFD
- enable_receive_flow=1: å¯ç”¨RFD
注æ„事项åQ?/p>
- 当å¯ç”¨æ—¶åQŒåœ¨å½“å‰çš„实玎ͼŒRFD完全覆盖RPS½{–ç•¥åQŒåƈ使得RPSæ— æ•ˆã€‚è‹¥ä½¿ç”¨RPSåQŒè¯·¼›ç”¨æ¤ç‰¹æ€?
- ç”׃ºŽRFDåªä¼šå¯¹è¯¸å¦‚代ç†åº”用程åºæœ‰åˆ©ï¼Œæˆ‘们å»ø™®®åœ¨WebæœåŠ¡å™¨ä¸Š¼›ç”¨æ¤ç‰¹æ€?
以上åQŒç¿»è¯‘完毕ã€?/em>
æºç ½Ž€å•æ¢³ç?/h3>
fastsocketçš„å†…æ ¸æ¨¡å—相对èµ\径äØ“fastsocket/module/åQŒé™¤äº†README.md外，ž®±æ˜¯ä¸¤ä¸ªè½¯è¿žæŽ¥æ–‡ä»¶äº†åQ?/p>
- fastsocket.c ../kernel/net/fastsocket/fastsocket.c 真实环境下ä¸å˜åœ¨˜q™ä¸ªæ–‡äšgåQŒå¯èƒ½æ˜¯½E‹åºBUG
- fastsocket.h ../kernel/net/fastsocket/fastsocket.h 有对应头文äšgå˜åœ¨
æ¢ç§è¯´æ³•åQŒfastsocketå†…æ ¸æ¨¡å—真æ£è·¯å¾„ä¸?code>fastsocket/kernel/net/fastsocketåQŒå…·ä½“文件列表äØ“åQ?/p>
- Kconfig
- Makefile
- fastsocket.h å®šä¹‰å†…æ ¸æ¨¡å—所使用到å˜é‡å’Œæ–ÒŽ³•
- fastsocket_core.c è´Ÿè´£æ–ÒŽ³•å®žçŽ°åQŒä¾›fastsocket_api.c调用
- fastsocket_api.c å†…æ ¸æ¨¡å—åŠ è²/å¸è²½{‰æ“作，处ç†å‰ç«¯åŠ¨æ€é“¾æŽ¥åº“¾lç”±ioctlä¼ é€’çš„æ•°æ®
fastsocket_api.cå®žçŽ°å†…æ ¸æ¨¡å—接å£åQŒåœ¨æºç 里é¢æ³¨å†Œäº†å¥½å¤šæ–‡æ¡£æš‚时没有公开的å¯é…ç½®™å¹ç›®åQ?/p>
int enable_fastsocket_debug = 3;
/* Fastsocket feature switches */
int enable_listen_spawn = 2;
int enable_receive_flow_deliver;
int enable_fast_epoll = 1;
int enable_skb_pool;
int enable_rps_framework;
int enable_receive_cpu_selection = 0;
int enable_direct_tcp = 0;
int enable_socket_pool_size = 0;
module_param(enable_fastsocket_debug,int, 0);
module_param(enable_listen_spawn, int, 0);
module_param(enable_receive_flow_deliver, int, 0);
module_param(enable_fast_epoll, int, 0);
module_param(enable_direct_tcp, int, 0);
module_param(enable_skb_pool, int, 0);
module_param(enable_receive_cpu_selection, int, 0);
module_param(enable_socket_pool_size, int, 0);
MODULE_PARM_DESC(enable_fastsocket_debug, " Debug level [Default: 3]" );
MODULE_PARM_DESC(enable_listen_spawn, " Control Listen-Spawn: 0 = Disabled, 1 = Process affinity required, 2 = Autoset process affinity[Default]");
MODULE_PARM_DESC(enable_receive_flow_deliver, " Control Receive-Flow-Deliver: 0 = Disabled[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_fast_epoll, " Control Fast-Epoll: 0 = Disabled, 1 = Enabled[Default]");
MODULE_PARM_DESC(enable_direct_tcp, " Control Direct-TCP: 0 = Disbale[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_skb_pool, " Control Skb-Pool: 0 = Disbale[Default], 1 = Receive skb pool, 2 = Send skb pool, 3 = Both skb pool");
MODULE_PARM_DESC(enable_receive_cpu_selection, " Control RCS: 0 = Disabled[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_socket_pool_size, "Control socket pool size: 0 = Disabled[Default], other are the pool size");
接收用户æ€çš„libfsocket.so通过ioctlä¼ é€’è¿‡æ¥çš„æ•°æ®åQŒæ ¹æ®å‘½ä»¤è¿›è¡Œæ•°æ®åˆ†å‘：
static long fastsocket_ioctl(struct file *filp, unsigned int cmd, unsigned long __user u_arg)
{
struct fsocket_ioctl_arg k_arg;
if (copy_from_user(&k_arg, (struct fsocket_ioctl_arg *)u_arg, sizeof(k_arg))) {
EPRINTK_LIMIT(ERR, "copy ioctl parameter from user space to kernel failed\n");
return -EFAULT;
}
switch (cmd) {
case FSOCKET_IOC_SOCKET:
return fastsocket_socket(&k_arg);
case FSOCKET_IOC_LISTEN:
return fastsocket_listen(&k_arg);
case FSOCKET_IOC_SPAWN_LISTEN:
return fastsocket_spawn_listen(&k_arg);
case FSOCKET_IOC_ACCEPT:
return fastsocket_accept(&k_arg);
case FSOCKET_IOC_CLOSE:
return fastsocket_close(&k_arg);
case FSOCKET_IOC_SHUTDOWN_LISTEN:
return fastsocket_shutdown_listen(&k_arg);
//case FSOCKET_IOC_EPOLL_CTL:
// return fastsocket_epoll_ctl((struct fsocket_ioctl_arg *)arg);
default:
EPRINTK_LIMIT(ERR, "ioctl [%d] operation not support\n", cmd);
break;
}
return -EINVAL;
}
fastsocket/library/libsocket.h头文件定义的FSOCKET_IOC_* æ“作状æ€ç ž®Þpƒ½å¤Ÿä¸€ä¸€å¯¹åº”的上ã€?ioctlä¼ è¾“æ•°æ®ä»Žç”¨æˆäh€?>å†…æ ¸æ€ï¼Œéœ€è¦ç»˜q‡ä¸€‹Æ¡æ‹·è´è¿‡½E‹ï¼ˆcopy_from_useråQ‰ï¼Œç„¶åŽæ ÒŽ®cmd命ä×o˜q›è¡ŒåŠŸèƒ½è·¯ç”±ã€?/p>
å‰è¨€
本篇å¦ä¹ Fastsocketå†…æ ¸æ¨¡å—fastsocket.soåQŒä½œä¸ºç”¨æˆäh€?code>libfsocket.soçš„å†…æ ¸æ€çš„支æŒåQŒå¤„ç?code>ioctlä¼ é€’åˆ°/dev/fastsocketçš„æ•°æ®ï¼Œéžå¸¸æ ¸å¿ƒå’ŒåŸº¼‹€ã€‚å—¯åQŒè¿˜æ˜¯å…ˆ¾˜»è¯‘åQŒéšåŽæŒŸå¸¦äº›ç‚¹è¯„˜q›æ¥ã€?/p>
模å—介ç»
Fastsocketå†…æ ¸æ¨¡å— (fastsocket.ko) æ供若干ç‰ÒŽ€§ï¼Œòq¶å„自具有开å¯å’Œå…³é—½{‰ä¸°å¯Œé€‰é¡¹å¯é…¾|®ã€?/p>
VFS 优化
CentOS 6.5带æ¥çš„å†…æ ”R”竞争处处å¯è§åQŒå¯¼è‡´æ— 论如何优化TCP/IP¾|‘ç»œå †æ ˆéƒ½ä¸èƒ½å¤Ÿå¸¦æ¥å¾ˆå¥½çš„性能扩展。比较严é‡é”竞争例ååQ?code>inode_lockå’?code>dcache_lockåQŒé’ˆå¯¹å¥—接å—æ–‡äšg¾pÈ»Ÿsockfs而言åQŒåƈä¸æ˜¯å¿…须。fastsocket通过在VFSåˆå§‹åŒ–结构时æä¾›fastpath快速èµ\径用以解å†Ïx¤™åšw—®é¢˜ï¼Œå·²ç»å‘代å·äؓ香è‰åQˆvanillaåQ‰çš„å†…æ ¸æ交了两处修改：
a209dfc vfs: dont chain pipe/anon/socket on superblock s_inodes list
4b93688 fs: improve scalability of pseudo filesystems
æ¤é¡¹ä¿®æ”¹æ²¡æœ‰æ供选项å¯ä¾›é…ç½®åQŒå› æ¤æ‰€æœ‰fastsocket创å¾çš„套接å—sockets都会强制¾lç”±fastpathä¼ è¾“ã€?/p>
å†…æ ¸æ¨¡å—å‚æ•°
enable_listen_spawn
fastsocket为æ¯ä¸ªCPU创å¾äº†ä¸€ä¸ªæœ¬åœ°socket监å¬è¡¨ï¼ˆlocal listen tableåQ‰ï¼Œåº”用½E‹åºå¯ä»¥å†›_®šåœ¨ä¸€ä¸ªç‰¹å®šCPUå†…æ ¸ä¸Šå¤„ç†æŸä¸ªæ–°çš„连接，具体ž®±æ˜¯é€šè¿‡æ‹¯‚´åŽŸå§‹ç›‘å¬å¥—接å—socketåQŒç„¶åŽæ’入到本地套接å—socket监å¬è¡¨ä¸ã€‚当新徘qžæŽ¥åœ¨æŸCPU处ç†æ—Óž¼Œ¾pÈ»Ÿå†…æ ¸ž®è¯•åŒšw…本地socket监å¬è¡¨ï¼ŒåŒšw…æˆåŠŸä¼šæ’入到本地accept队列ä¸ã€‚ç¨åŽï¼ŒCPU会从本地accept队列ä¸èŽ·å–进行处ç†ã€?/p>
˜q™ç§æ–¹å¼æ¯ä¸€ä¸ªç½‘¾lœèÊYä¸æ–都会有隶属于自己本地套接å—队列当新的˜qžæŽ¥˜q›æ¥æ—¶å¯ä»¥åŽ‹å…¥ï¼Œæ¯ä¸€ä¸ªè¿›½E‹ä»Žæœ¬åœ°é˜Ÿåˆ—ä¸å¼¹å‡ø™¿žæŽ¥è¿›è¡Œå¤„ç†ã€‚当˜q›ç¨‹å’ŒCPU˜q›è¡Œ¾l‘定åQŒä¸€æ—¦æœ‰¾|‘å¡æŽ¥å£å†›_®šæŠ•é€’到æŸä¸ªCPUå†…æ ¸ä¸Šï¼Œé‚£ä¹ˆåŒ…æ‹¬¼‹¬ä¸æ–ã€èÊYä¸æ–ã€ç³»¾lŸè°ƒç”¨ä»¥åŠç”¨æˆ¯‚¿›½E‹ï¼Œéƒ½ä¼šæœ‰è¿™ä¸ªCPU全程负责。好处就是客æˆïL«¯è¯äh±‚˜qžæŽ¥åœ¨æ²¡æœ‰é”的竞争环境下分散到å„个CPU上被动处ç†æœ¬åœ°è¿žæŽ¥ã€?/p>
本特性更适åˆä»¥ä¸‹æƒ…况åQ?/p>
- ž®½å¯èƒ½å¤šçš„网å¡Rx接收队列和CPUæ ¸æ•°
- 应用½E‹åºå·¥ä½œ˜q›ç¨‹è¢«é™æ€ç»‘定到æ¯ä¸€ä¸ªCPUä¸?
½W¬ä¸€¿U情况下åQŒRPSå¯ä»¥åœ¨ç½‘å¡æŽ¥æ”‰™˜Ÿåˆ—å°äºŽCPUæ ¸æ•°æ—¶è¢«ä½¿ç”¨ã€‚ç¬¬äºŒç§æ–ÒŽ¡ˆå¯ä»¥æ»¡èƒö两个斚w¢åQ?/p>
- 应用½E‹åºåœ¨å¯åŠ¨æ—¶è‡ªå·±¾l‘定工作˜q›ç¨‹å’ŒCPU亲和æ€?
- å…许fastsocket自动为工作进½E‹ç»‘定CPU亲和æ€?
å› æ¤åQ?code>enable_listen_spawnå…ähœ‰ä¸‰ä¸ªå€¼å¯ä¾›é…¾|®ï¼š
- enable_listen_spawn=0: å½Õdº•¼›æ¢
- enable_listen_spawn=1: å¯ç”¨åQŒä½†è¦æ±‚应用½E‹åºè‡ªå·±¾l‘定CPU
- enable_listen_spawn=2 (默认å€?: å¯ç”¨æ¤ç‰¹æ€§ï¼Œå…许fastsocket为æ¯ä¸€ä¸ªå·¥ä½œè¿›½E‹ç»‘定到CPUä¸?
enable_fast_epoll
一旦开å¯ï¼Œéœ€è¦äØ“æ–‡äšg¾l“æž„é¢å¤–æ·ÕdŠ 一å—段用以ä¿å˜æ–‡äšg与epitemçš„æ˜ ž®„å…³¾p»ï¼Œ˜q™æ ·å¯çœåŽÕdœ¨epoll_ctlæ–ÒŽ³•è¢«è°ƒç”¨æ—¶ä»Žepoll¾U¢é»‘æ ‘æŸ¥æ‰¾epitem的开销ã€?/p>
虽然æ¤é¡¹ä¼˜åŒ–有所修改epollè¯ä¹‰åQŒä½†å¸¦æ¥äº†å¥—接å—性能æå‡ã€‚å¼€å¯çš„å‰æ是一个套接å—åªå…è®¸æ·»åŠ åˆ°ä¸€ä¸ªepoll实例ä¸ï¼Œä½†ä¸åŒ…括监å¬å¥—接å—。默认å€égØ“trueå¯ä»¥é€‚用于ç»å¤§å¤šæ•°åº”用程åºï¼Œè‹¥ä½ 的程åºä¸æ»¡èƒöæ¡äšgž®±å¾—需è¦ç¦ç”¨äº†ã€?/p>
enable_fast_epoll 为布ž®”åž‹boolean选项:
- enable_fast_epoll=0: ¼›ç”¨fast-epoll
- enable_fast_epoll=1 (默认å€?: å¼€å¯fast-epoll
enable_receive_flow_deliver
RFDåQˆReceive Flow DeliveråQ‰ä¼šæŠŠäؓ新徘qžæŽ¥åˆ†é…çš„CPU IDž®è£…到其˜qžæŽ¥çš„端å£å·ä¸ï¼Œè€Œä¸æ˜¯éšæœºé€‰æ‹©æ–°åˆ›å»ºçš„ä¸ÕdŠ¨˜qžæŽ¥çš„æºç«¯å£˜q›è¡Œåˆ†é…到CPU上ã€?/p>
当应用从‹zÕdŠ¨˜qžæŽ¥æ”¶åˆ°æ•°æ®åŒ…RFD解ç æ—Óž¼Œä¼šä»Žç›®çš„地端å£ä¸Šè§£æžå‡ºå¯¹åº”çš„CPUå†…æ ¸IDåQŒç‘ô而è{å‘给对应的CPUå†…æ ¸ã€‚å†åŠ 上listen_spawnåQŒä¿è¯äº†ä¸€ä¸ªè¿žæŽ¥CPU处ç†çš„完全本地化ã€?/p>
enable_receive_flow是一个布ž®”型选项:
- enable_receive_flow=0 (默认å€?: ¼›ç”¨RFD
- enable_receive_flow=1: å¯ç”¨RFD
注æ„事项åQ?/p>
- 当å¯ç”¨æ—¶åQŒåœ¨å½“å‰çš„实玎ͼŒRFD完全覆盖RPS½{–ç•¥åQŒåƈ使得RPSæ— æ•ˆã€‚è‹¥ä½¿ç”¨RPSåQŒè¯·¼›ç”¨æ¤ç‰¹æ€?
- ç”׃ºŽRFDåªä¼šå¯¹è¯¸å¦‚代ç†åº”用程åºæœ‰åˆ©ï¼Œæˆ‘们å»ø™®®åœ¨WebæœåŠ¡å™¨ä¸Š¼›ç”¨æ¤ç‰¹æ€?
以上åQŒç¿»è¯‘完毕ã€?/em>
æºç ½Ž€å•æ¢³ç?/h3>
fastsocketçš„å†…æ ¸æ¨¡å—相对èµ\径äØ“fastsocket/module/åQŒé™¤äº†README.md外，ž®±æ˜¯ä¸¤ä¸ªè½¯è¿žæŽ¥æ–‡ä»¶äº†åQ?/p>
- fastsocket.c ../kernel/net/fastsocket/fastsocket.c 真实环境下ä¸å˜åœ¨˜q™ä¸ªæ–‡äšgåQŒå¯èƒ½æ˜¯½E‹åºBUG
- fastsocket.h ../kernel/net/fastsocket/fastsocket.h 有对应头文äšgå˜åœ¨
æ¢ç§è¯´æ³•åQŒfastsocketå†…æ ¸æ¨¡å—真æ£è·¯å¾„ä¸?code>fastsocket/kernel/net/fastsocketåQŒå…·ä½“文件列表äØ“åQ?/p>
- Kconfig
- Makefile
- fastsocket.h å®šä¹‰å†…æ ¸æ¨¡å—所使用到å˜é‡å’Œæ–ÒŽ³•
- fastsocket_core.c è´Ÿè´£æ–ÒŽ³•å®žçŽ°åQŒä¾›fastsocket_api.c调用
- fastsocket_api.c å†…æ ¸æ¨¡å—åŠ è²/å¸è²½{‰æ“作，处ç†å‰ç«¯åŠ¨æ€é“¾æŽ¥åº“¾lç”±ioctlä¼ é€’çš„æ•°æ®
fastsocket_api.cå®žçŽ°å†…æ ¸æ¨¡å—接å£åQŒåœ¨æºç 里é¢æ³¨å†Œäº†å¥½å¤šæ–‡æ¡£æš‚时没有公开的å¯é…ç½®™å¹ç›®åQ?/p>
int enable_fastsocket_debug = 3;
/* Fastsocket feature switches */
int enable_listen_spawn = 2;
int enable_receive_flow_deliver;
int enable_fast_epoll = 1;
int enable_skb_pool;
int enable_rps_framework;
int enable_receive_cpu_selection = 0;
int enable_direct_tcp = 0;
int enable_socket_pool_size = 0;
module_param(enable_fastsocket_debug,int, 0);
module_param(enable_listen_spawn, int, 0);
module_param(enable_receive_flow_deliver, int, 0);
module_param(enable_fast_epoll, int, 0);
module_param(enable_direct_tcp, int, 0);
module_param(enable_skb_pool, int, 0);
module_param(enable_receive_cpu_selection, int, 0);
module_param(enable_socket_pool_size, int, 0);
MODULE_PARM_DESC(enable_fastsocket_debug, " Debug level [Default: 3]" );
MODULE_PARM_DESC(enable_listen_spawn, " Control Listen-Spawn: 0 = Disabled, 1 = Process affinity required, 2 = Autoset process affinity[Default]");
MODULE_PARM_DESC(enable_receive_flow_deliver, " Control Receive-Flow-Deliver: 0 = Disabled[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_fast_epoll, " Control Fast-Epoll: 0 = Disabled, 1 = Enabled[Default]");
MODULE_PARM_DESC(enable_direct_tcp, " Control Direct-TCP: 0 = Disbale[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_skb_pool, " Control Skb-Pool: 0 = Disbale[Default], 1 = Receive skb pool, 2 = Send skb pool, 3 = Both skb pool");
MODULE_PARM_DESC(enable_receive_cpu_selection, " Control RCS: 0 = Disabled[Default], 1 = Enabled");
MODULE_PARM_DESC(enable_socket_pool_size, "Control socket pool size: 0 = Disabled[Default], other are the pool size");
接收用户æ€çš„libfsocket.so通过ioctlä¼ é€’è¿‡æ¥çš„æ•°æ®åQŒæ ¹æ®å‘½ä»¤è¿›è¡Œæ•°æ®åˆ†å‘：
static long fastsocket_ioctl(struct file *filp, unsigned int cmd, unsigned long __user u_arg)
{
struct fsocket_ioctl_arg k_arg;
if (copy_from_user(&k_arg, (struct fsocket_ioctl_arg *)u_arg, sizeof(k_arg))) {
EPRINTK_LIMIT(ERR, "copy ioctl parameter from user space to kernel failed\n");
return -EFAULT;
}
switch (cmd) {
case FSOCKET_IOC_SOCKET:
return fastsocket_socket(&k_arg);
case FSOCKET_IOC_LISTEN:
return fastsocket_listen(&k_arg);
case FSOCKET_IOC_SPAWN_LISTEN:
return fastsocket_spawn_listen(&k_arg);
case FSOCKET_IOC_ACCEPT:
return fastsocket_accept(&k_arg);
case FSOCKET_IOC_CLOSE:
return fastsocket_close(&k_arg);
case FSOCKET_IOC_SHUTDOWN_LISTEN:
return fastsocket_shutdown_listen(&k_arg);
//case FSOCKET_IOC_EPOLL_CTL:
// return fastsocket_epoll_ctl((struct fsocket_ioctl_arg *)arg);
default:
EPRINTK_LIMIT(ERR, "ioctl [%d] operation not support\n", cmd);
break;
}
return -EINVAL;
}
fastsocket/library/libsocket.h头文件定义的FSOCKET_IOC_* æ“作状æ€ç ž®Þpƒ½å¤Ÿä¸€ä¸€å¯¹åº”的上ã€?ioctlä¼ è¾“æ•°æ®ä»Žç”¨æˆäh€?>å†…æ ¸æ€ï¼Œéœ€è¦ç»˜q‡ä¸€‹Æ¡æ‹·è´è¿‡½E‹ï¼ˆcopy_from_useråQ‰ï¼Œç„¶åŽæ ÒŽ®cmd命ä×o˜q›è¡ŒåŠŸèƒ½è·¯ç”±ã€?/p>
libfsocket.so如何与fastsocketå†…æ ¸æ¨¡å—交互
通过指定的设备通é“/dev/fastsocket˜q›è¡Œäº¤äº’åQ?/p>
- fastsocketå†…æ ¸æ¨¡å—注册è¦ç›‘å¬çš„通é“讑֤‡å称ä¸?code>/dev/fastsocket
- libfsocket打开
/dev/fastsocket讑֤‡èŽ·å¾—æ–‡äšgå¥æŸ„åQŒå¼€å§?code>ioctlæ•°æ®ä¼ é€?
ž®ç»“
½Ž€å•æ¢³ç†äº†fastsocketå†…æ ¸æ¨¡å—åQŒä½†ä¸€æ ähœ‰å¾ˆå¤šçš„点没有涉åŠåQŒåŽé¢å¯èƒ½ä¼šåœ¨Fastsocketå†…æ ¸½‹‡ä¸å†æ¬¡æ¢³ç†ä¸€ä¸‹ã€?/p>
å‰è¨€
å‰é¢¾~–译安装好了包å«æœ‰fastsocketçš„å†…æ ¸æ¨¡å—，以åŠfastsocket的动æ€é“¾æŽ¥åº“libfsocket.soåQŒä¸‹é¢å…¶å®žå°±å¯ä»¥è®„¡½®¾|‘å¡äº†ã€?/p>
下é¢ä¸ÞZ¸€äº›åè¯è§£é‡Šï¼Œä¸Šä¸‹æ–‡ä¸éœ€è¦ä‹É用到åQ?/p>
- RxåQšæŽ¥æ”‰™˜Ÿåˆ?
- TxåQšå‘é€é˜Ÿåˆ?
本文¾|‘å¡è®„¡½®½W”记内容åQŒå¤§éƒ¨åˆ†æ¥è‡ªäºŽfastsocketæºç 相对路径fastsocket/scripts/åQ›è€è§„矩，先翻译ã€?/p>
¾|‘å¡è®„¡½®½‹‡ç¿»è¯‘原æ–?/h3>
介ç»
nic.sh脚本负责¾|‘å¡é…置以尽å¯èƒ½çš„最大化å—益于fastsocket带æ¥çš„问题。给定一个网å¡æŽ¥å£ï¼Œ 它调整接å£çš„å„ç§ç‰ÒŽ€§ä»¥åŠä¸€äº›ç³»¾lŸé…¾|®ã€?/p>
相关é…ç½®
ä¸æ–å’ŒCPU的亲和æ€?/h5>
æ¯ä¸ª¾|‘塼‹¬äšg队列åŠå…¶å…Œ™”ä¸æ–¾l‘定åˆîC¸åŒçš„CPUæ ¸å¿ƒã€‚è‹¥¼‹¬äšg队列数大于CPUæ ¸æ•°åQŒé˜Ÿåˆ—需è¦é…¾|®æˆå¾ªçŽ¯round-robinæ–¹å¼åQ?IrqbalanceæœåŠ¡éœ€è¦è¢«¼›ç”¨ä»¥é˜²å…¶æ›´æ”šw…¾|®ã€?/p>
ä¸æ–阀速率
nic.sh脚本通过ethtool命ä×o讄¡½®æ¯ç§’ä¸æ–æ•îC¸Šé™ï¼Œé˜²æ¢ä¸æ–风暴。两个Rxä¸æ–间隔讄¡½®æˆè‡³ž®?33usåQŒçº¦3000个ä¸æ–毿U’ã€?/p>
RPS
为æ¯ä¸ªCPUæ ¸å¿ƒä¸Žä¸åŒçš„¾|‘塼‹¬äšgé˜Ÿåˆ—ä¹‹é—´å»ºç«‹ä¸€ä¸€æ˜ å°„å¯¹åº”å…³ç³»åQŒè¿™æ ·CPUæ ¸å¿ƒž®±å¯ä»¥å¾ˆå‡åŒ€åœ°å¤„ç†ç½‘¾lœæ•°æ®åŒ…。当¾|‘塼‹¬äšg队列ž®äºŽCPUå†…æ ¸æ•ŽÍ¼Œnic.sh脚本利用RPS (Receive Packet Steering)软äšgæ–¹å¼òqŒ™¡¡˜q›å…¥‹¹é‡è´Ÿè²åQŒè¿™æ ·CPU和硬仉™˜Ÿåˆ—ä¸å˜åœ¨å¯¹åº”关系。RPS机制å¯ä»¥è®©è¿›å…¥çš„æ•°æ®åŒ…自由分å‘到ä»ÖM¸€CPUæ 怸Šã€?/p>
¾|‘å¡æŽ¥æ”¶äº§ç”Ÿçš„ä¸æ–å¯ä»¥å‡è¡¡åˆ†é…到对应CPU上ã€?/p>
XPS
XPS (Transmit Packet Steering) 建立CPUå†…æ ¸å’ŒTxå‘é€é˜Ÿåˆ—æ˜ ž®„对应关¾p»ï¼ŒæŽŒæŽ§å‡ºç«™æ•°æ®åŒ…。系¾lŸæœ‰N个CPUæ ¸å¿ƒåQŒè„šæœ¬ä¼šè®„¡½®XPS臛_°‘å˜åœ¨N个Tx队列在网å¡æŽ¥å£ä¸ŠåQŒè¿™æ ·å°±å¯ä»¥å»ºç«‹CPUå†…æ ¸å’ŒTx队列1å¯?çš„æ˜ ž®„å…³¾p…R€?/p>
¾|‘å¡ä¼ é€æ•°æ®äñ”生的ä¸æ–ä¸€æ ·å¯ä»¥å‡å¾ˆåˆ†é…到CPU上，é¿å…å•ä¸ªCPUæ ¸å¿ƒ˜q‡äºŽ¾Jå¿™ã€?/p>
IPTABLES
压测æ—Óž¼Œé˜²ç«å¢™iptables的规则会å 用更多的CPU周期åQŒæœ‰æ‰€é™ä½Ž¾|‘ç»œå †æ ˆæ€§èƒ½ã€‚å› æ?code>nic.sh脚本若检‹¹‹åˆ°iptablesåŽå°˜qè¡Œä¸ä¼šç›´æŽ¥è¾“出报è¦ä¿¡æ¯åQŒæ½Cºå…³é—之ã€?/p>
nic.sh脚本脚本分æž
¾l过验è¯å¥½ç”¨çš„Intelå’Œåšé€šç³»åˆ—åƒå…†å’Œä¸‡å…†¾|‘å¡åˆ—表åQ?/p>
# igb
"Intel Corporation 82576 Gigabit Network Connection (rev 01)"
"Intel Corporation I350 Gigabit Network Connection (rev 01)"
# ixgbe
"Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)"
"Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)"
# tg3
"Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe"
"Broadcom Corporation NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)"
# bnx2
"Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)"
"Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)"
若当å‰æœåŠ¡å™¨æ²¡æœ‰ä»¥ä¸Š¾|‘å¡åQŒä¼šè¦å‘Šä¸€ä¸‹ï¼Œæ— ç¢ã€?/p>
˜q™é‡ŒæŠŠä¸€äº›å¸¸è§„性的CPUã€ç½‘å¡é©±åŠ¨ã€ç½‘¾lœé˜Ÿåˆ—情冉|£€æŸ¥å•ç‹¬æŠ½å–出æ¥ï¼Œé‡æ¸©å¥½å¤šå·²ç»é—忘的命令，有改å˜ï¼Œ˜q™æ ·å†™è¾ƒ½Ž€å•å˜›åQŒä¾¿äºŽä»¥åŽä‹É用：
- 直接查看CPUæ ¸æ•°åQ?code>grep -c processor /proc/cpuinfo
- 查看¾|‘å¡è½¯æŽ¥æ”‰™˜Ÿåˆ—æ•°åQ?code>ls /sys/class/net/eth0/queues | grep -c rx
- 查看¾|‘å¡è½¯å‘生队列数åQ?code>ls /sys/class/net/eth0/queues | grep -c tx
- 查看当剾|‘塼‹¬äšg队列敎ͼš
egrep -c eth0 /proc/interrupts - 查看¾|‘å¡å称和版本å·åQ?code>lspci | grep Ethernet | sed "s/Ethernet controller: //g"
- 查看¾|‘å¡é©±åŠ¨å称åQ?code>ethtool -i eth0 | grep driver
脚本先是获å–CPUã€ç½‘å¡ç‰ä¿¡æ¯åQŒæŽ¥ç€è®„¡½®ä¸æ–å•ä½¿U’内åžåé‡ï¼š ethtool -C eth0 rx-usecs 333 > /dev/null 2>&1
å¯ç”¨XPSåQŒå……分借助¾|‘å¡å‘é€é˜Ÿåˆ—，æ凾|‘å¡å‘é€åžåé‡åQŒæ˜¯æœ‰æ¡ä»‰™™åˆ¶çš„åQŒå‘é€é˜Ÿåˆ—æ•°è¦å¤§äºŽCPUæ ¸æ•°åQ?/p>
if [[ $TX_QUEUES -ge $CORES ]]; then
for i in $(seq 0 $((CORES-1))); do
cpuid_to_mask $((i%CORES)) | xargs -i echo {} > /sys/class/net/$IFACE/queues/tx-$i/xps_cpus
done
info_msg " XPS enabled"
fi
接ç€åˆ¤æ–是å¦å¯ä»¥å¯ç”¨PRSåQŒçœåŽÀL‰‹åŠ¨è®¾¾|®çš„éºÈƒ¦åQŒä½†å¯ç”¨RPSå‰æ是CPUæ ¸æ•°ä¸Žç½‘å¡ç¡¬ä»‰™˜Ÿåˆ—ä¸ç›¸ç‰åQ?/p>
if [[ ! $HW_QUEUES == $CORES ]]; then
for i in /sys/class/net/$IFACE/queues/rx-*; do
printf "%x\n" $((2**CORES-1)) | xargs -i echo {} > $i/rps_cpus;
done
info_msg " RPS enabled"
else
for i in /sys/class/net/$IFACE/queues/rx-*; do
echo 0 > $i/rps_cpus;
done
info_msg " RPS disabled"
fi
若没有ä‹É用fastsocketåQŒå•¾U¯å€ŸåŠ©äºŽRPSåQŒä¼šå¸¦æ¥å¤„ç†ä¸æ–çš„CPU和处ç†å½“å‰æ•°æ®åŒ…çš„CPUä¸æ˜¯åŒä¸€ä¸ªï¼Œè‡ªç„¶ä¼šé€ æˆCPU Cache MissåQˆCPU¾~“å˜ä¸¢å¤±åQ‰ï¼Œé€ 戞®‘许的性能影å“åQŒäؓ了é¿å…è¿™¿U情况，äºÞZ»¬ä¼šä¾èµ–于RFSåQˆReceive Flow SteeringåQ‰ã€?/p>
使用了fastsocketåŽï¼Œž®×ƒ¸ç”¨è¿™ä¹ˆéº»çƒ¦äº†ã€?/p>
irqbalanceå’Œfastsocket有冲½H，会强制ç¦ç”¨ï¼š
if ps aux | grep irqbalance | grep -v grep; then
info_msg "Disable irqbalance..."
# XXX Do we have a more moderate way to do this?
killall irqbalance > /dev/null 2>&1
fi
脚本也包å«äº†è®„¡½®ä¸æ–å’ŒCPU的亲和性：
i=0
intr_list $IFACE $DRIVER | while read irq; do
cpuid_to_mask $((i%CORES)) | xargs -i echo {} > /proc/irq/$irq/smp_affinity
i=$((i+1))
done
è‹¥iptablesæœåŠ¡å˜åœ¨åQŒä¼šå‹å–„å»ø™®®¼›ç”¨ä¼šå¥½ä¸€äº›ï¼Œæ¯•ç«Ÿä¼šå¸¦æ¥æ€§èƒ½æŸè€—。文件打开å¥æŸ„ä¸å¤§äº?024åQŒè„šæœ¬åŒæ ·ä¼šæ醒åQŒæ€Žä¹ˆè®„¡½®æ–‡äšg打开å¥æŸ„åQŒå¯ä»¥å‚考以å‰åšæ–‡ã€?/p>
Linux¾pÈ»Ÿ¾|‘ç»œå †æ ˆçš„å¸¸è§„æ‰©å±•ä¼˜åŒ–æŽªæ–?/h3>
针对ä¸ä‹É用fastsocketçš„æœåŠ¡å™¨åQŒå½“å‰æ¯”较æµè¡Œçš„针对¾|‘å¡çš„网¾lœå †æ ˆæ€§èƒ½æ‰©å±•ã€ä¼˜åŒ–措施，一般会使用到RSSã€RPSã€RFSã€XFS½{‰æ–¹å¼ï¼Œä»¥ä¾¿å……分利用CPUå¤šæ ¸å’Œç¡¬ä»¶ç½‘å¡ç‰è‡ªèín性能åQŒè¾¾åˆ°åƈè¡?òq¶å‘处ç†çš„目的。下é¢æ€È»“一个表æ û|¼Œå¯ä»¥å‡‘åˆçœ‹ä¸€ä¸‹ã€?/p>
| RSS (Receive Side Scaling) |
RPS (Receive Packet Steering) |
RFS (Receive Flow Steering) |
Accelerated RFS (Accelerated Receive Flow Steering) |
XPS (Transmit Packet Steering) | |
|---|---|---|---|---|---|
| 解决问题 | ¾|‘å¡å’Œé©±åŠ¨æ”¯æŒ?/td> | 软äšgæ–¹å¼å®žçŽ°RSS | æ•°æ®åŒ…äñ”生的ä¸æ–和应用处ç†åœ¨åŒä¸€ä¸ªCPUä¸?/td> | åŸÞZºŽRFS¼‹¬äšgåŠ é€Ÿçš„è´Ÿè²òqŒ™¡¡æœºåˆ¶ | æ™ø™ƒ½é€‰æ‹©¾|‘å¡å¤šé˜Ÿåˆ—的队列快速å‘åŒ?/td> |
| å†…æ ¸æ”¯æŒ | 2.6.36开始引入，需è¦ç¡¬ä»¶æ”¯æŒ?/td> | 2.6.35 | 2.6.35 | 2.6.35 | 2.6.38 |
| å»ø™®® | ¾|‘å¡é˜Ÿåˆ—数和物ç†æ ¸æ•°ä¸€ç›?/td> | è‡Ïx¤å¤šé˜Ÿåˆ—çš„¾|‘å¡è‹¥RSSå·²ç»é…置了，则ä¸éœ€è¦RPSäº?/td> | 需è¦rps_sock_flow_entrieså’Œrps_flow_cnt属æ€?/td> | 需è¦ç½‘å¡è®¾å¤‡å’Œé©±åŠ¨éƒ½æ”¯æŒåŠ 速。åƈ且è¦æ±‚ntuple˜q‡æ×oå·²ç»é€šè¿‡ethtoolå¯ç”¨ | å•ä¼ 输队列的¾|‘å¡æ— 效åQŒè‹¥é˜Ÿåˆ—比CPUž®‘，å…׃ín指定队列的CPU最好是与处ç†ä¼ 输硬ä¸æ–çš„CPUå…׃ín¾~“å˜çš„CPU |
| fastsocket | ¾|‘å¡ç‰ÒŽ€?/td> | 改进版RPSåQŒæ€§èƒ½æå‡ | æºç 包å«åQŒæ–‡æ¡£æ²¡æœ‰æ¶‰å?/td> | æ–‡æ¡£æ²¡æœ‰æ¶‰åŠ | è¦æ±‚å‘é€é˜Ÿåˆ—æ•°è¦å¤§äºŽCPUæ ¸æ•° |
| ä¼ é€æ–¹å?/td> | ¾|‘å¡æŽ¥æ”¶ | å†…æ ¸æŽ¥æ”¶ | CPUæŽ¥æ”¶å¤„ç† | åŠ é€Ÿåƈ接收 | ¾|‘å¡å‘é€æ•°æ?/td> |
更具体优化措施，å¯ä»¥å‚考文档：Scaling in the Linux Networking Stackã€?/p>
å¦ï¼Œè‹¥ç½‘å¡æ”¯æŒ?code>Flow Director Filtersç‰ÒŽ€§ï¼ˆ˜q™é‡Œæœ‰ä¸€ä¸ªéžå¸¸æœ‰‘£çš„动画介ç»åQ?a >Intel® Ethernet Flow DirectoråQŒå€¼å¾—一看）åQŒé‚£ä¹ˆå¯ä»¥ç»“åˆFastsocketä¸€èµ·åŠ é€Ÿã€‚æ¯”å¦‚ï¼Œåœ¨å…¶æ‰€ä½œRedis长连接测试ä¸åQŒå¯ç”¨Flow-Directorç‰ÒŽ€§è¦æ¯”ç¦ç”¨å¯ä»¥å¸¦æ?5%的性能æå‡ã€?/p>
自然软硬¾l“åˆåQŒå¯ä»¥åšçš„更好一些嘛ã€?/p>
延äŽ×阅读åQ?a >多队列网å¡ç®€ä»?/a>
ž®ç»“
以上记录了å¦ä¹ fastsocket的网å¡è®¾¾|®è„šæœ¬æ–¹é¢ç¬”è®°ã€?/p>
ä¸è¿‡å‘¢ï¼Œnic.sh脚本åQŒå€¼å¾—收è—åQŒæ— è®ÞZ‹Éä¸ä‹É用fastsocketåQŒå¯¹¾U¿ä¸ŠæœåŠ¡å™¨ç½‘å¡è°ƒä¼˜éƒ½æ˜¯ä¸é”™é€‰æ‹©å“¦ã€?/p>
å‰è¨€
˜q行环境为Centos 6.5¾pÈ»ŸåQŒé»˜è®¤å†…æ æ€Ø“2.6.32-431.el6.x86_64åQŒä¸‹é¢æ‰€æœ‰ç¼–译安装æ“作是ä»?code>root用户æƒé™˜q›è¡Œæ“作ã€?/p>
¾~–译安装fastsocketå†…æ ¸
½W¬ä¸€æ¥éœ€è¦ä¸‹è½½ä»£ç ，当然˜q™æ˜¯åºŸè¯äº†ï¼Œä¸‹è²åˆ?opt目录下：
git clone https://github.com/fastos/fastsocket.git
¾~–译安装
下è²ä¹‹åŽåQŒéœ€è¦è¿›å…¥å…¶ç›®å½•ä¸ï¼š
cd fastsocket/kernel
å› äؓ是涉åŠåˆ°å†…æ ¸å˜›ï¼Œ¾~–译之å‰éœ€è¦åšä¸€äº›å‚数选项é…ç½®åQŒä‹Éç”?code>make config会ç¯æÖMhçš„ï¼Œå¥½å‡ åƒä¸ªé€‰é¡¹å‚数需è¦ä½ 一一é…ç½®åQŒå¤§éƒ¨åˆ†æ—‰™—´åQŒé»˜è®¤é…¾|®å°±æŒºå¥½çš„：
make defconfig
然åŽå˜›ï¼Œ¾~–è¯‘å†…æ ¸çš„èŠ‚å¥ï¼š
make
å†…æ ¸¾~–译相当耗费旉™—´åQŒè‡³ž®?0分钟旉™—´ã€‚之åŽç´§æŽ¥ç€æ˜¯ç¼–è¯‘æ‰€éœ€çš„å†…æ ¸æ¨¡å—，fastsocket模å—åQ?/p>
make modules_install
¾~–译完æˆä¹‹åŽåQŒæœ€åŽä¸€æ¡è¾“出，会看刎ͼš
DEPMOD 2.6.32-431.17.1.el6.FASTSOCKET
fastsocketå†…æ ¸æ¨¡å—¾~–译好之åŽï¼Œéœ€è¦å®‰è£…å†…æ ¸ï¼š
make install
上é¢å‘½ä×o其实执行shell脚本˜q›è¡Œå®‰è£…åQ?/p>
sh /opt/fastsocket/kernel/arch/x86/boot/install.sh 2.6.32-431.17.1.el6.FASTSOCKET arch/x86/boot/bzImage \ System.map "/boot"
基本上，fastsocketå†…æ ¸æ¨¡å—å·²ç»æž„å¾å®‰è£…完毕了，但需è¦å‘ŠçŸ¥Linux¾pÈ»Ÿåœ¨ä¸‹‹Æ¡å¯åŠ¨çš„时候切æ¢åˆ°æ–°ç¼–译的ã€åŒ…å«æœ‰fastsocket模å—çš„å†…æ ¸ã€?/p>
é…ç½®å¯åŠ¨æ—‰™œ€è¦åˆ‡æ¢çš„å†…æ ¸
˜q™éƒ¨åˆ†éœ€è¦åœ¨/etc/grup.confä¸é…¾|®ï¼ŒçŽ°åœ¨çœ‹ä¸€ä¸‹å…¶æ–‡äšg内容åQ?/p>
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-431.17.1.el6.FASTSOCKET)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.17.1.el6.FASTSOCKET ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS rd_NO_MD rd_LVM_LV=vg_centos6/lv_swap crashkernel=auto LANG=zh_CN.UTF-8 rd_LVM_LV=vg_centos6/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.17.1.el6.FASTSOCKET.img
title CentOS (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS rd_NO_MD rd_LVM_LV=vg_centos6/lv_swap crashkernel=auto LANG=zh_CN.UTF-8 rd_LVM_LV=vg_centos6/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.el6.x86_64.img
defautl=1åQŒè¡¨½Cºç›®å‰ç³»¾lŸé€‰æ‹©çš„ä»¥åŽŸå…ˆå†…æ ¸ä½œä½œä¸ºå¯åŠ¨é¡¹åQŒåŽŸå…ˆä½äºŽç¬¬äºŒä¸ªroot (hd0,0)åŽé¢åQŒéœ€è¦åˆ‡æ¢åˆ°æ–°çš„å†…æ ¸ä¸‹é¢åQŒéœ€è¦ä¿®æ”?code>default=0åQŒä¿å˜åŽåQŒrebooté‡å¯¾pÈ»ŸåQŒä‹É之生效ã€?/p>
‹‚€‹¹‹ç”Ÿæ•?/h4>
¾pÈ»Ÿé‡å¯åŽï¼Œéœ€è¦åŠ è½½fastsocket模å—到系¾lŸè¿è¡Œä¸åŽ»ï¼Œä¸‹é¢ä»¥é»˜è®¤é€‰é¡¹å‚æ•°æ–¹å¼åŠ è²åQ?/p>
modprobe fastsocket
åŠ è²ä¹‹åŽåQŒåˆ—出当å‰ç³»¾lŸæ‰€åŠ è²æ¨¡å—列表åQŒæ£€æŸ¥æ˜¯å¦æˆåŠ?/p>
lsmod | grep fastsocket
若能看到¾cÖM¼¼è¾“出信æ¯åQŒè¡¨½CºOKåQ?/p>
fastsocket 39766 0
开始构建libfastsocket.so链接库文�/h3>
上é¢å†…æ ¸æ¨¡å—安装好之åŽï¼Œå¯ä»¥æž„å¾fastsocket的动æ€é“¾æŽ¥åº“æ–‡äšg了：
cd /opt/fastsocket/library/
make
å¯èƒ½ä¼šæ”¶åˆîC¸€äº›è¦å‘Šä¿¡æ¯ï¼Œæ— ç¢åQ?/p>
gcc -g -shared -ldl -fPIC libsocket.c -o libfsocket.so -Wall
libsocket.c: 在函æ•?#8216;fastsocket_init’ä¸?
libsocket.c:59: è¦å‘ŠåQšéšå¼å£°æ˜Žå‡½æ•?#8216;open’
libsocket.c: 在函æ•?#8216;fastsocket_expand_fdset’ä¸?
libsocket.c:109: è¦å‘ŠåQšéšå¼å£°æ˜Žå‡½æ•?#8216;ioctl’
libsocket.c: 在函æ•?#8216;accept’ä¸?
libsocket.c:186: è¦å‘ŠåQšå¯¹æŒ‡é’ˆèµ‹å€¼æ—¶ç›®æ ‡ä¸ŽæŒ‡é’ˆç¬¦å·ä¸ä¸€è‡?
libsocket.c: 在函æ•?#8216;accept4’ä¸?
libsocket.c:214: è¦å‘ŠåQšå¯¹æŒ‡é’ˆèµ‹å€¼æ—¶ç›®æ ‡ä¸ŽæŒ‡é’ˆç¬¦å·ä¸ä¸€è‡?
最åŽï¼Œå¯ä»¥çœ‹åˆ°gcc¾~–译之åŽç”Ÿæˆçš?code>libfsocket.so库文ä»Óž¼Œè¯´æ˜Ž¾~–译æˆåŠŸã€?/p>
ž®ç»“
OKåQŒç¼–译安装到æ¤ç»“æŸï¼ŒåŽé¢ž®±æ˜¯å¦‚何使用fastsocket的示范程åºè¿›è¡Œæµ‹è¯•äº†ã€?/p>
å‰è¨€
上篇介ç»äº†å¦‚何构建安装fastsocketå†…æ ¸æ¨¡å—åQŒä¸‹é¢å°†åŸÞZºŽfastsocket/demo/README.mdæ–‡äšg¾˜»è¯‘æ•´ç†è€Œæˆã€?/p>
嗯，下题q›å…¥¾˜»è¯‘½‹‡ã€?/p>
介ç»
½Cø™Œƒä¸ÞZ¸€ä¸ªç®€å•TCP ServeræœåŠ¡å™¨ç¨‹åºï¼Œç”¨äºŽåŸºå‡†‹¹‹è¯•å’Œå‰–æžLiunxå†…æ ¸¾|‘ç»œå †æ ˆæ€§èƒ½è¡¨çŽ°åQŒå½“然也是äؓ了演½CºFastsocketå¯æ‰©å±•å’Œå…¶æ€§èƒ½æ”¹è¿›ã€?/p>
½Cø™Œƒåº”用åŸÞZºŽepoll模型和éžé˜Õd¡žæ€§IOåQŒå¤„ç†ç½‘¾lœè¿žæŽ¥ï¼Œä½†åªæœ‰åœ¨å¤šæ ¸çš„模å¼ä¸‹æ‰èƒ½å¤Ÿå·¥ä½œå¾—很好åQšç¨‹åºçš„æ¯ä¸€ä¸ªè¿›½E‹è¢«¾l‘定到CPUçš„ä¸åŒæ ¸åQŒè“v始于CPU core 0åQŒå„自独立处ç†å®¢æˆïL«¯˜qžæŽ¥è¯äh±‚ã€?/p>
½Cø™Œƒ½E‹åºå…ähœ‰ä¸¤ç§å·¥ä½œæ¨¡å¼åQ?/p>
- æœåŠ¡å™¨æ¨¡å¼?/strong>åQšä“Q何请求都会直接返回HTTP 200 OK
- 代ç†æ¨¡å¼åQšæœåŠ¡å™¨æŽ¥æ”¶åˆ°å®¢æˆïL«¯è¯äh±‚åQŒè{å‘ç»™åŽç«¯æœåŠ¡å™¨ï¼ŒåŒæ—¶è½¬å‘åŽç«¯å“应¾l™å®¢æˆïL«¯ã€?
˜q™æ˜¯ä¸€ä¸ªç®€å•å‚»ç“œåÅžå¼çš„Tcp ServeråQŒä»…仅用于测试ä‹É用，使用时è¦æ±‚客æˆïL«¯å’ŒæœåŠ¡å™¨ç«¯åªèƒ½å¤Ÿæºå¸¦ä¸€ä¸ªpacket包大ž®çš„æ•°æ®åQŒå¦åˆ™ç¨‹åºä¼šå¤„ç†ä¸äº†ã€?/p>
æž„å¾
以下é¢æ–¹å¼è¿›è¡Œæž„建：
cd demo && make
用法
最½Ž€å•æ–¹å¼ä»¥é»˜è®¤é…ç½®æ— å‚æ•°åÅžå¼è¿è¡Œï¼š
./server
å‚数如下:
- -w worker_num: 定义˜q›ç¨‹æ•?
- 默认å€égؓ当å‰å¯ç”¨CPUæ ¸å¿ƒæ•îC¸ª˜q›ç¨‹.
- -c start_core: 指定˜q›ç¨‹¾l‘定CPUæ ¸çš„å¼€å§‹çƒ¦å¼•å€?
- 默认å€égØ“ 0.
- -o log_file: 定义日志文äšgå称
- 默认å€égØ“ ./demo.log
- -a listen_address: 指定监å¬åœ°å€åQŒ[ip:port]å—符串组åˆåÅžå¼ï¼Œæ”¯æŒæ·ÕdŠ 多个地å€
- 默认å€égØ“ 0.0.0.0:80
- -x backend_address: å¯åŠ¨ä»£ç†æ¨¡å¼åQŒéœ€è¦å¡«å†™[ip:port]¾l„åˆå½¢å¼åœ°å€åQŒæ”¯æŒå¤šä¸ªä»£ç†åœ°å€
- 默认ä¸å¼€å?
- -v: å¯ç”¨è¯¦ç»†¾lŸè®¡æ•°æ®è¾“出
- 默认为ç¦ç”?
- -d: å¯åŠ¨Debug调试模å¼åQŒè°ƒè¯•ä¿¡æ¯è¢«å†™å…¥æ—¥å¿—æ–‡äšgä¸?
- 默认¼›ç”¨
- -k: å¯ç”¨HTTP keepalive机制åQŒå½“å‰åªèƒ½å¤Ÿå·¥ä½œåœ¨æœåŠ¡å™¨æ¨¡å¼ä¸?
- 默认被ç¦ç”?
实例
在è¿è¡Œä¹‹å‰ï¼Œéœ€è¦æ³¨æ„两点：
- ä¸ÞZº†è·‘满CPUåQŒéœ€è¦ç¡®ä¿å®¢æˆïL«¯å’ŒåŽç«¯æœåŠ¡å™¨éƒ½ä¸åº”该æˆäؓ瓉™¢ˆåQŒä¸¤¿Uå¯è¡Œæ–¹æ¡ˆï¼š
- æä¾›‘›_¤Ÿå¤šæœºå™¨ç”¨ä»¥å……当客æˆïL«¯å’ŒåŽç«¯æœåŠ¡å™¨è§’色
- 或在一å°æœºå™¨ä¸Šå……当客户端和åŽç«¯æœåŠ¡å™¨ï¼Œä½¿ç”¨fastsocketåQˆæŽ¨è方案，较äؓ节çœæœåŠ¡å™¨ï¼‰
- æ£ç¡®é…ç½®¾|‘å¡åQŒè‹¥ä¸çŸ¥é“如何åšåQŒå¯ä»¥å‚考æºç ä¸
script目录
æœåŠ¡å™¨æ¨¡å¼ç¤ºèŒ?/h4>
æœåŠ¡å™¨æ¨¡å¼è‡³ž®‘需è¦ä¸¤åîC¸»æœºï¼š
- ä¸ÀLœºA作äؓ客户端äñ”生HTTPè¯äh±‚
- ä¸ÀLœºB为WebæœåŠ¡å™?
讑֮šæ¯å°ä¸ÀLœºCPU 12æ ¸ï¼Œ¾|‘络大概讄¡½®å¦‚下åQ?/p>
+--------------------+ +--------------------+
| Host A | | Host B |
| | | |
| 10.0.0.1/24 |-----| 10.0.0.2/24 |
| | | |
+--------------------+ +--------------------+
下é¢æ˜¯è¿è¡Œä¸¤åîC¸»æœºçš„æ¥éª¤åQ?/p>
ä¸ÀLœºBåQ?/p>
-
WebæœåŠ¡å™¨æ¨¡å¼å•ç‹¬è¿è¡Œï¼Œå¼€å?2个工作进½E‹ï¼Œå’ŒCPUæ ¸å¿ƒæ•îC¸€è‡ß_¼š
./server -w 12 -a 10.0.0.2:80 -
或者测试借助于Fastsocket所带æ¥çš„性能
LD_PRELOAD=../library/libfsocket.so ./server -w 12 -a 10.0.0.2:80
ä¸ÀLœºAåQ?/p>
- ˜qè¡ŒApache ab½E‹åºä½œäØ“è¯äh±‚è€?br />
ab -n 1000000 -c 100 http://10.0.0.2:80/ - å•ä¸ªApache ab½E‹åºä¸èƒ½å¤Ÿä½“现æœåŠ¡å™¨è´Ÿè²èƒ½åŠ›åQŒå¤šä¸ªab实例åŒæ—¶òq¶å‘˜qè¡Œå¯èƒ½ä¼šå¥½å¾ˆå¤šåQŒå¼€12个实例和CPUæ ¸å¿ƒæ•îC¸€è‡ß_¼š
N=12; for i in $(seq 1 $N); do ab -n 1000000 -c 100 http://10.0.0.2:80/ > /dev/null 2>&1; done
代ç†æ¨¡å¼½Cø™Œƒ
代ç†æ¨¡å¼ä¸‹ï¼Œéœ€è¦ä¸‰å°æœºå™¨ï¼š
- ä¸ÀLœºA作äؓ客户端äñ”生HTTPè¯äh±‚
- ä¸ÀLœºB作äؓ代ç†è§’色
- ä¸ÀLœºC则需è¦åŽç«¯æœåŠ¡å™¨
讑֮šæ¯å°æœºå™¨CPUå†…æ ¸æ•?2åQŒç½‘¾lœç»“构如下：
+--------------------+ +--------------------+ +--------------------+
| Host A | | Host B | | Host C |
| | | | | |
| 10.0.0.1/24 | | 10.0.0.2/24 | | 10.0.0.3/24 |
+---------+----------+ +---------+----------+ +----------+---------+
| | |
+---------+--------------------------+---------------------------+---------+
| switch |
+--------------------------------------------------------------------------+
下é¢ä¸ºå…·ä½“çš„˜qè¡Œæ¥éª¤åQ?/p>
ä¸ÀLœºBåQ?/p>
- ä¸ÞZ»£ç†æœåŠ¡å™¨å¯åŠ¨12个进½E?br />
./server -w 12 -a 10.0.0.2:80 -x 10.0.0.3:80 - 或者以Fastsocketæ–¹å¼å¯åŠ¨
LD_PRELOAD=../library/libsocket.so ./server -w 12 -a 10.0.0.2:80 -x 10.0.0.3:80
ä¸ÀLœºCåQ?/p>
- ç†è®ºä¸Šä“Q何WEBæœåŠ¡å™¨éƒ½å¯ä»¥å……当åŽç«¯æœåŠ¡å™¨ï¼Œ˜q™é‡Œå……分利用½Cø™Œƒ½E‹åºå¥½äº†åQ?br />
./server -w 12 -a 10.0.0.3:80
ä¸ÀLœºAåQ?/p>
- 作äؓ客户端请求生æˆå™¨åQŒåŒæ ·å¯åŠ?2个Apache ab实例åQ?br />
N=12; for i in $(seq 1 $N); do ab -n 1000000 -c 100 http://10.0.0.2:80/ > /dev/null 2>&1; done
动手实践
以上¾˜»è¯‘完毕åQŒä¸‹é¢å°†æ˜¯æ ¹æ®ä¸Šé¢å†…容进行动手测试æ˜q°å§ã€?/p>
安装Apache ab命ä×o
‹‚€æŸ¥ä¸€ä¸‹åŒ…å«Apache ab命ä×oçš„èÊY件包åQ?/p>
yum provides /usr/bin/ab
å¯ä»¥çœ‹åˆ°¾cÖM¼¼äºŽå¦‚下å—æ øP¼š
httpd-tools-2.2.15-39.el6.centos.x86_64 : Tools for use with the Apache HTTP Server
安装它就å¯ä»¥äº?/p>
yum install httpd-tools
虚拟机测�/h4>
Windows 7专业版跑VMware Workstation 10.04虚拟机，两个Centos 6.5¾pÈ»ŸåQŒé…¾|®ä¸€è‡ß_¼Œ2G内å˜åQ?个CPU逻辑处ç†å™¨æ ¸å¿ƒã€?/p>
客户端安装Apache ab命ä×o‹¹‹è¯•åQŒè·‘8个实例： for i in $(seq 1 8); do ab -n 10000 -c 100 http://192.168.192.16:80/ > /dev/null 2>&1; done
æœåŠ¡å™¨ç«¯åQŒåˆ†åˆ«è®°å½•ï¼š/opt/fast/server -w 8 LD_PRELOAD=../library/libfsocket.so ./server -w 8
æœåŠ¡å™¨æ¨¡å¼å¯¹æ¯?/h4>
两组数æ®å¯ÒŽ¯”åQ?/p>
| ˜qè¡Œæ–¹å¼ | 处ç†æ¶ˆè€—æ—¶é—?¿U? | 处ç†æ€ÀL•° | òq›_‡æ¯ç§’处ç†æ•?/th> | 最大å€?/th> |
|---|---|---|---|---|
| å•ç‹¬˜qè¡Œ | 34s | 80270 | 2361 | 2674 |
| åŠ è²fasocket | 28s | 80399 | 2871 | 2964 |
代ç†æ¨¡å¼æ•°æ®
‹¹‹è¯•æ–¹å¼å¦‚上åQŒä¸‰å°æœåŠ¡å™¨åQˆæµ‹è¯•ç«¯+代ç†ç«?æœåŠ¡å™¨ç«¯åQ‰é…¾|®ä¸€æ —÷€‚第一‹Æ¡ä»£ç†å•ç‹¬å¯åŠ¨ï¼Œ½W¬äºŒ‹Æ¡ä»£ç†é¢„åŠ è²fastsocketæ–¹å¼ã€?/p>
| ˜qè¡Œæ–¹å¼ | 处ç†æ¶ˆè€—æ—¶é—?¿U? | 处ç†æ€ÀL•° | òq›_‡æ¯ç§’处ç†æ•?/th> | 最大å€?/th> |
|---|---|---|---|---|
| ½W¬ä¸€‹Æ¡æµ‹è¯•åŽç«?/td> | 44s | 80189 | 1822 | 2150 |
| ½W¬ä¸€‹Æ¡æµ‹è¯•ä»£ç?/td> | 44s | 80189 | 1822 | 2152 |
| ½W¬äºŒ‹Æ¡æµ‹è¯•åŽç«?/td> | 42s | 80051 | 1906 | 2188 |
| ½W¬äºŒ‹Æ¡æµ‹è¯•ä»£ç?/td> | 42s | 80051 | 1906 | 2167 |
备注åQšè™šæ‹Ÿæœºä¸Šæ•°æ®ï¼Œä¸ä»£è¡¨çœŸå®žæœåŠ¡å™¨ä¸Šæ•°æ®ï¼Œä»…ä¾›å‚考ã€?/p>
虽然åŸÞZºŽè™šæ‹Ÿæœºï¼Œ‹¹‹è¯•çŽ¯å¢ƒå—é™åQŒä½†ä¸€æ ·å¯ä»¥çœ‹åˆ°åŸºäºŽfastsocketæœåŠ¡å™¨æ¨¡åž‹ï¼Œå¤„ç†æ€§èƒ½æœ‰æ‰€æå‡åQšæ€ÖM½“处ç†æ—‰™—´åQŒæ¯¿U’åã^å‡å¤„ç†æ•°åQŒä»¥åŠå¤„ç†ä¸Šé™ç‰ã€?/p>
关于LD_PRELOAD注æ„事项
动æ€é“¾æŽ¥é¢„å…ˆåŠ è½½LD_PRELOAD虽是利器åQŒä½†ä¸æ˜¯ä¸‡èƒ½è¯ï¼ŒLD_PRELOADé‡åˆ°ä¸‹é¢æƒ…况会失效：
- é™æ€é“¾æŽ¥ä‹É用gcc -staticå‚数把libc.so.6é™æ€é“¾å…¥æ‰§è¡Œç¨‹åºä¸
- 讄¡½®æ‰§è¡Œæ–‡äšgçš„SUIDæƒé™åQŒå¯èƒ½ä¹Ÿä¼šå¯¼è‡´LD_PRELOAD失效åQˆå¦‚åQšchmod 4755 daemonåQ?
情况很å¤æ‚，ž®å¿ƒä¸ÞZ¸Šã€?/p>
ž®ç»“
å¦ä¹ òq¶æµ‹è¯•äº†fastsocketçš„æºç 示范部分，å‰åŽå¯ÒŽ¯”å¯ä»¥çœ‹åˆ°fastsocket带æ¥äº†å¤„ç†æ€§èƒ½çš„æå‡ã€?/p>
¾|‘络延迟是客观å˜åœ¨çš„åQŒä½†¾|‘络游æˆè¡Œä¸šå·²ç»¿U¯ç¯äº†å¤§é‡ä¼˜è´¨ç»éªŒï¼Œä½¿ç”¨ä¸€äº›ç–ç•¥ã€æŠ€æœ¯æ‰‹ŒDµåœ¨å®¢æˆ·ç«¯æ¶ˆé™?éšè—掉åšg˜qŸå¸¦æ¥çš„ä¸ä¾¿åQŒä»¥ž®½å¯èƒ½çš„掩盖实际å˜åœ¨çš„åšgæ—Óž¼ŒåŒæ—¶å®žçŽ°å®žæ—¶æ¸²æŸ“åQŒå°†ç”¨æˆ·å¸¦å…¥å¿«é€Ÿçš„交互å¼å®žæ—¶æ¸¸æˆä¸åQŒä½“验完¾ŸŽçš„互动å¨×ƒ¹ä¸ã€?/p>
˜q™æ ·å¤„熾l“æžœåQŒç¨é«˜åšg˜qŸçš„玩家也ä¸ä¼šå› 为网¾lœä¸æ˜¯é‚£ä¹ˆå¥½åQŒä¹Ÿèƒ½å¤Ÿå¾ˆå’Œè°çš„与其它网¾lœå‚å·®ä¸åŠçŽ©å®¶ä¸€èµäh¸¸æˆä¸ã€?/p>
虽然延时军_®šäº†å®žæ—¶æ¸¸æˆçš„最低å应时é—ß_¼Œä½†æœ€é‡è¦çš„是客户端看èµäh¥è¦æµç•…。第一人称设计游æˆåQˆFPSåQ‰å¯å·§å¦™çš„化解与规é¿åQŒæœ€¾lˆåœ¨é€‚åˆæ™®é用户¾|‘络环境ä¸?200ms)åQŒå®žçŽ°å®žæ—¶å¿«é€Ÿäº’动游æˆã€?/p>
嗯，下颞®±æ˜¯˜q‘期脑补¾l“æžœã€?/p>
¾|‘游P2P & CS¾l“æž„
æ—©å…ˆ¾|‘游使用P2P¾|‘络拓扑在玩家之间进行交æ¢æ•°æ®é€šä¿¡ã€‚但P2P模型引è“v的高延迟在FPS游æˆä¸æ— 法被很好掩盖åQŒæ‰€æœ‰çŽ©å®¶çš„延迟å–决于当å‰çŽ©å®¶ä¸å»¶è¿Ÿæœ€çƒ‚的那个。好比木桶ç†è®ºï¼Œä½Žåšg˜qŸç½‘¾lœå¥½çš„玩家会被高延迟å网¾lœçš„玩家拖ç¯ã€‚最¾lˆç»“果导è‡ß_¼Œæ‰€æœ‰çŽ©å®‰™ƒ½ä¸å¤ªå¼€å¿ƒäº†ã€‚但在局域网环境下，ä¸ä¼šæ„Ÿè§‰åˆ°åšg˜qŸå¸¦æ¥çš„问题。å¦åQŒæ¸¸æˆé€»è¾‘大部分都集ä¸åœ¨å®¢æˆïL«¯äº†ï¼Œå¾ˆéš¾é¿å…作弊行äØ“ã€?/p>
C/S¾l“æž„¾|‘游åQ?/p>
- C/S¾l“构在æœåŠ¡å™¨ç«¯è·‘所有的游æˆé€»è¾‘和输入å“应，客户端åªéœ€è¦æ¸²æŸ“以åŠæŠŠè‡ªå·±éœ€è¦ä¸€äº›çŠ¶æ€åŒæ¥ä¸‹æ¥ï¼ŒæŠŠç”¨æˆ¯‚¾“入呾l™æœåŠ¡å™¨ç«¯ï¼Œç„¶åŽæ˜„¡¤º¾l“æžœž®±å¯ä»¥äº†
- C/S¾l“æž„¾|‘游最大优点就是把延迟从玩家之间最å¡çŽ©å®¶çš„延迟改å˜ä¸ºçŽ©å®¶å’ŒæœåŠ¡å™¨è¿žæŽ¥çš„延迟åQŒç»“果就是客æˆïL«¯åœ¨å¸¦å®½ä¸Šçš„è¦æ±‚也低了ä¸å°‘åQŒå› 为åªéœ€è¦æŠŠè¾“å…¥å‘ç»™æœåŠ¡å™¨ç«¯ž®×ƒ»¥åŠæŽ¥æ”¶æœåŠ¡å™¨å“应ž®±å¤Ÿäº?
- C/S¾l“æž„¾|‘游虽然转移了网¾lœåšg˜qŸçŸ›ç›„¡‚¹åQŒä½†çŽ°å®ž¾|‘ç»œçŽ¯å¢ƒä¸€æ ·ä¼šå¸¦æ¥è¾ƒé«˜çš„网¾lœåšg˜qŸã€‚客æˆïL«¯æ¯æ‰§è¡Œä¸€‹Æ¡æ“作，都需è¦ç‰å¾…æœåŠ¡å™¨ç«¯å‘½ä»¤ï¼Œé‚£ä¼šç”¨æˆ·æ“ä½œä¼šé€ æˆæ“纵å¡é¡¿çŽ°è±¡ã€‚如何解军_‘¢åQŒå®¢æˆïL«¯ä¸€èˆ¬é‡‡ç”¨é¢„‹¹‹å’Œæ’值ç‰æ–¹å¼åœ¨æ¸²æŸ“层éšè—¾|‘络延迟
客户端预‹¹‹å’Œæ’å€?/h3>
æœåŠ¡å™¨å¯ä»¥å…许æŸäº›æƒ…况下客户端本地å³æ—¶æ‰§è¡Œç§»åŠ¨æ“作，˜q™ç§æ–ÒŽ³•å¯ä»¥¿UîCؓ客户端预‹¹‹ã€?/p>
比如游æˆä¸é”®ç›˜æŽ§åˆ¶è§’色行赎ͼŒ˜q™ä¸ªæ—¶å€™å¯ä»¥åœ¨å¾ˆå°çš„时间段åQˆæ—¶é—´å¾ˆçŸï¼Œæ¯”如1-3¿U’）内预‹¹‹ç”¨æˆ¯‚¡ŒåŠ¨è½¨˜q¹ï¼ˆæ–¹å‘+åŠ é€Ÿåº¦åQŒè§’色行走结果）åQŒè¿™éƒ¨åˆ†çš„命令客æˆïL«¯ä¼šå…¨éƒ¨å‘é€åˆ°æœåŠ¡å™¨ç«¯æ ¡éªŒæ£ç¡®ä¸Žå¦åQˆé¿å…瞬间è{¿Uȉ外挂åQ‰ã€‚但客户端预‹¹‹æœ‰æ—¶ä¹Ÿä¸æ˜¯ç™‘Öˆ†ç™‘Ö‡†¼‹®ï¼Œéœ€è¦æœåŠ¡å™¨˜q›è¡Œ¾U æ£åQˆæ‰€è°“æœåŠ¡å™¨ž®±æ˜¯ä¸Šå¸åQŒThe sever is the manåQï¼‰ã€‚çº æ£ç»“æžœå¯èƒ½å°±æ˜¯æ¸¸æˆè§’色行走轨˜q¹å’Œå®¢æˆ·ç«¯é¢„‹¹‹è½¨˜qÒŽœ‰æ‰€åå·®åQŒå®¢æˆïL«¯å¯ä»¥ä½¿ç”¨æ’值方å¼ï¼ˆ¾_—ç•¥æ¥è®²åQŒå°±æ˜¯è§’色在两点之间¿UÕdŠ¨æ¸²æŸ“çš„æ–¹å¼ï¼‰æ¸²æŸ“游æˆè§’色在游æˆä¸–ç•Œä¸çš„ä½¾|®è{¿UÕdã^滑一些，é¿å…游æˆè§’色从一个佾|®çž¬é—´æ‹‰å›žåˆ°å¦ä¸€ä¸ªä½¾|®ï¼Œè®©äh有些莫å其妙ã€?/p>
æ’å€û|¼Œæœ‰äh也称之äؓ路径补å¿åQŒéƒ½æ˜¯ä¸€å›žäº‹ã€‚æ’值的æ–ÒŽ³•ä¼šæ¶‰åŠåˆ°å¾ˆå¤šæ•°å¦å…¬å¼åQŒçº¿æ€§æ’倹{€ä¸‰‹Æ¡çº¿æ€§æ’值ç‰åQŒæ¯”如这½‹‡æ–‡ç« 所讲到çš?a >æ’值那些事ã€?/p>
ž®ç»“åQšå®¢æˆïL«¯é¢„测åQŒæœåŠ¡å™¨ç«¯çº æ£ï¼Œå®¢æˆ·ç«¯é‡‡ç”¨æ’值方å¼å¾®è°ƒã€?/p>
针对交互的一¾Ÿ¤çŽ©å®Óž¼Œ¾|‘络好å层次ä¸é½åQŒæ¸¸æˆçš„一些æ“作效果å¯èƒ½éœ€è¦?#8221;延迟补忓½{–ç•¥˜q›è¡Œ
延迟补å¿
延迟补å¿æ˜¯æ¸¸æˆæœåŠ¡å™¨ç«¯æ‰§è¡Œçš„一¿Uç–略，处ç†ç”¨æˆ·å‘½ä×o回退到客æˆïL«¯å‘é€å‘½ä»¤çš„准确旉™—´åQˆåšg˜qŸå¯¼è‡ß_¼‰åQŒæ ¹æ®å®¢æˆïL«¯çš„具体情况进行修æ£ï¼Œä»¥ç‰ºç‰²æ¸¸æˆåœ¨ä¼¤å®³åˆ¤å®šæ–šw¢çš„真实感æ¥å×I补攻击行为ç‰æ–šw¢çœŸå®žæ„Ÿï¼Œæœ¬è´¨ä¸Šæ˜¯ä¸€¿U折ä¸é€‰æ‹©ã€?/p>
主è¦æ³¨æ„åQŒåšg˜qŸè¡¥å¿ä¸æ˜¯å‘生在客户端ã€?/p>
关于延迟补å¿çš„一个例å：
- 在FPS游æˆä¸ï¼ŒçŽ©å®¶Aåœ?0.5¿U’æ—¶å‘ç›®æ ‡å¯¹è±¡çŽ©å®¶Bž®„击òq¶ä¸”å‡ÖM¸åQŒå°„å‡ÖM¿¡æ¯è¢«æ‰“包å‘é€ï¼ˆ¾|‘络延迟100毫秒åQ‰ï¼ŒæœåŠ¡å™¨äºŽ10.6¿U’收刎ͼŒæ¤æ—¶çŽ©å®¶Bå¯èƒ½å·²è·‘到å¦å¤–一个佾|®ã€?
- è‹¥æœåŠ¡å™¨ä»…ä»…åŸÞZºŽæŽ¥æ”¶æ—¶åˆ»åQ?0.6¿U’）˜q›è¡Œåˆ¤æ–åQŒé‚£ä¹ˆçŽ©å®¶B没有收到伤害åQŒæˆ–许å¯èƒ½ä¼šå‡ÖM¸çŽ©å®¶BåŽé¢ç´§è·Ÿçš„玩家CåQ?00msåŽçŽ©å®¶C完全由å¯èƒ½å·²å¤„于玩家Açš„å°„å‡È›®æ ‡ä½¾|®ï¼‰
- ä¸ÞZº†å¼¥è¡¥ç”׃ºŽå»¶è¿Ÿé€ æˆçš„问题，æœåŠ¡å™¨ç«¯éœ€è¦å¼•å…?#8220;延迟补忔½{–略用于修æ£å› åšg˜qŸé€ æˆé”™äØ•å‡è±¡
- æœåŠ¡å™¨è®¡½Ž—执行设计命令时é—ß_¼Œç„¶åŽæ‰‘Ö‡ºå½“å‰ä¸–ç•Œ10.5¿U’æ—¶åˆÈŽ©å®¶ä¿¡æ¯ï¼Œæ ÒŽ®ž®„击½Ž—法模拟得出是å¦å‘½ä¸åˆ¤æ–åQŒä»¥è¾‘Öˆ°ž®½å¯èƒ½ç²¾¼‹?
若游æˆåšg˜qŸè¡¥å¿è¢«¼›ç”¨åQŒé‚£ä¹ˆå°±ä¼šæœ‰è®¸å¤šçŽ©å®¶æŠ±æ€¨è‡ªå·±æ˜Žæ˜Žæ‰“ä¸äº†å¯ÒŽ–¹å´æ²¡æœ‰é€ æˆä»ÖM½•ä¼¤å®³ã€‚ã€?/p>
有所得，有所失：但这对低延时玩家貌似有些ä¸å…¬òq»I¼Œ¿UÕdŠ¨é€Ÿåº¦å¿«ï¼Œå¯èƒ½å·²ç»è·‘到角è½é‡Œåƈ且已íy²åœ¨ä¸€ä¸ªç®±ååŽé¢éšè—è“væ¥æ—¶è¢«å¯¹æ‰‹å‡»ä¸çš„错觉åQˆåå¼ÒŽ— 视掩体，玩家隔ç€å¢™è¢«ž®„击åQ‰ï¼Œ¼‹®å®žæœ‰äº›ä¸ä¹æ„ã€?/p>
延迟补å¿åQŒç½‘¾lœé«˜å»¶è¿Ÿçš„玩家有利，低åšg˜qŸçš„玩家优势å¯èƒ½ä¼šè¢«é™ä½ŽåQˆä½Žå»¶è¿ŸçŽ©å®¶åˆ©ç›Šå—æŸåQ‰ï¼Œä½†å¯¹¾l´æŠ¤æ¸¸æˆä¸–界的åã^衡还是有利的ã€?/p>
å¯ÒŽ—¶&阀å€?/h3>
客户端和æœåŠ¡å™¨éœ€è¦å¯¹æ—Óž¼Œäº’相知é“å½¼æ¤å»¶è¿Ÿæƒ…况åQŒæ¯”如云风定义的æŸä¸ªæ¥éª¤åQ?/p>
客户端å‘é€ä¸€ä¸ªæœ¬åœ°æ—¶é—´é‡¾l™æœåŠ¡å™¨åQŒæœåŠ¡æ”¶åˆ°åŒ…åŽï¼Œå¤¹å¸¦ä¸€ä¸ªæœåŠ¡å™¨æ—‰™—´˜q”回¾l™å®¢æˆïL«¯ã€‚当客户端收到这个包åŽï¼Œå¯ä»¥ä¼°ç®—出包在èµ\½E‹ä¸Š¾l过的时间。åŒæ—¶æŠŠæœ¬åœ°æ–°æ—¶é—´å¤¹å¸¦è¿›åŽ»ï¼Œå†æ¬¡å‘é€ç»™æœåŠ¡å™¨ã€‚æœåŠ¡å™¨ä¹Ÿå¯ä»¥è¿›ä¸€æ¥çš„了解å“应旉™—´ã€?/p>
C/S两端通过¾cÖM¼¼æ¥éª¤˜q›è¡Œè®¡ç®—å½¼æ¤å»¶æ—¶/时差åQŒåŒæ—¶ä¼šå¯¹å®žæ—¶åŒæ¥è®¾¾|®ä¸€ä¸ªé˜€å€û|¼Œæ¯”如对åšg˜qŸä½Žäº?0msåQ?.01¿U’）的交互认为是åÏx—¶åŒæ¥å‘生åQŒä¸ä¼šè®¤ä¸ºæ˜¯å»¶è¿Ÿã€?/p>
UDP或TCP
ä¸åŒ¾cÕdž‹çš„游æˆä¼šé’Ÿçˆ±ä¸åŒçš„å议呢åQŒä¸ä¸€è€ŒèƒöåQ?/p>
- 客户端间æ‡æ€§çš„å‘è“væ— çŠ¶æ€çš„查询åQŒåƈ且嶞®”å‘生åšg˜qŸæ˜¯å¯ä»¥å®¹å¿åQŒé‚£ä¹ˆä‹É用HTTP/HTTPSå?
- 客户端和æœåŠ¡å™¨éƒ½å¯ä»¥ç‹¬ç«‹å‘包åQŒå¶ž®”å‘生åšg˜qŸå¯ä»¥å®¹å¿ï¼ˆæ¯”如åQšåœ¨¾U¿çš„¾U¸ç‰Œæ¸¸æˆåQŒè®¸å¤šMMO¾cÈš„游æˆåQ‰ï¼Œé‚£ä¹ˆä½¿ç”¨TCP长连接å§
- 客户端和æœåŠ¡å™¨éƒ½å¯ä»¥ç‹¬ç«‹å‘包åQŒè€Œä¸”æ— æ³•å¿å—延迟åQˆæ¯”如：大多数的多ähFPS动作¾cÀL¸¸æˆQuakeã€CS½{‰ï¼Œä»¥åŠä¸€äº›MMO¾cÀL¸¸æˆï¼‰åQŒé‚£ä¹ˆä‹É用UDPå?
TCPä¼šè®¤å®šä¸¢åŒ…æ˜¯å› äؓ本地带宽ä¸èƒöå¯ÆD‡´åQˆæœ¬åœ°å¸¦å®½ä¸‘Ïx˜¯ä¸¢åŒ…çš„ä¸€éƒ¨åˆ†åŽŸå› åQ‰ï¼Œä½†å›½å†…ISPå¯èƒ½ä¼šåœ¨è‡ªèín机房¾|‘络拥挤时丢弃数æ®åŒ…åQŒè¿™æ—¶å€™å¯èƒ½éœ€è¦å¿«é€Ÿå‘包争抢通é“åQŒè€ŒéžTCP½H—å£æ”¶ç¾ƒåQŒUDP没有TCP½H—å£æ”¶ç¾ƒçš„负担，å¯ä»¥å¾ˆå®¹æ˜“åšåˆ°è¿™ä¸€ç‚V€?/p>
è¦æ±‚实时性放在第一ä½çš„FPS游æˆåQˆegåQšQuakeåQŒCSåQ‰ï¼Œòq¿åŸŸ¾|‘一般采用UDPåQŒå› å¯å®¹è®¸æœ‰ä¸¢å¤±æ•°æ®åŒ…å˜åœ¨ï¼ˆå¦å®¢æˆïL«¯è‹¥ç‰å¾…一ŒD‰|—¶é—´ä¸é—´ä¸¢åŒ…，å¯ä»¥é€šè¿‡æ’值ç‰æ‰‹æ®µå¿½ç•¥æŽ‰ï¼‰åQŒä¸€æ—¦æ£€‹¹‹åˆ°å¯ä»¥å¿«é€Ÿå‘é€ï¼Œå¦ä¸æ¶‰åŠåˆ°é‡å‘的时候UDP比TCPè¦å¿«ä¸€ç‚¹å˜›ã€‚但会在UDP应用层é¢æœ‰æ‰€å¢žåŠ å议控制åQŒæ¯”如ACK½{‰ã€?/p>
很多时候åè®®æ؜用，比如MMO客户端也讔R¦–å…ˆä‹É用HTTP去获å–上一‹Æ¡çš„更新内容åQ?é‡è¦ä¿¡æ¯å¦‚角色获得的物å“å’Œç»éªŒéœ€è¦é€šè¿‡TCPä¼ è¾“åQŒè€Œå‘¨å›´äh物的动å‘ã€NPC¿UÕdŠ¨ã€æŠ€èƒ½åŠ¨ç”ÀLŒ‡ä»¤ç‰åˆ™å¯ä»¥ä‹É用UDPä¼ è¾“åQŒè™½ç„¶å¯èƒ½ä¸¢åŒ…，但媄å“ä¸å¤§ã€?/p>
ž®ç»“
¾|‘游通过客户端预‹¹‹ã€æ’值和æœåŠ¡å™¨ç«¯å»¶è¿Ÿè¡¥è„“½{‰ï¼ŒåŒ–解/消除用户端网¾lœåšg˜qŸé€ æˆçš„圙åÑ€‚我们虽然å¯èƒ½æ²¡æœ‰æœºä¼šæŽ¥è§¦æ¸¸æˆå¼€å‘，å¦ä¹ 跨界的优良ç»éªŒå’Œå®žè·µåQŒè¯´ä¸å‡†ä¼šå¯¹å½“å‰å·¥ä½œæŸäº›ä¸šåŠ¡ç‚¹çš„处ç†æœ‰æ‰€å¯å‘å‘¢ã€?/p>
本集由韩国宇航局赞助æ’出åQšæˆ‘们è¦åŽ»è¿œæ–¹çœ‹çœ‹ï¼Œ˜q˜æœ‰ä»€ä¹ˆæ˜¯æˆ‘们的æ€å¯†è¾¾ã€?------ 《万万没惛_ˆ°ã€‹çŽ‹å¤§é”¤
¾U¿ä¸Šæƒ…况åQ?/p>
- ¾U¿ä¸ŠRedis集群åQŒå¤šä¸ªTwemproxy代ç†åQˆnutcrackeråQ‰ï¼ŒLVS DR路由å‡è¡¡è°ƒåº¦
- 客户端ä‹É用Jedisæ“作Redis集群åQŒä¸€ä¸ªç¨‹åºè¿›½E‹å®žä¾‹ä‹É用原å…?024个工作线½E‹å¤„ç†è¯·æ±‚，若干个进½E‹å®žä¾?
- 一天超˜q?2亿次è¯äh±‚åQŒç½‘¾lœä¸€èˆ¬æƒ…况下åQŒä¸€å¤©è¶…˜q‡ä¸Šä¸‡ä¸ª˜qžæŽ¥å¤ÞpÓ|异常
- ˜qç»´åŒå¦å‘ŠçŸ¥åQŒLVS压力较大
改进工作åQ?/p>
- 工作¾U¿ç¨‹ç”±åŽŸå…?024改用16ä¸?
- æ¯ä¸ª¾U¿ç¨‹æ¯æ¬¡æœ€å¤šæ“ä½?000个Redis命ä×o扚w‡æ交
实际效果åQ?/p>
- 一天ä¸åˆîC¸€äº¿æ¬¡çš„请求é‡
- LVS压力大å‡
- CPU压力é™ä½Žåˆ°åŽŸå…?/3以下
- å•ä¸ªè¯äh±‚æŠ½æ ·è°ƒç ”òq›_‡å‡å°‘1-90毫秒旉™—´åQˆå°¤å…¶æ˜¯è·¨æœºæˆ¿å¤„ç†ï¼‰
Redis支æŒæ‰šw‡æ交
原生支æŒæ‰šw‡æ“作方å¼
一般命令剾~€è‹¥æ·»åŠ 上må—符ä¸ÔŒ¼Œè¡¨ç¤ºæ”¯æŒå¤šä¸ªã€æ‰¹é‡å‘½ä»¤æ交了ã€?/p>
昑ּ�..
MSET key value [key value ...]
MSETNX key value [key value ...]
HMGET key field [field ...]
HMSET key field value [field value ...]
一般方å¼çš„...
HDEL key field [field ...]
SREM key member [member ...]
RPUSH key value [value ...]
......
更多åQŒè¯·å‚考：http://redis.cn/commands.html
pipeline½Ž¡é“æ–¹å¼
官方文档åQ?a >http://redis.io/topics/pipelining
- Redis Client把所有命令一èµäh‰“包å‘é€åˆ°Redis ServeråQŒç„¶åŽé˜»å¡žç‰å¾…处ç†ç»“æž?
- Redis Server必须在处ç†å®Œæ‰€æœ‰å‘½ä»¤å‰å…ˆç¼“å˜è“v所有命令的处熾l“æžœ
- 打包的命令越多，¾~“å˜æ¶ˆè€—内å˜ä¹Ÿ‘Šå¤š
- ä¸æ˜¯æ‰“包的命令越多越å¥?
- 实际环境需è¦æ ¹æ®å‘½ä»¤æ‰§è¡Œæ—¶é—´ç‰å„ç§å› ç´ é€‰æ‹©åˆåƈ命ä×o的个敎ͼŒä»¥åŠ‹¹‹è¯•æ•ˆæžœ½{?
Java队列支æŒ
一般业务ã€æŽ¥å…¥å‰ç«¯è¯·æ±‚釘q‡å¤§åQŒç”Ÿäº§è€…速度˜q‡å¿«åQŒè¿™æ—¶å€™ä‹É用队列暂时缓å˜ä¼šæ¯”较好一些，消费者直接直接从队列获å–ä»ÕdŠ¡åQŒé€šè¿‡é˜Ÿåˆ—让生产者和消费者进行分¼›»è¿™ä¹Ÿæ˜¯ä¸šç•Œæ™®é€šé‡‡ç”¨çš„æ–¹å¼ã€?/p>
监控队列
有的时候，若å¯ä»¥ç›‘控一下队列消è´ÒŽƒ…况，å¯ä»¥ç›‘控一下，ž®±å¾ˆç›´è§‚。åŒäº‹äؓ队列æ·ÕdŠ 了一个监控线½E‹ï¼Œæ¸…晰明了了解队列消费情况ã€?/p>
½Cø™Œƒ
½Cø™Œƒä½¿ç”¨äº†Redis PipelineåQŒçº¿½E‹æ± åQŒå‡†å¤‡æ•°æ®ï¼Œç”Ÿäñ”è€?消费者队列，队列监控½{‰ï¼Œæ¶ˆè´¹å®Œæ¯•åQŒç¨‹åºå…³é—ã€?/p>
/**
* 以下‹¹‹è¯•åœ¨Jedis 2.6下测试通过
*
* @author nieyong
*
*/
public class TestJedisPipeline {
private static final int NUM = 512;
private static final int MAX = 1000000; // 100W
private static JedisPool redisPool;
private static final ExecutorService pool = Executors.newCachedThreadPool();
protected static final BlockingQueue<String> queue = new ArrayBlockingQueue<String>(
MAX); // 100W
private static boolean finished = false;
static {
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxActive(64);
config.setMaxIdle(64);
try {
redisPool = new JedisPool(config, "192.168.192.8", 6379, 10000,
null, 0);
} catch (Exception e) {
System.err.println("Init msg redis factory error! " + e.toString());
}
}
public static void main(String[] args) throws InterruptedException {
System.out.println("prepare test data 100W");
prepareTestData();
System.out.println("prepare test data done!");
// 生äñ”者，模拟è¯äh±‚100W‹Æ?
pool.execute(new Runnable() {
@Override
public void run() {
for (int i = 0; i < MAX; i++) {
if (i % 3 == 0) {
queue.offer("del_key key_" + i);
} else {
queue.offer("get_key key_" + i);
}
}
}
});
// CPUæ ¸æ•°*2 个工作者线½E?
int threadNum = 2 * Runtime.getRuntime().availableProcessors();
for (int i = 0; i < threadNum; i++)
pool.execute(new ConsumerTask());
pool.execute(new MonitorTask());
Thread.sleep(10 * 1000);// 10sec
System.out.println("going to shutdown server ...");
setFinished(true);
pool.shutdown();
pool.awaitTermination(1, TimeUnit.MILLISECONDS);
System.out.println("colse!");
}
private static void prepareTestData() {
Jedis redis = redisPool.getResource();
Pipeline pipeline = redis.pipelined();
for (int i = 0; i < MAX; i++) {
pipeline.set("key_" + i, (i * 2 + 1) + "");
if (i % (NUM * 2) == 0) {
pipeline.sync();
}
}
pipeline.sync();
redisPool.returnResource(redis);
}
// queue monitoråQŒç”Ÿäº§è€?消费队列监控
private static class MonitorTask implements Runnable {
@Override
public void run() {
while (!Thread.interrupted() && !isFinished()) {
System.out.println("queue.size = " + queue.size());
try {
Thread.sleep(500); // 0.5 second
} catch (InterruptedException e) {
break;
}
}
}
}
// consumeråQŒæ¶ˆè´¹è€?
private static class ConsumerTask implements Runnable {
@Override
public void run() {
while (!Thread.interrupted() && !isFinished()) {
if (queue.isEmpty()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
}
continue;
}
List<String> tasks = new ArrayList<String>(NUM);
queue.drainTo(tasks, NUM);
if (tasks.isEmpty()) {
continue;
}
Jedis jedis = redisPool.getResource();
Pipeline pipeline = jedis.pipelined();
try {
List<Response<String>> resultList = new ArrayList<Response<String>>(
tasks.size());
List<String> waitDeleteList = new ArrayList<String>(
tasks.size());
for (String task : tasks) {
String key = task.split(" ")[1];
if (task.startsWith("get_key")) {
resultList.add(pipeline.get(key));
waitDeleteList.add(key);
} else if (task.startsWith("del_key")) {
pipeline.del(key);
}
}
pipeline.sync();
// 处熘q”回列表
for (int i = 0; i < resultList.size(); i++) {
resultList.get(i).get();
// handle value here ...
// System.out.println("get value " + value);
}
// è¯Õd–完毕åQŒç›´æŽ¥åˆ 除之
for (String key : waitDeleteList) {
pipeline.del(key);
}
pipeline.sync();
} catch (Exception e) {
redisPool.returnBrokenResource(jedis);
} finally {
redisPool.returnResource(jedis);
}
}
}
}
private static boolean isFinished(){
return finished;
}
private static void setFinished(boolean bool){
finished = bool;
}
}
代ç 作äØ“½Cø™Œƒã€‚è‹¥¾U¿ä¸Šåˆ™éœ€è¦å¤„ç†ä¸€äº›å¼‚常ç‰ã€?/p>
ž®ç»“
若能够批é‡è¯·æ±‚进行åˆòq¶æ“作，自然å¯ä»¥èŠ‚çœå¾ˆå¤šçš„网¾lœå¸¦å®½ã€CPU½{‰èµ„æºã€‚有¾cÖM¼¼é—®é¢˜çš„åŒå¦ï¼Œä¸å¦¨è€ƒè™‘一下ã€?/p>
新申è¯ïLš„æœåŠ¡å™¨å†…æ æ€Ø“2.6.32åQŒåŽŸå…ˆçš„TCP Serverç›´æŽ¥åœ¨æ–°å†…æ ¸çš„LinxuæœåŠ¡å™¨ä¸Š˜qè¡ŒåQŒè¿è¡Œdmesg命ä×oåQŒå¯ä»¥çœ‹åˆ°å¤§é‡çš„SYN floodingè¦å‘ŠåQ?/p>
possible SYN flooding on port 8080. Sending cookies.
原先çš?.6.18å†…æ ¸çš„å‚数在2.6.32å†…æ ¸ç‰ˆæœ¬æƒ…å†µä¸‹ï¼Œ½Ž€å•è°ƒæ•?net.ipv4.tcp_max_syn_backlog"å·²ç»æ²¡æœ‰ä½œç”¨ã€?/p>
怎么办，åªèƒ½å†æ¬¡é˜…读2.6.32æºç åQŒä»¥ä¸‹å³æ˜¯ã€?/p>
最åŽå°¾l“å¤„æœ‰ç›´æŽ¥ç»“è®ºï¼Œå¿ƒæ€¥çš„ä½ å¯ä»¥ç›´æŽ¥é˜…è¯ÀL€È»“好了ã€?/p>
linuxå†…æ ¸2.6.32有关backlog值分æž?/h3>
net/Socket.c:
SYSCALL_DEFINE2(listen, int, fd, int, backlog)
{
struct socket *sock;
int err, fput_needed;
int somaxconn;
sock = sockfd_lookup_light(fd, &err, &fput_needed);
if (sock) {
somaxconn = sock_net(sock->sk)->core.sysctl_somaxconn;
if ((unsigned)backlog > somaxconn)
backlog = somaxconn;
err = security_socket_listen(sock, backlog);
if (!err)
err = sock->ops->listen(sock, backlog);
fput_light(sock->file, fput_needed);
}
return err;
}
net/ipv4/Af_inet.c:
/*
* Move a socket into listening state.
*/
int inet_listen(struct socket *sock, int backlog)
{
struct sock *sk = sock->sk;
unsigned char old_state;
int err;
lock_sock(sk);
err = -EINVAL;
if (sock->state != SS_UNCONNECTED || sock->type != SOCK_STREAM)
goto out;
old_state = sk->sk_state;
if (!((1 << old_state) & (TCPF_CLOSE | TCPF_LISTEN)))
goto out;
/* Really, if the socket is already in listen state
* we can only allow the backlog to be adjusted.
*/
if (old_state != TCP_LISTEN) {
err = inet_csk_listen_start(sk, backlog);
if (err)
goto out;
}
sk->sk_max_ack_backlog = backlog;
err = 0;
out:
release_sock(sk);
return err;
}
inet_listen调用inet_csk_listen_start函数åQŒæ‰€ä¼ 入的backlogå‚数改头æ¢é¢åQŒå˜æˆäº†ä¸å¯ä¿®æ”¹çš„常é‡nr_table_entries了ã€?/p>
net/ipv4/Inet_connection_sock.c:
int inet_csk_listen_start(struct sock *sk, const int nr_table_entries)
{
struct inet_sock *inet = inet_sk(sk);
struct inet_connection_sock *icsk = inet_csk(sk);
int rc = reqsk_queue_alloc(&icsk->icsk_accept_queue, nr_table_entries);
if (rc != 0)
return rc;
sk->sk_max_ack_backlog = 0;
sk->sk_ack_backlog = 0;
inet_csk_delack_init(sk);
/* There is race window here: we announce ourselves listening,
* but this transition is still not validated by get_port().
* It is OK, because this socket enters to hash table only
* after validation is complete.
*/
sk->sk_state = TCP_LISTEN;
if (!sk->sk_prot->get_port(sk, inet->num)) {
inet->sport = htons(inet->num);
sk_dst_reset(sk);
sk->sk_prot->hash(sk);
return 0;
}
sk->sk_state = TCP_CLOSE;
__reqsk_queue_destroy(&icsk->icsk_accept_queue);
return -EADDRINUSE;
}
下é¢å¤„ç†çš„是TCP SYN_RECV状æ€çš„˜qžæŽ¥åQŒå¤„于æ¡æ‰‹é˜¶ŒDµï¼Œä¹Ÿå¯ä»¥è¯´æ˜¯åŠ˜qžæŽ¥æ—Óž¼Œ½{‰å¾…瀘qžæŽ¥æ–¹ç¬¬ä¸‰æ¬¡æ¡æ‰‹ã€?/p>
/*
* Maximum number of SYN_RECV sockets in queue per LISTEN socket.
* One SYN_RECV socket costs about 80bytes on a 32bit machine.
* It would be better to replace it with a global counter for all sockets
* but then some measure against one socket starving all other sockets
* would be needed.
*
* It was 128 by default. Experiments with real servers show, that
* it is absolutely not enough even at 100conn/sec. 256 cures most
* of problems. This value is adjusted to 128 for very small machines
* (<=32Mb of memory) and to 1024 on normal or better ones (>=256Mb).
* Note : Dont forget somaxconn that may limit backlog too.
*/
int reqsk_queue_alloc(struct request_sock_queue *queue,
unsigned int nr_table_entries)
{
size_t lopt_size = sizeof(struct listen_sock);
struct listen_sock *lopt;
nr_table_entries = min_t(u32, nr_table_entries, sysctl_max_syn_backlog);
nr_table_entries = max_t(u32, nr_table_entries, 8);
nr_table_entries = roundup_pow_of_two(nr_table_entries + 1);
lopt_size += nr_table_entries * sizeof(struct request_sock *);
if (lopt_size > PAGE_SIZE)
lopt = __vmalloc(lopt_size,
GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO,
PAGE_KERNEL);
else
lopt = kzalloc(lopt_size, GFP_KERNEL);
if (lopt == NULL)
return -ENOMEM;
for (lopt->max_qlen_log = 3;
(1 << lopt->max_qlen_log) < nr_table_entries;
lopt->max_qlen_log++);
get_random_bytes(&lopt->hash_rnd, sizeof(lopt->hash_rnd));
rwlock_init(&queue->syn_wait_lock);
queue->rskq_accept_head = NULL;
lopt->nr_table_entries = nr_table_entries;
write_lock_bh(&queue->syn_wait_lock);
queue->listen_opt = lopt;
write_unlock_bh(&queue->syn_wait_lock);
return 0;
}
关键è¦çœ‹nr_table_entrieså˜é‡åQŒåœ¨reqsk_queue_alloc函数ä¸nr_table_entrieså˜æˆäº†æ— ½W¦å·å˜é‡åQŒå¯ä¿®æ”¹çš„，å˜åŒ–å—é™ã€?/p>
æ¯”å¦‚å®žé™…å†…æ ¸å‚æ•°å€égØ“åQ?/p>
net.ipv4.tcp_max_syn_backlog = 65535
æ‰€ä¼ å…¥çš„backlogåQˆä¸å¤§äºŽnet.core.somaxconn = 65535åQ‰äØ“8102åQŒé‚£ä¹?/p>
// å–listen函数的backlogå’Œsysctl_max_syn_backlog最ž®å€û|¼Œ¾l“æžœä¸?102
nr_table_entries = min_t(u32, nr_table_entries, sysctl_max_syn_backlog);
// å–nr_table_entrieså’?˜q›è¡Œæ¯”较的最大å€û|¼Œ¾l“æžœä¸?102
nr_table_entries = max_t(u32, nr_table_entries, 8);
// å¯çœ‹å?nr_table_entries*2åQŒç»“æžœäØ“8102*2=16204
nr_table_entries = roundup_pow_of_two(nr_table_entries + 1);
计算¾l“æžœåQŒmax_qlen_log = 14
2.6.18å†…æ ¸ä¸max_qlen_log的计½Ž—æ–¹æ³?/h4>for (lopt->max_qlen_log = 6;
(1 << lopt->max_qlen_log) < sysctl_max_syn_backlog;
lopt->max_qlen_log++);
- 很显ç„Óž¼Œsysctl_max_syn_backlogå‚与了追Ž—，sysctl_max_syn_backlog值很大的è¯ä¼šå¯ÆD‡´max_qlen_log值相å¯ÒŽ¯”也很å¤?
- è‹¥sysctl_max_syn_backlog=65535åQŒé‚£ä¹ˆmax_qlen_log=16
- 2.6.18å†…æ ¸ä¸åŠ˜qžæŽ¥é•¿åº¦ä¸?^16=65536
for (lopt->max_qlen_log = 6;
(1 << lopt->max_qlen_log) < sysctl_max_syn_backlog;
lopt->max_qlen_log++);
作äØ“listen_sock¾l“构定义了需è¦å¤„ç†çš„处ç†åŠè¿žæŽ¥çš„é˜Ÿåˆ—å…ƒç´ ä¸ªæ•°ä¸ºnr_table_entriesåQŒæ¤ä¾‹ä¸ä¸?6204长度ã€?/p>
/** struct listen_sock - listen state
*
* @max_qlen_log - log_2 of maximal queued SYNs/REQUESTs
*/
struct listen_sock {
u8 max_qlen_log;
/* 3 bytes hole, try to use */
int qlen;
int qlen_young;
int clock_hand;
u32 hash_rnd;
u32 nr_table_entries;
struct request_sock *syn_table[0];
};
¾læ˜q°è€ŒçŸ¥åQ?^max_qlen_log = åŠè¿žæŽ¥é˜Ÿåˆ—长度qlen倹{€?/p>
å†å›žå¤´çœ‹çœ‹æŠ¥å‘ŠSYN flooding的函敎ͼš
net/ipv4/Tcp_ipv4.c
#ifdef CONFIG_SYN_COOKIES
static void syn_flood_warning(struct sk_buff *skb)
{
static unsigned long warntime;
if (time_after(jiffies, (warntime + HZ * 60))) {
warntime = jiffies;
printk(KERN_INFO
"possible SYN flooding on port %d. Sending cookies.\n",
ntohs(tcp_hdr(skb)->dest));
}
}
#endif
被调用的处，已精½Ž€è‹¥å¹²ä»£ç åQ?/p>
int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
{
......
#ifdef CONFIG_SYN_COOKIES
int want_cookie = 0;
#else
#define want_cookie 0 /* Argh, why doesn't gcc optimize this :( */
#endif
......
/* TW buckets are converted to open requests without
* limitations, they conserve resources and peer is
* evidently real one.
*/
// 判æ–åŠè¿žæŽ¥é˜Ÿåˆ—是å¦å·²æ»?&& !0
if (inet_csk_reqsk_queue_is_full(sk) && !isn) {
#ifdef CONFIG_SYN_COOKIES
if (sysctl_tcp_syncookies) {
want_cookie = 1;
} else
#endif
goto drop;
}
/* Accept backlog is full. If we have already queued enough
* of warm entries in syn queue, drop request. It is better than
* clogging syn queue with openreqs with exponentially increasing
* timeout.
*/
if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1)
goto drop;
req = inet_reqsk_alloc(&tcp_request_sock_ops);
if (!req)
goto drop;
......
if (!want_cookie)
TCP_ECN_create_request(req, tcp_hdr(skb));
if (want_cookie) {
#ifdef CONFIG_SYN_COOKIES
syn_flood_warning(skb);
req->cookie_ts = tmp_opt.tstamp_ok;
#endif
isn = cookie_v4_init_sequence(sk, skb, &req->mss);
} else if (!isn) {
......
}
......
}
判æ–åŠè¿žæŽ¥é˜Ÿåˆ—已满的函数很关键，å¯ä»¥çœ‹çœ‹˜q算法则åQ?/p>
include/net/Inet_connection_sock.h:
static inline int inet_csk_reqsk_queue_is_full(const struct sock *sk)
{
return reqsk_queue_is_full(&inet_csk(sk)->icsk_accept_queue);
}
include/net/Rquest_sock.h:
static inline int reqsk_queue_is_full(const struct request_sock_queue *queue)
{
// å‘峿UÖM½max_qlen_log个å•ä½?
return queue->listen_opt->qlen >> queue->listen_opt->max_qlen_log;
}
˜q”回1åQŒè‡ªç„¶è¡¨½CºåŠ˜qžæŽ¥é˜Ÿåˆ—已满ã€?/p>
以上仅仅是分æžäº†åŠè¿žæŽ¥é˜Ÿåˆ—已满的判æ–æ¡äšgåQŒæ€ÖM¹‹åº”用½E‹åºæ‰€ä¼ 入的backlog很关键，如值太ž®ï¼Œå¾ˆå®¹æ˜“å¾—åˆ?.
è‹?somaxconn = 128åQŒsysctl_max_syn_backlog = 4096åQŒbacklog = 511 则最¾l?nr_table_entries = 256åQŒmax_qlen_log = 8。那么超˜q?56个劘qžæŽ¥çš„队列，257 >> 8 = 1åQŒé˜Ÿåˆ—已满ã€?/p>
如何讄¡½®backlogåQŒè¿˜å¾—需è¦ç»“åˆå…·ä½“应用程åºï¼Œéœ€è¦äؓ其调用listenæ–ÒŽ³•èµ‹å€¹{€?/p>
Netty backlog处ç†
Tcp Server使用Netty 3.7 版本åQŒç‰ˆæœ¬è¾ƒä½Žï¼Œåœ¨å¤„ç†backlogåQŒè‹¥æˆ‘们ä¸æ‰‹åŠ¨æŒ‡å®šbacklogå€û|¼ŒJDK 1.6默认ä¸?0ã€?/p>
有è¯å¦‚下åQ?java.net.ServerSocket:
public void bind(SocketAddress endpoint, int backlog) throws IOException {
if (isClosed())
throw new SocketException("Socket is closed");
if (!oldImpl && isBound())
throw new SocketException("Already bound");
if (endpoint == null)
endpoint = new InetSocketAddress(0);
if (!(endpoint instanceof InetSocketAddress))
throw new IllegalArgumentException("Unsupported address type");
InetSocketAddress epoint = (InetSocketAddress) endpoint;
if (epoint.isUnresolved())
throw new SocketException("Unresolved address");
if (backlog < 1)
backlog = 50;
try {
SecurityManager security = System.getSecurityManager();
if (security != null)
security.checkListen(epoint.getPort());
getImpl().bind(epoint.getAddress(), epoint.getPort());
getImpl().listen(backlog);
bound = true;
} catch(SecurityException e) {
bound = false;
throw e;
} catch(IOException e) {
bound = false;
throw e;
}
}
nettyä¸ï¼Œå¤„ç†backlog的地方：
org/jboss/netty/channel/socket/DefaultServerSocketChannelConfig.java:
@Override
public boolean setOption(String key, Object value) {
if (super.setOption(key, value)) {
return true;
}
if ("receiveBufferSize".equals(key)) {
setReceiveBufferSize(ConversionUtil.toInt(value));
} else if ("reuseAddress".equals(key)) {
setReuseAddress(ConversionUtil.toBoolean(value));
} else if ("backlog".equals(key)) {
setBacklog(ConversionUtil.toInt(value));
} else {
return false;
}
return true;
}
既然需è¦æˆ‘们手动指定backlogå€û|¼Œé‚£ä¹ˆå¯ä»¥˜q™æ ·åšï¼š
bootstrap.setOption("backlog", 8102); // 讄¡½®å¤§ä¸€äº›æ²¡æœ‰å…³¾p»ï¼Œ¾pÈ»Ÿå†…æ ¸ä¼šè‡ªåŠ¨ä¸Žnet.core.somaxconn相比较，å–最低å€?
相对比Netty 4.0åQŒæœ‰äº›ä¸æ™ø™ƒ½åQŒå¯å‚考：http://www.aygfsteel.com/yongboy/archive/2014/07/30/416373.html
ž®ç»“
在linuxå†…æ ¸2.6.32åQŒè‹¥åœ¨æ²¡æœ‰éå—到SYN floodingæ”Õd‡»çš„情况下åQŒå¯ä»¥é€‚当调整åQ?/p>
sysctl -w net.core.somaxconn=32768
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -p
å¦åƒä¸‡åˆ«å¿˜è®°ä¿®æ”¹TCP Serverçš„listen接å£æ‰€ä¼ 入的backlogå€û|¼Œè‹¥ä¸è®„¡½®æˆ–者过ž®ï¼Œéƒ½ä¼šæœ‰å¯èƒ½é€ æˆSYN floodingçš„è¦å‘Šä¿¡æ¯ã€‚开始ä¸å¦¨è®¾¾|®æˆ1024åQŒç„¶åŽè§‚察一ŒD‰|—¶é—´æ ¹æ®å®žé™…情况需è¦å†æ…¢æ…¢å¾€ä¸Šè°ƒã€?/p>
æ— è®ºä½ å¦‚ä½•è®¾¾|®ï¼Œæœ€¾lˆbacklogå€ÆDŒƒå›´äØ“åQ?/p>
backlog <= net.core.somaxconn
åŠè¿žæŽ¥é˜Ÿåˆ—长度约为：
åŠè¿žæŽ¥é˜Ÿåˆ—é•¿åº?≈ 2 * min(backlog, net.ipv4.tcpmax_syn_backlog)
å¦ï¼Œè‹¥å‡ºçŽ°SYN floodingæ—Óž¼Œæ¤æ—¶TCP SYN_RECVæ•°é‡è¡¨ç¤ºåŠè¿žæŽ¥é˜Ÿåˆ—å·²¾l满åQŒå¯ä»¥æŸ¥çœ‹ä¸€ä¸‹ï¼š
ss -ant | awk 'NR>1 {++s[$1]} END {for(k in s) print k,s[k]}'
æ„Ÿè°¢˜q维书夞®ä¼™æ供的比较好用查看命令ã€?/p>
最˜q‘线上æœåŠ¡å™¨åQŒdmesg会给å‡ÞZ¸€äº›è¦å‘Šä¿¡æ¯ï¼š
possible SYN flooding on port 8080. Sending cookies.
åˆçœ‹ä»¥äؓ是å—到DOSæ‹’ç»æ€§æ”»å‡»ï¼Œä½†ä»”¾l†ä¸€åˆ†æžåQŒä¸€å¤©é‡ä¹Ÿå°±æ˜¯åœ¨1000多æ¡å·¦å³åQŒæ„Ÿè§‰ä¸Šå±žäºŽæ£å¸¸å¯æŽ¥å—范围ã€?/p>
下é¢éœ€è¦æ‰¾å‡ºæ¥æºï¼Œä»¥åŠåŽŸå› åQŒä»¥ä¸‹å†…容基于Linux 2.6.18å†…æ ¸ã€?/p>
è¦å‘Šè¾“出æºå¤´
net/ipv4/Tcp_ipv4.c:
#ifdef CONFIG_SYN_COOKIES
static void syn_flood_warning(struct sk_buff *skb)
{
static unsigned long warntime; // ½W¬ä¸€‹Æ¡åŠ è½½åˆå§‹åŒ–为零åQŒåŽ¾lwarntime = jiffies
if (time_after(jiffies, (warntime + HZ * 60))) {
warntime = jiffies;
printk(KERN_INFO
"possible SYN flooding on port %d. Sending cookies.\n",
ntohs(skb->h.th->dest));
}
}
#endif
很显ç„Óž¼ŒCONFIG_SYN_COOKIES在Linux¾pÈ»Ÿ¾~–译æ—Óž¼Œå·²è¢«è®„¡½®trueã€?/p>
time_afterå®å®šä¹‰ï¼š
#define time_after(a,b) \
(typecheck(unsigned long, a) && \
typecheck(unsigned long, b) && \
((long)(b) - (long)(a) < 0))
ä¸¤ä¸ªæ— ç¬¦åïLš„æ—‰™—´æ¯”较åQŒç¡®å®šå…ˆåŽé¡ºåºã€?/p>
jiffies真èínåQ?/p>
# define jiffies raid6_jiffies()
#define HZ 1000
......
static inline uint32_t raid6_jiffies(void)
{
struct timeval tv;
gettimeofday(&tv, NULL);
return tv.tv_sec*1000 + tv.tv_usec/1000; // ¿U?1000 + 微秒/1000
}
回过头æ¥åQŒå†çœ‹çœ‹syn_flood_warning函数åQ?/p>
static void syn_flood_warning(struct sk_buff *skb)
{
static unsigned long warntime; // ½W¬ä¸€‹Æ¡åŠ è½½åˆå§‹åŒ–为零åQŒåŽ¾lwarntime = jiffies
if (time_after(jiffies, (warntime + HZ * 60))) {
warntime = jiffies;
printk(KERN_INFO
"possible SYN flooding on port %d. Sending cookies.\n",
ntohs(skb->h.th->dest));
}
}
warntime为static¾cÕdž‹åQŒç¬¬ä¸€‹Æ¡è°ƒç”¨æ—¶è¢«åˆå§‹åŒ–为零åQŒä¸‹‹Æ¡è°ƒç”¨å°±æ˜¯ä¸Š‹Æ¡çš„jiffieså€égº†åQŒå‰åŽé—´éš”å€ÆD¶…˜q‡HZ*60ž®×ƒ¸ä¼šè¾“å‡ø™¦å‘Šä¿¡æ¯äº†ã€?/p>
有关time_afterå’ŒjiffiesåQŒåˆ†äº«å‡ ½‹‡æ–‡ç« ：
http://wenku.baidu.com/view/c75658d480eb6294dd886c4e.html
è¦å‘Šè¾“出需è¦æ»¡‘³çš„æ¡äšg
注æ„观察want_cookie=1时的æ¡äšgã€?/p>
net/ipv4/Tcp_ipv4.c:
int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
{
struct inet_request_sock *ireq;
struct tcp_options_received tmp_opt;
struct request_sock *req;
__u32 saddr = skb->nh.iph->saddr;
__u32 daddr = skb->nh.iph->daddr;
__u32 isn = TCP_SKB_CB(skb)->when; // when在tcp_v4_rcv()ä¸ä¼šè¢«ç½®ä¸?
struct dst_entry *dst = NULL;
#ifdef CONFIG_SYN_COOKIES
int want_cookie = 0;
#else
#define want_cookie 0 /* Argh, why doesn't gcc optimize this :( */
#endif
/* Never answer to SYNs send to broadcast or multicast */
if (((struct rtable *)skb->dst)->rt_flags &
(RTCF_BROADCAST | RTCF_MULTICAST))
goto drop;
/* TW buckets are converted to open requests without
* limitations, they conserve resources and peer is
* evidently real one.
*/
// if(判æ–åŠè¿žæŽ¥é˜Ÿåˆ—å·²æ»?&& !0)
if (inet_csk_reqsk_queue_is_full(sk) && !isn) {
#ifdef CONFIG_SYN_COOKIES
if (sysctl_tcp_syncookies) { // net.ipv4.tcp_syncookies = 1
want_cookie = 1;
} else
#endif
goto drop;
}
/* Accept backlog is full. If we have already queued enough
* of warm entries in syn queue, drop request. It is better than
* clogging syn queue with openreqs with exponentially increasing
* timeout.
*/
// if(˜qžæŽ¥é˜Ÿåˆ—是å¦å·²æ»¡ && åŠè¿žæŽ¥é˜Ÿåˆ—丘q˜æœ‰æœªé‡ä¼ ACKåŠè¿žæŽ¥æ•°å?> 1)
if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1)
goto drop;
......
tcp_openreq_init(req, &tmp_opt, skb);
ireq = inet_rsk(req);
ireq->loc_addr = daddr;
ireq->rmt_addr = saddr;
ireq->opt = tcp_v4_save_options(sk, skb);
if (!want_cookie)
TCP_ECN_create_request(req, skb->h.th);
if (want_cookie) { // åŠè¿žæŽ¥é˜Ÿåˆ—已满会触å‘
#ifdef CONFIG_SYN_COOKIES
syn_flood_warning(skb);
#endif
isn = cookie_v4_init_sequence(sk, skb, &req->mss);
} else if (!isn) {
......
}
/* Kill the following clause, if you dislike this way. */
// net.ipv4.tcp_syncookies未设¾|®æƒ…况下åQŒsysctl_max_syn_backlogå‘生的作ç”?
else if (!sysctl_tcp_syncookies &&
(sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
(sysctl_max_syn_backlog >> 2)) &&
(!peer || !peer->tcp_ts_stamp) &&
(!dst || !dst_metric(dst, RTAX_RTT))) {
/* Without syncookies last quarter of
* backlog is filled with destinations,
* proven to be alive.
* It means that we continue to communicate
* to destinations, already remembered
* to the moment of synflood.
*/
LIMIT_NETDEBUG(KERN_DEBUG "TCP: drop open "
"request from %u.%u.%u.%u/%u\n",
NIPQUAD(saddr),
ntohs(skb->h.th->source));
dst_release(dst);
goto drop_and_free;
}
isn = tcp_v4_init_sequence(sk, skb);
}
tcp_rsk(req)->snt_isn = isn;
if (tcp_v4_send_synack(sk, req, dst))
goto drop_and_free;
if (want_cookie) {
reqsk_free(req);
} else {
inet_csk_reqsk_queue_hash_add(sk, req, TCP_TIMEOUT_INIT);
}
return 0;
drop_and_free:
reqsk_free(req);
drop:
return 0;
}
ž®ç»“
æ€ÖM¹‹åQŒå¦‚¾pÈ»Ÿå‡ºçŽ°åQ?/p>
possible SYN flooding on port 8080. Sending cookies.
è‹¥é‡ä¸å¤§åQŒæ˜¯åœ¨æé†’ä½ éœ€è¦å…³å¿ƒä¸€ä¸‹sysctl_max_syn_backlog其值是å¦è¿‡ä½?
sysctl -a | grep 'max_syn_backlog'
ä¸å¦¨æˆå€å¢žåŠ 一ä¸?/p>
sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sysctl -p
若进½E‹æ— 法åšåˆ°é‡æ–°åŠ 载，那就需è¦é‡å¯åº”ç”¨ï¼Œä»¥é€‚åº”æ–°çš„å†…æ ¸å‚数。进而挾l观察一ŒD‰|—¶é—´ã€?/p>
貌似tcp_max_syn_backlogå‚数其完整作用域˜q˜æ²¡æœ‰ç†è§£å®Œæ•ß_¼Œä¸‹æ¬¡æœ‰æ—¶é—´å†å†™å§ã€?/p>
有些东西æ€ÀL˜¯å¾ˆå®¹æ˜“é—忘，一时记得了åQŒè¿‡ä¸¤å¤©ž®ÞqœŸæ£è¿˜¾l™å‘¨å…¬äº†ã€‚零零碎¼„Žçš„ä¸å¦‚一òq¶è®°ä¸‹æ¥åQŒä»¥åŽå¯ä»¥ç›´æŽ¥æ‹¿˜q‡æ¥æŸ¥è¯¢å›_¯ã€?/p>
以下内容åŸÞZºŽLinux 2.6.18å†…æ ¸ã€?/p>
一。listenæ–ÒŽ³•ä¼ 入的backlogå‚æ•°åQŒnet.core.somaxconn
˜q™ä¸ªå‚数具体æ„义åQŒå…ˆçœ‹çœ‹Linux Socketçš„listen解释
man listen
#include <sys/socket.h>
int listen(int sockfd, int backlog);
int¾cÕdž‹çš„backlogå‚æ•°åQŒlistenæ–ÒŽ³•çš„backlogæ„义为，已ç»å®Œæˆä¸‰æ¬¡æ¡æ‰‹ã€å·²¾læˆåŠŸå¾ç«‹è¿žæŽ¥çš„套接å—å°†è¦è¿›å…¥é˜Ÿåˆ—的长度ã€?/p>
一般我们自己定义设定backlogå€û|¼Œè‹¥æˆ‘们设¾|®çš„backlog值大于net.core.somaxconnå€û|¼Œž®†è¢«¾|®äØ“net.core.somaxconn值大ž®ã€‚è‹¥ä¸æƒ³ç›´æŽ¥¼‹¬æ€§æŒ‡å®šï¼Œè·Ÿéš¾pÈ»Ÿè®‘Ö®šåQŒåˆ™éœ€è¦è¯»å?proc/sys/net/core/somaxconnã€?/p>
net\Socket.c :
/*
* Perform a listen. Basically, we allow the protocol to do anything
* necessary for a listen, and if that works, we mark the socket as
* ready for listening.
*/
int sysctl_somaxconn = SOMAXCONN;
asmlinkage long sys_listen(int fd, int backlog)
{
struct socket *sock;
int err, fput_needed;
if ((sock = sockfd_lookup_light(fd, &err, &fput_needed)) != NULL) {
if ((unsigned) backlog > sysctl_somaxconn)
backlog = sysctl_somaxconn;
err = security_socket_listen(sock, backlog);
if (!err)
err = sock->ops->listen(sock, backlog);
fput_light(sock->file, fput_needed);
}
return err;
}
比如¾l常使用的netty(4.0)框架åQŒåœ¨Linux下å¯åŠ¨æ—¶åQŒä¼šç›´æŽ¥è¯Õd–/proc/sys/net/core/somaxconn值然åŽä½œä¸ºlistençš„backlogå‚æ•°˜q›è¡Œè°ƒç”¨Linux¾pÈ»Ÿçš„listen˜q›è¡Œåˆå§‹åŒ–ç‰ã€?/p>
int somaxconn = 3072;
BufferedReader in = null;
try {
in = new BufferedReader(new FileReader("/proc/sys/net/core/somaxconn"));
somaxconn = Integer.parseInt(in.readLine());
logger.debug("/proc/sys/net/core/somaxconn: {}", somaxconn);
} catch (Exception e) {
// Failed to get SOMAXCONN
} finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
// Ignored.
}
}
}
SOMAXCONN = somaxconn;
......
private volatile int backlog = NetUtil.SOMAXCONN;
一般ç¨å¾®å¢žå¤§net.core.somaxconn值就昑־—很有必è¦ã€?/p>
讄¡½®å…¶å€¼æ–¹æ³•ï¼š
sysctl -w net.core.somaxconn=65535
较大内å˜çš„LinuxåQ?5535æ•°å€ég¸€èˆ¬å°±å¯ä»¥äº†ã€?/p>
若让其生效，sysctl -p å›_¯åQŒç„¶åŽé‡å¯ä½ çš„Server应用å›_¯ã€?/p>
二。网å¡è®¾å¤‡å°†è¯äh±‚攑օ¥é˜Ÿåˆ—的长度，netdev_max_backlog
å†…æ ¸ä»£ç ä¸sysctl.cæ–‡äšg解释åQ?/p>
number of unprocessed input packets before kernel starts dropping them, default 300
我所ç†è§£çš„å«ä¹‰ï¼Œæ¯ä¸ª¾|‘络接å£æŽ¥æ”¶æ•°æ®åŒ…çš„é€ŸçŽ‡æ¯”å†…æ ¸å¤„ç†è¿™äº›åŒ…的速率快时åQŒå…è®”R€åˆ°é˜Ÿåˆ—的最大数目，一旦超˜q‡å°†è¢«ä¸¢å¼ƒã€?/p>
所起作用处åQŒnet/core/Dev.cåQ?/p>
int netif_rx(struct sk_buff *skb)
{
struct softnet_data *queue;
unsigned long flags;
/* if netpoll wants it, pretend we never saw it */
if (netpoll_rx(skb))
return NET_RX_DROP;
if (!skb->tstamp.off_sec)
net_timestamp(skb);
/*
* The code is rearranged so that the path is the most
* short when CPU is congested, but is still operating.
*/
local_irq_save(flags);
queue = &__get_cpu_var(softnet_data);
__get_cpu_var(netdev_rx_stat).total++;
if (queue->input_pkt_queue.qlen <= netdev_max_backlog) {
if (queue->input_pkt_queue.qlen) {
enqueue:
dev_hold(skb->dev);
__skb_queue_tail(&queue->input_pkt_queue, skb);
local_irq_restore(flags);
return NET_RX_SUCCESS;
}
netif_rx_schedule(&queue->backlog_dev);
goto enqueue;
}
__get_cpu_var(netdev_rx_stat).dropped++;
local_irq_restore(flags);
kfree_skb(skb);
return NET_RX_DROP;
}
以上代ç 看一下，大概会明白netdev_max_backlog会在什么时候è“v作用ã€?/p>