??xml version="1.0" encoding="utf-8" standalone="yes"?> 在几个月前改造dubboӞnetty4已经E_很久?jin),一时手痒,按照netty3-rpc的源码克隆了(jin)一套netty4Q在修正?jin)大量的包、类型不同之后,基本保持?jin)netty3的风|q发量小或者数据包很小Ӟ一切都很ok, 在进行大q发试Ӟl果和netty3完全不同Q基本用惨不忍睹来Ş宏V由于当时急于开发php客户端,把netty4-rpc当做一个失败的lg存档hQ?nbsp;前几天php-dubbo开发基本完成之后,q回q来思考netty4-rpc的问题,l过仔细分析数据包的解析q程Q单步跟t源?/p> NettyCodecAdapter, TelnetCodec, ExchangeCoedecQ发现ByteBuf的缓冲区?024,当数据超q?024Ӟ?x)调用多ơDecoder.messageReceived函数Q第一ơ分析dubbo的协议头Ӟ是正的Q第二次之后数据错误了(jin)Q然后dubbo内部~冲区的数据来长Q但是仍然分析不C个完整的dubbo数据?/p> 因此ȝnetty4的源码,发现AbstractNioByteChannel中有|络数据接收的代码时q么处理ByteBuf?br style="padding: 0px; margin: 0px;" /> ByteBuf byteBuf = null; 看见没,内核是需要ByteBuf.release的,l箋(hu)通过byteBuf的一个实现PooledByteBuf分析源码Q原来是实现?jin)一个基于简单计数应用计数的循环使用的缓冲区Q一旦计数变?sh)?Q该~冲归还到netty4内核Q被后面的数据读取线E重C?/p> 而我们InternalDecoder的代码ؓ(f) message = com.alibaba.dubbo.remoting.buffer.ChannelBuffers.wrappedBuffer( 直接引用?jin)ByteBuf.toByteBufferQl查看源码UnpooledHeapByteBuf, 其toByteBuffer实际是对内部数据?/p> 一个nio装而已Q因此,使用上述函数ӞDdubbo的decode保存?sh)(jin)一个某一个ByteBuffer的内部数据,但是虽有?/p> buffer被归q到netty4~冲Z被@环引用,下一ơ可能被其他dU程重新改写数据Q因此,高ƈ发下当缓冲区被重复用时Qbytebuf由于计数问题(sh)断被使用Q而解码器中缺d{待?/p> 解决Ҏ(gu) 1.通过byteBuf的retain和release函数保证计数的有效性,通过E序例外或者缓冲区被用完成时候归qByteBuf到netty4内核 2.拯数据到dubbo的缓冲区?/p> 思考:(x) netty3 是否也有该问题呢Q?Q?/p> 同事刘阳使用dubbo服务器中配置mina作ؓ(f)|络传输层,发现大ƈ发情况下Q解码发生如下异?/p> 014-12-01 18:00:44,652 [DubboServerHandler-10.1.19.13:20880-thread-164] WARN alibaba.dubbo.remoting.exchange.codec.ExchangeCodec (ExchangeCodec.java:596) - [DUBBO] Fail to encode response: Response [id=8119, version=2.0.0, status=40, event=false, error=Fail to decode request due to: RpcInvocation [methodName=null, parameterTypes=null, arguments=null, attachments={input=242}, headers=null], result=null], send bad_response info instead, cause: null, dubbo version: 2.5.5, current host: 127.0.0.1 l过Ҏ(gu)netty3和netty4作ؓ(f)传输层,却都没有发现cM的问题?/p> 首先排除不是mina本n的问题,mina也没有爆出有q个问题Q初步判断dubbo在用mina时机制有问题 l过Ҏ(gu)发现 1.netty是ؓ(f)每一个channel分配?jin)一个NettyCodecAdapter, mina实在服务器监听前配|了(jin)MinaCodecAdapter 2.也就是说Qnetty的每一个独立的通道的Codec(encoder/decoderQ是通道安全?/p> 3.mina的所有通道是共享相同的codec(encoder/decoder)的,因此Q解码器中的实例数据旉channel安全?br style="padding: 0px; margin: 0px;" /> 因此解码器中与netty相同的解码器的缓冲数据算法在q发情况下将?x)生数据覆盖问题?/p> 4.解决Ҏ(gu) 1.配置a(chn)cceptor的监听器 codecAdapter = new MinaCodecAdapter(getCodec(), getUrl(), this); acceptor.addListener(new IoServiceListener(){ 2.监听session的create和destroy事gQ传递到decoder中,decoder中,通过session和buffer的键值对保存对不同通道的数据的~存Q?/p> private Map<IoSession, ChannelBuffer> buffers = new ConcurrentHashMap<IoSession, ChannelBuffer>(); 3.解码旉过session获得当前channel的数?/p> ChannelBuffer buffer = buffers.get(session); l过试Q问题得以解?/p>
int messages = 0;
boolean close = false;
try {
int totalReadAmount = 0;
boolean readPendingReset = false;
do {
byteBuf = allocHandle.allocate(allocator);
int writable = byteBuf.writableBytes();
int localReadAmount = doReadBytes(byteBuf);
if (localReadAmount <= 0) {
// not was read release the buffer
byteBuf.release();
close = localReadAmount < 0;
break;
}
if (!readPendingReset) {
readPendingReset = true;
setReadPending(false);
}
pipeline.fireChannelRead(byteBuf);
byteBuf = null;
input.toByteBuffer());
java.lang.NullPointerException
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.encodeResponseData(DubboCodec.java:301)
at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encodeResponse(ExchangeCodec.java:560)
at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:104)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:39)
at com.alibaba.dubbo.remoting.transport.mina.MinaCodecAdapter$InternalEncoder.encode(MinaCodecAdapter.java:79)
at org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:214)
at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:361)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1300(AbstractIoFilterChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:659)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite(AbstractIoFilterChain.java:587)
at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:361)
at org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(AbstractIoFilterChain.java:355)
at org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSessionImpl.java:166)
at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:177)
at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:168)
at com.alibaba.dubbo.remoting.transport.mina.MinaChannel.send(MinaChannel.java:95)
at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
at
acceptor.getFilterChain().addLast("codec", new ProtocolCodecFilter(codecAdapter));
@Override
public void serviceActivated(IoService service,
SocketAddress serviceAddress, IoHandler handler,
IoServiceConfig config) {
}
@Override
public void serviceDeactivated(IoService service,
SocketAddress serviceAddress, IoHandler handler,
IoServiceConfig config) {
}
@Override
public void sessionCreated(IoSession session) {
codecAdapter.sessionCreated(session);
}
@Override
public void sessionDestroyed(IoSession session) {
codecAdapter.sessionDestroyed(session);
}
});
// ChannelBuffers.EMPTY_BUFFER;
public void sessionCreated(IoSession session) {
buffers.put(session, ChannelBuffers.EMPTY_BUFFER);
}
public void sessionDestroyed(IoSession session) {
buffers.remove(session);
}
if(buffer == null) return;
1.字符串的格式定义?nbsp;字母S或者R + 两个字节的数据长度(MSBQ? utf8格式的字节数l,S表示最后一个块Q上面那个长度是unicode
字符串的长度Q不是自己数l的长度Q即"中国"q个词,长度?,utf8表示的字节数l确?个字?br style="padding: 0px; margin: 0px;" />
2.我的php客户端是用c混合c++~写的php扩展Q输出字W串旉先调用libmbfl库计unicode字符串的长度Q然后输出utf8数据,因ؓ(f)在php中,我们默认采用utf8格式Q字W串zval已经是utf8格式?jin),q且附带一个utf8长度的整敎ͼ
3.d应答分析字符串时Q根据上面的长度q不知道该分配多内存来接受整个字符Ԍ因ؓ(f)长度和utf8的字节长度根本没有关p,q有多个节时防止utf8字母被分配到多个不同的chunk上面
4.因此Q我们重新定义了(jin)一个字W串数据协议Q?E +4字节UTF8字节长度QMSBQ? utf8表示一个完整的字符Ԍ4字节Ӟ表示的数据有2^31-1个utf8字节Q可以表C好几百兆的汉字Q够用了(jin)Q这h们可以用c语言的memcpy函数q行快速数据拷贝和数据~冲区分配了(jin)
5.l过试Q大量字W串的数据传输性能能提?%
from:http://pfmiles.github.io/blog/recently-hessian-deserialize-problem-and-thread-pool-executor-experience/
最q工作中遇到一个诡异的问题Q别E调用我们的pȝ暴露的服务,同步调用Q底层用hessian协议做序列化Q?br />调用方系l报I指针,反序列化p|Q?/p>
2013-04-18 16:52:10,308 [AvatarRuleChargeService.java:74] [com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService] ERROR com.alibaba.itbu.billing.biz.adaptor.crm.ChargeProxy :: avatar charge sys error com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method match in the service com.alibaba.china.ruleservice.RuleService. Tried 3 times of the providers [172.22.6.83:20980, 172.22.6.80:20980, 172.22.9.76:20980] (3/3) from the registry dubbo-reg1.hst.xyi.cn.alidc.net:9090 on the consumer 172.30.118.26 using the dubbo version 2.4.9. Last error is: Failed to invoke remote method: match, provider: dubbo://172.22.6.83:20980/com.alibaba.china.ruleservice.RuleService?anyhost=true&application=billing&check=false&default.reference.filter=dragoon&dubbo=2.4.9&interface=com.alibaba.china.ruleservice.RuleService&methods=match&pid=18616&revision=1.0-SNAPSHOT&side=consumer&timeout=5000×tamp=1366275108588&version=1.0.0, cause: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275) at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155) at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99) at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83) at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271) ... 28 more Caused by: java.lang.NullPointerException at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.<init>(RuleServiceImpl.java:163) ... 33 more at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101) at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:226) at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72) at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52) at com.alibaba.dubbo.common.bytecode.proxy1.match(proxy1.java) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.alibaba.itbu.billing.framework.aop.OpenApiLogAspect.logExecuteTime(OpenApiLogAspect.java:38) at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy107.match(Unknown Source) at com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService.chargeByFactor(AvatarRuleChargeService.java:72) at com.alibaba.itbu.billing.biz.charge.times.RuleChargeByTimesProcessor.getChargeResult(RuleChargeByTimesProcessor.java:62) at com.alibaba.itbu.billing.biz.charge.times.ChargeByTimesProcessor.charge(ChargeByTimesProcessor.java:117) at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask.charge(ChargeByIncInstantTimesTask.java:174) at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask$1.run(ChargeByIncInstantTimesTask.java:109) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: com.alibaba.dubbo.remoting.RemotingException: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275) at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155) at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990) at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94) at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99) at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83) at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128) at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271) ... 28 more Caused by: java.lang.NullPointerException at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.<init>(RuleServiceImpl.java:163) ... 33 more at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:190) at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:110) at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84) at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96) at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144) at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74) at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke0(DragoonFilter.java:82) at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke(DragoonFilter.java:34) at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53) at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48) at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77) ... 32 more
看到q个日志W一反映是觉?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">RuleServiceImpl.java:163数据错误Q抛I指针,但检查那块代码发现那个地Ҏ(gu)本不可能抛空指针 —— 所有用到的变量都是new
出来的;
而且Q依据调用方提供的错误日志的抛出旉Q我在被调用方系l的所有机器的日志里查找了(jin)一遍,没有发现对应的服务端日志Q照理说服务端抛I指针,服务端也?x)有对应日志Q但没有MU烦(ch)…
因此只好d?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">JavaDeserializer.java:275作检查,发现有这么一D:(x)
public JavaDeserializer(Class cl) { _type = cl; _fieldMap = getFieldMap(cl); _readResolve = getReadResolve(cl); if (_readResolve != null) { _readResolve.setAccessible(true); } Constructor []constructors = cl.getDeclaredConstructors(); long bestCost = Long.MAX_VALUE; for (int i = 0; i < constructors.length; i++) { Class []param = constructors[i].getParameterTypes(); long cost = 0; for (int j = 0; j < param.length; j++) { cost = 4 * cost; if (Object.class.equals(param[j])) cost += 1; else if (String.class.equals(param[j])) cost += 2; else if (int.class.equals(param[j])) cost += 3; else if (long.class.equals(param[j])) cost += 4; else if (param[j].isPrimitive()) cost += 5; else cost += 6; } if (cost < 0 || cost > (1 << 48)) cost = 1 << 48; cost += param.length << 48; if (cost < bestCost) { _constructor = constructors[i]; bestCost = cost; } } if (_constructor != null) { _constructor.setAccessible(true); Class []params = _constructor.getParameterTypes(); _constructorArgs = new Object[params.length]; for (int i = 0; i < params.length; i++) { _constructorArgs[i] = getParamArg(params[i]); } } }
看完q段后,再结合远E调用的q回l果cd恍然大?zhn)Q?br />上面q段代码Q是hessian在反序列化的时候,用于在被反序列化的类里面找一?#8220;得分最?#8221;的构造函敎ͼ反序列化时会(x)加以调用;
构造函数的“得分”规则大致是:(x)参数少得分低Q参C数相同时Q参数类型越接近JDK内置cd分越?br />而我们的调用q回的类只有一个构造函敎ͼ当然只有q个构造函C(x)被选中
但是Qhessian反序列化调用被选中的构造函数时Q是q样来创造该构造函数需要的参数的:(x)
protected static Object getParamArg(Class cl) { if (! cl.isPrimitive()) return null; else if (boolean.class.equals(cl)) return Boolean.FALSE; else if (byte.class.equals(cl)) return new Byte((byte) 0); else if (short.class.equals(cl)) return new Short((short) 0); else if (char.class.equals(cl)) return new Character((char) 0); else if (int.class.equals(cl)) return new Integer(0); else if (long.class.equals(cl)) return new Long(0); else if (float.class.equals(cl)) return new Float(0); else if (double.class.equals(cl)) return new Double(0); else throw new UnsupportedOperationException(); }
可以看到Q如果参C?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">primitivecdQ会(x)?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">null代替Q但不y的是我们的构造函数内部对该参数调用了(jin)一个方法,因此抛出?jin)空指?#8230;
也就是说q个I指针是抛在客户端反序列化的时候而不是服务端内部Q因此服务端当然找不到对应的错误日志?jin)?br />解决的办法也很简单:(x)可以新增一个无参构造函?无参构造函数肯?#8220;得分”最低一定会(x)被选中)Q或者修改代码保证Q意参Cؓ(f)null
的时候都不会(x)出问?/p>
下面q个问题更有意思,说的?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">ThreadPoolExecutor?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">RejectedExecutionHandler的用:(x)
threadPool = new ThreadPoolExecutor(5, maxThreadNum, 5, TimeUnit.MINUTES, new SynchronousQueue<Runnable>(), tf, new RejectedExecutionHandler() { public void rejectedExecution(Runnable r, ThreadPoolExecutor e) { logger.error("Ep thread pool exhausted, epId: " + id + "!!"); r.run(); } });
q个代码是说Q当我这个ThreadPool不够用,又不能再新增U程数的时候,p用方U程自己来执行这?code style="margin: -1px 0px; padding: 0px 0.3em; border: 1px solid #dddddd; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: 0.8em; line-height: 1.5em; font-family: Menlo, Monaco, 'Andale Mono', 'lucida console', 'Courier New', monospace; vertical-align: baseline; display: inline-block; color: #555555; border-radius: 0.4em; background: #ffffff;">Runnabled…
本来q看上去没什么问题,问题出在q个Runnable
本n的实C —— 它内部将执行它的U程blockC(jin)一个blocking queue上面Q当调用方主U程亲自来执行它Ӟ使得ȝE再也回不去做它自己该做的事情了(jin)Q因此会(x)出大问题…
所以这么看来,“当线E池不够用就让调用方U程自己来干”的这个策略在实际使用时要非常谨慎
q个问题是通过一个方便的thread dump分析工具: tda来查扄Q因为出问题的这个应用是个多U程E序Q有1600多个帔RU程Qthread dump非常之大Q肉眼直接看很不方便Q但tda能将thread dump变得更友好易读,方便排查问题Q在此推荐一?/p>