WRITE AS 堵住 应用系统瓶颈排查和分析的思考

大家好，今天小编来为大家解答WRITE AS 堵住这个问题，应用系统瓶颈排查和分析的思考很多人还不知道，现在让我们一起来看看吧！

背景

业务应用系统接入流程引擎来处理业务应用的流程执行，流程引擎提供多线程高性能异步化来执行流程元素的执行，但是如何设置流程引擎的线程池线程数执行，以及执行线程数和任务数，应用机器资源使用情况之间的关系如何，目前只能通过接入方的人工经验评估，比较粗泛评估，参数的合理性也很难直观的评估，现通过现有的应用系统默认的参数配置进行问题说明。

性能监控

查看CPU核数

应用机器的配置是2核CPU配置：

查看应用机器的负载情况

应用程序的近15分钟内的负载严重的超载(平均超出CPU执行的10倍之多)，应用程序对应的CPU和MEM的利用率都很高，按照本次单机模拟线上的流量（QPS：10)不至于出现如此的负载情况吧。

查看应用机器的综合统计

CPU运行队列中也严重的堵筛(Procs列中指标），应用的CPU有效利用率其实并不算太高（US+SY),但是系统出现了严重的CS(上下文切换)，配合BI，BO执行，也间接反馈出一些端倪，具体反馈到应用程序中，还需要进一步分析。

查看应用服务(PID=512)的进程CPU执行情况

查看应用机器的磁盘IO使用情况

应用系统的线程统计概要

应用系统中居然有如此多的线程(1285),应用系统接入流程SDK中的线程池数据设置也不会超过80个线程，为何有如何大的偏差，考虑到应用系统中还存在RPC，kafka等相关的技术栈，对于应用系统的线程分布还需要进一步的分析。

应用系统执行dashboard

GC回收线程监控统计

应用系统线程执行栈的统计

阻塞线程ThreadDump信息

org.apache.log4j.spi.RootLoggerisblocking379threads.

Pigeon-Server-Request-Processor-46-thread-318StackTraceis:java.lang.Thread.State:RUNNABLEatjava.lang.Throwable.printStackTrace(Throwable.java:665)

locked<0x000000075332fde8>(ajava.io.PrintWriter)java.lang.Throwable.printStackTrace(Throwable.java:721)

atorg.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)atorg.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)

locked<0x000000074ae96550>(aorg.apache.log4j.spi.ThrowableInformation)org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)

atorg.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)atorg.apache.log4j.WriterAppender.append(WriterAppender.java:162)atcom.dianping.combiz.misc.ExtendedConsoleAppender.append(ExtendedConsoleAppender.java:82)atorg.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)

eliminated<0x000000075f880030>(acom.dianping.combiz.misc.ExtendedConsoleAppender)com.dianping.combiz.misc.ExtendedConsoleAppender.doAppend(ExtendedConsoleAppender.java:75)locked<0x000000075f880030>(acom.dianping.combiz.misc.ExtendedConsoleAppender)org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)

atorg.apache.log4j.Category.callAppenders(Category.java:206)-locked<0x000000075ed2bd58>(aorg.apache.log4j.spi.RootLogger)atorg.apache.log4j.Category.forcedLog(Category.java:391)atorg.apache.log4j.Category.log(Category.java:856)atorg.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)atcom.meituan.mafka.client.producer.DefaultProducerProcessor.doSend(DefaultProducerProcessor.java:832)atcom.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:603)atcom.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:584)atcom.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:567)atcom.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:562)atcom.dianping.poi.flowcenter.mq.WorkflowEventMQProducer.sendSyncMessage(WorkflowEventMQProducer.java:98)atcom.dianping.poi.flowcenter.WorkflowEngine.createFlowInstance(WorkflowEngine.java:642)atcom.dianping.poi.flowcenter.WorkflowEngine.startWorkflow0(WorkflowEngine.java:560)atcom.dianping.poi.flowcenter.WorkflowEngine.startWorkflow(WorkflowEngine.java:504)atcom.dianping.poi.mainflow.clearengine.engine.create.impl.CreateWorkflowServiceImpl.createWorkflow(CreateWorkflowServiceImpl.java:66)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:277)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:183)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:176)atcom.dianping.poi.mainflow.core.business.WorkflowBusinessFastClassBySpringCGLIBf3bb47a4.invoke()atorg.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)atorg.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)atcom.dianping.poi.mainflow.core.business.WorkflowBusinessEnhancerBySpringCGLIBe721a2aa.createWorkflow()atcom.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:220)atcom.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:179)atcom.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpFeedbackBase(FeedbackForPoiServiceImpl.java:1233)atcom.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpTerminalFeedback(FeedbackForPoiServiceImpl.java:486)atsun.reflect.GeneratedMethodAccessor144.invoke(UnknownSource)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:497)atcom.dianping.pigeon.remoting.provider.service.method.ServiceMethod.invoke(ServiceMethod.java:193)atcom.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:79)atcom.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:34)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:242)atcom.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:51)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:31)atcom.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:41)atcom.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:24)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:44)atcom.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:27)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:43)atcom.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:28)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:36)atcom.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:26)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:34)atcom.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:14)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:29)atcom.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:101)atcom.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:35)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:37)atcom.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:28)atcom.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:18)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:254)atcom.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)atjava.lang.Thread.run(Thread.java:745)Lockedownablesynchronizers:

<0x00000007ab2b6228>(ajava.util.concurrent.ThreadPoolExecutor$Worker)

Pigeon阻塞线程ThreadDump信息

priority:5-threadId:0x00007fa599636000-nativeId:0x2ec8-nativeId(decimal):11976-state:BLOCKEDstackTrace:java.lang.Thread.State:BLOCKED(onobjectmonitor)atdeps.redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:207)atdeps.redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:47)atdeps.redis.clients.jedis.Protocol.process(Protocol.java:204)atdeps.redis.clients.jedis.Protocol.read(Protocol.java:271)atdeps.redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:305)atdeps.redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:236)atdeps.redis.clients.jedis.BinaryJedis.hget(BinaryJedis.java:788)atdeps.redis.clients.jedis.BinaryJedisCluster$13.execute(BinaryJedisCluster.java:202)atdeps.redis.clients.jedis.BinaryJedisCluster$13.execute(BinaryJedisCluster.java:199)atdeps.redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:64)atdeps.redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:41)atdeps.redis.clients.jedis.BinaryJedisCluster.hget(BinaryJedisCluster.java:199)atcom.dianping.squirrel.client.impl.redis.RedisStoreClientImpl$41.excute(RedisStoreClientImpl.java:1022)atcom.dianping.squirrel.client.impl.AbstractStoreClient$MonitorCommand.run(AbstractStoreClient.java:1066)atcom.dianping.squirrel.client.impl.redis.RedisStoreClientImpl.hget(RedisStoreClientImpl.java:1018)atcom.dianping.poi.mainflow.core.redisdao.TimestampRedisDao.getUpdateTime(TimestampRedisDao.java:18)atcom.dianping.poi.mainflow.common.RefreshableCache.checkUpdate(RefreshableCache.java:99)

locked<0x00000007623197a0>(acom.dianping.poi.mainflow.core.cache.FlowTemplateRuleCache)com.dianping.poi.mainflow.common.RefreshableCache.getValue(RefreshableCache.java:53)

atcom.dianping.poi.mainflow.common.RefreshableCache.getValue(RefreshableCache.java:43)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.getTemplateRule(WorkflowBusiness.java:160)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.setTemplateByWorkflow(WorkflowBusiness.java:134)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:253)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:183)atcom.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:176)atcom.dianping.poi.mainflow.core.business.WorkflowBusinessFastClassBySpringCGLIBf3bb47a4.invoke()atorg.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)atorg.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)atcom.dianping.poi.mainflow.core.business.WorkflowBusinessEnhancerBySpringCGLIBe721a2aa.createWorkflow()atcom.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:220)atcom.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:179)atcom.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpFeedbackBase(FeedbackForPoiServiceImpl.java:1233)atcom.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpTerminalFeedback(FeedbackForPoiServiceImpl.java:486)atsun.reflect.GeneratedMethodAccessor144.invoke(UnknownSource)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:497)atcom.dianping.pigeon.remoting.provider.service.method.ServiceMethod.invoke(ServiceMethod.java:193)atcom.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:79)atcom.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:34)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:242)atcom.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:51)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:31)atcom.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:41)atcom.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:24)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:44)atcom.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:27)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:43)atcom.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:28)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:36)atcom.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:26)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:34)atcom.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:14)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:29)atcom.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:101)atcom.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:35)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:37)atcom.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:15)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:28)atcom.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:18)atcom.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)atcom.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:254)atcom.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)atjava.lang.Thread.run(Thread.java:745)Lockedownablesynchronizers:-<0x0000000735ab3bc0>(ajava.util.concurrent.ThreadPoolExecutor$Worker)

问题说明通过对应用系统的模拟上线应用运行状态分析，目标找出系统的瓶颈点，以及通过对应用系统的资源使用情况来评估应用系统的相关参数设置是否合理；如何根据应用系统的运行时的性能监控，动态的评估应用系统潜在的瓶颈点，以及预测此瓶颈点对应用的影响，是否可以动态的调配相关应用参数，来自适用应用系统性能，充分利用应用系统资源(CPU,Memory,IO)等，结合应用系统的任务使用特性(CPU密集型，IO密集型，MiX混合性）以及多个线程池任务执行特性，是否能够动态预测。

作者|一啦米

本文为阿里云原创内容，未经允许不得转载。

好了，本文到此结束，如果可以帮助到大家，还望关注本站哦！