[alibaba/nacos]nacos客户端上万线程com.alibaba.nacos.client.naming.updater

1、版本 nacos-client:1.4.1 nacos-server:2.0.4 spring-cloud-alibaba:2021.1

2、问题描述 1）gateway微服务出现上万的线程主要是下面三个线程 com.alibaba.nacos.client.naming.updater com.alibaba.nacos.naming.failover com.alibaba.nacos.naming.push.receiver 2）NacosNamingService只有一个对象 3）NamingProxy有上百个对象 4）NacosConfigService有8个对象 5）大量增加线程时，nacos-server出现过连不上的问题 2023-03-21 05:28:54.366 [http-nio-7050-exec-2] ERROR com.alibaba.nacos.client.naming:617 - [NA] failed to request java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933) at com.alibaba.nacos.common.http.client.request.JdkHttpClientRequest.execute(JdkHttpClientRequest.java:112) at com.alibaba.nacos.common.http.client.NacosRestTemplate.execute(NacosRestTemplate.java:482) at com.alibaba.nacos.common.http.client.NacosRestTemplate.exchangeForm(NacosRestTemplate.java:427) at com.alibaba.nacos.client.naming.net.NamingProxy.callServer(NamingProxy.java:603) at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:526) at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:498) at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:493) at com.alibaba.nacos.client.naming.net.NamingProxy.serverHealthy(NamingProxy.java:445) at com.alibaba.nacos.client.naming.NacosNamingService.getServerStatus(NacosNamingService.java:512) at com.alibaba.cloud.nacos.discovery.actuate.health.NacosDiscoveryHealthIndicator.doHealthCheck(NacosDiscoveryHealthIndicator.java:43) at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82) at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37) at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:87) at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:45) at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:130) at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateContribution(HealthEndpointSupport.java:141) at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:126) at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:95) at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:66) at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:73) at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:62) at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74) at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:353) at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:458) at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)

3、现在就是gateway微服务产生大量的线程不释放，导致发生OOM，linux本地线程被耗尽。可能就是连不上nacos注册中心导致的，但是只有gateway一个微服务会产生大量线程，其它的微服务并未发现产生大量线程。请帮忙看看这个问题如何解决，感谢。

RoyWong0220

com.alibaba.nacos.client.naming.updater 这个线程池和NacosNamingServie个数是绑定的，如果NacosNamingServie创建多个并且不shutdown销毁，这个线程池会不停增加，应该是spring cloud gateway的问题，没有正确的使用nacos-client

KomachiSion

可以提交到spring cloud alibaba 社区，看下是否NacosNamingServie创建销毁的时候没有shutdown

KomachiSion

1、看了heap dump，NacosNamingServie对象只有一个，这个对象是NamingFactory通过反射创建，只创建了一次，只有在NacosNamingServie初始化的时候会创建NamingProxy、BeatReactor等对象，这些对象都有非常多的数量，和各自的线程数是对应的，现在的问题是，为什么NacosNamingServie对象只有一个而NamingProxy等对象会有那么多？ 2、gateway里面只是单纯依赖了nacos，没有显式的去操作nacos中的类，所以可能和gateway没啥关系 3、现在主要想知道为啥NacosNamingServie对象只有一个而NamingProxy等对象会有那么多？什么样的场景可以模拟复现？暂时没啥思路，请帮忙再看看呢，感谢。

RoyWong0220

比如是否一直有new NacosNamingServie, 并且把旧的引用替换成新的，这样旧的NacosNamingServie对象会被GC掉，但线程池不会被关闭。

比如：

NacosNamingServie service = null;

start() {
service = NamingFactory.newNamingServie();
}

....

refresh() {
service = NamingFactory.newNamingServie();
}

KomachiSion

这个问题必然是用法上的问题，因为NacosNamingServie内，对应的线程池一次初始化只有一个。

只可能是创建多个NacosNamingServie，且旧的NacosNamingServie没有调用shutdown命令关闭线程池和连接池导致。

KomachiSion

以前dubbo， Spring Cload alibaba、 sentitnel都有类似错误

dubbo是创建很多个NacosNamingServie，

Spring Cload alibaba和 sentitnel是动态刷新配置后重新创建NacosNamingServie，但是旧的没有shutdown。

具体的issue忘记是哪个。

KomachiSion

以前dubbo， Spring Cload alibaba、 sentitnel都有类似错误

dubbo是创建很多个NacosNamingServie，

Spring Cload alibaba和 sentitnel是动态刷新配置后重新创建NacosNamingServie，但是旧的没有shutdown。

具体的issue忘记是哪个。

好的，那就必然是NacosNamingServie被创建了多次，我再看下是不是nacos短暂连不上后重连导致动态刷新没有shutdown，感谢。

RoyWong0220

以前dubbo， Spring Cload alibaba、 sentitnel都有类似错误 dubbo是创建很多个NacosNamingServie， Spring Cload alibaba和 sentitnel是动态刷新配置后重新创建NacosNamingServie，但是旧的没有shutdown。具体的issue忘记是哪个。

好的，那就必然是NacosNamingServie被创建了多次，我再看下是不是nacos短暂连不上后重连导致动态刷新没有shutdown，感谢。

请问下，最终是什么问题呢，我这边也出现同样的问题。nacos-client:1.4.2

onlymate918

已经排查到问题，项目中使用NacosFactory.createNamingService多次创建了对象导致的

onlymate918

[alibaba/nacos]nacos客户端上万线程com.alibaba.nacos.client.naming.updater

回答