SpringBoot + Dubbo的項目如何優雅停機

什麼是優雅停機？

在web服務（Http協議）上線的時候，會經過kill命令殺死進程,這個時候在已經accept的請求還在線程池裏面，咱們要保證這部分請求正常處理而且返回數據以後再停機.

dubbo服務（Tcp協議）也是一樣的道理.

優雅停機包括：線程池的優雅關閉，數據庫鏈接池的關閉，數據源的關閉，kafka鏈接的關閉....

沒有優雅停機的現象

1.客戶端拿不到數據

2.數據源報已經關閉

Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed

JVM的鉤子

Runtime.addShutDownHook

咱們的服務是基於SpringBoot+Dubbo+業務線程池來實現的.

dubbo框架提供了destroyAll來實現自身的優雅關閉，Spring容器對Bean的優雅關閉是經過@PreDestory來實現的，咱們本身建立的ThreadPool也能夠經過Runtime.addShutDownHook來實現優雅關閉。

可是這裏有一個關鍵的問題是這三者是並行執行的，dubbo在進入優雅停機狀態中的時候已經中止接收新的業務請求,然而已經接收的請求須要繼續處理，可是有可能此時Spring的優雅關閉已經執行完成，致使在處理請求的時候出現異常（好比DataSource已經close了）。

那如何保證Spring容器等待dubbo優雅關閉執行完成之後再執行bean的@PreDestory方法（銷燬bean）呢？下面有請Spring的ApplicationListener！

 

Spring應用生命週期

Spring能夠經過繼承下面這個接口來實現對應用聲明週期的回調

public interface ApplicationListener<E extends ApplicationEvent> extends EventListener {
    void onApplicationEvent(E var1);
}

ContextClosedEvent是在全部bean執行PreDestory以前發出的事件廣播.咱們在這個事件回調中執行Dubbo的優雅關閉，就不會出現數據源已經關閉的異常.

SpringBoot + Dubbo 服務的優雅關閉

思路很簡單，咱們只要監聽Spring容器的聲明週期,在容器啓動的時候把Dubbo註冊到JVM的shutdown hook刪除，而後在ContextClosedEvent中執行Dubbo的優雅關閉.

@Configuration
@Slf4j
public class DefaultSpringDubboConfigurations {

    @Bean
    DubboShutdownListener dubboShutdownListener() {
        return new DubboShutdownListener();
    }

    public static class DubboShutdownListener implements ApplicationListener, PriorityOrdered {

        @Override
        public void onApplicationEvent(ApplicationEvent event) {
            if (event instanceof ApplicationStartedEvent) {
                Runtime.getRuntime().removeShutdownHook(DubboShutdownHook.getDubboShutdownHook());
                log.info("dubbo default shutdown hook removed,will be managed by spring");
            } else if (event instanceof ContextClosedEvent) {
                log.info("start destroy dubbo on spring close event");
                DubboShutdownHook.getDubboShutdownHook().destroyAll();
                log.info("dubbo destroy finished");
            }
        }

        @Override
        public int getOrder() {
            return 0;
        }
    }
}

 

業務線程池如何優雅關閉？

錯誤的作法：

Runtime.getRuntime().addShutdownHook(new Thread(){
           // 線程池雖然關閉，可是隊列中的任務任然繼續執行，因此用 shutdown()方式關閉線程池時須要考慮是不是你想要的效果
            //若是但願當即中止，拋棄隊列中的任務，可使用shutdownNow()
            threadPoolExecutor.shutdown();
        });

上面的代碼忽視了多個鉤子函數是併發執行的問題,線程池的業務邏輯可能須要數據源連接、redis連接等，可是這個時候有可能數據源已經關閉了。

正確的作法：

@PostConstruct
    public void afterPropertiesSet() throws Exception {
        DEAL_EVENT_THREAD_POOL = ThreadPoolUtils.newExecutor(CORE_POOL_SIZE, MAXIMUM_POOL_SIZE, KEEP_ALIVE_TIME, TimeUnit.SECONDS,
                QUEUE_MAX_SIZE, "DealEventLogTask-service");
    }

    @PreDestroy
    public void destroy() throws Exception {
        ThreadPoolUtils.stop(DEAL_EVENT_THREAD_POOL);
    }

SpringBoot內嵌Tomcat的優雅停機

@Configuration
public class DefaultTomcatFactoryConfigurations {

    @Bean
    TomcatServletWebServerFactory tomcatServletWebServerFactory(@Value("${server.port}") int port,
            @Qualifier("gracefulShutdownListener") GracefulShutdownListener gracefulShutdownListener) {

        TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();
        factory.setPort(port);
        factory.setProtocol("org.apache.coyote.http11.Http11Nio2Protocol");

        List<LifecycleListener> contextLifecycleListeners = Lists.newArrayList(new VersionLoggerListener(),
                new JreMemoryLeakPreventionListener(), new ThreadLocalLeakPreventionListener());
        factory.setContextLifecycleListeners(contextLifecycleListeners);

        factory.addConnectorCustomizers(gracefulShutdownListener);

        //  RFC 6265 對cookie domain有較多限制，其中一條是不能以 "." 開頭，與com.kuaikan.common.utils.web.CookieUtils有衝突
        factory.addContextCustomizers(context -> context.setCookieProcessor(new LegacyCookieProcessor()));

        return factory;
    }

    @Bean
    GracefulShutdownListener gracefulShutdownListener(
            @Value("${server.shutdown-wait-seconds:30}") int shutdownWaitSeconds) {
        GracefulShutdownListener gracefulShutdownListener = new GracefulShutdownListener();
        gracefulShutdownListener.setShutdownWaitSeconds(shutdownWaitSeconds);
        return gracefulShutdownListener;
    }

    @Slf4j
    public static class GracefulShutdownListener
            implements TomcatConnectorCustomizer, ApplicationListener<ContextClosedEvent>, PriorityOrdered {

        private Connector connector;
        private int shutdownWaitSeconds;

        public void setShutdownWaitSeconds(int shutdownWaitSeconds) {
            this.shutdownWaitSeconds = shutdownWaitSeconds;
        }

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent contextClosedEvent) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                log.info("start to shutdown tomcat server,executor:{}", executor);
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(shutdownWaitSeconds, TimeUnit.SECONDS)) {
                        log.warn(
                                "tomcat thread pool did not shutdown gracefully within {} seconds. Proceeding with forceful shutdown",
                                shutdownWaitSeconds);
                    }
                } catch (Exception e) {
                    log.error("stop tomcat graceful exception", e);
                    Thread.currentThread().interrupt();
                }
                log.info("tomcat server stopped");
            }
        }

        @Override
        public int getOrder() {
            return Ordered.HIGHEST_PRECEDENCE;
        }
    }
}