請勿過分依賴Redis的過時監聽

Redis過時監聽場景

業務中有相似等待必定時間以後執行某種行爲的需求 , 好比30分鐘以後關閉訂單 . 網上有不少使用Redis過時監聽的Demo , 可是其實這是個大坑 , 由於Redis不能確保key在指定時間被刪除 , 也就形成了通知的延期 . 很少說 , 跑個測試

測試狀況

先說環境 , redis 運行在Docker容器中 ,分配了 一個cpu以及512MB內存, 在Docker中執行 redis-benchmark -t set -r 100000 -n 1000000 結果以下:

====== SET ======
  1000000 requests completed in 171.03 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 3600 1 300 100 60 10000
  host configuration "appendonly": no
  multi-thread: no複製代碼

其實這裏有些不嚴謹 benchmark 線程不該該在Docker容器內部運行 . 跑分的時候大概 benchmark 和redis 主線程各自持有50%CPU

測試代碼以下:

@Service
@Slf4j
public class RedisJob {
    @Autowired
    private StringRedisTemplate stringRedisTemplate;

    public DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
    public LocalDateTime end = LocalDateTime.of(LocalDate.of(2020, 5, 12), LocalTime.of(8, 0));

    @Scheduled(cron = "0 56 * * * ?")
    public void initKeys() {
        LocalDateTime now = LocalDateTime.now();
        ValueOperations<String, String> operations = stringRedisTemplate.opsForValue();
        log.info("開始設置key");
        LocalDateTime begin = now.withMinute(0).withSecond(0).withNano(0);
        for (int i = 1; i < 17; i++) {
            setExpireKey(begin.plusHours(i), 8, operations);
        }
        log.info("設置完畢: " + Duration.between(now, LocalDateTime.now()));
    }

    private void setExpireKey(LocalDateTime expireTime, int step, ValueOperations<String, String> operations) {
        LocalDateTime localDateTime = LocalDateTime.now().withNano(0);
        String nowTime = dateTimeFormatter.format(localDateTime);
        while (expireTime.getMinute() < 55) {
            operations.set(nowTime + "@" + dateTimeFormatter.format(expireTime), "A", Duration.between(expireTime, LocalDateTime.now()).abs());
            expireTime = expireTime.plusSeconds(step);
        }
    }
}
複製代碼

大概意思就是每小時56分的時候 , 會增長一批在接下來16小時過時的key , 過時時間間隔8秒 , 且過時時間都在55分以前

@Slf4j
@Component
public class RedisKeyExpirationListener extends KeyExpirationEventMessageListener {

    public RedisKeyExpirationListener(RedisMessageListenerContainer listenerContainer) {
        super(listenerContainer);
    }

    public DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
    @Autowired
    private StringRedisTemplate stringRedisTemplate;


    @Override
    public void onMessage(Message message, byte[] pattern) {
        String keyName = new String(message.getBody());
        LocalDateTime parse = LocalDateTime.parse(keyName.split("@")[1], dateTimeFormatter);
        long seconds = Duration.between(parse, LocalDateTime.now()).getSeconds();
        stringRedisTemplate.execute((RedisCallback<Object>) connection -> {
            Long size = connection.dbSize();
            log.info("過時key:" + keyName + " ,當前size:" + size + " ,滯後時間" + seconds);
            return null;
        });
    }
}複製代碼

這裏是監測到過時以後打印當前的dbSize 以及滯後時間

@Bean
public RedisMessageListenerContainer configRedisMessageListenerContainer(RedisConnectionFactory connectionFactory) {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(100);
    executor.setMaxPoolSize(100);
    executor.setQueueCapacity(100);
    executor.setKeepAliveSeconds(3600);
    executor.setThreadNamePrefix("redis");
    // rejection-policy：當pool已經達到max size的時候，如何處理新任務
    // CALLER_RUNS：不在新線程中執行任務，而是由調用者所在的線程來執行
    executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
    executor.initialize();
    RedisMessageListenerContainer container = new RedisMessageListenerContainer();
    // 設置Redis的鏈接工廠
    container.setConnectionFactory(connectionFactory);
    // 設置監聽使用的線程池
    container.setTaskExecutor(executor);
    // 設置監聽的Topic
    return container;
}複製代碼

設置Redis的過時監聽 以及線程池信息 , 

最後的測試結果是當key數量小於1萬的時候 , 基本上均可以在10s內完成過時通知 , 可是若是數量到3萬 , 就有部分key會延遲120s . 順便貼一下我最新的日誌

2020-05-13 22:16:48.383  : 過時key:2020-05-13 11:56:02@2020-05-13 22:14:08 ,當前size:57405 ,滯後時間160
2020-05-13 22:16:49.389  : 過時key:2020-05-13 11:56:02@2020-05-13 22:14:32 ,當前size:57404 ,滯後時間137
2020-05-13 22:16:49.591  : 過時key:2020-05-13 10:56:02@2020-05-13 22:13:20 ,當前size:57403 ,滯後時間209
2020-05-13 22:16:50.093  : 過時key:2020-05-13 20:56:00@2020-05-13 22:12:32 ,當前size:57402 ,滯後時間258
2020-05-13 22:16:50.596  : 過時key:2020-05-13 07:56:03@2020-05-13 22:13:28 ,當前size:57401 ,滯後時間202
2020-05-13 22:16:50.697  : 過時key:2020-05-13 20:56:00@2020-05-13 22:14:32 ,當前size:57400 ,滯後時間138
2020-05-13 22:16:50.999  : 過時key:2020-05-13 19:56:00@2020-05-13 22:13:44 ,當前size:57399 ,滯後時間186
2020-05-13 22:16:51.199  : 過時key:2020-05-13 20:56:00@2020-05-13 22:14:40 ,當前size:57398 ,滯後時間131
2020-05-13 22:16:52.205  : 過時key:2020-05-13 15:56:01@2020-05-13 22:16:24 ,當前size:57397 ,滯後時間28
2020-05-13 22:16:52.808  : 過時key:2020-05-13 06:56:03@2020-05-13 22:15:04 ,當前size:57396 ,滯後時間108
2020-05-13 22:16:53.009  : 過時key:2020-05-13 06:56:03@2020-05-13 22:16:40 ,當前size:57395 ,滯後時間13
2020-05-13 22:16:53.110  : 過時key:2020-05-13 20:56:00@2020-05-13 22:14:56 ,當前size:57394 ,滯後時間117
2020-05-13 22:16:53.211  : 過時key:2020-05-13 06:56:03@2020-05-13 22:13:44 ,當前size:57393 ,滯後時間189
2020-05-13 22:16:53.613  : 過時key:2020-05-13 15:56:01@2020-05-13 22:12:24 ,當前size:57392 ,滯後時間269
2020-05-13 22:16:54.317  : 過時key:2020-05-13 15:56:01@2020-05-13 22:16:00 ,當前size:57391 ,滯後時間54
2020-05-13 22:16:54.517  : 過時key:2020-05-13 18:56:00@2020-05-13 22:15:44 ,當前size:57390 ,滯後時間70
2020-05-13 22:16:54.618  : 過時key:2020-05-13 21:56:00@2020-05-13 22:14:24 ,當前size:57389 ,滯後時間150
2020-05-13 22:16:54.819  : 過時key:2020-05-13 17:56:00@2020-05-13 22:14:40 ,當前size:57388 ,滯後時間134
2020-05-13 22:16:55.322  : 過時key:2020-05-13 10:56:02@2020-05-13 22:13:52 ,當前size:57387 ,滯後時間183
2020-05-13 22:16:55.423  : 過時key:2020-05-13 07:56:03@2020-05-13 22:14:16 ,當前size:57386 ,滯後時間159複製代碼

能夠看到 ,當數量到達5萬的時候 , 大部分都已經滯後了兩分鐘 , 對於業務方來講已經徹底沒法忍受了

總結

可能到這裏 , 你會說Redis 給你挖了一個大坑 , 但其實這些都在文檔上寫的明明白白

• How Redis expires keys 
  
• Timing of expired events 
  

尤爲是在 Timing of expired events  中 , 明確的說明了 "Basically expired events are generated when the Redis server deletes the key and not when the time to live theoretically reaches the value of zero."  , 這兩個文章讀下來你會感受 ,  臥槽Redis的過時策略其實也挺'Low'的

其實公衆號看多了 , 你會發現大部分Demo都是互相抄來抄去 , 以及翻譯官方Demo . 建議你們仍是謹慎一些 , 真要使用的話 , 最好讀一下官方文檔 , 哪怕用百度翻譯也要有一些本身的理解 .

文章比較枯燥 , 感謝你們耐心閱讀 ,  若有建議 懇請留言.