kafka CommitFailedError

線上kafka出現以下異常

2018-09-16 23:59:58,712-pid:20908-MainThread-consumer.py[line:782]-WARNING:Auto offset commit failed for group robot: CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

根據提示，老老實實修改max_poll_interval_ms，出現如上緣由多是消息的處理時間超過max_poll_interval_ms時間間隔，致使kafka認爲該消費者掛掉，從新進行reblancing。

修改後max_poll_interval_ms>消息處理的時間便可

max.poll.interval.ms ，它表示最大的poll數據間隔，若是超過這個間隔沒有發起pool請求，但heartbeat仍舊在發，就認爲該consumer處於 livelock狀態，進行reblancing

session.timeout.ms 消費者超時時間，若是消費者再超過了這個時間尚未發送心跳，那麼kafka則會認爲該消費者掛掉了，會從新進行reblancing。session.timeout.ms與消息的處理時間沒有關係