python分佈式調度celery沒法啓動或自動退出排查步驟

時間 2019-11-11

標籤 python 分佈式調度 celery 沒法啓動自動退出排查步驟欄目 Python 简体版

原文原文鏈接

Celery是個好東西,可是不具體去分析,等出現錯誤會顯得手忙腳亂node

1) 啓動調試方法python

celery worker -c 2 -A project -B -l debug -f /var/log/project.loggit

日誌基本沒有,控制檯就是一些debug和warning日誌json

[2017-01-16 13:06:35,449: INFO/Beat] beat: Starting...
[2017-01-16 13:06:35,461: INFO/MainProcess] Connected to amqp://username:**@127.0.0.1:5672/
[2017-01-16 13:06:35,482: DEBUG/Beat] Current schedule:
<Entry: 任務名稱 <crontab: */3 * * * * (m/h/d/dM/MY)>
[2017-01-16 13:06:35,482: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes
[2017-01-16 13:06:35,483: INFO/MainProcess] mingle: searching for neighbors
[2017-01-16 13:06:35,503: INFO/Beat] Scheduler: Sending due task 任務名稱 (任務名稱)
[2017-01-16 13:06:36,500: INFO/MainProcess] mingle: all alone
[2017-01-16 13:06:36,528: WARNING/MainProcess] celery@localhost ready.
[2017-01-16 13:06:36,559: DEBUG/MainProcess] Canceling task consumer...
[2017-01-16 13:06:36,561: INFO/MainProcess] beat: Shutting down...
[2017-01-16 13:06:37,587: DEBUG/MainProcess] Canceling task consumer...
[2017-01-16 13:06:37,588: DEBUG/MainProcess] Closing consumer channel...vim

而後怎麼辦,懵逼了...python2.7

這個時候,不能隨便猜了,不信你可要猜一夜試試oop

去git上下celery的源碼,搜索關鍵字this

Canceling task consumerspa

估計緣由是在xx地方報錯了,而後打印了2行日誌出來..net

找到代碼在worker/consumer.py裏面

def shutdown(self, c):
if c.task_consumer:
import traceback
debug("start shutdown traceback ====")
traceback.print_stack()
self.stop(c)
debug('Closing consumer channel...')
ignore_errors(c, c.task_consumer.close)
c.task_consumer = None

添加一個traceback,打印出從哪裏調用進來的.

再啓動一下celery,你就會發現不同了

[2017-01-16 13:06:37,579: WARNING/MainProcess] File "/usr/local/bin/celery", line 9, in <module>
[2017-01-16 13:06:37,580: WARNING/MainProcess] load_entry_point('celery==3.1.23', 'console_scripts', 'celery')()
[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
[2017-01-16 13:06:37,580: WARNING/MainProcess] main()
[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
[2017-01-16 13:06:37,581: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
[2017-01-16 13:06:37,581: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
[2017-01-16 13:06:37,581: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
[2017-01-16 13:06:37,582: WARNING/MainProcess] return self.execute(command, argv)
[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute
[2017-01-16 13:06:37,582: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
[2017-01-16 13:06:37,583: WARNING/MainProcess] return self(*args, **options)
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
[2017-01-16 13:06:37,583: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
[2017-01-16 13:06:37,584: WARNING/MainProcess] state_db=self.node_format(state_db, hostname), **kwargs
[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 211, in start
[2017-01-16 13:06:37,584: WARNING/MainProcess] self.stop()
[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 252, in stop
[2017-01-16 13:06:37,584: WARNING/MainProcess] self._shutdown(warm=True)
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 266, in _shutdown
[2017-01-16 13:06:37,585: WARNING/MainProcess] self.blueprint.stop(self, terminate=not warm)
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 183, in stop
[2017-01-16 13:06:37,585: WARNING/MainProcess] self.on_stopped()
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 164, in on_stopped
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.consumer.shutdown()
[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 306, in shutdown
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.blueprint.shutdown(self)
[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 171, in shutdown
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.send_all(parent, 'shutdown')
[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 155, in send_all
[2017-01-16 13:06:37,587: WARNING/MainProcess] fun(parent, *args)
[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 628, in shutdown
[2017-01-16 13:06:37,587: WARNING/MainProcess] traceback.print_stack()

懷疑: start以後爲啥立刻就stop了.奇怪.

vim /usr/local/lib/python2.7/site-packages/celery/worker/__init__.py

到211行

def start(self):
"""Starts the workers main loop."""
try:
self.blueprint.start(self)
except WorkerTerminate:
self.terminate()
except Exception as exc:
logger.error('Unrecoverable error: %r', exc, exc_info=True)
self.stop()
except (KeyboardInterrupt, SystemExit):
self.stop()

爲啥這地方使用了logger.error,咱們沒有看到日誌呢,艹

剩下的就是想辦法吧日誌全打印出來,

經過代碼跟蹤須要配置2個日誌名稱celery.worker和celery.task

LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'direct': {
'format': '%(asctime)s - %(message)s'
},
'json': {
'format': '%(message)s'
},
},
'handlers': {
'celery.worker': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/celery.worker.log',
'formatter': 'direct'
},
'celery.task': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/celery.task.log',
'formatter': 'direct'
},
},
'loggers': {
'celery.worker': {
'handlers': ['celery.worker'],
'level': 'DEBUG'
},
'celery.task': {
'handlers': ['celery.task'],
'level': 'DEBUG'
},
}
}

配置上面,

而後啓動celery,

[2017-01-16 13:06:36,550: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
self.blueprint.start(self)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
step.start(parent)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
return self.obj.start()
File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
blueprint.start(self)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
step.start(parent)
File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 841, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 48, in asynloop
raise WorkerLostError('Could not start worker processes')
WorkerLostError: Could not start worker processes

而後你會發現不一樣環境的這個錯誤是不同的.

celery worker -c 2 -A project -l debug -P gevent

添加一個參數-P gevent可以正常啓動了

查看一下-B參數的說明 -B, --beat Also run the celery beat periodic task scheduler. Please note that there must only be one instance of this service