django+celery+redis實現運行定時任務

時間 2020-07-18

標籤 django+celery+redis django celery redis 實現運行定時任務欄目 Python 简体版

原文原文鏈接

0、目的

在開發項目中，常常有一些操做時間比較長（生產環境中超過了nginx的timeout時間），或者是間隔一段時間就要執行的任務。

在這種狀況下，使用celery就是一個很好的選擇。

celery是一個異步任務隊列/基於分佈式消息傳遞的做業隊列。

Celery經過消息（message）進行通訊，使用代理（broker）在客戶端和工做執行者之間進行交互。

當開始一個任務時，客戶端發送消息到隊列並由代理將其發往響應的工做執行者處。

準備使用redis做爲消息代理（broker），Django數據庫做爲結果存儲（ResultStore）。

一、安裝

redis：

windows：

https://github.com/MSOpenTech/redis/releases/

linux：

yum install redis-server

PS：須要在cmd中運行，不能再powercmd。很奇怪。

pip install celery

pip install celery-with-redis

pip install django-celery

二、django代碼（whthas_home爲project，portal爲app）

修改代碼，whthas_home/__init__.py

1 from __future__ import absolute_import
2 
3 from .celery import app as celery_app

修改代碼，whthas_home/setting.pyhtml

 1 # Celery settings
 2 import djcelery
 3 djcelery.setup_loader()
 4 
 5 BROKER_URL = 'redis://127.0.0.1:6379/0'
 6 CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
 7 CELERY_ACCEPT_CONTENT = ['json']
 8 CELERY_TASK_SERIALIZER = 'json'
 9 CELERY_RESULT_SERIALIZER = 'json'
10 CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
11 CELERY_TIMEZONE = 'Asia/Shanghai'
12 
13 # Application definition
14 
15 INSTALLED_APPS = [
16     'django.contrib.auth',
17     'django.contrib.contenttypes',
18     'django.contrib.sessions',
19     'django.contrib.messages',
20     'django.contrib.staticfiles',
21     'suit',
22     'django.contrib.admin',
23     'DjangoUeditor',
24     'portal',
25     'djcelery',
26 ]

增長文件， whthas_home/celery.pypython

 1 from __future__ import absolute_import
 2 
 3 import os
 4 from celery import Celery
 5 from django.conf import settings
 6 
 7 os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'whthas_home.settings')
 8 
 9 app = Celery('portal')
10 
11 app.config_from_object('django.conf:settings')
12 app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
13 
14 
15 @app.task(bind=True)
16 def debug_task(self):
17     print('Request: {0!r}'.format(self.request))

增長文件，portal/tasks.py

 1 from celery import task
 2 from time import sleep
 3 
 4 
 5 @task()
 6 def Task_A(message):
 7     Task_A.update_state(state='PROGRESS', meta={'progress': 0})
 8     sleep(10)
 9     Task_A.update_state(state='PROGRESS', meta={'progress': 30})
10     sleep(10)
11     return message
12 
13 
14 def get_task_status(task_id):
15     task = Task_A.AsyncResult(task_id)
16 
17     status = task.state
18     progress = 0
19 
20     if status == u'SUCCESS':
21         progress = 100
22     elif status == u'FAILURE':
23         progress = 0
24     elif status == 'PROGRESS':
25         progress = task.info['progress']
26     
27     return {'status': status, 'progress': progress}

三、測試

啓動broker：python manage.py celeryd -l info

進入shell：python manage.py shell，測試程序是否正常。

>>> from portal.tasks import *
>>> t = TaskA.delay("heel2")
>>> get_task_status(t.id)
{'status': u'PROGRESS', 'progress': 0}
>>> get_task_status(t.id)
{'status': u'PROGRESS', 'progress': 0}
>>> get_task_status(t.id)
{'status': u'PROGRESS', 'progress': 30}
>>> get_task_status(t.id)
{'status': u'PROGRESS', 'progress': 30}
>>> get_task_status(t.id)
{'status': u'SUCCESS', 'progress': 100}
>>>

同時broker側能看到： linux

[2017-04-21 16:38:47,023: INFO/MainProcess] Received task: portal.tasks.Task_A[da948495-c64b-4ff9-882b-876721cd5017]
[2017-04-21 16:39:07,035: INFO/MainProcess] Task portal.tasks.Task_A[da948495-c64b-4ff9-882b-876721cd5017] succeeded in 20.0099999905s: heel

表示代碼能正常運行。nginx

使用redis-client看到任務狀態：

127.0.0.1:6379> get "celery-task-meta-da948495-c64b-4ff9-882b-876721cd5017"
"{\"status\": \"SUCCESS\", \"traceback\": null, \"result\": \"heel\", \"children\": []}"

相關redis命令：keys *git

四、django後臺定義任務

進入後臺

設置任務

五、執行任務

啓動broker：python manage.py celeryd -l infogithub

由於這裏是個定時任務，因此還須要啓動心跳：python manage.py celery beat

broker側能看到：

[2017-04-21 16:56:33,216: INFO/MainProcess] Received task: portal.tasks.Task_A[d8a26977-8413-4bf0-b518-b53052af4cee]
[2017-04-21 16:56:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[00f1cab7-eb56-4cc3-9979-a8a68aaaf2de]
[2017-04-21 16:56:53,220: INFO/MainProcess] Task portal.tasks.Task_A[d8a26977-8413-4bf0-b518-b53052af4cee] succeeded in 20.003000021s: heel2
[2017-04-21 16:57:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[aa652612-8525-4110-94ea-9010085ec20b]
[2017-04-21 16:57:13,223: INFO/MainProcess] Task portal.tasks.Task_A[00f1cab7-eb56-4cc3-9979-a8a68aaaf2de] succeeded in 20.0080001354s: heel2
[2017-04-21 16:57:33,213: INFO/MainProcess] Received task: portal.tasks.Task_A[9876890e-6a71-4501-bdae-775492ebae88]
[2017-04-21 16:57:33,219: INFO/MainProcess] Task portal.tasks.Task_A[aa652612-8525-4110-94ea-9010085ec20b] succeeded in 20.0050001144s: heel2
[2017-04-21 16:57:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[c12fcffc-3910-4a22-93b3-0df740910728]
[2017-04-21 16:57:53,221: INFO/MainProcess] Task portal.tasks.Task_A[9876890e-6a71-4501-bdae-775492ebae88] succeeded in 20.0069999695s: heel2
[2017-04-21 16:58:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[ccfad575-c0b4-48f5-9385-85ff5dac76fc]
[2017-04-21 16:58:13,217: INFO/MainProcess] Task portal.tasks.Task_A[c12fcffc-3910-4a22-93b3-0df740910728] succeeded in 20.003000021s: heel2
[2017-04-21 16:58:33,211: INFO/MainProcess] Received task: portal.tasks.Task_A[0d6f77a6-a29c-4ead-9428-e3e758c754e1]
[2017-04-21 16:58:33,221: INFO/MainProcess] Task portal.tasks.Task_A[ccfad575-c0b4-48f5-9385-85ff5dac76fc] succeeded in 20.007999897s: heel2
[2017-04-21 16:58:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[34b2b3a0-771c-4a28-94a1-92e25763fae1]
[2017-04-21 16:58:53,217: INFO/MainProcess] Task portal.tasks.Task_A[0d6f77a6-a29c-4ead-9428-e3e758c754e1] succeeded in 20.0039999485s: heel2
[2017-04-21 16:59:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[d4be920f-376a-46a2-9edd-095234d29ef2]
[2017-04-21 16:59:13,217: INFO/MainProcess] Task portal.tasks.Task_A[34b2b3a0-771c-4a28-94a1-92e25763fae1] succeeded in 20.0039999485s: heel2