Airflow 配置celery+rabbitmq和celery+redis

Airflow 配置celery+rabbitmq

一、安裝celery和rabbitmq組件

pip3 install apache-airflow[celery]

pip3 install apache-airflow[rabbitmq]

二、系統安裝rabbit

  1）安裝 erlang

yum install -y erlang

  若是報錯以下：

[ ]# yum install erlang
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
No package erlang available.
Error: Nothing to do

  則須要安裝第三方軟件庫 epel：

# 下載
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# 安裝完成便可
rpm -ivh epel-release-latest-7.noarch.rpm

  再次執行則能夠成功安裝erlang了！

  2）安裝 rabbitmq

wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-3.6.15-1.el6.noarch.rpm
rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
yum install -y rabbitmq-server-3.6.15-1.el6.noarch.rpm

 

  3）配置rabbitmq（設置用戶名root，密碼rabbitmq，建立虛擬主機airflow-rabbitmq）

[root@centos-master airflow]# /sbin/service rabbitmq-server stop
Stopping rabbitmq-server (via systemctl):                  [  OK  ]
[root@centos-master airflow]# /sbin/service rabbitmq-server start
Starting rabbitmq-server (via systemctl):                  [  OK  ]
[root@centos-master airflow]# rabbitmqctl add_user root rabbitmq
Creating user "root"
[root@centos-master airflow]# rabbitmqctl add_vhost airflow-rabbitmq
Creating vhost "airflow-rabbitmq"
[root@centos-master airflow]# rabbitmqctl set_user_tags root airflow-rabbitmq
Setting tags for user "root" to ['airflow-rabbitmq']
[root@centos-master airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"
Setting permissions for user "root" in vhost "airflow-rabbitmq"
[root@centos-master airflow]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  amqp_client
  cowlib
  cowboy
  rabbitmq_web_dispatch
  rabbitmq_management_agent
  rabbitmq_management

 

  4）修改airflow配置

[root@centos-master airflow]# vi airflow.cfg 
    executor = CeleryExecutor
    broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq
    result_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

　[root@centos-master airflow]# echo "export C_FORCE_ROOT="True"" >> /etc/profile
　[root@centos-master airflow]# source /etc/profile

 

 

Airflow 配置celery+redis

 

1.安裝redis

yum -y install redis

 

 

 

2.修改配置文件

sudo vim /etc/redis.conf
#bind 127.0.0.1     // 註釋掉，使redis容許遠程訪問
requirepass redis  // 修改這行，redis登陸密碼（自定義）

 

 

3.開啓redis服務

systemctl start redis

 

4.python下安裝redis庫

pip3 install redis

 

5.安裝celery

pip3 install celery

 

6.配置airflow.cfg

executor = CeleryExecutor
broker_url = redis://127.0.0.1:6379/0
result_backend = redis://127.0.0.1:6379/0