Gearman + Nodejs + MySQL UDF異步實現 MySQL 到 Redis 的數據同步

目錄

• 1, 環境
• 2, Redis簡介
• 3, Gearman簡介
• 4, MySQL - Redis配合使用方案
• 4, 軟件安裝
• 5, MySQL UDF + Trigger同步數據到Gearman
• 6, 參考資料

1, 環境

CentOS, MySQL, Redis, Nodejs

2, Redis簡介

Redis是一個開源的K-V內存數據庫，它的key能夠是string/set/hash/list/...，由於是基於內存的，所在訪問速度至關快。

3, Gearman簡介

Gearman是一個開源的Map/Reduce分佈式計算框架，具備豐富的client sdk，並且它支持MySQL UDF。

Gearman工做圖

Gearman調用流程

Gearman集羣

從圖中能夠看出貌似Gearman的集羣功能比較弱，Job Server之間沒啥關係好像。

4, MySQL - Redis配合使用方案

首先咱們以MySQL數據爲主，將insert/update/delete交給MySQL，而select交給redis；
當有數據發生變化時，經過MySQL Trigger實時異步調用Gearman的UDF提交一個job給Job Server，當job執行的時候會去更新redis；從而保證redis與MySQL中的數據是同步的。

4, 軟件安裝

安裝gearman

//安裝依賴
$ yum install -y boost-devel gperf libevent-devel libuuid-devel

//下載gearman
$ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz
$ tar zxvf gearmand-1.1.12.tar.gz

//編譯安裝，指定mysqlclient的連接路徑
$ ./configure LDFLAGS=-L/usr/lib64/mysql/
$ make
$ make install

//啓動gearmand服務端
$ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d

//查看是否安裝成功，查看gearman版本
$ /usr/local/sbin/gearmand -V

gearman client

gearman client有不少種：PHP, Perl, Python, Ruby, Nodejs, Go, Java, C#...，咱們選擇GearmaNode(Nodejs庫)做爲實驗用的gearman client

Nodejs的安裝省略

GearmaNode的安裝及測試

安裝GearmaNode

//安裝GearmaNode
$ npm install gearmanode

啓動Gearman Job Server

//確保gearman的job server已經啓動, job server默認監聽4730端口
$ netstat -nplt | grep 4730

//若是沒有，則啓動job server（-d表示啓動, -l指定log的位置）
$ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d

worker.js (Gearman Job Worker)

var gearmanode = require('gearmanode');
var worker = gearmanode.worker();
//對job client傳遞過來的字符串執行reverse倒序操做
worker.addFunction('reverse', function (job) {
    job.sendWorkData(job.payload); // mirror input as partial result
    job.workComplete(job.payload.toString().split("").reverse().join(""));
});

//啓動job worker
$ node worker.js

client.js (Gearman Job Client)

var gearmanode = require('gearmanode');
var client = gearmanode.client();
//提交job
var job = client.submitJob('reverse', 'hello world!');
job.on('workData', function(data) {
    console.log('WORK_DATA >>> ' + data);
});
//結果回調函數
job.on('complete', function() {
    console.log('RESULT >>> ' + job.response);
    client.close();
});

//執行client.js
$ node client.js
//查看控制檯打印輸出

WORK_DATA >>> hello world!
RESULT >>> !dlrow olleh

能夠看到gearman安裝成功，運行正常。

5, MySQL UDF + Trigger同步數據到Gearman

安裝lib_mysqludf_json

lib_mysqludf_json能夠把MySQL表的數據以json數據格式輸出

//下載lib_mysqludf_json
$ git clone https://github.com/mysqludf/lib_mysqludf_json.git
$ cd lib_mysqludf_json

//編譯
$ gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

//拷貝lib_mysqludf_json.so到MySQL的plugin目錄
//能夠登錄MySQL，輸入命令"show variables like '%plugin%'"查看plugin位置
$ cp lib_mysqludf_json.so /usr/lib64/mysql/plugin/

//演示lib_mysqludf_json功能
$ mysql -uname -hhost -ppwd
//首先註冊UDF函數
mysql> CREATE FUNCTION json_object RETURNS STRING 
       SONAME "lib_mysqludf_json.so";
//json_array|json_members|json_values函數註冊方式與json_object同樣.
mysql> use test;
mysql> select * from user_list;
+------+----------+
| NAME | PASSWORD |
+------+----------+
| troy | pwd      |
+------+----------+
mysql> select json_object(name,password) as user from user_list;
+----------------------------------+
| user                             |
+----------------------------------+
| {"name":"troy","password":"pwd"} |
+----------------------------------+

安裝gearman-mysql-udf

//下載
$ wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz
$ tar zxvf gearman-mysql-udf-0.6.tar.gz
$ cd gearman-mysql-udf-0.6

//安裝libgearman-devel
$ yum install libgearman-devel

//編譯安裝
$ ./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
$ make && make install

//登陸MySQL註冊UDF函數
mysql> CREATE FUNCTION gman_do_background RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
mysql> CREATE FUNCTION gman_servers_set RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
//函數gman_do|gman_do_high|gman_do_low|gman_do_high_background|gman_do_low_background|gman_sum註冊方式相似，請參考gearman-mysql-udf-0.6/README

//指定gearman job server地址
mysql> SELECT gman_servers_set('127.0.0.1:4730');

若是出現異常信息：
ERROR 1126 (HY000): Can't open shared library 'libgearman_mysql_udf.so' (errno: 11 libgearman.so.8: cannot open shared object file: No such file or directory)
表示系統找不到 libgearman.so 文件，通常so都在/usr/local/lib目錄下，修改配置文件/etc/ld.so.conf，將/usr/local/lib目錄加入進去便可:

$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
/usr/local/lib
$ /sbin/ldconfig -v | grep gearman*

MySQL Trigger調用Gearman UDF實現同步

DELIMITER $$
CREATE TRIGGER user_list_data_to_redis AFTER UPDATE ON user_list
  FOR EACH ROW BEGIN
    SET @ret=gman_do_background('syncToRedis', json_object(NEW.name as 'name', NEW.password as 'password')); 
  END$$
DELIMITER ;

Gearman Nodejs Worker將數據異步複製到redis

redis的安裝步驟省略
啓動redis-server

$ redis-server

安裝redis的Nodejs client:

npm install hiredis redis

redis-worker.js (Gearman Nodejs worker)

var redis = require("redis");
var gearmanode = require('gearmanode');
var worker = gearmanode.worker();

//添加gearman函數syncToRedis
//當MySQL表記錄更改時，此函數會被調用
worker.addFunction('syncToRedis', function (job) {
    job.sendWorkData(job.payload);
    console.log("-------job.payload: " + job.payload.toString());
    //將字符串轉換成json object, 而後調用更新redis
    updateRedis(eval('(' + job.payload.toString() + ')'));
    job.workComplete("Successed!");
});

//些函數只是簡單的將MySQL表中的一行的記錄按單個字段更新到redis中。可根據實際狀況自行擴展
function updateRedis(json)
{
    var client = redis.createClient();
    client.on("error", function (err) {
        console.log("Error " + err);
    });
    for(var key in json)
    {
        client.set(key, json[key], redis.print);
    }
    client.quit();
}

啓動worder:

node redis-worker.js

更新MySQL

mysql> select * from user_list;
+------+----------+
| NAME | PASSWORD |
+------+----------+
| troy | jane     |
+------+----------+
mysql> update user_list set name='hello', password='world';

redis-worker控制檯輸出：

-------job.payload: {"name":"hello","password":"world"}

查看redis數據

$ redis-cli
127.0.0.1:6379> keys *
1) "password"
2) "name"
127.0.0.1:6379> get name
"hello"
127.0.0.1:6379> get password
"world"

能夠看到只要MySQL中user_list表的數據被update了，redis中的數據也會被更新。

6, 參考資料

Gearman+PHP+MySQL UDF的組合異步實現MySQL到Redis的數據複製