gearman mysql持久化

時間 2019-12-01

原文原文鏈接

gearman 建立Mysql持久化隊列的方式以下：php

啓動gearman，命令以下：html

gearmand命令：mysql

-b, –backlog=BACKLOG 鏈接請求隊列的最大值
-d, –daemon Daemon 守護進程化
-f, –file-descriptors=FDS 可打開的文件描述符數量
-h, –help
-l, –log-file=FILE Log 日誌文件
-L, –listen=ADDRESS 開啓監聽的地址
-p, –port=PORT 開啓監聽的端口
-P, –pid-file=FILE File pid file
-r,–protocol=PROTOCOL 使用的協議
-q, –queue-type=QUEUE 持久化隊列類型 -t, –threads=THREADS I/O線程數量
-u, –user=USER 進程的有效用戶名
libdrizzle Options:
--libdrizzle-host=HOST Host of server.
--libdrizzle-port=PORT Port of server.
--libdrizzle-uds=UDS Unix domain socket for server.
--libdrizzle-user=USER User name for authentication.
--libdrizzle-password=PASSWORD Password for authentication.
--libdrizzle-db=DB Database to use.
--libdrizzle-table=TABLE Table to use.
--libdrizzle-mysql Use MySQL protocol.
libmemcached Options:
--libmemcached-servers=SERVER_LIST List of Memcached servers to use.
libsqlite3 Options:
--libsqlite3-db=DB Database file to use.
--libsqlite3-table=TABLE Table to use.
libpq Options:
--libpq-conninfo=STRING PostgreSQL connection information string.
--libpq-table=TABLE Table to use.
http Options:
--http-port=PORT Port to listen on.

/usr/local/gearman/sbin/gearmand -p 4730 -L 0.0.0.0 --log-file=/tmp/gearmand-4730.log --pid-file=/tmp/gearmand-4730.pid -q MySQL --mysql-host=localhost --mysql-user=root --mysql-db=gearman --verbose DEBUG-d

我本身看gearmand 參數與上面不一樣：

[root@localhost sbin]# ./gearmand  -h
Allowed options:

Allowed options:

General options:
  -b [ --backlog ] arg (=32)                         Number of backlog 
                                                     connections for listen.
  -d [ --daemon ]                                    Daemon, detach and run in 
                                                     the background.
  --exceptions                                       Enable protocol exceptions
                                                     by default.
  -f [ --file-descriptors ] arg                      Number of file descriptors
                                                     to allow for the process 
                                                     (total connections will be
                                                     slightly less). Default is
                                                     max allowed for user.
  -h [ --help ]                                      Print this help menu.
  -j [ --job-retries ] arg (=0)                      Number of attempts to run 
                                                     the job before the job 
                                                     server removes it. This is
                                                     helpful to ensure a bad 
                                                     job does not crash all 
                                                     available workers. Default
                                                     is no limit.
  --job-handle-prefix arg                            Prefix used to generate a 
                                                     job handle string. If not 
                                                     provided, the default 
                                                     "H:<host_name>" is used.
  --hashtable-buckets arg (=991)                     Number of buckets in the 
                                                     internal job hash tables. 
                                                     The default of 991 works 
                                                     well for about three 
                                                     million jobs in queue. If 
                                                     the number of jobs in the 
                                                     queue at any time will 
                                                     exceed three million, use 
                                                     proportionally larger 
                                                     values (991 * # of jobs / 
                                                     3M). For example, to 
                                                     accomodate 2^32 jobs, use 
                                                     1733003. This will consume
                                                     ~26MB of extra memory. 
                                                     Gearmand cannot support 
                                                     more than 2^32 jobs in 
                                                     queue at this time.
  --keepalive                                        Enable keepalive on 
                                                     sockets.
  --keepalive-idle arg (=-1)                         If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPIDLE for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  --keepalive-interval arg (=-1)                     If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPINTVL for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  --keepalive-count arg (=-1)                        If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPCNT for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  -l [ --log-file ] arg (=/opt/var/log/gearmand.log) Log file to write errors 
                                                     and information to. If the
                                                     log-file parameter is 
                                                     specified as 'stderr', 
                                                     then output will go to 
                                                     stderr. If 'none', then no
                                                     logfile will be generated.
  -L [ --listen ] arg                                Address the server should 
                                                     listen on. Default is 
                                                     INADDR_ANY.
  -P [ --pid-file ] arg (=/opt/var/gearmand.pid)     File to write process ID 
                                                     out to.
  -r [ --protocol ] arg                              Load protocol module.
  -R [ --round-robin ]                               Assign work in round-robin
                                                     order per worker 
                                                     connection. The default is
                                                     to assign work in the 
                                                     order of functions added 
                                                     by the worker.
  -q [ --queue-type ] arg (=builtin)                 Persistent queue type to 
                                                     use.
  --config-file arg (=/opt/etc/gearmand.conf)        Can be specified with 
                                                     '@name', too
  --syslog                                           Use syslog.
  --coredump                                         Whether to create a core 
                                                     dump for uncaught signals.
  -t [ --threads ] arg (=4)                          Number of I/O threads to 
                                                     use, 0 means that gearmand
                                                     will try to guess the 
                                                     maximum number it can use.
                                                     Default=4.
  -u [ --user ] arg                                  Switch to given user after
                                                     startup.
  --verbose arg (=ERROR)                             Set verbose level (FATAL, 
                                                     ALERT, CRITICAL, ERROR, 
                                                     WARNING, NOTICE, INFO, 
                                                     DEBUG).
  -V [ --version ]                                   Display the version of 
                                                     gearmand and exit.
  -w [ --worker-wakeup ] arg (=0)                    Number of workers to 
                                                     wakeup for each job 
                                                     received. The default is 
                                                     to wakeup all available 
                                                     workers.

HTTP:
  --http-port arg (=8080) Port to listen on.

Gear:
  -p [ --port ] arg (=4730) Port the server should listen on.
  --ssl                     Enable ssl connections.
  --ssl-ca-file arg         CA file.
  --ssl-certificate arg     SSL certificate.
  --ssl-key arg             SSL key for certificate.

builtin:

MySQL:
  --mysql-host arg (=localhost)      MySQL host.
  --mysql-port arg (=3306)           Port of server. (by default 3306)
  --mysql-user arg                   MySQL user.
  --mysql-password arg               MySQL user password.
  --mysql-db arg                     MySQL database.
  --mysql-table arg (=gearman_queue) MySQL table name.

/sbin/gearmand -p 4730 -L 0.0.0.0 -p 4730  -q MySQL --mysql-host=localhost  --mysql-port 3306 
--mysql-user=root --mysql-password pwd --mysql-db=gearman --verbose DEBUG -d

爲了安全起見，能夠把-L指定爲127.0.0.0

執行上面的錯誤：./gearmand: Error while initializing the queue : MySQLlinux

具體緣由查看日誌：gearmand.log。顯示：redis

DEBUG 2014-12-05 06:49:31.300168 [ main ] THREADS: 4 -> libgearman-server/gearmand.cc:263
INFO 2014-12-05 06:49:31.300259 [ main ] Initializing MySQL module
ERROR 2014-12-05 06:49:31.341972 [ main ] Failed to connect to database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) -> libgearman-server/plugins/queue/mysql/queue.cc:228
ERROR 2014-12-05 06:49:31.342008 [ main ] Failed to initialize mysql: initialize(QUEUE_ERROR) -> libgearman-server/queue.cc:246
DEBUG 2014-12-05 06:49:31.342081 [ main ] Shutting down all threads -> libgearman-server/gearmand.cc:276
DEBUG 2014-12-05 06:49:31.342093 [ main ] MySQL queue flush -> libgearman-server/plugins/queue/mysql/queue.cc:370
DEBUG 2014-12-05 06:49:31.342103 [ main ] removing queue: FUNCTION -> libgearman-server/gearmand.cc:178
INFO 2014-12-05 06:49:31.342112 [ main ] Shutdown complete
[root@localhost log]# pwdsql

我去/var/lib/mysql/這個目錄沒有mysql.sock這個文件。數據庫

因爲mysql 默認的mysql.sock 是在/var/lib/mysql/mysql.sock，建立符號鏈接：、（參考文章：安全

Can't connect to local MySQL server through socket '/tmp/mysql.sock' (11)服務器

解決不能經過mysql .sock鏈接MySQL問題 這個問題主要提示是，不能經過 '/tmp/mysql .sock'連到服務器，而php標準配置正是用過'/tmp/mysql .sock'，可是一些mysql 安裝方法 將 mysql .sock放在/var/lib/mysql .sock或者其餘的什麼地方，你能夠經過修改/etc/my.cnf文件來修正它，打開文件，能夠看到以下的東東： 
　　[mysql d] 
　　socket=/var/lib/mysql .sock 
　　改一下就行了，但也會引發其餘的問題，如mysql 程序連不上了，再加一點： 
　　[mysql ] 
　　socket=/tmp/mysql .sock 
　　或者還能夠經過修改php.ini中的配置來使php用其餘的mysql .sock來連，這個你們本身去找找
　　
　　或者用這樣的方法:
　　ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock 

成功了,就是這樣ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock 

）

locate mysql.sock所在目錄。而後建立鏈接。less

ln -s /data/xx/data/mysql.sock /var/lib/mysql/mysql.sock

在 /var/lib/mysql/建立鏈接後，再運行，果真成功。成功自動建立gearman_queue表。表結構：

create table `gearman_queue` (
`unique_key` varchar(64) NOT NULL,
`function_name` varchar(255) NOT NULL,
`priority` int(11) NOT NULL,
`data` LONGBLOB NOT NULL,
`when_to_run` INT, PRIMARY KEY  (`unique_key`)
)

具體的參數，根據本身的服務器情況去修改就好了。

3. 再次登入mysql命令行，執行：

use gearman;show tables;

就能夠看到下面多了個「gearman_queue」的表。

這樣，gearman就變成了持久化的方式。

=============================================================================

gearman用mysql持久化以後，其實會帶來一些問題。

1. 每一個任務都會寫入數據庫，這樣會帶來磁盤IO的損耗，而且gearman的性能瓶頸又多了一個可能，就是由數據庫引發的性能問題。

2. mysql有個「wait_timeout」的參數，在mysql命令行中運行

show variables like "%timeout%";

能夠看到wait_timeout的值，默認是28800。也就是說，若是一個mysql的鏈接，超過28800s沒有任何響應，就會斷開。

3. gearman持久化的方式，若是超過了mysql的wait_timeouts的時間沒有任何響應，和數據庫的鏈接就會被mysql斷開，並且gearman目前是沒有mysql從新鏈接的，結果就是，會致使以下錯誤，必須重啓gearman才能從新正常工做。

gearman報錯

ERROR 2014-04-01 02:10:02.897899 [ proc ] mysql_stmt_execute failed: -> libgearman-server/plugins/queue/mysql/queue.cc:357
ERROR 2014-04-01 02:10:02.897910 [ proc ] gearman_server_job_add gearman_server_run_command(QUEUE_ERROR) -> libgearman-server/server.cc:301

因此，gearman持久化方式帶來的缺點顯而易見，該方式下，要避免gearman對mysql鏈接超時斷開，能夠將mysql的wait_timeout參數改大。

或者，直接放棄用mysql的持久化方式。

持久化隊列將後臺做業存放在一個外部持久的隊列中。持久化隊列只對後臺jobs有效，由於前臺jobs依附於客戶端。若是job服務器擋掉了，客戶端會檢測到，將會從其餘地方從新啓動這個前臺job或者返回錯誤。然後臺jobs沒有依附於客戶端，若是要想讓它運行則須要提交。

The persistent queue works by calling a module callback function right before putting a new job in the internal queue for pending jobs to be run. This allows the module to store the job about to be run in some persistent way so that it can later be replayed during a restart. Once it is stored through the module, the job is put onto the active runnable queue, waking up available workers if needed. Once the job has been successfully completed by a worker, another module callback function is called to notify the module the job is done and can be removed. If a job server crashes or is restarted between these two calls for a job, the jobs are reloaded during the next job server start. When the job server starts up, it will call a replay callback function in the module to provide a list of all jobs that were not complete. This is used to populate the internal memory queue of jobs to be run. Once this replay is complete, the job server finishes its initialization and the jobs are now runnable once workers connect (the queue should be in the same state as when it crashed). These jobs are removed from the persistent queue when completed as normal. NOTE: Deleting jobs from the persistent queue storage will not remove them from the in-memory queue while the server is running.

建立一個後臺job

 
     ./gearman -f sayHello -b  xx00 
   
    -f函數名 -b 後臺  xx00爲數據， 
   
    執行後。 
   
     數據表多了一條記錄： 
   
執行隊列中的job：
 ./gearman -f sayHello  -w

    gearmanMonitor能夠可視化顯示：