安裝ruby

首先經過 /etc/issue 命令查看當前使用centos是哪一個版本：mysql

[hadoop@hadoop03 ~]$ cat /etc/issue

因爲centos版本是6.6，安裝ruby時就要選擇在centos 6.X環境，具體安裝步驟參考以下所示便可！linux

yum install gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison iconv-devel wget tar

cd ~/

wget https://ruby.taobao.org/mirrors/ruby/ruby-2.2.3.tar.gz

tar xvf ruby-2.2.3.tar.gz

cd ruby-2.2.3

./configure

makemake install

查看驗證c++

[root@hadoop02 ~]# ruby -v

ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]

安裝fluent-plugin-sql插件(輸入源)

[root@hadoop03 ~]# gem install fluent-plugin-sql

64a08fad70766b661e4505903ec8e023b1b45128

準備MySQL表及數據

在test數據庫建立一張表，建表語句以下：

use test; 進入test數據庫裏操做

create table test_fluent

(

id int unsigned not null auto_increment,

sex varchar(1),

name varchar(225),

primary key(id)

)engine=innodb default charset=utf8 auto_increment=1;

其中id是主鍵，自增

插入數據，插入語句以下命令：

insert into test_fluent(sex,name) values('f','dongdong');

insert into test_fluent(sex,name) values('m','heihei');

insert into test_fluent(sex,name) values('m','qingsong');

insert into test_fluent(sex,name) values('f','jiafu');

insert into test_fluent(sex,name) values('m','angrybaby');

insert into test_fluent(sex,name) values('f','jack');

insert into test_fluent(sex,name) values('f','helloword');

insert into test_fluent(sex,name) values('m','sunlongfei');

insert into test_fluent(sex,name) values('m','donglang');

insert into test_fluent(sex,name) values('f','deguang');

insert into test_fluent(sex,name) values('m','yuanijng');

insert into test_fluent(sex,name) values('f','yangqun');

備註：

因爲我操做的MySQL數據庫位於172.16.1.156機器上，用戶名是dong，密碼是123456

而我安裝的fluent位於172.16.1.158機器上，不在一臺機器上，若是要從158機器遠程訪問156機器上MySQL會受限，禁止訪問。

爲此，須要在156機器上執行如下命令給指定IP受權。

受權158機器上的dong用戶能夠遠程訪問156上MySQL，dong用戶登陸密碼是123456

grant ALL PRIVILEGES ON *.* to dong@"172.16.1.158" identified by "123456" WITH GRANT OPTION;

使剛授予權限當即生效

flush privileges;

準備ODPS測試表

建立ODPS 表爲 demo_access_log，其建表語句爲：

drop table if exists demo_access_log;

create table demo_access_log(

sex string,

name string)

into 5 shards hublifecycle 7;

編輯fluent.conf配置文件

配置mysql輸入源、ODPS輸出源：

state_file /var/run/fluentd/sql_state 配置項 (path to a file to store last rows該文件默認不存在，須要提早建立好！)sql

state_file stores last selected rows to a file (named state_file) to not forget last row when Fluentd restarts.數據庫

[root@hadoop03 ~]# vi /etc/fluent/fluent.conf --編輯fluent.conf配置文件centos

@type sql

host 172.16.1.156

port 3306

database test

adapter mysql

username dong

password 123456

select_interval 10s

select_limit 10

state_file /var/run/fluentd/sql_state

<table>

table test_fluent

tag in.sql

update_column id

</table>

</source>

type aliyun_odps

aliyun_access_id UQV2yoSSWNgquhhe

aliyun_access_key bG8xSLwhmKYRmtBoE3HbhOBYXvknG6

aliyun_odps_endpoint http://service.odps.aliyun.com/api

aliyun_odps_hub_endpoint http://dh.odps.aliyun.com

buffer_chunk_limit 2m

buffer_queue_limit 128

flush_interval 5s

project dtstack_dev

table demo_access_log

fields sex,name

shard_number 5

</table>

</match>

啓動fluent

fluentd啓動時會自動加載/etc/fluent/fluent.conf中讀取fluent.conf配置文件

fluentd --啓動命令

大概 5 分鐘後，實時導入數據會被同步到離線表，可使用 select count(*) from demo_access_log這樣sql 語句進行驗證。

若是安裝Fluentd 用的是Ruby Gem，能夠建立一個配置文件運行下面命令。發出一個終止信號將會從新安裝配置文件。（若是修改了配置文件—fluent.conf 文件，ctrl c 終止進程，而後在配置文件下從新啓動）api

$ ctrl c ruby

$ fluentd -c fluent.confide

若是有相似以下輸出，就能夠說明數據實時寫入Datahub服務已經成功。

30b7a6d16985950389ab9e0989c8815653bb3927

運行過程遇到異常及排查

(1) 異常描述：fluent.conf文件沒有配置正確

0f23223c004226c6e901b2f4e83e9693c8405e0f

異常產生緣由：輸入端沒有配置 tag，輸出端table上也沒有制定對應tag。輸入tag，在輸出match時要能

匹配上，在輸出table 要能對應上才行。

解決方法：在mysql輸入源上添加上tag標籤，即 tag in.sql

(2)在fluent.conf配置正確基礎上運行fluentd啓動命令，又報如下異常：

/usr/local/lib/ruby/gems/2.2.0/gems/activerecord- 4.2.6/lib/active_record/connection_adapters/connection_specification.rb:177:in `rescue in spec': Specified 'mysql' for database adapter, but the gem is not loaded. Add `gem 'mysql'` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord). (Gem::LoadError)

這個問題是mysql插件須要用到mysql adapter適配器，須要安裝mysql adapter適配器，執行如下命令：

[root@hadoop03 fluent]# yum install mysql-devel

[root@hadoop03 fluent]# gem install mysql

Building native extensions. This could take a while...

Successfully installed mysql-2.9.1

Parsing documentation for mysql-2.9.1

Installing ri documentation for mysql-2.9.1

Done installing documentation for mysql after 2 seconds

1 gem installed

鏈接數據庫適配器路徑：

mysql2_adapter.rb、mysql_adapter.rb、 postgresql_adapter.rb在

/usr/local/lib/ruby/gems/2.2.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters目錄下

gem安裝插件時遇到異常及排查

ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)

Errno::ECONNRESET: Connection reset by peer - SSL_connect (https://api.rubygems.org/quick/Marshal.4.8/cool.io-1.4.4.gemspec.rz)

87e96aeca2f6d3af42fc917d1aa087dbbb056add

異常產生緣由：

因爲gem源引發，須要加上淘寶源後要把原來那個rubygems那個刪掉

解決方法：

# 刪除默認的官方源

gem sources -r https://rubygems.org/

# 添加淘寶源

gem sources -a https://ruby.taobao.org/

具體解決截圖展現以下：

a8c4b9469fe22a92ee1de6a1fb3269c44b251ba3

第二種異常

[hadoop@hadoop03 ~]$ gem install fluent-plugin-aliyun-odps

ERROR: While executing gem ... (Gem::FilePermissionError)

You don't have write permissions for the /usr/local/lib/ruby/gems/2.2.0 directory.