channel 介紹

!!!1.Memory Channel 內存通道

事件將被存儲在內存中的具備指定大小的隊列中。

很是適合那些須要高吞吐量可是失敗是會丟失數據的場景下。

 

屬性說明：

!type – 類型，必須是「memory」

capacity 100 事件存儲在信道中的最大數量

transactionCapacity 100 每一個事務中的最大事件數

keep-alive 3 添加或刪除操做的超時時間

byteCapacityBufferPercentage 20 Defines the percent of buffer between byteCapacity and the estimated total size of all events in the channel, to account for data in headers. See below.

byteCapacity see description Maximum total bytes of memory allowed as a sum of all events in this channel. The implementation only counts the Event body, which is the reason for providing the byteCapacityBufferPercentage configuration parameter as well. Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of the -Xmx value passed on the command line). Note that if you have multiple memory channels on a single JVM, and they happen to hold the same physical events (i.e. if you are using a replicating channel selector from a single source) then those event sizes may be double-counted for channel byteCapacity purposes. Setting this value to 0 will cause this value to fall back to a hard internal limit of about 200 GB.

 

案例:參看入門案例

2.JDBC Channel

事件被持久存儲在可靠的數據庫中。目前支持嵌入式的Derby數據庫。若是可恢復性很是的重要能夠使用這種方式。

 

!!!3.File Channel

性能會比較低下，可是即便程序出錯數據不會丟失

屬性說明:

!type – 類型，必須是「file」

checkpointDir ~/.flume/file-channel/checkpoint 檢查點文件存放的位置

useDualCheckpoints false Backup the checkpoint. If this is set to true, backupCheckpointDir must be set

backupCheckpointDir – The directory where the checkpoint is backed up to. This directory must not be the same as the data directories or the checkpoint directory

dataDirs ~/.flume/file-channel/data 逗號分隔的目錄列表，用以存放日誌文件。使用單獨的磁盤上的多個目錄能夠提升文件通道效率。

transactionCapacity 10000 The maximum size of transaction supported by the channel

checkpointInterval 30000 Amount of time (in millis) between checkpoints

maxFileSize 2146435071 一個日誌文件的最大尺寸

minimumRequiredSpace 524288000 Minimum Required free space (in bytes). To avoid data corruption, File Channel stops accepting take/put requests when free space drops below this value

capacity 1000000 Maximum capacity of the channel

keep-alive 3 Amount of time (in sec) to wait for a put operation

use-log-replay-v1 false Expert: Use old replay logic

use-fast-replay false Expert: Replay without using queue

checkpointOnClose true Controls if a checkpoint is created when the channel is closed. Creating a checkpoint on close speeds up subsequent startup of the file channel by avoiding replay.

encryption.activeKey – Key name used to encrypt new data

encryption.cipherProvider – Cipher provider type, supported types: AESCTRNOPADDING

encryption.keyProvider – Key provider type, supported types: JCEKSFILE

encryption.keyProvider.keyStoreFile – Path to the keystore file

encrpytion.keyProvider.keyStorePasswordFile – Path to the keystore password file

encryption.keyProvider.keys – List of all keys (e.g. history of the activeKey setting)

encyption.keyProvider.keys.*.passwordFile – Path to the optional key password file

!!!4.Spillable Memory Channel -- 內存溢出通道

事件被存儲在內存隊列和磁盤中。

內存隊列做爲主存儲，而磁盤做爲溢出內容的存儲。

內存存儲經過embedded File channel來進行管理。

當內存隊列已滿時，後續的事件將被存儲在文件通道中。 

這個通道適用於正常操做期間適用內存通道已期實現高效吞吐，而在高峯期間適用文件通道實現高耐受性。經過下降吞吐效率提升系統可耐受性。

若是Agent崩潰，則只有存儲在文件系統中的事件能夠被恢復。

此通道處於試驗階段，不建議在生產環境中使用。 

 

屬性說明：

!type – 類型，必須是"SPILLABLEMEMORY"

memoryCapacity 10000 內存中存儲事件的最大值，若是想要禁用內存緩衝區將此值設置爲0。

overflowCapacity 100000000 能夠存儲在磁盤中的事件數量最大值。設置爲0能夠禁用磁盤存儲。 

overflowTimeout 3 The number of seconds to wait before enabling disk overflow when memory fills up.

byteCapacityBufferPercentage 20 Defines the percent of buffer between byteCapacity and the estimated total size of all events in the channel, to account for data in headers. See below.

byteCapacity see description Maximum bytes of memory allowed as a sum of all events in the memory queue. The implementation only counts the Event body, which is the reason for providing the byteCapacityBufferPercentage configuration parameter as well. Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of the -Xmx value passed on the command line). Note that if you have multiple memory channels on a single JVM, and they happen to hold the same physical events (i.e. if you are using a replicating channel selector from a single source) then those event sizes may be double-counted for channel byteCapacity purposes. Setting this value to 0 will cause this value to fall back to a hard internal limit of about 200 GB.

avgEventSize 500 Estimated average size of events, in bytes, going into the channel

<file channel properties> see file channel Any file channel property with the exception of ‘keep-alive’ and ‘capacity’ can be used. The keep-alive of file channel is managed by Spillable Memory Channel. Use ‘overflowCapacity’ to set the File channel’s capacity.

 

5.自定義渠道

自定義渠道須要本身實現Channel接口。

自定義Channle類及其依賴類必須在Flume啓動前放置到類加載的目錄下。

 

參數說明：

type - 本身實現的Channle類的全路徑名稱

 

 

 

----------------------------------------------

課後練習

flume從多個netcat中獲取日誌

經過Interceptors在事件中增長host信息

經過selector實現多路複用

不一樣的ip進行扇出，進入不一樣的channel和sink進行發送

存儲到本地文件系統 存儲到hdfs 存儲到logger中

 

總體結構參看圖