Namenode |
Defaultweb |
Descriptionapp |
channelless |
–ide |
|
typeoop |
–ui |
The component type name, needs to be hdfsthis |
hdfs.pathspa |
–線程 |
HDFS directory path (eg hdfs://namenode/flume/webdata/) |
hdfs.filePrefix |
FlumeData |
HDSFSink產生文件的前綴名稱,若是定義爲 MyFile,則生成文件將會是/hdfspath/MyFile.後綴 |
hdfs.fileSuffix |
– |
定義產生文件的後綴。例如要生成文本類型的.txt做爲後綴 |
hdfs.inUsePrefix |
– |
寫入HDFS過程當中,會生成一個臨時文件,inUsePrefix則定義了臨時文件的前綴 |
hdfs.inUseSuffix |
.tmp |
定義寫入HDFS過程當中,文件的後綴 |
hdfs.rollInterval |
30 |
Number of seconds to wait before rolling current file (0 = never roll based on time interval) 按時間生成HDFS文件 |
hdfs.rollSize |
1024 |
File size to trigger roll, in bytes (0: never roll based on file size) 按文件大小觸發roll,生成新的HDFS文件 |
hdfs.rollCount |
10 |
Number of events written to file before it rolled (0 = never roll based on number of events) 按寫入的event的個數觸發roll,生成新的HDFS文件 |
hdfs.idleTimeout |
0 |
Timeout after which inactive files get closed (0 = disable automatic closing of idle files) 當達到idleTimeout時,關閉空閒的文件(默認爲0,不自動關閉空閒的文件) |
hdfs.batchSize |
100 |
number of events written to file before it is flushed to HDFS HDFSEventSink以batchSize的event個數做爲一個事務處理,統一flush到HDFS中,提交事務 |
hdfs.codeC |
– |
Compression codec. one of following : gzip, bzip2, lzo, lzop, snappy 設置具體的壓縮格式,在配置fileType爲CompressedStream時,須要設置CodeC |
hdfs.fileType |
SequenceFile |
File format: currently SequenceFile, DataStream or CompressedStream (1)DataStream will not compress output file and please don’t set codeC (2)CompressedStream requires set hdfs.codeC with an available codeC 在生成HDFSWriter的具體實現時,經過該參數制定HDFSWriter的實現類。 三個格式分別對應HDFSSequenceFile,HDFSDataStream,HDFSCompressedDataStream |
hdfs.maxOpenFiles |
5000 |
Allow only this number of open files. If this number is exceeded, the oldest file is closed. HDFSSink中維護了與HDFS建立的文件的鏈接(每一個文件對應一個BucketWriter), 若是超過該值,HDFSSink將會關閉建立最久的BucketWriter |
hdfs.minBlockReplicas |
– |
Specify minimum number of replicas per HDFS block. If not specified, it comes from the default Hadoop config in the classpath. 設置HDFS塊的副本數,若是沒有特殊設置,默認採用Hadoop的配置 |
hdfs.writeFormat |
Writable |
Format for sequence file records. One of Text or Writable. Set to Text before creating data files with Flume, otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive. |
hdfs.callTimeout |
10000 |
Number of milliseconds allowed for HDFS operations, such as open, write, flush, close. This number should be increased if many HDFS timeout operations are occurring. |
hdfs.threadsPoolSize |
10 |
Number of threads per HDFS sink for HDFS IO ops (open, write, etc.) 處理HDFS相關操做的線程數 |
hdfs.rollTimerPoolSize |
1 |
Number of threads per HDFS sink for scheduling timed file rolling 對HDFS文件進行roll操做的線程數 |
hdfs.kerberosPrincipal |
– |
Kerberos user principal for accessing secure HDFS |
hdfs.kerberosKeytab |
– |
Kerberos keytab for accessing secure HDFS |
hdfs.proxyUser |
|
|
hdfs.round |
false |
Should the timestamp be rounded down (if true, affects all time based escape sequences except %t) |
hdfs.roundValue |
1 |
Rounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than current time. |
hdfs.roundUnit |
second |
The unit of the round down value - second, minute or hour. |
hdfs.timeZone |
Local Time |
Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles. |
hdfs.useLocalTimeStamp |
false |
Use the local time (instead of the timestamp from the event header) while replacing the escape sequences. |
hdfs.closeTries |
0 |
Number of times the sink must try renaming a file, after initiating a close attempt. If set to 1, this sink will not re-try a failed rename (due to, for example, NameNode or DataNode failure), and may leave the file in an open state with a .tmp extension. If set to 0, the sink will try to rename the file until the file is eventually renamed (there is no limit on the number of times it would try). The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart. |
hdfs.retryInterval |
180 |
Time in seconds between consecutive attempts to close a file. Each close call costs multiple RPC round-trips to the Namenode, so setting this too low can cause a lot of load on the name node. If set to 0 or less, the sink will not attempt to close the file if the first attempt fails, and may leave the file open or with a 」.tmp」 extension. |
serializer |
TEXT |
Other possible options include avro_event or the fully-qualified class name of an implementation of the EventSerializer.Builder interface. |
serializer.* |