Druid學習之路 （四）Druid的數據採集格式

做者：Syn良子 出處：http://www.javashuo.com/article/p-yxqufjvn-hq.html 轉載請註明出處

Druid的數據採集格式

---

Druid能夠採集非標準化的數據諸如JSON,CSV或者以某種分隔符隔開的TSV格式,固然還支持自定義格式.雖然大部分的文檔使用JSON格式,可是經過druid來配置支持其餘的限定格式也不是很難.

當前支持的格式化數據

---

1. 列表項

JSON

{"timestamp": "2013-08-31T01:02:33Z", "page": "Gypsy Danger", "language" : "en", "user" : "nuclear", "unpatrolled" : "true", "newPage" : "true", "robot": "false", "anonymous": "false", "namespace":"article", "continent":"North America", "country":"United States", "region":"Bay Area", "city":"San Francisco", "added": 57, "deleted": 200, "delta": -143}

CSV

2013-08-31T01:02:33Z,"Gypsy Danger","en","nuclear","true","true","false","false","article","North America","United States","Bay Area","San Francisco",57,200,-143

TSV

2013-08-31T01:02:33Z    "Gypsy Danger"  "en"    "nuclear"   "true"  "true"  "false" "false" "article"   "North America" "United States" "Bay Area"  "San Francisco" 57  200 -143

須要注意的是CSV,TSV不能包含列頭,這點在數據採集的時候必定要注意

自定義格式

---

Druid支持使用正則解析和JavaScript來自定義數據格式.可是這種方式並無本身實現的Java解析器或者額外的流式處理工具效率更高.

配置數據採集的schema

---

什麼是data schema?其實就是Druid的index數據攝取任務須要的數據源的描述的元數據.它主要描述要採集的數據類型,數據由哪些列構成,哪些是指標列,哪些是維度列,時間的粒度等.

以CSV格式舉例

"parseSpec": {
"format" : "csv",
"timestampSpec" : {
  "column" : "timestamp"
},
"columns" : ["timestamp","page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city","added","deleted","delta"],
"dimensionsSpec" : {
  "dimensions" : ["page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city"]
}}

parseSpec指明瞭數據源格式,這裏是format中代表是CSV格式,而後說明時間戳字段名是timestamp,數據字段名是columns裏面那一堆,dimensionsSpec則表明哪些字段能夠做爲維度.

參考資料:Druid的數據格式