Hadoop-MR實現日誌清洗（一）

1.日誌內容樣式

目前所接觸到的日誌一種是網頁請求日誌，一種是埋點日誌，一種後端系統日誌。

1.1請求日誌

請求日誌是用戶訪問網站時，打開網址或點擊網站上了項目元素時，向服務器發送或提交的資源請求。

（論壇日誌）

27.38.53.84 - - [30/May/2013:23:37:57 +0800] "GET /uc_server/data/avatar/000/00/50/90_avatar_small.jpg HTTP/1.1" 200 1828
218.28.247.140 - - [30/May/2013:23:37:57 +0800] "GET /static/image/common/swfupload.swf?preventswfcaching=1369928282717 HTTP/1.1" 200 13333
123.147.245.79 - - [30/May/2013:23:37:57 +0800] "GET /static/js/swfupload.queue.js?y7a HTTP/1.1" 304 -
182.242.227.232 - - [30/May/2013:23:37:56 +0800] "GET /misc.php?mod=patch&action=ipnotice&inajax=1&ajaxtarget=ip_notice HTTP/1.1" 200 65
183.67.254.204 - - [30/May/2013:23:37:56 +0800] "POST /forum.php?mod=post&action=newthread&fid=72&extra=&topicsubmit=yes&inajax=1 HTTP/1.1" 200 425
110.255.113.85 - - [30/May/2013:23:37:59 +0800] "GET /uc_server/avatar.php?uid=26294&size=middle HTTP/1.1" 301 -
111.37.4.243 - - [30/May/2013:23:37:58 +0800] "POST /source/plugin/pcmgr_url_safeguard/url_api.inc.php HTTP/1.1" 200 1300
125.82.229.229 - - [30/May/2013:23:38:05 +0800] "GET /uc_server/data/avatar/000/07/18/34_avatar_middle.jpg HTTP/1.1" 200 3790
122.70.237.247 - - [30/May/2013:23:38:03 +0800] "GET /forum.php?mod=image&aid=18696&size=300x300&key=3e12991ed5ff7ecd&nocache=yes&type=fixnone&ramdom=dZqQb HTTP/1.1" 200 39594
111.37.4.243 - - [30/May/2013:23:38:04 +0800] "GET /forum.php?mod=misc&action=postreview&do=support&tid=11228&pid=44989&hash=29c64660&infloat=yes&handlekey=login&referer=http%3A%2F%2Fbbs.itcast.cn%2Fforum.php%3Fmod%3Dviewthread%26tid%3D11228&inajax=1&ajaxtarget=fwin_content_login HTTP/1.1" 302 -
49.5.1.14 - - [30/May/2013:23:38:09 +0800] "GET /api/connect/like.php HTTP/1.1" 200 722

 

（商城日誌）

183.49.46.228 - - [18/Sep/2013:06:49:23 +0000] "-" 400 0 "-" "-"
163.177.71.12 - - [18/Sep/2013:06:49:33 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0"
163.177.71.12 - - [18/Sep/2013:06:49:36 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0"
60.208.6.156 - - [18/Sep/2013:06:49:48 +0000] "GET /wp-content/uploads/2013/07/rcassandra.png HTTP/1.0" 200 185524 "http://cos.name/category/software/packages/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36"
222.68.172.190 - - [18/Sep/2013:06:49:57 +0000] "GET /images/my.jpg HTTP/1.1" 200 19939 "http://www.angularjs.cn/A00n" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36"
222.68.172.190 - - [18/Sep/2013:06:50:08 +0000] "-" 400 0 "-" "-"
58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] "GET /nodejs-socketio-chat/ HTTP/1.1" 200 10818 "http://www.google.com/url?sa=t&rct=j&q=nodejs%20%E5%BC%82%E6%AD%A5%E5%B9%BF%E6%92%AD&source=web&cd=1&cad=rja&ved=0CCgQFjAA&url=%68%74%74%70%3a%2f%2f%62%6c%6f%67%2e%66%65%6e%73%2e%6d%65%2f%6e%6f%64%65%6a%73%2d%73%6f%63%6b%65%74%69%6f%2d%63%68%61%74%2f&ei=rko5UrylAefOiAe7_IGQBw&usg=AFQjCNG6YWoZsJ_bSj8kTnMHcH51hYQkAA&bvm=bv.52288139,d.aGc" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0"
58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] "GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1" 304 0 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0"
58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] "GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1" 304 0 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0"
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1" 200 786 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)"
180.168.34.26 - - [18/Sep/2013:07:11:08 +0000] "-" 400 0 "-" "-"
180.168.34.26 - - [18/Sep/2013:07:11:08 +0000] "-" 400 0 "-" "-"
50.116.27.194 - - [18/Sep/2013:07:11:29 +0000] "POST /wp-cron.php?doing_wp_cron=1379488288.8893849849700927734375 HTTP/1.0" 200 0 "-" "WordPress/3.6; http://blog.fens.me"
222.35.232.69 - - [18/Sep/2013:16:14:17 +0000] "GET /wp-content/uploads/2013/05/favicon.ico HTTP/1.1" 200 1150 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36"
114.252.89.91 - - [18/Sep/2013:16:14:20 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 58 "http://blog.fens.me/wp-admin/post.php?post=2445&action=edit&message=10" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
58.209.132.183 - - [18/Sep/2013:16:29:17 +0000] "GET /images/2.jpg HTTP/1.1" 200 105089 "http://image.baidu.com/i?ct=503316480&z=&tn=baiduimagedetail&ipn=d&word=%E6%B5%99%E6%B1%9F%E5%AE%89%E5%90%89&step_word=&ie=utf-8&in=17038&cl=2&lm=-1&st=&pn=0&rn=1&di=47839122900&ln=1998&fr=&&fmq=1379521091792_R&ic=&s=&se=&sme=0&tab=&width=&height=&face=&is=&istype=&ist=&jit=&objurl=http%3A%2F%2Fnews.eastday.com%2Feastday%2F06news%2Fchina%2Fzh2green%2Fanji%2Fnode327399%2Fimages%2F01517676.jpg" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)"

 

1.2埋點日誌

埋點日誌是電商網站採用的技術手段，當用戶瀏覽曝光的商時，主動記錄曝光的商品列表、停留時間、點擊的商品、點擊的組件等信息，服務運營，優化商城佈局，常見的埋點日誌有瀏覽、點擊、曝光日誌。

（瀏覽）

2018-08-28 11:59:58,263 - site: leeyk99, ip: 188.133.207.46, refer: https://m.leeyk99.com/ru/user/login?redirection=%2Fru%2FSneakers-c-1913.html%3Ficn%3Dsneakers%26ici%3Dmru_navbar15menu01dir02&prot=1, agent: Mozilla/5.0 (Linux; Android 5.1.1; SM-G531H Build/LMY48B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.91 Mobile Safari/537.36, body: {"device_type":"m","home_site":"leeyk99","sub_site":"mru","language":"ru","money_type":"RUB","device_country":"","app_versions":"","network_type":"","ip":"","screen_pixel":"360X640","screen_size":"","device_class":0,"device_brand":"","device_name":"","device_model":"","os_type":0,"os_name":"Android","os_versions":"5.1.1","browser_name":"Chrome","browser_versions":"68.0.3440.91","session_id":"","timestamp":1535428798994,"local_time":"2018/8/28 10:59:58","device_id":"","cookie_id":"5BCE0E1F_DAFD_2E64_F24E_B3B6D5D6BAC5","member_id":"","login":0,"page_id":3,"page_name":"page_real_class","page_param":{"category_id":"1913","source_category_id":"1745"},"start_time":1535428764401,"end_time":1535428798994,"tab_page_id":"page_real_class1535428764401"}
2018-08-28 11:59:58,272 - site: leeyk99, ip: 74.205.199.213, refer: https://m.leeyk99.com/us/Watermelon-Print-Round-Beach-Blanket-p-365584-cat-1866.html, agent: Mozilla/5.0 (Linux; Android 6.0; HUAWEI CAM-L21 Build/HUAWEICAM-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.91 Mobile Safari/537.36, body: {"device_type":"m","home_site":"leeyk99","sub_site":"mus","language":"en","money_type":"USD","device_country":"","app_versions":"","network_type":"","ip":"","screen_pixel":"360X640","screen_size":"","device_class":0,"device_brand":"","device_name":"","device_model":"","os_type":0,"os_name":"Android","os_versions":"6.0","browser_name":"Chrome","browser_versions":"68.0.3440.91","session_id":"","timestamp":1535428797165,"local_time":"2018/8/27 20:59:57","device_id":"","cookie_id":"B66A47CF_5522_DC84_F221_F0848C812BCA","member_id":"","login":0,"page_id":7,"page_name":"page_goods_detail","page_param":{"goods_id":365584,"traceid":"sm`1535428371336`B66A47CF_5522_DC84_F221_F0848C812BCA"},"start_time":1535428797165,"end_time":"","tab_page_id":"page_goods_detail1535428797165"}
2018-08-28 11:59:58,274 - site: leeyk99, ip: 99.174.207.56, refer: https://m.leeyk99.com/us/Striped-Ringer-Tee-p-469810-cat-1738.html?rrec=true, agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1, body: {"device_type":"m","home_site":"leeyk99","sub_site":"mus","language":"en","money_type":"USD","device_country":"","app_versions":"","network_type":"","ip":"","screen_pixel":"375X667","screen_size":"","device_class":0,"device_brand":"","device_name":"","device_model":"","os_type":0,"os_name":"iOS","os_versions":"11.4.1","browser_name":"Mobile Safari","browser_versions":"11.0","session_id":"","timestamp":1535428797977,"local_time":"2018/8/27 22:59:57","device_id":"","cookie_id":"D56B15A4_37D3_9164_CA60_3B4CDB382F2D","member_id":"","login":0,"page_id":7,"page_name":"page_goods_detail","page_param":{"goods_id":469810,"traceid":"sm`1535428730780`D56B15A4_37D3_9164_CA60_3B4CDB382F2D"},"start_time":1535428797977,"end_time":"","tab_page_id":"page_goods_detail1535428797977"}
2018-08-28 11:59:58,293 - site: leeyk99, ip: 172.56.35.21, refer: https://m.leeyk99.com/us/FB-US-Striped-20180402-A-D7-vc-64042.html?utm_source=facebook.com&utm_medium=cpc&utm_campaign=fbadsus_20180408_mobmpa_Food_FB-US-Striped-20180402-A-D7-vc-64042_3554_&url_from=fbadsus_20180408_mobmpa_Food_FB-US-Striped-20180402-A-D7-vc-64042_3554_, agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15G77 Instagram 24.0.0.14.205 (iPhone7,2; iOS 11_4_1; en_US; en-US; scale=2.00; gamut=normal; 750x1334), body: {"device_type":"m","home_site":"leeyk99","sub_site":"mus","language":"en","money_type":"USD","device_country":"","app_versions":"","network_type":"","ip":"","screen_pixel":"375X667","screen_size":"","device_class":0,"device_brand":"","device_name":"","device_model":"","os_type":0,"os_name":"iOS","os_versions":"11.4.1","browser_name":"WebKit","browser_versions":"605.1.15","session_id":"","timestamp":1535428797377,"local_time":"2018/8/27 23:59:57","device_id":"","cookie_id":"3BFCB287_A97B_AA24_DBCC_86DC346D3100","member_id":"","login":0,"page_id":2,"page_name":"page_virtual_class","page_param":{"category_id":"64042"},"start_time":1535428797377,"end_time":"","tab_page_id":"page_virtual_class1535428797377"}

 

點擊、曝光的日誌內容與瀏覽的相似，根據埋點需求不一樣，採集記錄的數據略有不一樣，記錄的核心內容就是body裏的內容。

埋點日誌是根據需求設計記錄的內容，格式齊整，內容規範，通常使用Hive-正則便可進行過濾入庫，像這個瀏覽日誌，只須要建立一張表，指定如下正則格式，便可入庫使用日誌：

'input.regex'='([0-9\\.\\- :,]+) \\- site: ([\\w]+), ip: ([0-9\\.\\- :,]+), refer: (.*), agent: (.*), body: ([\\[\\{].*[\\}\\]])'

 

1.3後端系統日誌

後端系統日誌是系統本身主動記錄的，一般是前端或其餘系統向後端系統請求接口數據，後端系統記錄接口請求信息或接口返回結果信息。這種數據一般是系統間約定好的，所以是格式很是規範的日誌數據，也能夠直接使用Hive的正則技術處理數據。

例如：

格式一：（結果信息）

2018-07-03 06:50:00,142 [XNIO-2 task-28] INFO  com.leeyk99.bi.abt.rest.CoreApiController - 1A42F7C6_B904_A334_AB87_5A69A7034DA0  leeyk99PcRealClass 66 158

 

格式二：（接口信息）

2018-07-03 20:39:46,043 [XNIO-2 task-211] INFO  com.leeyk99.bi.abt.filter.LogFilter - GET  /api/v1/bi/abt?cid=973EA838_E20E_74E4_41AB_E218DA91D73E&uid=&site=mtw&terminal=leeyk99-M&lan=zh-tw took 1ms and returned 200

 

(1.2\1.3中的leeyk99是對源數據中某個公司品牌的替換)

 

關於Hive正則技術處理比較規範的日誌數據，能夠查看： http://www.javashuo.com/article/p-pengbsua-w.html （博客園）或者 ABT日誌入庫記錄 （印象筆記）

本篇學習使用Hadoop-MR清洗請求日誌。

 

2.請求日誌採集入庫

對於日誌文件的採集，咱們數倉通常不會直接去生產系統去採集，而是由運維或者專門的小組負責日誌採集，通常是採集落到HDFS或S3文件系統上或者某臺接口機上，數倉採集入庫這些文件，進行清洗加工。

ELK結構（Elasticsearch , Logstash, Kibana ）提供了一整套解決方案，而且都是開源軟件，之間互相配合使用，完美銜接，高效的知足了不少場合的應用，這個結構是面向平臺或系統用戶的，用來查看監視日誌，跟蹤系統運行情況的。

Flume是Cloudera提供的一個高可用的，高可靠的，分佈式的海量日誌採集、聚合和傳輸的系統。

• Flume+Kafka+Storm+mysql構建大數據實時系統
• Flume+HDFS+KafKa+Strom實現實時推薦，反爬蟲服務等服務
• Flume+Hadoop+Hive的離線分析網站用戶瀏覽行爲路徑
• Flume+Logstash+Kafka+Spark Streaming進行實時日誌處理分析
• Flume+Spark + ELK數據系統實時監控平臺

ftp文件傳輸也是一種很是重要的文件服務方式，但對於大量的日誌可能不太適用。除非是日誌離線歸檔收集好，再傳輸到接口機上供第三方取用。

關於實時收集等模式，暫無涉獵。

 

3.配置Maven-Hadoop環境

3.1.項目初始化

<groupId>com.leeyk99.udp</groupId>
<artifactId>hadoop-mapreduce</artifactId>
<version>1.0-SNAPSHOT</version>

 

目標：建立一個Maven項目，配置Hadoop運行環境須要的Jar文件。

 

3.2.配置pom.xml

配置Hadoop運行須要的JAR文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.leeyk99.udp</groupId>
    <artifactId>hadoop-mapreduce</artifactId>
    <version>1.0-SNAPSHOT</version>
 
   <!-- <packaging>jar</packaging>-->
 
    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.7.6</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
    </dependencies>
    <!--<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>-->
</project>

 

關於IDEA上Maven項目JAR文件自動下載配置，參考筆記 Maven

自動下載後，IDEA給該Maven項目下載了不少JAR文件（External Libraries下），除了咱們本身配置的核心文件，還有相關必要的文件也被下載了， 省去了咱們逐個下載的麻煩。