hive學習06-youtube數據集

---hive練習第6天

1.hive基礎：order by，sort by，distribute by，cluster by order by 全局排序 全部的map結果提交至一個reduce裏，若是在嚴格模式下，order by 須要指定 limit 數據條數，否則數據量巨大的狀況下 會形成崩潰無輸出結果。 涉及屬性：set hive.mapred.mode=nonstrict/strict 設置爲嚴格模式後。order by 後面須要接 limit sort by 多個reduce，reduce內部有序，不保證全局有序。 distribute by distribute by 通常要和 sort by 一塊兒使用，即將某一塊數據歸給(distribute by)某一個reducer 處理，而後在指定的 reducer 中進行 sort by 排序。 cluster by =distribute by + sort by (分桶字段和排序字段爲同一個字段) create table company_info ( personid string, company string, money float )row format delimited fields terminated by "\t"; load data local inpath "/home/dip/company.txt"

 [dip@dip005 lzm]$ cat company.txt p1 公司1 150 p2 公司2 230 p1 公司3 160 p3 公司4 30 p4 公司4 20 p4 公司6 90 p5 公司5 500 練習 #全局排序 select  * from company_info order by money desc ; #reduce內部有序，若是啓動多個reduce，reduce之間沒有有序 select * from company_info sort by money desc; #默認第一個字段分桶，這裏咱們用第二個字段分桶distribute by 而後用第三個字段作sort by 2、hive基礎：行轉列、列轉行 UDAF UDTF 2.1；列轉行 concat()函數用於將多個字符串鏈接成一個字符串 SELECT CONCAT(id, ‘，’, name) AS con FROM info conact_ws()函數是caoncat的特殊形式，第一個參數是分隔符號。 SELECT CONCAT("_",id, name) AS con FROM info group_concat()函數返回一個字符串結果 hive中collect_list()函數和collect_set()函數將某列轉爲一個數組返回 [dip@dip005 lzm]$ cat person_info.tsv 孫悟空 白羊座 A 大海 射手座 A 宋宋 白羊座 B 豬八戒 白羊座 A 鳳姐 射手座 A create table person_info( name string, constellation string, blood_type string) row format delimited fields terminated by "\t"; load data local inpath "/home/dip/lzm/person_info.tsv" into table person_info #請將具備相同星座和血型的人返回 select t1.base,collect_set(name) from (select name,concat_ws(",",constellation,blood_type) base from person_info)t1 group by t1.base; 將collect_set()返回的列表進行處理： select base,concat_ws("|",collect_set(name)) from (select name,concat_ws(",",constellation,blood_type) base from person_info)t1 group by base; 2.2行轉列： 「fields terminated by」：字段與字段之間的分隔符。 「collection items terminated by」：一個字段中各個子元素 item 的分隔符。 [dip@dip005 lzm]$ head movie_info.tsv 《疑犯追蹤》 懸疑，動做，科幻，劇情 《lie to me》 懸疑，警匪，動做，心理，劇情 《戰狼2》 戰爭，動做，災難 create table movie_info( movie string, category array<string> )row format delimited fields terminated by "\t" collection items terminated by ","; load data local inpath "/home/dip/lzm/movie_info.tsv" into table movie_info; lateral view 與 explode函數的使用 youtube數據實戰： 源數據解釋:youtube_video_ori 視頻表 video_id 視頻id uploader 視頻上傳者 age 視頻年齡 category 視頻類別 length 視頻長度 views 觀看次數 rate 視頻評分 ratings 流量 comments 評論數 related_ids 相關視頻id （很少於20個） 用戶表 uploader 視頻上傳者 videos 上傳視頻數 friends 朋友的數量 練習 1、各類 TopN 指標： 01統計視頻觀看數 top10 select * from youtube_video_ori order by views  desc limit 10; 02統計視頻類別熱度 top10 （視頻數最多的十個類別） 考點：把category進行 一行轉多行，而後聚合統計次數 方法一： select t1.category_name,count(*) as cnt from (select video_id,category_name from youtube_video_ori lateral view explode(category) youtube_video_ori as category_name)t1 group by category_name order by cnt desc limit 10 ; 方法二: select t1.category_name ,count(*) as cnt from (select explode(category) as category_name from youtube_video_ori)t1 group by category_name order by cnt desc limit 10; 03統計視頻觀看數 Top20 所屬類別包含這 Top20 視頻的個數 ①計算觀看視頻數top20 select * from youtube_video_ori order by views  desc limit 20; ②類別一行轉多行 select video_id,category_name from (select * from youtube_video_ori order by views  desc limit 20)t1 lateral view explode(category) t1 as category_name; ③統計每一個類別包含視頻個數視頻 select category_name,count(*) from (select video_id,category_name from (select * from youtube_video_ori order by views  desc limit 20)t1 lateral view explode(category) t1 as category_name)t2 group by category_name; 04統計視頻觀看數 Top50 所關聯視頻的所屬類別Rank ①top前50 select * from youtube_video_ori order by views  desc limit 50; ②關聯視頻id一行轉多行： select video_id,related_ids_name from (select * from youtube_video_ori order by views  desc limit 50)t1 lateral view explode(related_ids)t1 as related_ids_name; ③video_id作關聯，與表youtube_video_ori關聯,統計關聯視頻所屬id select t2.related_ids_name,youtube_video_ori.category from (select video_id,related_ids_name from (select * from youtube_video_ori order by views  desc limit 50)t1 lateral view explode(related_ids)t1 as related_ids_name)t2 join youtube_video_ori on t2.video_id =youtube_video_ori.video_id; ④類別一行轉多行 ，爲key，聚類統計數量 select category_name,count(*) as cnt from (select t2.related_ids_name,youtube_video_ori.category from (select video_id,related_ids_name from (select * from youtube_video_ori order by views  desc limit 50)t1 lateral view explode(related_ids)t1 as related_ids_name)t2 join youtube_video_ori on t2.video_id =youtube_video_ori.video_id)t3 lateral view explode(category) t3 as category_name group by category_name order by cnt; 05統計每一個類別中視頻觀看數的 Top10 查詢有多少類別 hive -e "use hue_test1;select distinct(category_name) from youtube_video_ori lateral view explode(category)youtube_video_ori as category_name"|  grep -v WARN>a.txt cat a.txt|while read line do hive -e "use hue_test1;select video_id,category_name,views from youtube_video_ori lateral view explode(category)youtube_video_ori as category_name where category_name='$line' order by views desc limit 10" |grep -v WARN>>b.txt done 06統計上傳視頻最多的用戶 Top10 以及他們上傳的視頻 select youtube_video_ori.uploader,youtube_video_ori.video_id from ( select uploader,count(*)as up_cnt from youtube_video_ori group by uploader order by up_cnt desc limit 10)t1 join  youtube_video_ori  on t1.uploader = youtube_video_ori.uploader;