pandas實現hive的lag和lead函數 以及 first_value和last_value函數

lag和lead VS shift 

該函數的格式以下:

• 第一個參數爲列名，
• 第二個參數爲往上第n行（可選，默認爲1），
• 第三個參數爲默認值（當往上第n行爲NULL時候，取默認值，如不指定，則爲NULL）

lag

lag(字段名,N，默認值) over(partition by 分組字段 order by 排序字段 排序方式)

lead

lead(字段名,N，默認值) over(partition by 分組字段 order by 排序字段 排序方式)

案例:

select cookieid, createtime, url, row_number() over (partition by cookieid order by createtime) as rn, LAG(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as last_1_time, LAG(createtime,2) over (partition by cookieid order by createtime) as last_2_time from cookie.cookie4

select cookieid, createtime, url, row_number() over (partition by cookieid order by createtime) as rn, LEAD(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as next_1_time, LEAD(createtime,2) over (partition by cookieid order by createtime) as next_2_time from cookie.cookie4;

窗口函數的pandas實現

pandas中使用shift函數來實現lag/lead函數

import pandas as pd df=pd.read_csv('c:/Users/WQBin/Desktop/data.csv',engine='python', names=['cookieid','createtime','url']) df['last_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(1) df['next_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(-1) df.sort_values(by=['cookieid','createtime'])

 

first_value和 last_value VS first()和last()

• FIRST_VALUE 返回組中數據窗口的第一個值
  • 　　FIRST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause ] order_by_clause )
• LAST_VALUE 返回組中數據窗口的最後一個值
  • 　　 LAST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause order_by_clause )

 

select cookieid, createtime, url, row_number() over (partition by cookieid order by createtime) as rn, first_value(url) over (partition by cookieid order by createtime) as first1 from cookie.cookie4;

 

 

 

select cookieid, createtime, url, row_number() over (partition by cookieid order by createtime) as rn, last_value(url) over (partition by cookieid order by createtime) as last1 from cookie.cookie4;

 

 

窗口函數的pandas實現

 

 

df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).first()

 

 

df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).last()