利用pandas隨機切分csv文件

 

把數據集隨機切分爲訓練集和測試集

 

method 1:

df = pd.read_csv('data/tgnb_merge.csv', encoding='utf-8')
df.drop_duplicates(keep='first', inplace=True)  # 去重，只保留第一次出現的樣本
df_test = df.sample(frac=0.1)
df_train = pd.concat([df, df_test], axis=0)   # 拼接
df_train.drop_duplicates(keep=False, inplace=True)  # 去除全部重複的樣本
print df.shape, df_test.shape, df_train.shape  # (3045, 12) (305, 12) (2740, 12)

 

method 2（推薦）:

df = pd.read_csv('data/tgnb_merge.csv', encoding='utf-8')
# df.drop_duplicates(keep='first', inplace=True)  # 去重，只保留第一次出現的樣本
df = df.sample(frac=1.0)  # 所有打亂
cut_idx = int(round(0.1 * df.shape[0]))
df_test, df_train = df.iloc[:cut_idx], df.iloc[cut_idx:]
print df.shape, df_test.shape, df_train.shape  # (3184, 12) (318, 12) (2866, 12)