機器學習模型評估與超參數調優詳解

機器學習分爲兩類基本問題----迴歸與分類。在以前的文章中，也介紹了不少基本的機器學習模型。可在 Datawhale機器學習專輯 中 查看。 可是，當咱們創建好了相關模型之後咱們怎麼評價咱們創建的模型的好壞以及優化咱們創建的模型呢？那本次分享的內容就是關於機器學習模型評估與超參數調優的。本次分享的內容包括：

• 用管道簡化工做流

• 使用k折交叉驗證評估模型性能

• 使用學習和驗證曲線調試算法

• 經過網格搜索進行超參數調優

• 比較不一樣的性能評估指標

1、用管道簡化工做流

在不少機器學習算法中，咱們可能須要作一系列的基本操做後才能進行建模，如：在創建邏輯迴歸以前，咱們可能須要先對數據進行標準化，而後使用PCA將維，最後擬合邏輯迴歸模型並預測。那有沒有什麼辦法能夠同時進行這些操做，使得這些操做造成一個工做流呢？下面請看代碼：

1. 加載基本工具庫

import numpy as npimport pandas as pdimport matplotlib.pyplot as plt%matplotlib inlineplt.style.use("ggplot")import warningswarnings.filterwarnings("ignore")

2. 加載數據，並作基本預處理

# 加載數據df = pd.read_csv("http://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data",header=None)# 作基本的數據預處理from sklearn.preprocessing import LabelEncoder
X = df.iloc[:,2:].valuesy = df.iloc[:,1].valuesle = LabelEncoder()    #將M-B等字符串編碼成計算機能識別的0-1y = le.fit_transform(y)le.transform(['M','B'])# 數據切分8：2from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,stratify=y,random_state=1)

3. 把全部的操做所有封在一個管道pipeline內造成一個工做流：標準化+PCA+邏輯迴歸

完成以上操做，共有兩種方式：

方式1：make_pipeline

# 把全部的操做所有封在一個管道pipeline內造成一個工做流：## 標準化+PCA+邏輯迴歸
### 方式1：make_pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAfrom sklearn.linear_model import LogisticRegressionfrom sklearn.pipeline import make_pipeline
pipe_lr1 = make_pipeline(StandardScaler(),PCA(n_components=2),LogisticRegression(random_state=1))pipe_lr1.fit(X_train,y_train)y_pred1 = pipe_lr.predict(X_test)print("Test Accuracy: %.3f"% pipe_lr1.score(X_test,y_test))

Test Accuracy: 0.956

方式2：Pipeline

# 把全部的操做所有封在一個管道pipeline內造成一個工做流：## 標準化+PCA+邏輯迴歸
### 方式2：Pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAfrom sklearn.linear_model import LogisticRegressionfrom sklearn.pipeline import Pipeline
pipe_lr2 = Pipeline([['std',StandardScaler()],['pca',PCA(n_components=2)],['lr',LogisticRegression(random_state=1)]])pipe_lr2.fit(X_train,y_train)y_pred2 = pipe_lr2.predict(X_test)print("Test Accuracy: %.3f"% pipe_lr2.score(X_test,y_test))

Test Accuracy: 0.956

2、使用k折交叉驗證評估模型性能

評估方式1：k折交叉驗證

• 
  
• 
  
• 
  
• 
  
• 
  
• 
  
• 
  

# 評估方式1：k折交叉驗證
from sklearn.model_selection import cross_val_score
scores1 = cross_val_score(estimator=pipe_lr,X = X_train,y = y_train,cv=10,n_jobs=1)print("CV accuracy scores:%s" % scores1)print("CV accuracy:%.3f +/-%.3f"%(np.mean(scores1),np.std(scores1)))

評估方式2：分層k折交叉驗證

# 評估方式2：分層k折交叉驗證
from sklearn.model_selection import StratifiedKFold
kfold = StratifiedKFold(n_splits=10,random_state=1).split(X_train,y_train)scores2 = []for k,(train,test) in enumerate(kfold):    pipe_lr.fit(X_train[train],y_train[train])    score = pipe_lr.score(X_train[test],y_train[test])    scores2.append(score)    print('Fold:%2d,Class dist.:%s,Acc:%.3f'%(k+1,np.bincount(y_train[train]),score))print('\nCV accuracy :%.3f +/-%.3f'%(np.mean(scores2),np.std(scores2)))

3、 使用學習和驗證曲線調試算法

若是模型過於複雜，即模型有太多的自由度或者參數，就會有過擬合的風險（高方差）；而模型過於簡單，則會有欠擬合的風險(高誤差)。

下面咱們用這些曲線去識別並解決方差和誤差問題：

1. 用學習曲線診斷誤差與方差

# 用學習曲線診斷誤差與方差from sklearn.model_selection import learning_curve
pipe_lr3 = make_pipeline(StandardScaler(),LogisticRegression(random_state=1,penalty='l2'))train_sizes,train_scores,test_scores = learning_curve(estimator=pipe_lr3,X=X_train,y=y_train,train_sizes=np.linspace(0.1,1,10),cv=10,n_jobs=1)train_mean = np.mean(train_scores,axis=1)train_std = np.std(train_scores,axis=1)test_mean = np.mean(test_scores,axis=1)test_std = np.std(test_scores,axis=1)plt.plot(train_sizes,train_mean,color='blue',marker='o',markersize=5,label='training accuracy')plt.fill_between(train_sizes,train_mean+train_std,train_mean-train_std,alpha=0.15,color='blue')plt.plot(train_sizes,test_mean,color='red',marker='s',markersize=5,label='validation accuracy')plt.fill_between(train_sizes,test_mean+test_std,test_mean-test_std,alpha=0.15,color='red')plt.xlabel("Number of training samples")plt.ylabel("Accuracy")plt.legend(loc='lower right')plt.ylim([0.8,1.02])plt.show()

2. 用驗證曲線解決欠擬合和過擬合

# 用驗證曲線解決欠擬合和過擬合from sklearn.model_selection import validation_curve
pipe_lr3 = make_pipeline(StandardScaler(),LogisticRegression(random_state=1,penalty='l2'))param_range = [0.001,0.01,0.1,1.0,10.0,100.0]train_scores,test_scores = validation_curve(estimator=pipe_lr3,X=X_train,y=y_train,param_name='logisticregression__C',param_range=param_range,cv=10,n_jobs=1)train_mean = np.mean(train_scores,axis=1)train_std = np.std(train_scores,axis=1)test_mean = np.mean(test_scores,axis=1)test_std = np.std(test_scores,axis=1)plt.plot(param_range,train_mean,color='blue',marker='o',markersize=5,label='training accuracy')plt.fill_between(param_range,train_mean+train_std,train_mean-train_std,alpha=0.15,color='blue')plt.plot(param_range,test_mean,color='red',marker='s',markersize=5,label='validation accuracy')plt.fill_between(param_range,test_mean+test_std,test_mean-test_std,alpha=0.15,color='red')plt.xscale('log')plt.xlabel("Parameter C")plt.ylabel("Accuracy")plt.legend(loc='lower right')plt.ylim([0.8,1.02])plt.show()

4、經過網格搜索進行超參數調優

若是隻有一個參數須要調整，那麼用驗證曲線手動調整是一個好方法，可是隨着須要調整的超參數愈來愈多的時候，咱們能不能自動去調整呢？！！！注意對比各個算法的時間複雜度。

（注意參數與超參數的區別：參數能夠經過優化算法進行優化，如邏輯迴歸的係數；超參數是不能用優化模型進行優化的，如正則話的係數。）

方式1：網格搜索GridSearchCV()

# 方式1：網格搜索GridSearchCV()from sklearn.model_selection import GridSearchCVfrom sklearn.svm import SVCimport time
start_time = time.time()pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]param_grid = [{'svc__C':param_range,'svc__kernel':['linear']},{'svc__C':param_range,'svc__gamma':param_range,'svc__kernel':['rbf']}]gs = GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring='accuracy',cv=10,n_jobs=-1)gs = gs.fit(X_train,y_train)end_time = time.time()print("網格搜索經歷時間：%.3f S" % float(end_time-start_time))print(gs.best_score_)print(gs.best_params_)

方式2：隨機網格搜索RandomizedSearchCV()

# 方式2：隨機網格搜索RandomizedSearchCV()from sklearn.model_selection import RandomizedSearchCVfrom sklearn.svm import SVCimport time
start_time = time.time()pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]param_grid = [{'svc__C':param_range,'svc__kernel':['linear']},{'svc__C':param_range,'svc__gamma':param_range,'svc__kernel':['rbf']}]# param_grid = [{'svc__C':param_range,'svc__kernel':['linear','rbf'],'svc__gamma':param_range}]gs = RandomizedSearchCV(estimator=pipe_svc, param_distributions=param_grid,scoring='accuracy',cv=10,n_jobs=-1)gs = gs.fit(X_train,y_train)end_time = time.time()print("隨機網格搜索經歷時間：%.3f S" % float(end_time-start_time))print(gs.best_score_)print(gs.best_params_)

方式3 ：嵌套交叉驗證

# 方式3：嵌套交叉驗證from sklearn.model_selection import GridSearchCVfrom sklearn.svm import SVCfrom sklearn.model_selection import cross_val_scoreimport time
start_time = time.time()pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]param_grid = [{'svc__C':param_range,'svc__kernel':['linear']},{'svc__C':param_range,'svc__gamma':param_range,'svc__kernel':['rbf']}]gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid,scoring='accuracy',cv=2,n_jobs=-1)scores = cross_val_score(gs,X_train,y_train,scoring='accuracy',cv=5)end_time = time.time()print("嵌套交叉驗證：%.3f S" % float(end_time-start_time))print('CV accuracy :%.3f +/-%.3f'%(np.mean(scores),np.std(scores)))

5、比較不一樣的性能評估指標

有時候，準確率不是咱們惟一須要考慮的評價指標，由於有時候會存在各種預測錯誤的代價不同。例如：在預測一我的的腫瘤疾病的時候，若是病人A真實得腫瘤可是咱們預測他是沒有腫瘤，跟A真實是健康可是預測他是腫瘤，兩者付出的代價很大區別（想一想爲何）。因此咱們須要其餘更加普遍的指標：

1. 繪製混淆矩陣

# 繪製混淆矩陣from sklearn.metrics import confusion_matrix
pipe_svc.fit(X_train,y_train)y_pred = pipe_svc.predict(X_test)confmat = confusion_matrix(y_true=y_test,y_pred=y_pred)fig,ax = plt.subplots(figsize=(2.5,2.5))ax.matshow(confmat, cmap=plt.cm.Blues,alpha=0.3)for i in range(confmat.shape[0]):    for j in range(confmat.shape[1]):        ax.text(x=j,y=i,s=confmat[i,j],va='center',ha='center')plt.xlabel('predicted label')plt.ylabel('true label')plt.show()

2. 各類指標的計算

• 
  
• 
  
• 
  
• 
  
• 
  
• 
  

# 各類指標的計算from sklearn.metrics import precision_score,recall_score,f1_score
print('Precision:%.3f'%precision_score(y_true=y_test,y_pred=y_pred))print('recall_score:%.3f'%recall_score(y_true=y_test,y_pred=y_pred))print('f1_score:%.3f'%f1_score(y_true=y_test,y_pred=y_pred))

3. 將不一樣的指標與GridSearch結合

• 
  
• 
  
• 
  
• 
  
• 
  
• 
  
• 
  

# 將不一樣的指標與GridSearch結合from sklearn.metrics import make_scorer,f1_scorescorer = make_scorer(f1_score,pos_label=0)gs = GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring=scorer,cv=10)gs = gs.fit(X_train,y_train)print(gs.best_score_)print(gs.best_params_)

4. 繪製ROC曲線

# 繪製ROC曲線from sklearn.metrics import roc_curve,aucfrom sklearn.metrics import make_scorer,f1_scorescorer = make_scorer(f1_score,pos_label=0)gs = GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring=scorer,cv=10)y_pred = gs.fit(X_train,y_train).decision_function(X_test)#y_pred = gs.predict(X_test)fpr,tpr,threshold = roc_curve(y_test, y_pred) ###計算真陽率和假陽率roc_auc = auc(fpr,tpr) ###計算auc的值plt.figure()lw = 2plt.figure(figsize=(7,5))plt.plot(fpr, tpr, color='darkorange',         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc) ###假陽率爲橫座標，真陽率爲縱座標作曲線plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')plt.xlim([-0.05, 1.0])plt.ylim([-0.05, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic ')plt.legend(loc="lower right")plt.show()