(原創)（三）機器學習筆記之Scikit Learn的線性迴歸模型初探

1、Scikit Learn中使用estimator三部曲

1. 構造estimator

2. 訓練模型：fit

3. 利用模型進行預測：predict

 

2、模型評價

模型訓練好後，度量模型擬合效果的常見準則有：

1.      均方偏差（mean squared error，MSE）：

2.      平均絕對偏差（mean absolute error，MAE）

3.      R2 score：scikit learn線性迴歸模型的缺省評價準則，既考慮了預測值與真值之間的差別，也考慮了問題自己真值之間的差別：

4.      檢測殘差的分佈

檢測殘差是不是均值爲0的正態分佈。

 

3、Scikit Learn中的線性迴歸

1. 線性迴歸，梯度降低法模型優化參數

sklearn.linear_model.LinearRegression(fit_intercept=True, normalize=False, copy_X=True, n_jobs=1)

參數：

Ordinary least squares Linear Regression.

Parameters:

fit_intercept : boolean, optional whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (e.g. data is expected to be already centered). normalize : boolean, optional, default False If True, the regressors X will be normalized before regression. This parameter is ignored when fit_intercept is set to False. When the regressors are normalized, note that this makes the hyperparameters learnt more robust and almost independent of the number of samples. The same property is not valid for standardized data. However, if you wish to standardize, please use preprocessing.StandardScaler before calling fit on an estimator with normalize=False. copy_X : boolean, optional, default True If True, X will be copied; else, it may be overwritten. n_jobs : int, optional, default 1 The number of jobs to use for the computation. If -1 all CPUs are used. This will only provide speedup for n_targets > 1 and sufficient large problems.

Attributes:

coef_ : array, shape (n_features, ) or (n_targets, n_features) Estimated coefficients for the linear regression problem. If multiple targets are passed during the fit (y 2D), this is a 2D array of shape (n_targets, n_features), while if only one target is passed, this is a 1D array of length n_features. residues_ : array, shape (n_targets,) or (1,) or empty Sum of residuals. Squared Euclidean 2-norm for each target passed during the fit. If the linear regression problem is under-determined (the number of linearly independent rows of the training matrix is less than its number of linearly independent columns), this is an empty array. If the target vector passed during the fit is 1-dimensional, this is a (1,) shape array. New in version 0.18. intercept_ : array Independent term in the linear model.

 

訓練：fit(X, y, sample_weight=None)

Fit linear model.

Parameters:	X : numpy array or sparse matrix of shape [n_samples,n_features] Training data y : numpy array of shape [n_samples, n_targets] Target values sample_weight : numpy array of shape [n_samples] Individual weights for each sample New in version 0.17: parameter sample_weight support to LinearRegression.
Returns:	self : returns an instance of self.

預測：predict(X)

Predict using the linear model

Parameters:	X : {array-like, sparse matrix}, shape = (n_samples, n_features) Samples.
Returns:	C : array, shape = (n_samples,) Returns predicted values.

 

評分：score(X, y, sample_weight=None) R2分數

Returns the coefficient of determination R^2 of the prediction.

The coefficient R^2 is defined as (1 - u/v), where u is the regression sum of squares ((y_true - y_pred) ** 2).sum() and v is the residual sum of squares ((y_true - y_true.mean()) ** 2).sum(). Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.

Parameters:	X : array-like, shape = (n_samples, n_features) Test samples. y : array-like, shape = (n_samples) or (n_samples, n_outputs) True values for X. sample_weight : array-like, shape = [n_samples], optional Sample weights.
Returns:	score : float R^2 of self.predict(X) wrt. y.

http://sklearn.lzjqsdd.com/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression.score

 

2. 線性迴歸，隨機梯度降低優化模型參數

sklearn.linear_model.SGDRegressor(loss='squared_loss', penalty='l2', alpha=0.0001, l1_ratio=0.15, fit_intercept=True, n_iter=5, shuffle=True, verbose=0, epsilon=0.1, random_state=None, learning_rate='invscaling', eta0=0.01, power_t=0.25, warm_start=False, average=False)

Parameters:

loss : str, ‘squared_loss’, ‘huber’, ‘epsilon_insensitive’, or ‘squared_epsilon_insensitive’ The loss function to be used. Defaults to ‘squared_loss’ which refers to the ordinary least squares fit. ‘huber’ modifies ‘squared_loss’ to focus less on getting outliers correct by switching from squared to linear loss past a distance of epsilon. ‘epsilon_insensitive’ ignores errors less than epsilon and is linear past that; this is the loss function used in SVR. ‘squared_epsilon_insensitive’ is the same but becomes squared loss past a tolerance of epsilon. penalty : str, ‘none’, ‘l2’, ‘l1’, or ‘elasticnet’ The penalty (aka regularization term) to be used. Defaults to ‘l2’ which is the standard regularizer for linear SVM models. ‘l1’ and ‘elasticnet’ might bring sparsity to the model (feature selection) not achievable with ‘l2’. alpha : float Constant that multiplies the regularization term. Defaults to 0.0001 Also used to compute learning_rate when set to ‘optimal’. l1_ratio : float The Elastic Net mixing parameter, with 0 <= l1_ratio <= 1. l1_ratio=0 corresponds to L2 penalty, l1_ratio=1 to L1. Defaults to 0.15. fit_intercept : bool Whether the intercept should be estimated or not. If False, the data is assumed to be already centered. Defaults to True. n_iter : int, optional The number of passes over the training data (aka epochs). The number of iterations is set to 1 if using partial_fit. Defaults to 5. shuffle : bool, optional Whether or not the training data should be shuffled after each epoch. Defaults to True. random_state : int seed, RandomState instance, or None (default) The seed of the pseudo random number generator to use when shuffling the data. verbose : integer, optional The verbosity level. epsilon : float Epsilon in the epsilon-insensitive loss functions; only if loss is ‘huber’, ‘epsilon_insensitive’, or ‘squared_epsilon_insensitive’. For ‘huber’, determines the threshold at which it becomes less important to get the prediction exactly right. For epsilon-insensitive, any differences between the current prediction and the correct label are ignored if they are less than this threshold. learning_rate : string, optional The learning rate schedule: ‘constant’: eta = eta0 ‘optimal’: eta = 1.0 / (alpha * (t + t0)) [default] ‘invscaling’: eta = eta0 / pow(t, power_t) where t0 is chosen by a heuristic proposed by Leon Bottou. eta0 : double, optional The initial learning rate [default 0.01]. power_t : double, optional The exponent for inverse scaling learning rate [default 0.25]. warm_start : bool, optional When set to True, reuse the solution of the previous call to fit as initialization, otherwise, just erase the previous solution. average : bool or int, optional When set to True, computes the averaged SGD weights and stores the result in the coef_ attribute. If set to an int greater than 1, averaging will begin once the total number of samples seen reaches average. So average=10 will begin averaging after seeing 10 samples.

Attributes:

coef_ : array, shape (n_features,) Weights assigned to the features. intercept_ : array, shape (1,) The intercept term. average_coef_ : array, shape (n_features,) Averaged weights assigned to the features. average_intercept_ : array, shape (1,) The averaged intercept term

 

訓練：fit(X, y, coef_init=None, intercept_init=None, sample_weight=None)

Fit linear model with Stochastic Gradient Descent.

Parameters:

X : {array-like, sparse matrix}, shape (n_samples, n_features) Training data y : numpy array, shape (n_samples,) Target values coef_init : array, shape (n_features,) The initial coefficients to warm-start the optimization. intercept_init : array, shape (1,) The initial intercept to warm-start the optimization. sample_weight : array-like, shape (n_samples,), optional Weights applied to individual samples (1. for unweighted).

Returns:

self : returns an instance of self.

 

預測：predict(X)

Predict using the linear model

Parameters:	X : {array-like, sparse matrix}, shape (n_samples, n_features)
Returns:	array, shape (n_samples,) : Predicted target values per element in X.

 

評分：score(X, y, sample_weight=None) R2分數

Returns the coefficient of determination R^2 of the prediction.

The coefficient R^2 is defined as (1 - u/v), where u is the regression sum of squares ((y_true - y_pred) ** 2).sum() and v is the residual sum of squares ((y_true - y_true.mean()) ** 2).sum(). Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.

Parameters:	X : array-like, shape = (n_samples, n_features) Test samples. y : array-like, shape = (n_samples) or (n_samples, n_outputs) True values for X. sample_weight : array-like, shape = [n_samples], optional Sample weights.
Returns:	score : float R^2 of self.predict(X) wrt. y.

http://sklearn.lzjqsdd.com/modules/generated/sklearn.linear_model.SGDRegressor.html#sklearn.linear_model.SGDRegressor

 

3. 嶺迴歸／L2正則

sklearn.linear_model.RidgeCV(alphas=(0.1, 1.0, 10.0), fit_intercept=True, normalize=False, scoring=None, cv=None, gcv_mode=None, store_cv_values=False)

參數：

Parameters:

alphas : numpy array of shape [n_alphas] Array of alpha values to try. Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to C^-1 in other linear models such as LogisticRegression or LinearSVC. fit_intercept : boolean Whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (e.g. data is expected to be already centered). normalize : boolean, optional, default False If True, the regressors X will be normalized before regression. This parameter is ignored when fit_intercept is set to False. When the regressors are normalized, note that this makes the hyperparameters learnt more robust and almost independent of the number of samples. The same property is not valid for standardized data. However, if you wish to standardize, please use preprocessing.StandardScaler before calling fit on an estimator with normalize=False. scoring : string, callable or None, optional, default: None A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y). cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: None, to use the efficient Leave-One-Out cross-validation integer, to specify the number of folds. An object to be used as a cross-validation generator. An iterable yielding train/test splits. For integer/None inputs, if y is binary or multiclass,sklearn.model_selection.StratifiedKFold is used, else, sklearn.model_selection.KFoldis used. Refer User Guide for the various cross-validation strategies that can be used here. gcv_mode : {None, ‘auto’, ‘svd’, eigen’}, optional Flag indicating which strategy to use when performing Generalized Cross-Validation. Options are: 'auto' : use svd if n_samples > n_features or when X is a sparse matrix, otherwise use eigen 'svd' : force computation via singular value decomposition of X (does not work for sparse matrices) 'eigen' : force computation via eigendecomposition of X^T X The ‘auto’ mode is the default and is intended to pick the cheaper option of the two depending upon the shape and format of the training data. store_cv_values : boolean, default=False Flag indicating if the cross-validation values corresponding to each alpha should be stored in the cv_values_ attribute (see below). This flag is only compatible with cv=None(i.e. using  Generalized Cross-Validation).

Attributes:

cv_values_ : array, shape = [n_samples, n_alphas] or shape = [n_samples, n_targets, n_alphas], optional Cross-validation values for each alpha (if store_cv_values=True and cv=None). After fit() has been called, this attribute will contain the mean squared errors (by default) or the values of the {loss,score}_func function (if provided in the constructor). coef_ : array, shape = [n_features] or [n_targets, n_features] Weight vector(s). intercept_ : float | array, shape = (n_targets,) Independent term in decision function. Set to 0.0 if fit_intercept = False. alpha_ : float Estimated regularization parameter.

訓練：fit(X, y, sample_weight=None)

Fit Ridge regression model

Parameters:	X : array-like, shape = [n_samples, n_features] Training data y : array-like, shape = [n_samples] or [n_samples, n_targets] Target values sample_weight : float or array-like of shape [n_samples] Sample weight
Returns:	self : Returns self.

 

預測：predict(X)

Predict using the linear model

Parameters:	X : {array-like, sparse matrix}, shape = (n_samples, n_features) Samples.
Returns:	C : array, shape = (n_samples,) Returns predicted values.

 

評分：score(X, y, sample_weight=None) R2分數

Returns the coefficient of determination R^2 of the prediction.

The coefficient R^2 is defined as (1 - u/v), where u is the regression sum of squares ((y_true - y_pred) ** 2).sum() and v is the residual sum of squares ((y_true - y_true.mean()) ** 2).sum(). Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.

Parameters:	X : array-like, shape = (n_samples, n_features) Test samples. y : array-like, shape = (n_samples) or (n_samples, n_outputs) True values for X. sample_weight : array-like, shape = [n_samples], optional Sample weights.
Returns:	score : float R^2 of self.predict(X) wrt. y.

http://sklearn.lzjqsdd.com/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV

 

4. Lasso／L1正則

sklearn.linear_model.LassoCV(eps=0.001, n_alphas=100, alphas=None, fit_intercept=True, normalize=False, precompute='auto', max_iter=1000, tol=0.0001, copy_X=True, cv=None, verbose=False, n_jobs=1, positive=False, random_state=None, selection='cyclic')

參數：

Parameters:

eps : float, optional Length of the path. eps=1e-3 means that alpha_min / alpha_max = 1e-3. n_alphas : int, optional Number of alphas along the regularization path alphas : numpy array, optional List of alphas where to compute the models. If None alphas are set automatically precompute : True | False | ‘auto’ | array-like Whether to use a precomputed Gram matrix to speed up calculations. If set to 'auto' let us decide. The Gram matrix can also be passed as argument. max_iter : int, optional The maximum number of iterations tol : float, optional The tolerance for the optimization: if the updates are smaller than tol, the optimization code checks the dual gap for optimality and continues until it is smaller than tol. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: None, to use the default 3-fold cross-validation, integer, to specify the number of folds. An object to be used as a cross-validation generator. An iterable yielding train/test splits. For integer/None inputs, KFold is used. Refer User Guide for the various cross-validation strategies that can be used here. verbose : bool or integer Amount of verbosity. n_jobs : integer, optional Number of CPUs to use during the cross validation. If -1, use all the CPUs. positive : bool, optional If positive, restrict regression coefficients to be positive selection : str, default ‘cyclic’ If set to ‘random’, a random coefficient is updated every iteration rather than looping over features sequentially by default. This (setting to ‘random’) often leads to significantly faster convergence especially when tol is higher than 1e-4. random_state : int, RandomState instance, or None (default) The seed of the pseudo random number generator that selects a random feature to update. Useful only when selection is set to ‘random’. fit_intercept : boolean, default True whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (e.g. data is expected to be already centered). normalize : boolean, optional, default False If True, the regressors X will be normalized before regression. This parameter is ignored when fit_intercept is set to False. When the regressors are normalized, note that this makes the hyperparameters learnt more robust and almost independent of the number of samples. The same property is not valid for standardized data. However, if you wish to standardize, please use preprocessing.StandardScaler before calling fit on an estimator with normalize=False. copy_X : boolean, optional, default True If True, X will be copied; else, it may be overwritten.

Attributes:

alpha_ : float The amount of penalization chosen by cross validation coef_ : array, shape (n_features,) | (n_targets, n_features) parameter vector (w in the cost function formula) intercept_ : float | array, shape (n_targets,) independent term in decision function. mse_path_ : array, shape (n_alphas, n_folds) mean square error for the test set on each fold, varying alpha alphas_ : numpy array, shape (n_alphas,) The grid of alphas used for fitting dual_gap_ : ndarray, shape () The dual gap at the end of the optimization for the optimal alpha (alpha_). n_iter_ : int number of iterations run by the coordinate descent solver to reach the specified tolerance for the optimal alpha.

 

訓練：fit(X, y)

Fit linear model with coordinate descent

Fit is on grid of alphas and best alpha estimated by cross-validation.

Parameters:

X : {array-like}, shape (n_samples, n_features) Training data. Pass directly as float64, Fortran-contiguous data to avoid unnecessary memory duplication. If y is mono-output, X can be sparse. y : array-like, shape (n_samples,) or (n_samples, n_targets) Target values

 

評分：score(X, y, sample_weight=None) R2分數

Returns the coefficient of determination R^2 of the prediction.

The coefficient R^2 is defined as (1 - u/v), where u is the regression sum of squares ((y_true - y_pred) ** 2).sum() and v is the residual sum of squares ((y_true - y_true.mean()) ** 2).sum(). Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.

Parameters:	X : array-like, shape = (n_samples, n_features) Test samples. y : array-like, shape = (n_samples) or (n_samples, n_outputs) True values for X. sample_weight : array-like, shape = [n_samples], optional Sample weights.
Returns:	score : float R^2 of self.predict(X) wrt. y.

http://sklearn.lzjqsdd.com/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV.score

 

4、應用舉例

1. 線性迴歸，梯度降低法模型優化參數

In [15]:

# 線性迴歸
from sklearn.linear_model import LinearRegression

# 使用默認配置初始化
# LogisticRegression(penalty='l2', dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver='liblinear', max_iter=100, multi_class='ovr', verbose=0, warm_start=False, n_jobs=1)
lr = LinearRegression()

# 訓練模型參數
lr.fit(X_train, y_train)

# 預測
lr_y_predict_train = lr.predict(X_train)
lr_y_predict_test  = lr.predict(X_test)

#顯示特徵的迴歸係數
lr.coef_

Out[15]:

array([-0.09500237,  0.10914806, -0.00112144,  0.08231846, -0.18840035,
        0.32519197,  0.00668076, -0.32625467,  0.27938842, -0.21152335,
       -0.22468133,  0.10905708, -0.3760732 ])

 

模型評價

In [16]:

f,ax = plt.subplots(figsize = (7,5))
f.tight_layout()
ax.hist(y_train - lr_y_predict_train, bins=40, label='Residuals Linear', color='b', alpha=.5) # 繪製殘差的直方圖
ax.set_title("Histogram of Residuals") 
ax.legend(loc='best') # 放置 legend 標籤，指定標籤位置，能夠是整形數，也能夠是形如'upper right'的字符串

Out[16]:

<matplotlib.legend.Legend at 0xcaebfd0>

 

 

梯度降低法的均方偏差：

In [22]:

lr_score_train  = lr.score(X_train, y_train) # 訓練集上的R2分數
lr_score_test   = lr.score(X_test, y_test)   # 測試集上的R2分數
print lr_score_train
print lr_score_test

 

0.743656187629
0.72084091647

2. 線性迴歸，隨機梯度降低優化模型參數

In [17]:

# 線性模型， 隨機梯度降低優化模型
from sklearn.linear_model import SGDRegressor

#初始化隨機梯度降低優化模型
sgdr = SGDRegressor()

#訓練
sgdr.fit(X_train, y_train)

#預測
sgdr_y_predict = sgdr.predict(X_test)

#參數係數
sgdr.coef_

Out[17]:

array([-0.06649891,  0.04454054, -0.05024994,  0.09958022, -0.06565413,
        0.37271481, -0.01055654, -0.20753276,  0.08342105, -0.05108732,
       -0.20211071,  0.10570011, -0.3417218 ])

 

隨機梯度降低法的均方偏差：

In [23]:

sgdr_score_test = sgdr.score(X_test, y_test) # 測試集上的R2分數
print sgdr_score_test

 

0.691701561498

3. 嶺迴歸／L2

In [30]:

#嶺迴歸／L2正則
from sklearn.linear_model import RidgeCV

alphas = [0.01, 0.1, 1, 10,20, 40, 80,100]  # L2正則係數範圍
reg = RidgeCV(alphas=alphas, store_cv_values=True) # 初始化，store_cv_values是否保存每次交叉驗證的係數到cv_values_
reg.fit(X_train, y_train) # 訓練
print reg.coef_  # 訓練後的係數
print reg.cv_values_.shape
# 使用LinearRegression模型自帶的評估模塊（r2_score），並輸出評估結果
print 'R2_score =',reg.score(X_test, y_test)

 

[-0.08561028  0.09153468 -0.02368205  0.08599496 -0.15456091  0.33109336
  0.00038359 -0.29225882  0.20834624 -0.15016529 -0.21559628  0.10703687
 -0.36385829]
(379L, 8L)
R2_score = 0.716205750923

In [33]:

# 畫偏差圖
mse_mean = np.mean(reg.cv_values_, axis=0) #以列爲單位求均值
mse_stds = np.std(reg.cv_values_, axis=0)  #以列爲單位求方差偏差

x_axis =np.log10(alphas)

plt.errorbar(x_axis, mse_mean, yerr = mse_stds)  # 線上爲均值偏差，垂直方向的偏差線爲方差偏差

plt.title("Ridge for Boston House Price")
plt.xlabel("log(alpha)")
plt.ylabel("mse")
plt.show()

 

In [34]:

# 畫最佳alpha值的位置
mse_mean = np.mean(reg.cv_values_, axis=0) #以列爲單位求均值
plt.plot(np.log10(alphas), mse_mean)
plt.plot(np.log10(reg.alpha_)*np.ones(3), [0.28, 0.29, 0.30]) # 畫最佳alpha值的位置

plt.xlabel('log(alpha)')
plt.ylabel('mse')
plt.show()

print ('alpha is:', reg.alpha_)

 

 

('alpha is:', 10.0)

4.Lasso／L1正則

In [38]:

#### Lasso／L1正則
from sklearn.linear_model import LassoCV

alphas = [0.01, 0.1, 1, 10,20, 30, 40,100]
lasso = LassoCV(alphas= alphas)
lasso.fit(X_train, y_train)

print lasso.mse_path_.shape # lassco.mse_path_ : 存儲每次迭代的偏差. shape（alpha係數個數，迭代次數）
mses = np.mean(lasso.mse_path_, axis=1)
plt.plot(np.log10(lasso.alphas_), mses)
plt.plot(np.log10(lasso.alpha_)*np.ones(3), [0.2, 0.4, 1.0]) # 畫最佳的alpha

plt.xlabel('log(alpha)')
plt.ylabel('mse')
plt.show()    
            
print ('alpha is:', lasso.alpha_)
lasso.coef_  # 最佳係數

# 使用LinearRegression模型自帶的評估模塊（r2_score），並輸出評估結果
print 'test R2 score:', lasso.score(X_test, y_test)
print 'trainR2 score:', lasso.score(X_train, y_train)

 

(8L, 3L)

 

 

('alpha is:', 0.01)
test R2 score: 0.71146557665
trainR2 score: 0.739125903229