python data analysis | python數據預處理（基於scikit-learn模塊）

原文：http://www.jianshu.com/p/94516a58314d

• Dataset transformations| 數據轉換
• Combining estimators|組合學習器
• Feature extration|特徵提取
• Preprocessing data|數據預處理

---

 

1 Dataset transformations

---

scikit-learn provides a library of transformers, which may clean (see Preprocessing data), reduce (see Unsupervised dimensionality reduction), expand (see Kernel Approximation) or generate (see Feature extraction) feature representations.

scikit-learn 提供了數據轉換的模塊，包括數據清理、降維、擴展和特徵提取。

Like other estimators, these are represented by classes with fit method, which learns model parameters (e.g. mean and standard deviation for normalization) from a training set, and a transform method which applies this transformation model to unseen data. fit_transform may be more convenient and efficient for modelling and transforming the training data simultaneously.

scikit-learn模塊有3種通用的方法：fit(X,y=None)、transform(X)、fit_transform(X)、inverse_transform(newX)。fit用來訓練模型；transform在訓練後用來降維；fit_transform先用訓練模型，而後返回降維後的X；inverse_transform用來將降維後的數據轉換成原始數據。

 

1.1 combining estimators

•  

  1.1.1 Pipeline:chaining estimators

  Pipeline 模塊是用來組合一系列估計器的。對固定的一系列操做很是便利，如：同時結合特徵選擇、數據標準化、分類。
  • Usage|使用
    代碼：

    from sklearn.pipeline import Pipeline from sklearn.svm import SVC from sklearn.decomposition import PCA from sklearn.pipeline import make_pipeline #define estimators #the arg is a list of (key,value) pairs,where the key is a string you want to give this step and value is an estimators object estimators=[('reduce_dim',PCA()),('svm',SVC())] #combine estimators clf1=Pipeline(estimators) clf2=make_pipeline(PCA(),SVC()) #use func make_pipeline() can do the same thing print(clf1,'\n',clf2)

    輸出：

    Pipeline(steps=[('reduce_dim', PCA(copy=True, n_components=None, whiten=False)), ('svm', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False))]) Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False))])

    能夠經過set_params()方法設置學習器的屬性，參數形式爲<estimator>_<parameter>

    clf.set_params(svm__C=10)

    上面的方法在網格搜索時很重要：

    from sklearn.grid_search import GridSearchCV params = dict(reduce_dim__n_components=[2, 5, 10],svm__C=[0.1, 10, 100]) grid_search = GridSearchCV(clf, param_grid=params)

    上面的例子至關於把pipeline生成的學習器做爲一個普通的學習器，參數形式爲<estimator>_<parameter>。
  • Note|說明
    1.可使用dir()函數查看clf的全部屬性和方法。例如step屬性就是每一個操做步驟的屬性。
    如
    >>> clf.steps[0] ('reduce_dim', PCA(copy=True, n_components=None, whiten=False))
    2.調用pipeline生成的學習器的fit方法至關於依次調用其包含的全部學習器的方法，transform輸入而後把結果扔向下一步驟。pipeline生成的學習器有着它包含的學習器的全部方法。若是最後一個學習器是分類，那麼生成的學習器就是分類，若是最後一個是transform，那麼生成的學習器就是transform，依次類推。

•  

  1.1.2 FeatureUnion: composite feature spaces

  與pipeline不一樣的是FeatureUnion只組合transformer，它們也能夠結合成更復雜的模型。

  FeatureUnion combines several transformer objects into a new transformer that combines their output. AFeatureUnion takes a list of transformer objects. During fitting, each of these is fit to the data independently. For transforming data, the transformers are applied in parallel, and the sample vectors they output are concatenated end-to-end into larger vectors.

  • Usage|使用
    代碼：

    from sklearn.pipeline import FeatureUnion from sklearn.decomposition import PCA from sklearn.decomposition import KernelPCA from sklearn.pipeline import make_union #define transformers #the arg is a list of (key,value) pairs,where the key is a string you want to give this step and value is an transformer object estimators=[('linear_pca)',PCA()),('Kernel_pca',KernelPCA())] #combine transformers clf1=FeatureUnion(estimators) clf2=make_union(PCA(),KernelPCA()) print(clf1,'\n',clf2) print(dir(clf1))

    輸出：

    FeatureUnion(n_jobs=1, transformer_list=[('linear_pca)', PCA(copy=True, n_components=None, whiten=False)), ('Kernel_pca', KernelPCA(alpha=1.0, coef0=1, degree=3, eigen_solver='auto', fit_inverse_transform=False, gamma=None, kernel='linear', kernel_params=None, max_iter=None, n_components=None, remove_zero_eig=False, tol=0))], transformer_weights=None) FeatureUnion(n_jobs=1, transformer_list=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('kernelpca', KernelPCA(alpha=1.0, coef0=1, degree=3, eigen_solver='auto', fit_inverse_transform=False, gamma=None, kernel='linear', kernel_params=None, max_iter=None, n_components=None, remove_zero_eig=False, tol=0))], transformer_weights=None)

    能夠看出FeatureUnion的用法與pipeline一致

  • Note|說明

    (A FeatureUnion has no way of checking whether two transformers might produce identical features. It only produces a union when the feature sets are disjoint, and making sure they are is the caller’s responsibility.)

    Here is a example python source code:feature_stacker.py

 

1.2 Feature extraction

The sklearn.feature_extraction module can be used to extract features in a format supported by machine learning algorithms from datasets consisting of formats such as text and image.

skilearn.feature_extraction模塊是用機器學習算法所支持的數據格式來提取數據，如將text和image信息轉換成dataset。
Note:
Feature extraction（特徵提取）與Feature selection（特徵選擇）不一樣，前者是用來將非數值的數據轉換成數值的數據，後者是用機器學習的方法對特徵進行學習（如PCA降維）。

•  

  1.2.1 Loading features from dicts

  The class DictVectorizer can be used to convert feature arrays represented as lists of standard Python dict
  objects to the NumPy/SciPy representation used by scikit-learn estimators.
  Dictvectorizer類用來將python內置的dict類型轉換成數值型的array。dict類型的好處是在存儲稀疏數據時不用存儲無用的值。

  代碼：

  measurements=[{'city': 'Dubai', 'temperature': 33.} ,{'city': 'London', 'temperature':12.} ,{'city':'San Fransisco','temperature':18.},] from sklearn.feature_extraction import DictVectorizer vec=DictVectorizer() x=vec.fit_transform(measurements).toarray() print(x) print(vec.get_feature_names())

  輸出：

  [[  1. 0. 0. 33.] [ 0. 1. 0. 12.] [ 0. 0. 1. 18.]] ['city=Dubai', 'city=London', 'city=San Fransisco', 'temperature'] [Finished in 0.8s]

•  

  1.2.2 Feature hashing

•  

  1.2.3 Text feature extraction

•  

  1.2.4 Image feature extraction

  以上三小節暫未考慮（設計到語言處理及圖像處理）見官方文檔

 

1.3 Preprogressing data

The sklearn.preprocessing
package provides several common utility functions and transformer classes to change raw feature vectors into a representation that is more suitable for the downstream estimators

sklearn.preprogressing模塊提供了幾種常見的數據轉換，如標準化、歸一化等。

•  

  1.3.1 Standardization, or mean removal and variance scaling

  Standardization of datasets is a common requirement for many machine learning estimators implemented in the scikit; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.

  不少學習算法都要求事先對數據進行標準化，若是不是像標準正太分佈同樣0均值1方差就可能會有不好的表現。

  • Usage|用法

  代碼：

  from sklearn import preprocessing import numpy as np X = np.array([[1.,-1., 2.], [2.,0.,0.], [0.,1.,-1.]]) Y=X Y_scaled = preprocessing.scale(Y) y_mean=Y_scaled.mean(axis=0) #If 0, independently standardize each feature, otherwise (if 1) standardize each sample|axis=0 時求每一個特徵的均值，axis=1時求每一個樣本的均值 y_std=Y_scaled.std(axis=0) print(Y_scaled) scaler= preprocessing.StandardScaler().fit(Y)#用StandardScaler類也能完成一樣的功能 print(scaler.transform(Y))

  輸出：

  [[ 0. -1.22474487 1.33630621] [ 1.22474487 0. -0.26726124] [-1.22474487 1.22474487 -1.06904497]] [[ 0. -1.22474487 1.33630621] [ 1.22474487 0. -0.26726124] [-1.22474487 1.22474487 -1.06904497]] [Finished in 1.4s]

  • Note|說明
    1.func scale
    2.class StandardScaler
    3.StandardScaler 是一種Transformer方法，可讓pipeline來使用。
    MinMaxScaler （min-max標準化[0,1]）類和MaxAbsScaler（[-1,1]）類是另外兩個標準化的方式，用法和StandardScaler相似。
    4.處理稀疏數據時用MinMax和MaxAbs很合適
    5.魯棒的數據標準化方法（適用於離羣點不少的數據處理）：

    the median and the interquartile range often give better results

  用中位數代替均值（使均值爲0），用上四分位數-下四分位數代替方差（IQR爲1？）。

•  

  1.3.2 Impution of missing values|缺失值的處理

  • Usage
    代碼：

    import scipy.sparse as sp from sklearn.preprocessing import Imputer X=sp.csc_matrix([[1,2],[0,3],[7,6]]) imp=preprocessing.Imputer(missing_value=0,strategy='mean',axis=0) imp.fit(X) X_test=sp.csc_matrix([[0, 2], [6, 0], [7, 6]]) print(X_test) print(imp.transform(X_test))

    輸出：

    (1, 0) 6 (2, 0) 7 (0, 1) 2 (2, 1) 6 [[ 4. 2. ] [ 6. 3.66666675] [ 7. 6. ]] [Finished in 0.6s]

  • Note
    1.scipy.sparse是用來存儲稀疏矩陣的
    2.Imputer能夠用來處理scipy.sparse稀疏矩陣

•  

  1.3.3 Generating polynomial features

  • Usage
    代碼：

    import numpy as np from sklearn.preprocessing import PolynomialFeatures X=np.arange(6).reshape(3,2) print(X) poly=PolynomialFeatures(2) print(poly.fit_transform(X))

    輸出：

    [[0 1] [2 3] [4 5]] [[ 1. 0. 1. 0. 0. 1.] [ 1. 2. 3. 4. 6. 9.] [ 1. 4. 5. 16. 20. 25.]] [Finished in 0.8s]

  • Note
    生成多項式特徵用在多項式迴歸中以及多項式核方法中 。

•  

  1.3.4 Custom transformers

  這是用來構造transform方法的函數

  • Usage:
    代碼：

    import numpy as np from sklearn.preprocessing import FunctionTransformer transformer = FunctionTransformer(np.log1p) x=np.array([[0,1],[2,3]]) print(transformer.transform(x))

    輸出：

    [[ 0. 0.69314718] [ 1.09861229 1.38629436]] [Finished in 0.8s]

  • Note

    For a full code example that demonstrates using a FunctionTransformer to do custom feature selection, see Using FunctionTransformer to select columns

文／houhzize（簡書做者） 原文連接：http://www.jianshu.com/p/94516a58314d 著做權歸做者全部，轉載請聯繫做者得到受權，並標註「簡書做者」。