[Scikit-learn] 1.4 Support Vector Classification

Ref: http://sklearn.lzjqsdd.com/modules/svm.html

Ref: CS229 Lecture notes - Support Vector Machines

Ref: Lecture 6 | Machine Learning (Stanford) youtube

Ref: 支持向量機通俗導論（理解SVM的三層境界）

Ref: 《Kernel Methods for Pattern Analysis》

Ref: SVM教程：支持向量機的直觀理解【插圖來源於此連接，寫得不錯】

 

支持向量機

 

 

其實就是最大間隔分類器，並且是軟間隔，加正則。

 

對於時間充裕的年輕人，建議SVM的原理推導一遍，過程當中設計了大部分的數學優化基礎，並且SVM是自成體系的sol，大有裨益。 

Figure, SVM試圖找到右圖中的「分割線」

 

Support vector machines (SVMs) are a set of supervised learning methods used for classification (分類), regression (迴歸) and outliers detection ( 異常檢測).

優點

• Effective in high dimensional spaces. 【高維好操做】
• Still effective in cases where number of dimensions is greater than the number of samples. 【高維，例如語言模型，但效果好很差是另外一碼事】
• Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient. 【經過子集判斷】
• Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.

 

劣勢

• If the number of features is much greater than the number of samples, the method is likely to give poor performances.
• SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation.　

 

 

什麼是支持向量

支持向量（support vector）：距離最接近的數據點。

間隔（margin）：支持向量定義的沿着分隔線的區域。

有間隔就會影響分類結果中的偏差大小

SVM容許咱們經過參數 C 指定願意接受多少偏差，讓咱們能夠指定如下二者的折衷：

• • 較寬的間隔。正確分類 訓練數據 。
  • C值較高，意味着訓練數據上允許的偏差較少

 

 

什麼是核

升維使其可分

通常而言，很難找到這樣的特定投影。

不過，感謝Cover定理，咱們確實知道，投影到高維空間後，數據 更可能線性可分。

 

誰來作高維投影

SVM將使用一種稱爲 核（kernels）的東西進行投影，這至關迅速。

 

升維且高效

須要幾回運算？在二維情形下計算內積須要2次乘法、1次加法，而後平方又是1次乘法。因此總共是 4次運算，僅僅是以前先投影后計算的 運算量的31% 。

看來用核函數計算所需內積要快得多。在這個例子中，這點提高可能不算什麼：4次運算和13次運算。然而，若是數據點有許多維度，投影空間的維度更高，在大型數據集上，核函數節省的算力將飛速累積。這是核函數的巨大優點。

大多數SVM庫內置了流行的核函數，好比 多項式（Polynomial）、 徑向基函數（Radial Basis Function，RBF） 、 Sigmoid 。當咱們不進行投影時（好比本文的第一個例子），咱們直接在原始空間計算點積——咱們把這叫作使用 線性核（linear kernel） 。

 

 

徑向基函數 RBF

例：RBF kernel 【徑向基函數 (Radial Basis Function 簡稱 RBF), 就是某種沿徑向對稱的標量函數，也是默認kernel】

""" ============== Non-linear SVM ============== Perform binary classification using non-linear SVC with RBF kernel. The target to predict is a XOR of the inputs. The color map illustrates the decision function learned by the SVC. """
print(__doc__) import numpy as np import matplotlib.pyplot as plt from sklearn import svm 
# 生成網格型數據 xx, yy = np.meshgrid(np.linspace(-3, 3, 500), np.linspace(-3, 3, 500))
 np.random.seed(0) X = np.random.randn(300, 2) Y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0) # fit the model
clf = svm.NuSVC() clf.fit(X, Y) # plot the decision function for each datapoint on the grid
Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()]) Z = Z.reshape(xx.shape) plt.imshow(Z, interpolation='nearest', extent=(xx.min(), xx.max(), yy.min(), yy.max()), aspect='auto', origin='lower', cmap=plt.cm.PuOr_r) contours = plt.contour(xx, yy, Z, levels=[0], linewidths=2, linetypes='--')
 plt.scatter(X[:, 0], X[:, 1], s=30, c=Y, cmap=plt.cm.Paired) plt.xticks(()) plt.yticks(()) plt.axis([-3, 3, -3, 3]) plt.show()

變量：xx, yy

xx
Out[140]: 
array([[-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ],
       [-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ],
       [-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ],
       ..., 
       [-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ],
       [-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ],
       [-3.        , -2.98797595, -2.9759519 , ...,  2.9759519 ,
         2.98797595,  3.        ]])

yy
Out[141]: 
array([[-3.        , -3.        , -3.        , ..., -3.        ,
        -3.        , -3.        ],
       [-2.98797595, -2.98797595, -2.98797595, ..., -2.98797595,
        -2.98797595, -2.98797595],
       [-2.9759519 , -2.9759519 , -2.9759519 , ..., -2.9759519 ,
        -2.9759519 , -2.9759519 ],
       ..., 
       [ 2.9759519 ,  2.9759519 ,  2.9759519 , ...,  2.9759519 ,
         2.9759519 ,  2.9759519 ],
       [ 2.98797595,  2.98797595,  2.98797595, ...,  2.98797595,
         2.98797595,  2.98797595],
       [ 3.        ,  3.        ,  3.        , ...,  3.        ,
         3.        ,  3.        ]])

View Code

np.c_ 降維後的元素的reconstruct

np.c_[np.array([1,2,3]), np.array([4,5,6])]
Out[142]: 
array([[1, 4],
       [2, 5],
       [3, 6]])

np.c_[np.array([[1,2,3]]), 0, 0, np.array([[4,5,6]])]
Out[143]: array([[1, 2, 3, 0, 0, 4, 5, 6]])

View Code

 

  

不一樣的核：Various kernels

可見，RBF 的分割更爲細緻。

""" ================================ SVM Exercise ================================ A tutorial exercise for using different SVM kernels. This exercise is used in the :ref:`using_kernels_tut` part of the :ref:`supervised_learning_tut` section of the :ref:`stat_learn_tut_index`. """
print(__doc__) import numpy as np import matplotlib.pyplot as plt from sklearn import datasets, svm iris = datasets.load_iris() X = iris.data y = iris.target X = X[y != 0, :2] y = y[y != 0] n_sample = len(X) np.random.seed(0) order = np.random.permutation(n_sample) X = X[order] y = y[order].astype(np.float) # shuffle
 X_train = X[:.9 * n_sample] y_train = y[:.9 * n_sample] X_test = X[ .9 * n_sample:] y_test = y[ .9 * n_sample:] # fit the model
for fig_num, kernel in enumerate(('linear', 'rbf', 'poly')): clf = svm.SVC(kernel=kernel, gamma=10) clf.fit(X_train, y_train) plt.figure(fig_num) plt.clf() plt.scatter(X[:, 0], X[:, 1], c=y, zorder=10, cmap=plt.cm.Paired) # Circle out the test data
    plt.scatter(X_test[:, 0], X_test[:, 1], s=80, facecolors='none', zorder=10) plt.axis('tight') x_min = X[:, 0].min() x_max = X[:, 0].max() y_min = X[:, 1].min() y_max = X[:, 1].max() XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j] Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()]) # Put the result into a color plot
    Z = Z.reshape(XX.shape) plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.Paired) plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'], levels=[-.5, 0, .5]) plt.title(kernel) plt.show()

Result:

 

End.