基於SVM的思想作CIFAR 10圖像分類

時間 2019-12-10

標籤基於 svm 思想 cifar 圖像分類简体版

原文原文鏈接

#SVM 回顧一下以前的SVM，找到一個間隔最大的函數，使得正負樣本離該函數是最遠的，是否最遠不是看哪一個點離函數最遠，而是找到一個離函數最近的點看他是否是和該分割函數離的最近的。python

使用large margin來regularization。 以前講SVM的算法：www.jianshu.com/p/8fd28df73… #線性分類線性SVM就是一種線性分類的方法。輸入

，輸出

，每個樣本的權重是

，偏置項bias是

。得分函數$$s = wx +b$$ 算出這麼多個類別，哪個類別的分數高，那就是哪一個類別。好比要作的圖像識別有三個類別

，假設這張圖片有4個像素，拉伸成單列：

獲得的結果很明顯是dog分數最大，cat的分數最低，可是圖片很明顯是貓，什麼分類器是錯誤的。通常來講習慣會把w和b合併了，x加上一個全爲1的列，因而有$$W=[w;b];X = [x;1]$$

#損失函數以前的SVM是把正負樣本離分割函數有足夠的空間，雖然正確的是貓，可是貓的得分是最低的，常規方法是將貓的分數提升，這樣才能夠提升貓的正確率。可是SVM裏面是要求一個間隔最大化，提到這裏來講，其實就是cat score不只僅是要大於其餘的分數，並且是要有一個最低閾值，cat score不能低於這個分數。因此正確的分類score應該是要大於其餘的分類score一個閾值：$$s_{y_i} >= s_j + \triangle$$ $s_{y_i}$ 就是正確分類的分數，

就是其餘分類的分數。因此，這個損失函數就是：$$Loss_{y_i} = \sum_{j != y_i}max(0, s_j - s_{y_i}+\triangle)$$只有正確的分數比其餘的都大於一個閾值才爲0，不然都是有損失的。

只有 $s_j-s_{y_i}+\triangle <= 0$ 損失函數纔是0的。這種損失函數稱爲合頁損失函數，用的就是SVM間隔最大化的思想解決，若是損失函數爲0，那麼不用求解了，若是損失函數不爲0，就能夠用梯度降低求解。max求解梯度降低有點不現實，因此天然就有了square的合頁損失函數。

Loss_{y_i} = \sum_{j != y_i}max(0, s_j - s_{y_i}+\triangle)^2

這種squared hinge loss SVM與linear hinge loss SVM相比較，特色是對違背間隔閾值要求的點加劇懲罰，違背的越大，懲罰越大。某些實際應用中，squared hinge loss SVM的效果更好一些。具體使用哪一個，能夠根據實際問題，進行交叉驗證再肯定。對於 $\triangle$ 的設置，以前SVM其實討論過，對於一個平面是能夠隨意伸縮的，只須要增大w和b就能夠隨意把 $\triangle$ 增大，因此把它定爲1，也就是設置 $\triangle = 1$ 。由於w的增加或縮小徹底能夠抵消 $\triangle$ 的影響。這個時候損失函數就是：算法

Loss_{y_i} = \sum_{j != y_i}max(0, s_j - s_{y_i}+1)

最後還要增長的就是過擬合，regularization的限制了。L2正則化：bash

加上正則化以後就是：app

N是訓練樣本的個數，取平均損失函數， $\lambda$ 就是懲罰的力度了，能夠小也能夠大，若是大了可能w不足以抵消正負樣本之間的間隔，可能會欠擬合，由於 $\triangle = 1$ 是在w能夠自由伸縮達到的條件，若是w過小，可能就不足以增加到1了。若是小了，可能就會形成overfit。對於參數b就沒有這麼講究了。 #代碼實現首先是對CIFAR10的數據讀取：dom

def load_pickle(f):
    version = platform.python_version_tuple()
    if version[0] == '2':
        return pickle.load(f)
    elif version[0] == '3':
        return pickle.load(f, encoding='latin1')
    raise ValueError("invalid python version: {}".format(version))

def loadCIFAR_batch(filename):
    with open(filename, 'rb') as f:
        datadict = load_pickle(f)
        x = datadict['data']
        y = datadict['labels']
        x = x.reshape(10000, 3, 32, 32).transpose(0, 3, 2, 1).astype('float')
        y = np.array(y)
        return x, y

def loadCIFAR10(root):
    xs = []
    ys = []
    for b in range(1, 6):
        f = os.path.join(root, 'data_batch_%d' % (b, ))
        x, y = loadCIFAR_batch(f)
        xs.append(x)
        ys.append(y)
    X = np.concatenate(xs)
    Y = np.concatenate(ys)
    x_test, y_test = loadCIFAR_batch(os.path.join(root, 'test_batch'))
    return X, Y, x_test, y_test
複製代碼

首先要讀入每個文件的數據，先用load_pickle把文件讀成字典形式，取出來。由於常規的圖片都是(數量，高，寬，RGB顏色)，在loadCIFAR_batch要用transpose來把維度調換一下。最後把每個文件的數據都集合起來。以後就是數據的格式調整了：

def data_validation(x_train, y_train, x_test, y_test):
    num_training = 49000
    num_validation = 1000
    num_test = 1000
    num_dev = 500
    mean_image = np.mean(x_train, axis=0)
    x_train -= mean_image
    mask = range(num_training, num_training + num_validation)
    X_val = x_train[mask]
    Y_val = y_train[mask]
    mask = range(num_training)
    X_train = x_train[mask]
    Y_train = y_train[mask]
    mask = np.random.choice(num_training, num_dev, replace=False)
    X_dev = x_train[mask]
    Y_dev = y_train[mask]
    mask = range(num_test)
    X_test = x_test[mask]
    Y_test = y_test[mask]
    X_train = np.reshape(X_train, (X_train.shape[0], -1))
    X_val = np.reshape(X_val, (X_val.shape[0], -1))
    X_test = np.reshape(X_test, (X_test.shape[0], -1))
    X_dev = np.reshape(X_dev, (X_dev.shape[0], -1))
    X_train = np.hstack([X_train, np.ones((X_train.shape[0], 1))])
    X_val = np.hstack([X_val, np.ones((X_val.shape[0], 1))])
    X_test = np.hstack([X_test, np.ones((X_test.shape[0], 1))])
    X_dev = np.hstack([X_dev, np.ones((X_dev.shape[0], 1))])
    return X_val, Y_val, X_train, Y_train, X_dev, Y_dev, X_test, Y_test
    pass
複製代碼

數據要變成一個長條。先看看數據長啥樣：函數

def showPicture(x_train, y_train):
    classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
    num_classes = len(classes)
    samples_per_classes = 7
    for y, cls in enumerate(classes):
        idxs = np.flatnonzero(y_train == y)
        idxs = np.random.choice(idxs, samples_per_classes, replace=False)
        for i, idx in enumerate(idxs):
            plt_index = i*num_classes +y + 1
            plt.subplot(samples_per_classes, num_classes, plt_index)
            plt.imshow(x_train[idx].astype('uint8'))
            plt.axis('off')
            if i == 0:
                plt.title(cls)
    plt.show()
複製代碼

而後就是使用谷歌的公式了：

def loss(self, x, y, reg):
        loss = 0.0
        dw = np.zeros(self.W.shape)
        num_train = x.shape[0]
        scores = x.dot(self.W)
        correct_class_score = scores[range(num_train), list(y)].reshape(-1, 1)
        margin = np.maximum(0, scores - correct_class_score + 1)
        margin[range(num_train), list(y)] = 0
        loss = np.sum(margin)/num_train + 0.5 * reg * np.sum(self.W*self.W)

        num_classes = self.W.shape[1]
        inter_mat = np.zeros((num_train, num_classes))
        inter_mat[margin > 0] = 1
        inter_mat[range(num_train), list(y)] = 0
        inter_mat[range(num_train), list(y)] = -np.sum(inter_mat, axis=1)

        dW = (x.T).dot(inter_mat)
        dW = dW/num_train + reg*self.W
        return loss, dW
        pass
複製代碼

操做都是常規操做，算出score而後求loss最後SGD求梯度更新W。ui

def train(self, X, y, learning_rate=1e-3, reg=1e-5, num_iters=100,batch_size=200, verbose=False):
        num_train, dim = X.shape
        num_classes = np.max(y) + 1
        if self.W is None:
            self.W = 0.001 * np.random.randn(dim, num_classes)
        # Run stochastic gradient descent to optimize W
        loss_history = []
        for it in range(num_iters):
            X_batch = None
            y_batch = None
            idx_batch = np.random.choice(num_train, batch_size, replace = True)
            X_batch = X[idx_batch]
            y_batch = y[idx_batch]
            # evaluate loss and gradient
            loss, grad = self.loss(X_batch, y_batch, reg)
            loss_history.append(loss)
            self.W -=  learning_rate * grad
            if verbose and it % 100 == 0:
                print('iteration %d / %d: loss %f' % (it, num_iters, loss))
        return loss_history
        pass
複製代碼

預測：lua

def predict(self, X):
        y_pred = np.zeros(X.shape[0])
        scores = X.dot(self.W)
        y_pred = np.argmax(scores, axis = 1)
        return y_pred
複製代碼

最後運行函數：spa

svm = LinearSVM()
    tic = time.time()
    cifar10_name = '../Data/cifar-10-batches-py'
    x_train, y_train, x_test, y_test = loadCIFAR10(cifar10_name)
    X_val, Y_val, X_train, Y_train, X_dev, Y_dev, X_test, Y_test = data_validation(x_train, y_train, x_test, y_test)
    loss_hist = svm.train(X_train, Y_train, learning_rate=1e-7, reg=2.5e4,
                          num_iters=3000, verbose=True)
    toc = time.time()
    print('That took %fs' % (toc - tic))
    plt.plot(loss_hist)
    plt.xlabel('Iteration number')
    plt.ylabel('Loss value')
    plt.show()
    y_test_pred = svm.predict(X_test)
    test_accuracy = np.mean(Y_test == y_test_pred)
    print('accuracy: %f' % test_accuracy)
    w = svm.W[:-1, :]  # strip out the bias
    w = w.reshape(32, 32, 3, 10)
    w_min, w_max = np.min(w), np.max(w)
    classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
    for i in range(10):
        plt.subplot(2, 5, i + 1)
        wimg = 255.0 * (w[:, :, :, i].squeeze() - w_min) / (w_max - w_min)
        plt.imshow(wimg.astype('uint8'))
        plt.axis('off')
        plt.title(classes[i])
    plt.show()

複製代碼