【CV知識學習】early stop、regularation、fine-tuning and some other trick to be known

深度學習有很多的trick，並且這些trick有時還挺管用的，因此，瞭解一些trick仍是必要的。上篇說的normalization、initialization就是trick的一種，下面再總結一下本身看Deep Learning Summer School, Montreal 2016 總結的一些trick。請路過大牛指正~~~

 

early stop

「早中止」很好理解，就是在validation的error開始上升以前，就把網絡的訓練中止了。說到這裏，把數據集分紅train、validation和test也算是一種trick吧。看一張圖就明白了：

 

 

L一、L2 regularization

第一次接觸正則化是在學習PRML的迴歸模型，那時還不太明白，如今能夠詳細講講其中的原理了。額~~~~(I will write the surplus of this article in English)

At the very outset, we take a insight on L1, L2 regularization. Assume the loss function of a linear regression model as . In fact, L1, L2 regularization can be seen as introducing prior distribution for the parameters. Here, L1 regularization can be interpreted as Laplace prior, and Guass prior for L2 regularization.Such tricks are used to reduce the complexity of a model.

 

L2 regularization

As before, we make a hypothesis that the target variable  is determined by the function  with an additive guass noise ,  resulting in, where is a zero mean Guassian random variable with precision . Thus we know  and its loglikelihood function form can be wrote as 

Finding the optimal solution for this problem is equal to minimize the least squre function .   If a zero mean  variance Guass prior is introduced for ,  we get the posterior distribution for  from the function (need to know prior + likelihood = posterior)

rewrite the equaltion in loglikelihood form, we will get 

 

 at the present,  you see . This derivation process accounts for why L2 regularization could be interpreted as Guass Prior.

 

L1 regularization

Generally, solving the the L1 regularization is called LASSO problem, which will force some coefficients to zero thus create a sparse model. You can get reference from here.

  

From the above images(two-dimensionality), something can be discovered that for the L1 regularization, the interaction points always locate on the axis, and this will forces some of the variables to be zero. Addtionally, we can learn that L2 regularization doesn't produce a sparse model(the right side image).

 

Fine tuning

It's easy to understand fine tuning, which could be interpreted as a way to initailizes the weight parameters then proceed a supervised learning. Parameters value can be migrated from a well-trained network, such as ImageNet. However, sometimes you may need an autoencoder, which is an unsupervised learning method and seems not very popular now. An autoencoder will learn the latent structure of the data in hidden layers, whose input and output should be same.

In fact, supervised learning of fine tune just performs regular feed forward process as in a feed-forwad network.

Then, I want to show you an image about autoencoder, also you can get more information in detail through this website.

 

Data argumentation

 If training data is not large enough, it's a neccessary to expand the data by fliping, transferring or rotating to avoid overfit. However, according to the paper "Visualizing and Understanding Convolutional Networks",  CNN may not be as stable when fliping for image as transferring and rotating, except a symmetrical image. 

 

Pooling

I have introduced why pooling works well in the ealier article in this blog, if interested, you may need patience to look through the catalogue. In the paper "Visualizing and Understanding Convolutional Networks", you will see pooling sometimes a good way to control the invariance, I want to write down my notes about this classical paper in next article.

 

Dropout and Drop layer

Dropout now has become a prevalent technique to prevent overfitting, proposed by Hinton, etc. The dropout program will inactivate some nodes according to a probability p, which has been fixed before training. It makes the network more thinner. But why does it works ?  There may exist two main reasons, (1) for a batch of input data, nodes can be trained better in a thinner network, for they trained with more data in average.  (2) dropout inject noise into network which will do help.

Inspired by dropout and ResNet, Huang, etc propose drop layer, which make use of the shortcut connection in ResNet. It will enforce the information flow through the shortcut connection directly and obstruct the normal way according to a probability p just like what dropout does. This technique makes the network  shorter in training, and can be used to trained a deeper network even more than 1000 layers.

 

Depth is important

The chart below interprets the reason why depth is important,

as the number of layer increasing, the classification accuracy arises. While visualizing the feature extracted from layers using decovnet("Visualizing and Understanding Convolutional Networks"), the deeper layer the more abstract feature extracted. And abstract feature will strengthen the  robustness, i.e. the invariance to transformation.