TensorFlow學習筆記（一）

TF-Slim的優點：slim做爲一種輕量級的tensorflow庫，使得模型的構建，訓練，測試都變得更加簡單。

1. 使用方法：

    import tensorflow.contrib.slim as slim

2. 組成部分：

    arg_scope: 使得用戶能夠在同一個arg_scope中使用默認的參數

    data，evaluation，layers，learning，losses，metrics，nets，queues，regularizers，variables

3. 定義模型

    在slim中，組合使用variables, layers和scopes能夠簡潔的定義模型。

    （1）variables: 定義於variables.py。生成一個weight變量, 用truncated normal初始化它, 並使用l2正則化，並將其放置於CPU上, 只需下面的代碼便可:

           weights = slim.variable('weights', shape=[10, 10, 3 , 3], 

                                initializer=tf.truncated_normal_initializer(stddev=0.1), 

                             regularizer=slim.l2_regularizer(0.05),

                             device='/CPU:0')

        原生tensorflow包含兩類變量：普通變量和局部變量。大部分變量都是普通變量，它們一旦生成就能夠經過食用saver存入硬盤，局部變量只在session中存在，不會保存。

        slim進一步的區分了變量類型，定義了model variables，這種變量表明瞭模型的參數。模型變量經過訓練活着微調而獲得學習，活着在評測或前傳中能夠從checkpoint文件中載入。

        非模型參數在實際前傳中不須要的參數，好比global_step。一樣的，移動平均反應了模型參數，但它自己不是模型參數。例子見下：

        

　　# Model Variables 　　weights = slim.model_variable('weights', 　　 shape=[10, 10, 3 , 3], 　　initializer=tf.truncated_normal_initializer(stddev=0.1), 　　regularizer=slim.l2_regularizer(0.05), 　　device='/CPU:0') 　　model_variables = slim.get_model_variables() 　　# model_variables包含weights 　　# Regular variables 　　my_var = slim.variable('my_var', 　　 shape=[20, 1], 　　initializer=tf.zeros_initializer()) 　　regular_variables_and_model_variables = slim.get_variables()
　　＃get_variables()獲得模型參數和常規參數

當咱們經過slim的layers或着直接使用slim.model_variable建立變量時，tf會將此變量加入tf.GraphKeys.MODEL_VARIABLES這個集合中,當你須要構建本身的變量時，能夠經過如下代碼
將其加入模型參數。

　　my_model_variable = CreateViaCustomCode()

　　# Letting TF-Slim know about the additional variable. 　　slim.add_model_variable(my_model_variable)

（2）layers：抽象並封裝了經常使用的層，而且提供了repeat和stack操做，使得定義網絡更加方便。
repeat不只只實現了相同操做相同參數的重複，它還將scope進行了展開，例子中的scope被展開爲 
'conv3/conv3_1', 'conv3/conv3_2' and 'conv3/conv3_3'。
 操做使得咱們能夠重複的講同一個操做以不一樣參數一次做用於一些層，這些層的輸入輸出時串聯起來的。
首先，
slim.stack

input = ...
net = slim.conv2d(input, 128, [3, 3], scope='conv1_1'

但這個不是重要的，由於tenorflow目前也有大部分層的簡單實現，這裏比較吸引人的是slim中的repeat和stack操做：

假設定義三個相同的卷積層：

net = ...
net = slim.conv2d(net, 256, [3, 3], scope='conv3_1')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_2')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')

在slim中的repeat操做能夠減小代碼量：

net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')

而stack是處理卷積核或者輸出不同的狀況：

假設定義三層FC：

# Verbose way:
x = slim.fully_connected(x, 32, scope='fc/fc_1')
x = slim.fully_connected(x, 64, scope='fc/fc_2')
x = slim.fully_connected(x, 128, scope='fc/fc_3')

使用stack操做：

slim.stack(x, slim.fully_connected, [32, 64, 128], scope='fc')

同理卷積層也同樣：

# 普通方法:
x = slim.conv2d(x, 32, [3, 3], scope='core/core_1')
x = slim.conv2d(x, 32, [1, 1], scope='core/core_2')
x = slim.conv2d(x, 64, [3, 3], scope='core/core_3')
x = slim.conv2d(x, 64, [1, 1], scope='core/core_4')

# 簡便方法:
slim.stack(x, slim.conv2d, [(32, [3, 3]), (32, [1, 1]), (64, [3, 3]), (64, [1, 1])], scope='core');
（3）scopes：除了tensorflow中的name_scope和variable_scope, tf.slim新增了arg_scope操做，
這一操做符可讓定義在這一scope中的操做共享參數，即如不制定參數的話，則使用默認參數。且參數能夠被局部覆蓋。使得代碼更加簡潔，以下：

　　with slim.arg_scope([slim.conv2d], padding='SAME', 　　 weights_initializer=tf.truncated_normal_initializer(stddev=0.01) 　　 weights_regularizer=slim.l2_regularizer(0.0005)): 　　 net = slim.conv2d(inputs, 64, [11, 11], scope='conv1') 　　 net = slim.conv2d(net, 128, [11, 11], padding='VALID', scope='conv2') 　　 net = slim.conv2d(net, 256, [11, 11], scope='conv3')
並且，咱們也能夠嵌套多個arg_scope在其中使用多個操做,如：

　　with slim.arg_scope([slim.conv2d, slim.fully_connected],
   　　                   activation_fn=tf.nn.relu, 　　weights_initializer=tf.truncated_normal_initializer(stddev=0.01), 　　 weights_regularizer=slim.l2_regularizer(0.0005)): 　　 with slim.arg_scope([slim.conv2d], stride=1, padding='SAME'): 　　 net = slim.conv2d(inputs, 64, [11, 11], 4, padding='VALID', scope='conv1') 　　 net = slim.conv2d(net, 256, [5, 5], 　　 weights_initializer=tf.truncated_normal_initializer(stddev=0.03), 　　 scope='conv2') 　　 net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc')
在此例子中，第一個arg_scope將weights_initializer和

def vgg16(inputs): with slim.arg_scope([slim.conv2d, slim.fully_connected], activation_fn=tf.nn.relu, weights_initializer=tf.truncated_normal_initializer(0.0, 0.01), weights_regularizer=slim.l2_regularizer(0.0005)): net = slim.repeat(inputs, 2, slim.conv2d, 64, [3, 3], scope='conv1') net = slim.max_pool2d(net, [2, 2], scope='pool1') net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2') net = slim.max_pool2d(net, [2, 2], scope='pool2') net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3') net = slim.max_pool2d(net, [2, 2], scope='pool3') net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4') net = slim.max_pool2d(net, [2, 2], scope='pool4') net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5') net = slim.max_pool2d(net, [2, 2], scope='pool5') net = slim.fully_connected(net, 4096, scope='fc6') net = slim.dropout(net, 0.5, scope='dropout6') net = slim.fully_connected(net, 4096, scope='fc7') net = slim.dropout(net, 0.5, scope='dropout7') net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc8') return net
4. 訓練模型
Tensorflow的模型訓練須要模型，損失函數，梯度計算，以及根據loss的梯度迭代更新參數。
（1）losses
使用現有的loss：

loss = slim.losses.softmax_cross_entropy(predictions, labels)
對於多任務學習的loss，能夠使用：

# Define the loss functions and get the total loss. classification_loss = slim.losses.softmax_cross_entropy(scene_predictions, scene_labels) sum_of_squares_loss = slim.losses.sum_of_squares(depth_predictions, depth_labels) # The following two lines have the same effect: total_loss = classification_loss + sum_of_squares_loss total_loss = slim.losses.get_total_loss(add_regularization_losses=False)
若是使用了本身定義的loss，而又想使用slim的loss管理機制，能夠使用：

pose_loss = MyCustomLossFunction(pose_predictions, pose_labels)
slim.losses.add_loss(pose_loss) 
total_loss = slim.losses.get_total_loss()
＃total_loss中包涵了pose_loss
(2) 訓練循環
slim在learning.py中提供了一個簡單而有用的訓練模型的工具。咱們只需調用 和slim.learning.create_train_opslim.learning.train就能夠完成優化過程。

g = tf.Graph()

# Create the model and specify the losses... ... total_loss = slim.losses.get_total_loss() optimizer = tf.train.GradientDescentOptimizer(learning_rate) # create_train_op ensures that each time we ask for the loss, the update_ops # are run and the gradients being computed are applied too. train_op = slim.learning.create_train_op(total_loss, optimizer) logdir = ... # Where checkpoints are stored. slim.learning.train( train_op, logdir, number_of_steps=1000,＃迭代次數 save_summaries_secs=300,＃存summary間隔秒數 save_interval_secs=600)＃存模型建個秒數
（3）訓練的例子：

import tensorflow as tf slim = tf.contrib.slim vgg = tf.contrib.slim.nets.vgg ... train_log_dir = ... if not tf.gfile.Exists(train_log_dir): tf.gfile.MakeDirs(train_log_dir) with tf.Graph().as_default(): # Set up the data loading: images, labels = ... # Define the model: predictions = vgg.vgg16(images, is_training=True) # Specify the loss function: slim.losses.softmax_cross_entropy(predictions, labels) total_loss = slim.losses.get_total_loss() tf.summary.scalar('losses/total_loss', total_loss) # Specify the optimization scheme: optimizer = tf.train.GradientDescentOptimizer(learning_rate=.001) # create_train_op that ensures that when we evaluate it to get the loss, # the update_ops are done and the gradient updates are computed. train_tensor = slim.learning.create_train_op(total_loss, optimizer) # Actually runs training. slim.learning.train(train_tensor, train_log_dir)
5. 根據已有模型進行微調
（1）利用tf.train.Saver()從checkpoint恢復模型

# Create some variables. v1 = tf.Variable(..., name="v1") v2 = tf.Variable(..., name="v2") ... # Add ops to restore all the variables. restorer = tf.train.Saver() # Add ops to restore some variables. restorer = tf.train.Saver([v1, v2]) # Later, launch the model, use the saver to restore variables from disk, and # do some work with the model. with tf.Session() as sess: # Restore variables from disk. restorer.restore(sess, "/tmp/model.ckpt") print("Model restored.") # Do some work with the model ...
（2）部分恢復模型參數

# Create some variables. v1 = slim.variable(name="v1", ...) v2 = slim.variable(name="nested/v2", ...) ... # Get list of variables to restore (which contains only 'v2'). These are all # equivalent methods: variables_to_restore = slim.get_variables_by_name("v2") # or variables_to_restore = slim.get_variables_by_suffix("2") # or variables_to_restore = slim.get_variables(scope="nested") # or variables_to_restore = slim.get_variables_to_restore(include=["nested"]) # or variables_to_restore = slim.get_variables_to_restore(exclude=["v1"]) # Create the saver which will be used to restore the variables. restorer = tf.train.Saver(variables_to_restore) with tf.Session() as sess: # Restore variables from disk. restorer.restore(sess, "/tmp/model.ckpt") print("Model restored.") # Do some work with the model ...
（3）當圖的變量名與checkpoint中的變量名不一樣時，恢復模型參數
 當從checkpoint文件中恢復變量時，Saver在checkpoint文件中定位到變量名，而且把它們映射到當前圖中的變量中。以前的例子中，咱們建立了Saver，併爲其提供了變量列表做爲參數。這時，在checkpoint文件中定位的變量名，是隱含地從每一個做爲參數給出的變量的var.op.name而得到的。這一方式在圖與checkpoint文件中變量名字相同時，能夠很好的工做。而當名字不一樣時，必須給Saver提供一個將checkpoint文件中的變量名映射到圖中的每一個變量的字典，例子見下：

# Assuming that 'conv1/weights' should be restored from 'vgg16/conv1/weights' def name_in_checkpoint(var): return 'vgg16/' + var.op.name # Assuming that 'conv1/weights' and 'conv1/bias' should be restored from 'conv1/params1' and 'conv1/params2' def name_in_checkpoint(var): if "weights" in var.op.name: return var.op.name.replace("weights", "params1") if "bias" in var.op.name: return var.op.name.replace("bias", "params2") variables_to_restore = slim.get_model_variables() variables_to_restore = {name_in_checkpoint(var):var for var in variables_to_restore} restorer = tf.train.Saver(variables_to_restore) with tf.Session() as sess: # Restore variables from disk. restorer.restore(sess, "/tmp/model.ckpt")
（4）在一個不一樣的任務上對網絡進行微調
好比咱們要將1000類的imagenet分類任務應用於20類的Pascal VOC分類任務中，咱們只導入部分層，見下例：

image, label = MyPascalVocDataLoader(...) images, labels = tf.train.batch([image, label], batch_size=32) # Create the model predictions = vgg.vgg_16(images) train_op = slim.learning.create_train_op(...) # Specify where the Model, trained on ImageNet, was saved. model_path = '/path/to/pre_trained_on_imagenet.checkpoint' # Specify where the new model will live: log_dir = '/path/to/my_pascal_model_dir/' # Restore only the convolutional layers: variables_to_restore = slim.get_variables_to_restore(exclude=['fc6', 'fc7', 'fc8']) init_fn = assign_from_checkpoint_fn(model_path, variables_to_restore) # Start training. slim.learning.train(train_op, log_dir, init_fn=init_fn)