深度有趣 | 22 天馬行空的DeepDream

簡介

介紹DeepDream的原理並用TensorFlow實現

效果

先來看一下DeepDream的效果，原本是這樣一張圖片

通過DeepDream處理以後就有可能變成這樣

有點奇特和夢幻，也有點不明因此、精神污染

原理

大多時候咱們是根據給定的數據和標籤，去訓練和調整網絡的參數

不過也有時候，咱們是固定網絡的參數，根據某個損失函數調整輸入數據，例如在圖像風格遷移裏，根據內容損失函數和風格損失函數調整合成的圖片

對於常見的圖片分類模型，輸入一張圖片，網絡中的每一個tensor會輸出相應的響應值，值越大說明這個tensor越「喜歡」這張圖片

好比輸入一張狗的圖片，網絡中用於識別和分類狗的tensor就會輸出較大的響應值

把優化目標設爲最大化某個tensor的響應值，以此來調整輸入圖片，這就是DeepDream的原理

舉例來講，爲了知足一個喜歡狗的tensor，咱們將原始圖片中像狗的一些蛛絲馬跡進行調整和放大，從而使得這一tensor的響應值更大

實現

加載庫

# -*- coding: utf-8 -*-

import tensorflow as tf
import numpy as np
import cv2
from imageio import imread, imsave, mimsave
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.ndimage.filters import gaussian_filter

加載圖片分類模型，這裏使用inception5h

layer_names = ['conv2d0', 'conv2d1', 'conv2d2', 
               'mixed3a', 'mixed3b', 'mixed4a', 'mixed4b', 'mixed4c', 'mixed4d', 'mixed4e',
               'mixed5a', 'mixed5b']

graph = tf.Graph()
with graph.as_default():
    with tf.gfile.FastGFile('inception5h.pb', 'rb') as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())
        tf.import_graph_def(graph_def, name='')
    X = graph.get_tensor_by_name('input:0')
    layers = [graph.get_tensor_by_name(name + ':0') for name in layer_names]
    
    all_layers_names = [tensor.name for tensor in tf.get_default_graph().as_graph_def().node]
    print(all_layers_names)

sess = tf.Session(graph=graph)

定義獲取梯度tensor的函數、對原始圖片按塊計算梯度的函數

def get_gradient(tensor):
    with graph.as_default():
        return tf.gradients(tf.reduce_mean(tf.square(tensor)), X)[0]

def get_tile_size(num_pixels, tile_size=400):
    num_tiles = max(1, int(round(num_pixels / tile_size)))
    return int(np.ceil(num_pixels / num_tiles))
    
def tiled_gradient(gradient, image, tile_size=400):
    grad = np.zeros_like(image)
    H, W, _ = image.shape
    
    h = get_tile_size(H, tile_size)
    h_4 = h // 4
    w = get_tile_size(W, tile_size)
    w_4 = w // 4
    
    h_start = np.random.randint(-3 * h_4, -h_4)
    while h_start < H:
        h_end = h_start + h
        h_start_lim = max(h_start, 0)
        h_end_lim = min(h_end, H)
        
        w_start = np.random.randint(-3 * w_4, -w_4)
        while w_start < W:
            w_end = w_start + w
            w_start_lim = max(w_start, 0)
            w_end_lim = min(w_end, W)
            
            g = sess.run(gradient, feed_dict={X: [image[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :]]})[0]
            g /= (np.std(g) + 1e-8)
            
            grad[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :] = g
            
            w_start = w_end
        
        h_start = h_end
    
    return grad

根據梯度調整輸入圖片，即DeepDream

def dream(layer_tensor, image, iteration=10, step=3.0, tile_size=400):
    img = image.copy()
    gradient = get_gradient(layer_tensor)
    
    for i in range(iteration):
        grad = tiled_gradient(gradient, img)
        
        sigma = (i * 4.0) / iteration + 0.5
        grad = gaussian_filter(grad, 0.5 * sigma) + gaussian_filter(grad, sigma) + gaussian_filter(grad, 2 * sigma)
        
        scaled_step = step / (np.std(grad) + 1e-8)
        img += grad * scaled_step
        img = np.clip(img, 0, 255)
        
    return img

將原始圖片進行縮放，對多個尺度進行DeepDream處理併疊加

def recursive_dream(layer_tensor, image, repeat=3, scale=0.7, blend=0.2, iteration=10, step=3.0, tile_size=400):
    if repeat > 0:
        sigma = 0.5
        img_blur = gaussian_filter(image, (sigma, sigma, 0.0))
        
        h0 = img_blur.shape[0]
        w0 = img_blur.shape[1]
        h1 = int(scale * h0)
        w1 = int(scale * w0)
        img_downscaled = cv2.resize(img_blur, (w1, h1))
        
        img_dream = recursive_dream(layer_tensor, img_downscaled, repeat - 1, scale, blend, iteration, step, tile_size)
        img_upscaled = cv2.resize(img_dream, (w0, h0))
        
        image = blend * image + (1.0 - blend) * img_upscaled
        image = np.clip(image, 0, 255)
    
    return dream(layer_tensor, image, iteration, step, tile_size)

讀取一張圖片

image = imread('mountain.jpg')
image = image.astype(np.float32)

分別以12個tensor的響應值做爲優化目標，對原始圖片進行處理

for i in range(len(layers)):
    print(layer_names[i])
    result = recursive_dream(layers[i], image)
    plt.figure(figsize=(10, 15))
    plt.imshow(result / 255.)
    plt.show()
    imsave('imgs/%s.jpg' % layer_names[i], result)

conv2d2的DeepDream結果

mixed3a的DeepDream結果

mixed4c的DeepDream結果

mixed5a的DeepDream結果

隨着tensor所在的層數變深，DeepDream優化出來的圖形也更加複雜

除了將某個tensor整個做爲目標，也能夠僅選擇一個filter的響應值進行優化

例如選擇mixed4c的某個filter，能夠看到不一樣的filter偏好的圖形是不同的

for i in range(10):
    print('Filter %d of mixed4c' % i)
    result = recursive_dream(layers[7][:, :, :, i], image)
    plt.figure(figsize=(10, 15))
    plt.imshow(result / 255.)
    plt.show()
    imsave('imgs/mixed4c_filter_%d.jpg' % i, result)

mixed4c的filter0對應結果

mixed4c的filter8對應結果

固然，也能夠對一張圖片反覆執行DeepDream，優化出來的圖形會變得愈來愈明顯

img = image.copy()
imgs = []
for i in range(20):
    print('Iteration %d of mixed4c' % i)
    img = recursive_dream(layers[7], img)
    plt.figure(figsize=(10, 15))
    plt.imshow(img / 255.)
    plt.show()
    imgs.append(img)
mimsave('imgs/mixed4c多輪迭代結果.gif', imgs, fps=5)

結果有點鬼畜，多是mixed4c比較喜歡狗吧……

參考

• https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/14_DeepDream.ipynb

視頻講解課程

深度有趣（一）