在http://blog.csdn.net/fengbingchun/article/details/50814710中給出了CNN的簡單實現,這裏對每一步的實現做個說明:ios
共7層:依次爲輸入層、C1層、S2層、C3層、S4層、C5層、輸出層。C表明卷積層(特徵提取)。S表明降採樣層或池化層(Pooling),輸出層爲全鏈接層。c++
1. 各層權值、偏置(閾值)初始化:git
各層權值、偏置個數計算例如如下:github
(1)、輸入層:預處理後的32*32圖像數據。無權值和偏置;網絡
(2)、C1層:卷積窗大小5*5,輸出特徵圖數量6,卷積窗種類1*6=6。輸出特徵圖大小28*28,所以可訓練參數(權值+偏置):(5*5*1)*6+6=150+6。dom
(3)、S2層:卷積窗大小2*2。輸出下採樣圖數量6,卷積窗種類6,輸出下採樣圖大小14*14,所以可訓練參數(權值+偏置):1*6+6=6+6。ide
(4)、C3層:卷積窗大小5*5。輸出特徵圖數量16。卷積窗種類6*16=96,輸出特徵圖大小10*10。所以可訓練參數(權值+偏置):(5*5*6)*16+16=2400+16。函數
(5)、S4層:卷積窗大小2*2。輸出下採樣圖數量16,卷積窗種類16,輸出下採樣圖大小5*5,所以可訓練參數(權值+偏置):1*16+16=16+16。post
(6)、C5層:卷積窗大小5*5。輸出特徵圖數量120,卷積窗種類16*120=1920,輸出特徵圖大小1*1,所以可訓練參數(權值+偏置):(5*5*16)*120+120=48000+120;學習
(7)、輸出層:卷積窗大小1*1,輸出特徵圖數量10。卷積窗種類120*10=1200,輸出特徵圖大小1*1,所以可訓練參數(權值+偏置):(1*120)*10+10=1200+10.
代碼段例如如下:
#define num_map_input_CNN 1 //輸入層map個數 #define num_map_C1_CNN 6 //C1層map個數 #define num_map_S2_CNN 6 //S2層map個數 #define num_map_C3_CNN 16 //C3層map個數 #define num_map_S4_CNN 16 //S4層map個數 #define num_map_C5_CNN 120 //C5層map個數 #define num_map_output_CNN 10 //輸出層map個數 #define len_weight_C1_CNN 150 //C1層權值數,(5*5*1)*6=150 #define len_bias_C1_CNN 6 //C1層閾值數,6 #define len_weight_S2_CNN 6 //S2層權值數,1*6=6 #define len_bias_S2_CNN 6 //S2層閾值數,6 #define len_weight_C3_CNN 2400 //C3層權值數,(5*5*6)*16=2400 #define len_bias_C3_CNN 16 //C3層閾值數,16 #define len_weight_S4_CNN 16 //S4層權值數。1*16=16 #define len_bias_S4_CNN 16 //S4層閾值數。16 #define len_weight_C5_CNN 48000 //C5層權值數,(5*5*16)*120=48000 #define len_bias_C5_CNN 120 //C5層閾值數,120 #define len_weight_output_CNN 1200 //輸出層權值數。(1*120)*10=1200 #define len_bias_output_CNN 10 //輸出層閾值數,10 #define num_neuron_input_CNN 1024 //輸入層神經元數,(32*32)*1=1024 #define num_neuron_C1_CNN 4704 //C1層神經元數,(28*28)*6=4704 #define num_neuron_S2_CNN 1176 //S2層神經元數。(14*14)*6=1176 #define num_neuron_C3_CNN 1600 //C3層神經元數。(10*10)*16=1600 #define num_neuron_S4_CNN 400 //S4層神經元數。(5*5)*16=400 #define num_neuron_C5_CNN 120 //C5層神經元數。(1*1)*120=120 #define num_neuron_output_CNN 10 //輸出層神經元數,(1*1)*10=10
權值、偏置初始化:
(1)、權值使用函數uniform_real_distribution均勻分佈初始化。tiny-cnn中每次初始化權值數值都一樣。這裏做了調整,使每次初始化的權值均不一樣。每層權值初始化大小範圍都不同;
(2)、所有層的偏置均初始化爲0.
代碼段例如如下:
double CNN::uniform_rand(double min, double max) { //static std::mt19937 gen(1); std::random_device rd; std::mt19937 gen(rd()); std::uniform_real_distribution<double> dst(min, max); return dst(gen); } bool CNN::uniform_rand(double* src, int len, double min, double max) { for (int i = 0; i < len; i++) { src[i] = uniform_rand(min, max); } return true; } bool CNN::initWeightThreshold() { srand(time(0) + rand()); const double scale = 6.0; double min_ = -std::sqrt(scale / (25.0 + 150.0)); double max_ = std::sqrt(scale / (25.0 + 150.0)); uniform_rand(weight_C1, len_weight_C1_CNN, min_, max_); for (int i = 0; i < len_bias_C1_CNN; i++) { bias_C1[i] = 0.0; } min_ = -std::sqrt(scale / (4.0 + 1.0)); max_ = std::sqrt(scale / (4.0 + 1.0)); uniform_rand(weight_S2, len_weight_S2_CNN, min_, max_); for (int i = 0; i < len_bias_S2_CNN; i++) { bias_S2[i] = 0.0; } min_ = -std::sqrt(scale / (150.0 + 400.0)); max_ = std::sqrt(scale / (150.0 + 400.0)); uniform_rand(weight_C3, len_weight_C3_CNN, min_, max_); for (int i = 0; i < len_bias_C3_CNN; i++) { bias_C3[i] = 0.0; } min_ = -std::sqrt(scale / (4.0 + 1.0)); max_ = std::sqrt(scale / (4.0 + 1.0)); uniform_rand(weight_S4, len_weight_S4_CNN, min_, max_); for (int i = 0; i < len_bias_S4_CNN; i++) { bias_S4[i] = 0.0; } min_ = -std::sqrt(scale / (400.0 + 3000.0)); max_ = std::sqrt(scale / (400.0 + 3000.0)); uniform_rand(weight_C5, len_weight_C5_CNN, min_, max_); for (int i = 0; i < len_bias_C5_CNN; i++) { bias_C5[i] = 0.0; } min_ = -std::sqrt(scale / (120.0 + 10.0)); max_ = std::sqrt(scale / (120.0 + 10.0)); uniform_rand(weight_output, len_weight_output_CNN, min_, max_); for (int i = 0; i < len_bias_output_CNN; i++) { bias_output[i] = 0.0; } return true; }
2. 載入MNIST數據:
關於MNIST的介紹可以參考:http://blog.csdn.net/fengbingchun/article/details/49611549
使用MNIST庫做爲訓練集和測試集。訓練樣本集爲60000個,測試樣本集爲10000個。
(1)、MNIST庫中圖像原始大小爲28*28,這裏縮放爲32*32,數據取值範圍爲[-1,1],擴充值均取-1,做爲輸入層輸入數據。
代碼段例如如下:
static void readMnistImages(std::string filename, double* data_dst, int num_image) { const int width_src_image = 28; const int height_src_image = 28; const int x_padding = 2; const int y_padding = 2; const double scale_min = -1; const double scale_max = 1; std::ifstream file(filename, std::ios::binary); assert(file.is_open()); int magic_number = 0; int number_of_images = 0; int n_rows = 0; int n_cols = 0; file.read((char*)&magic_number, sizeof(magic_number)); magic_number = reverseInt(magic_number); file.read((char*)&number_of_images, sizeof(number_of_images)); number_of_images = reverseInt(number_of_images); assert(number_of_images == num_image); file.read((char*)&n_rows, sizeof(n_rows)); n_rows = reverseInt(n_rows); file.read((char*)&n_cols, sizeof(n_cols)); n_cols = reverseInt(n_cols); assert(n_rows == height_src_image && n_cols == width_src_image); int size_single_image = width_image_input_CNN * height_image_input_CNN; for (int i = 0; i < number_of_images; ++i) { int addr = size_single_image * i; for (int r = 0; r < n_rows; ++r) { for (int c = 0; c < n_cols; ++c) { unsigned char temp = 0; file.read((char*)&temp, sizeof(temp)); data_dst[addr + width_image_input_CNN * (r + y_padding) + c + x_padding] = (temp / 255.0) * (scale_max - scale_min) + scale_min; } } } }
(2)、對於Label,輸出層有10個節點,相應位置的節點值設爲0.8。其餘節點設爲-0.8,做爲輸出層數據。
代碼段例如如下:
static void readMnistLabels(std::string filename, double* data_dst, int num_image) { const double scale_max = 0.8; std::ifstream file(filename, std::ios::binary); assert(file.is_open()); int magic_number = 0; int number_of_images = 0; file.read((char*)&magic_number, sizeof(magic_number)); magic_number = reverseInt(magic_number); file.read((char*)&number_of_images, sizeof(number_of_images)); number_of_images = reverseInt(number_of_images); assert(number_of_images == num_image); for (int i = 0; i < number_of_images; ++i) { unsigned char temp = 0; file.read((char*)&temp, sizeof(temp)); data_dst[i * num_map_output_CNN + temp] = scale_max; } }static void readMnistLabels(std::string filename, double* data_dst, int num_image) { const double scale_max = 0.8; std::ifstream file(filename, std::ios::binary); assert(file.is_open()); int magic_number = 0; int number_of_images = 0; file.read((char*)&magic_number, sizeof(magic_number)); magic_number = reverseInt(magic_number); file.read((char*)&number_of_images, sizeof(number_of_images)); number_of_images = reverseInt(number_of_images); assert(number_of_images == num_image); for (int i = 0; i < number_of_images; ++i) { unsigned char temp = 0; file.read((char*)&temp, sizeof(temp)); data_dst[i * num_map_output_CNN + temp] = scale_max; } }
3. 前向傳播:主要計算每層的神經元值。當中C1層、C3層、C5層操做過程一樣。S2層、S4層操做過程一樣。
(1)、輸入層:神經元數爲(32*32)*1=1024。
(2)、C1層:神經元數爲(28*28)*6=4704,分別用每一個5*5的卷積圖像去乘以32*32的圖像,得到一個28*28的圖像。即相應位置相加再求和,stride長度爲1;一共6個5*5的卷積圖像,而後對每一個神經元加上一個閾值。最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
激活函數的做用:它是用來增長非線性因素的,解決線性模型所不能解決的問題。提供網絡的非線性建模能力。
假設沒有激活函數。那麼該網絡僅可以表達線性映射。此時即使有再多的隱藏層,其整個網絡跟單層神經網絡也是等價的。所以也可以以爲,惟獨增長了激活函數以後,深度神經網絡才具有了分層的非線性映射學習能力。
代碼段例如如下:
double CNN::activation_function_tanh(double x) { double ep = std::exp(x); double em = std::exp(-x); return (ep - em) / (ep + em); } bool CNN::Forward_C1() { init_variable(neuron_C1, 0.0, num_neuron_C1_CNN); for (int o = 0; o < num_map_C1_CNN; o++) { for (int inc = 0; inc < num_map_input_CNN; inc++) { int addr1 = get_index(0, 0, num_map_input_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN * num_map_input_CNN); int addr2 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN); int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN); const double* pw = &weight_C1[0] + addr1; const double* pi = data_single_image + addr2; double* pa = &neuron_C1[0] + addr3; for (int y = 0; y < height_image_C1_CNN; y++) { for (int x = 0; x < width_image_C1_CNN; x++) { const double* ppw = pw; const double* ppi = pi + y * width_image_input_CNN + x; double sum = 0.0; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { sum += *ppw++ * ppi[wy * width_image_input_CNN + wx]; } } pa[y * width_image_C1_CNN + x] += sum; } } } int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN); double* pa = &neuron_C1[0] + addr3; double b = bias_C1[o]; for (int y = 0; y < height_image_C1_CNN; y++) { for (int x = 0; x < width_image_C1_CNN; x++) { pa[y * width_image_C1_CNN + x] += b; } } } for (int i = 0; i < num_neuron_C1_CNN; i++) { neuron_C1[i] = activation_function_tanh(neuron_C1[i]); } return true; }
(3)、S2層:神經元數爲(14*14)*6=1176,對C1中6個28*28的特徵圖生成6個14*14的下採樣圖,相鄰四個神經元分別乘以同一個權值再進行相加求和,再求均值即除以4,而後再加上一個閾值,最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
代碼段例如如下:
bool CNN::Forward_S2() { init_variable(neuron_S2, 0.0, num_neuron_S2_CNN); double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN); assert(out2wi_S2.size() == num_neuron_S2_CNN); assert(out2bias_S2.size() == num_neuron_S2_CNN); for (int i = 0; i < num_neuron_S2_CNN; i++) { const wi_connections& connections = out2wi_S2[i]; neuron_S2[i] = 0; for (int index = 0; index < connections.size(); index++) { neuron_S2[i] += weight_S2[connections[index].first] * neuron_C1[connections[index].second]; } neuron_S2[i] *= scale_factor; neuron_S2[i] += bias_S2[out2bias_S2[i]]; } for (int i = 0; i < num_neuron_S2_CNN; i++) { neuron_S2[i] = activation_function_tanh(neuron_S2[i]); } return true; }
(4)、C3層:神經元數爲(10*10)*16=1600。C3層實現方式與C1層全然一樣。由S2中的6個14*14下採樣圖生成16個10*10特徵圖,對於生成的每一個10*10的特徵圖,是由6個5*5的卷積圖像去乘以6個14*14的下採樣圖,而後相應位置相加求和,而後對每一個神經元加上一個閾值,最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
也可依照Y.Lecun給出的表進行計算。即對於生成的每一個10*10的特徵圖,是由n個5*5的卷積圖像去乘以n個14*14的下採樣圖,當中n是小於6的,即不全然鏈接。這樣作的緣由:第一,不全然的鏈接機制將鏈接的數量保持在合理的範圍內。第二,也是最重要的,其破壞了網絡的對稱性。由於不一樣的特徵圖有不一樣的輸入,因此迫使他們抽取不一樣的特徵。
代碼段例如如下:
// connection table [Y.Lecun, 1998 Table.1] #define O true #define X false static const bool tbl[6][16] = { O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, X, O, O, O, X, O, O, O, X, X, O, O, O, O, X, X, O, X, O, O, X, X, O, O, O, X, X, O, O, O, O, X, O, O, X, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O }; #undef O #undef X bool CNN::Forward_C3() { init_variable(neuron_C3, 0.0, num_neuron_C3_CNN); for (int o = 0; o < num_map_C3_CNN; o++) { for (int inc = 0; inc < num_map_S2_CNN; inc++) { if (!tbl[inc][o]) continue; int addr1 = get_index(0, 0, num_map_S2_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C3_CNN * num_map_S2_CNN); int addr2 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN); int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN); const double* pw = &weight_C3[0] + addr1; const double* pi = &neuron_S2[0] + addr2; double* pa = &neuron_C3[0] + addr3; for (int y = 0; y < height_image_C3_CNN; y++) { for (int x = 0; x < width_image_C3_CNN; x++) { const double* ppw = pw; const double* ppi = pi + y * width_image_S2_CNN + x; double sum = 0.0; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { sum += *ppw++ * ppi[wy * width_image_S2_CNN + wx]; } } pa[y * width_image_C3_CNN + x] += sum; } } } int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN); double* pa = &neuron_C3[0] + addr3; double b = bias_C3[o]; for (int y = 0; y < height_image_C3_CNN; y++) { for (int x = 0; x < width_image_C3_CNN; x++) { pa[y * width_image_C3_CNN + x] += b; } } } for (int i = 0; i < num_neuron_C3_CNN; i++) { neuron_C3[i] = activation_function_tanh(neuron_C3[i]); } return true; }
(5)、S4層:神經元數爲(5*5)*16=400,S4層實現方式與S2層全然一樣。由C3中16個10*10的特徵圖生成16個5*5下採樣圖,相鄰四個神經元分別乘以同一個權值再進行相加求和,再求均值即除以4,而後再加上一個閾值。最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
代碼段例如如下:
bool CNN::Forward_S4() { double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN); init_variable(neuron_S4, 0.0, num_neuron_S4_CNN); assert(out2wi_S4.size() == num_neuron_S4_CNN); assert(out2bias_S4.size() == num_neuron_S4_CNN); for (int i = 0; i < num_neuron_S4_CNN; i++) { const wi_connections& connections = out2wi_S4[i]; neuron_S4[i] = 0.0; for (int index = 0; index < connections.size(); index++) { neuron_S4[i] += weight_S4[connections[index].first] * neuron_C3[connections[index].second]; } neuron_S4[i] *= scale_factor; neuron_S4[i] += bias_S4[out2bias_S4[i]]; } for (int i = 0; i < num_neuron_S4_CNN; i++) { neuron_S4[i] = activation_function_tanh(neuron_S4[i]); } return true; }
(6)、C5層:神經元數爲(1*1)*120=120,也可看爲全鏈接層,C5層實現方式與C一、C3層全然一樣。由S4中16個5*5下採樣圖生成120個1*1特徵圖,對於生成的每一個1*1的特徵圖,是由16個5*5的卷積圖像去乘以16個5*5的下採用圖,而後相加求和,而後對每一個神經元加上一個閾值,最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
代碼段例如如下:
bool CNN::Forward_C5() { init_variable(neuron_C5, 0.0, num_neuron_C5_CNN); for (int o = 0; o < num_map_C5_CNN; o++) { for (int inc = 0; inc < num_map_S4_CNN; inc++) { int addr1 = get_index(0, 0, num_map_S4_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C5_CNN * num_map_S4_CNN); int addr2 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN); int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN); const double *pw = &weight_C5[0] + addr1; const double *pi = &neuron_S4[0] + addr2; double *pa = &neuron_C5[0] + addr3; for (int y = 0; y < height_image_C5_CNN; y++) { for (int x = 0; x < width_image_C5_CNN; x++) { const double *ppw = pw; const double *ppi = pi + y * width_image_S4_CNN + x; double sum = 0.0; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { sum += *ppw++ * ppi[wy * width_image_S4_CNN + wx]; } } pa[y * width_image_C5_CNN + x] += sum; } } } int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN); double *pa = &neuron_C5[0] + addr3; double b = bias_C5[o]; for (int y = 0; y < height_image_C5_CNN; y++) { for (int x = 0; x < width_image_C5_CNN; x++) { pa[y * width_image_C5_CNN + x] += b; } } } for (int i = 0; i < num_neuron_C5_CNN; i++) { neuron_C5[i] = activation_function_tanh(neuron_C5[i]); } return true; }
(7)、輸出層:神經元數爲(1*1)*10=10。爲全鏈接層。輸出層中的每一個神經元均是由C5層中的120個神經元乘以相相應的權值。而後相加求和;而後對每一個神經元加上一個閾值。最後再經過tanh激活函數對每一神經元進行運算獲得終於每一個神經元的結果。
代碼段例如如下:
bool CNN::Forward_output() { init_variable(neuron_output, 0.0, num_neuron_output_CNN); for (int i = 0; i < num_neuron_output_CNN; i++) { neuron_output[i] = 0.0; for (int c = 0; c < num_neuron_C5_CNN; c++) { neuron_output[i] += weight_output[c * num_neuron_output_CNN + i] * neuron_C5[c]; } neuron_output[i] += bias_output[i]; } for (int i = 0; i < num_neuron_output_CNN; i++) { neuron_output[i] = activation_function_tanh(neuron_output[i]); } return true; }
4. 反向傳播:主要計算每層權值和偏置的偏差以及每層神經元的偏差;當中輸入層、S2層、S4層操做過程一樣。C1層、C3層操做過程一樣。
(1)、輸出層:計算輸出層神經元偏差;經過mse損失函數的導數函數和tanh激活函數的導數函數來計算輸出層神經元偏差,即a、已計算出的輸出層神經元值減去相應label值,b、1.0減去輸出層神經元值的平方,c、a與c的乘積和。
損失函數做用:在統計學中損失函數是一種衡量損失和錯誤(這樣的損失與」錯誤地」預計有關)程度的函數。損失函數在實踐中最重要的運用。在於協助咱們經過過程的改善而持續下降目標值的變異。並非只追求符合邏輯。
在深度學習中,對於損失函數的收斂特性。咱們指望是當偏差越大的時候。收斂(學習)速度應該越快。成爲損失函數需要知足兩點要求:非負性;預測值和指望值接近時,函數值趨於0.
代碼段例如如下:
double CNN::loss_function_mse_derivative(double y, double t) { return (y - t); } void CNN::loss_function_gradient(const double* y, const double* t, double* dst, int len) { for (int i = 0; i < len; i++) { dst[i] = loss_function_mse_derivative(y[i], t[i]); } } double CNN::activation_function_tanh_derivative(double x) { return (1.0 - x * x); } double CNN::dot_product(const double* s1, const double* s2, int len) { double result = 0.0; for (int i = 0; i < len; i++) { result += s1[i] * s2[i]; } return result; } bool CNN::Backward_output() { init_variable(delta_neuron_output, 0.0, num_neuron_output_CNN); double dE_dy[num_neuron_output_CNN]; init_variable(dE_dy, 0.0, num_neuron_output_CNN); loss_function_gradient(neuron_output, data_single_label, dE_dy, num_neuron_output_CNN); // 損失函數: mean squared error(均方差) // delta = dE/da = (dE/dy) * (dy/da) for (int i = 0; i < num_neuron_output_CNN; i++) { double dy_da[num_neuron_output_CNN]; init_variable(dy_da, 0.0, num_neuron_output_CNN); dy_da[i] = activation_function_tanh_derivative(neuron_output[i]); delta_neuron_output[i] = dot_product(dE_dy, dy_da, num_neuron_output_CNN); } return true; }
(2)、C5層:計算C5層神經元偏差、輸出層權值偏差、輸出層偏置偏差;經過輸出層神經元偏差乘以輸出層權值。求和。結果再乘以C5層神經元的tanh激活函數的導數(即1-C5層神經元值的平方)。得到C5層每一個神經元偏差。經過輸出層神經元偏差乘以C5層神經元得到輸出層權值偏差;輸出層偏置偏差即爲輸出層神經元偏差。
代碼段例如如下:
bool CNN::muladd(const double* src, double c, int len, double* dst) { for (int i = 0; i < len; i++) { dst[i] += (src[i] * c); } return true; } bool CNN::Backward_C5() { init_variable(delta_neuron_C5, 0.0, num_neuron_C5_CNN); init_variable(delta_weight_output, 0.0, len_weight_output_CNN); init_variable(delta_bias_output, 0.0, len_bias_output_CNN); for (int c = 0; c < num_neuron_C5_CNN; c++) { // propagate delta to previous layer // prev_delta[c] += current_delta[r] * W_[c * out_size_ + r] delta_neuron_C5[c] = dot_product(&delta_neuron_output[0], &weight_output[c * num_neuron_output_CNN], num_neuron_output_CNN); delta_neuron_C5[c] *= activation_function_tanh_derivative(neuron_C5[c]); } // accumulate weight-step using delta // dW[c * out_size + i] += current_delta[i] * prev_out[c] for (int c = 0; c < num_neuron_C5_CNN; c++) { muladd(&delta_neuron_output[0], neuron_C5[c], num_neuron_output_CNN, &delta_weight_output[0] + c * num_neuron_output_CNN); } for (int i = 0; i < len_bias_output_CNN; i++) { delta_bias_output[i] += delta_neuron_output[i]; } return true; }
(3)、S4層:計算S4層神經元偏差、C5層權值偏差、C5層偏置偏差;經過C5層權值乘以C5層神經元偏差。求和。結果再乘以S4層神經元的tanh激活函數的導數(即1-S4神經元的平方),得到S4層每一個神經元偏差。經過S4層神經元乘以C5層神經元偏差,求和,得到C5層權值偏差。C5層偏置偏差即爲C5層神經元偏差。
代碼段例如如下:
bool CNN::Backward_S4() { init_variable(delta_neuron_S4, 0.0, num_neuron_S4_CNN); init_variable(delta_weight_C5, 0.0, len_weight_C5_CNN); init_variable(delta_bias_C5, 0.0, len_bias_C5_CNN); // propagate delta to previous layer for (int inc = 0; inc < num_map_S4_CNN; inc++) { for (int outc = 0; outc < num_map_C5_CNN; outc++) { int addr1 = get_index(0, 0, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN); int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN); int addr3 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN); const double* pw = &weight_C5[0] + addr1; const double* pdelta_src = &delta_neuron_C5[0] + addr2; double* pdelta_dst = &delta_neuron_S4[0] + addr3; for (int y = 0; y < height_image_C5_CNN; y++) { for (int x = 0; x < width_image_C5_CNN; x++) { const double* ppw = pw; const double ppdelta_src = pdelta_src[y * width_image_C5_CNN + x]; double* ppdelta_dst = pdelta_dst + y * width_image_S4_CNN + x; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { ppdelta_dst[wy * width_image_S4_CNN + wx] += *ppw++ * ppdelta_src; } } } } } } for (int i = 0; i < num_neuron_S4_CNN; i++) { delta_neuron_S4[i] *= activation_function_tanh_derivative(neuron_S4[i]); } // accumulate dw for (int inc = 0; inc < num_map_S4_CNN; inc++) { for (int outc = 0; outc < num_map_C5_CNN; outc++) { for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { int addr1 = get_index(wx, wy, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN); int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN); int addr3 = get_index(wx, wy, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN); double dst = 0.0; const double* prevo = &neuron_S4[0] + addr1; const double* delta = &delta_neuron_C5[0] + addr2; for (int y = 0; y < height_image_C5_CNN; y++) { dst += dot_product(prevo + y * width_image_S4_CNN, delta + y * width_image_C5_CNN, width_image_C5_CNN); } delta_weight_C5[addr3] += dst; } } } } // accumulate db for (int outc = 0; outc < num_map_C5_CNN; outc++) { int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN); const double* delta = &delta_neuron_C5[0] + addr2; for (int y = 0; y < height_image_C5_CNN; y++) { for (int x = 0; x < width_image_C5_CNN; x++) { delta_bias_C5[outc] += delta[y * width_image_C5_CNN + x]; } } } return true; }
(4)、C3層:計算C3層神經元偏差、S4層權值偏差、S4層偏置偏差。經過S4層權值乘以S4層神經元偏差。求和,結果再乘以C3層神經元的tanh激活函數的導數(即1-S4神經元的平方),而後再乘以1/4。得到C3層每一個神經元偏差;經過C3層神經元乘以S4神經元偏差,求和。再乘以1/4。得到S4層權值偏差;經過S4層神經元偏差求和,來得到S4層偏置偏差。
代碼段例如如下:
bool CNN::Backward_C3() { init_variable(delta_neuron_C3, 0.0, num_neuron_C3_CNN); init_variable(delta_weight_S4, 0.0, len_weight_S4_CNN); init_variable(delta_bias_S4, 0.0, len_bias_S4_CNN); double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN); assert(in2wo_C3.size() == num_neuron_C3_CNN); assert(weight2io_C3.size() == len_weight_S4_CNN); assert(bias2out_C3.size() == len_bias_S4_CNN); for (int i = 0; i < num_neuron_C3_CNN; i++) { const wo_connections& connections = in2wo_C3[i]; double delta = 0.0; for (int j = 0; j < connections.size(); j++) { delta += weight_S4[connections[j].first] * delta_neuron_S4[connections[j].second]; } delta_neuron_C3[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C3[i]); } for (int i = 0; i < len_weight_S4_CNN; i++) { const io_connections& connections = weight2io_C3[i]; double diff = 0; for (int j = 0; j < connections.size(); j++) { diff += neuron_C3[connections[j].first] * delta_neuron_S4[connections[j].second]; } delta_weight_S4[i] += diff * scale_factor; } for (int i = 0; i < len_bias_S4_CNN; i++) { const std::vector<int>& outs = bias2out_C3[i]; double diff = 0; for (int o = 0; o < outs.size(); o++) { diff += delta_neuron_S4[outs[o]]; } delta_bias_S4[i] += diff; } return true; }
(5)、S2層:計算S2層神經元偏差、C3層權值偏差、C3層偏置偏差。經過C3層權值乘以C3層神經元偏差。求和,結果再乘以S2層神經元的tanh激活函數的導數(即1-S2神經元的平方),得到S2層每一個神經元偏差;經過S2層神經元乘以C3層神經元偏差。求和,得到C3層權值偏差;C3層偏置偏差即爲C3層神經元偏差和。
代碼段例如如下:
bool CNN::Backward_S2() { init_variable(delta_neuron_S2, 0.0, num_neuron_S2_CNN); init_variable(delta_weight_C3, 0.0, len_weight_C3_CNN); init_variable(delta_bias_C3, 0.0, len_bias_C3_CNN); // propagate delta to previous layer for (int inc = 0; inc < num_map_S2_CNN; inc++) { for (int outc = 0; outc < num_map_C3_CNN; outc++) { if (!tbl[inc][outc]) continue; int addr1 = get_index(0, 0, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN); int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN); int addr3 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN); const double *pw = &weight_C3[0] + addr1; const double *pdelta_src = &delta_neuron_C3[0] + addr2;; double* pdelta_dst = &delta_neuron_S2[0] + addr3; for (int y = 0; y < height_image_C3_CNN; y++) { for (int x = 0; x < width_image_C3_CNN; x++) { const double* ppw = pw; const double ppdelta_src = pdelta_src[y * width_image_C3_CNN + x]; double* ppdelta_dst = pdelta_dst + y * width_image_S2_CNN + x; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { ppdelta_dst[wy * width_image_S2_CNN + wx] += *ppw++ * ppdelta_src; } } } } } } for (int i = 0; i < num_neuron_S2_CNN; i++) { delta_neuron_S2[i] *= activation_function_tanh_derivative(neuron_S2[i]); } // accumulate dw for (int inc = 0; inc < num_map_S2_CNN; inc++) { for (int outc = 0; outc < num_map_C3_CNN; outc++) { if (!tbl[inc][outc]) continue; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { int addr1 = get_index(wx, wy, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN); int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN); int addr3 = get_index(wx, wy, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN); double dst = 0.0; const double* prevo = &neuron_S2[0] + addr1; const double* delta = &delta_neuron_C3[0] + addr2; for (int y = 0; y < height_image_C3_CNN; y++) { dst += dot_product(prevo + y * width_image_S2_CNN, delta + y * width_image_C3_CNN, width_image_C3_CNN); } delta_weight_C3[addr3] += dst; } } } } // accumulate db for (int outc = 0; outc < len_bias_C3_CNN; outc++) { int addr1 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN); const double* delta = &delta_neuron_C3[0] + addr1; for (int y = 0; y < height_image_C3_CNN; y++) { for (int x = 0; x < width_image_C3_CNN; x++) { delta_bias_C3[outc] += delta[y * width_image_C3_CNN + x]; } } } return true; }
(6)、C1層:計算C1層神經元偏差、S2層權值偏差、S2層偏置偏差;經過S2層權值乘以S2層神經元偏差,求和。結果再乘以C1層神經元的tanh激活函數的導數(即1-C1神經元的平方),而後再乘以1/4,得到C1層每一個神經元偏差;經過C1層神經元乘以S2神經元偏差,求和。再乘以1/4,得到S2層權值偏差;經過S2層神經元偏差求和,來得到S4層偏置偏差。
代碼段例如如下:
bool CNN::Backward_C1() { init_variable(delta_neuron_C1, 0.0, num_neuron_C1_CNN); init_variable(delta_weight_S2, 0.0, len_weight_S2_CNN); init_variable(delta_bias_S2, 0.0, len_bias_S2_CNN); double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN); assert(in2wo_C1.size() == num_neuron_C1_CNN); assert(weight2io_C1.size() == len_weight_S2_CNN); assert(bias2out_C1.size() == len_bias_S2_CNN); for (int i = 0; i < num_neuron_C1_CNN; i++) { const wo_connections& connections = in2wo_C1[i]; double delta = 0.0; for (int j = 0; j < connections.size(); j++) { delta += weight_S2[connections[j].first] * delta_neuron_S2[connections[j].second]; } delta_neuron_C1[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C1[i]); } for (int i = 0; i < len_weight_S2_CNN; i++) { const io_connections& connections = weight2io_C1[i]; double diff = 0.0; for (int j = 0; j < connections.size(); j++) { diff += neuron_C1[connections[j].first] * delta_neuron_S2[connections[j].second]; } delta_weight_S2[i] += diff * scale_factor; } for (int i = 0; i < len_bias_S2_CNN; i++) { const std::vector<int>& outs = bias2out_C1[i]; double diff = 0; for (int o = 0; o < outs.size(); o++) { diff += delta_neuron_S2[outs[o]]; } delta_bias_S2[i] += diff; } return true; }
(7)、輸入層:計算輸入層神經元偏差、C1層權值偏差、C1層偏置偏差;經過C1層權值乘以C1層神經元偏差。求和。結果再乘以輸入層神經元的tanh激活函數的導數(即1-輸入層神經元的平方),得到輸入層每一個神經元偏差;經過輸入層層神經元乘以C1層神經元偏差,求和。得到C1層權值偏差;C1層偏置偏差即爲C1層神經元偏差和。
bool CNN::Backward_input() { init_variable(delta_neuron_input, 0.0, num_neuron_input_CNN); init_variable(delta_weight_C1, 0.0, len_weight_C1_CNN); init_variable(delta_bias_C1, 0.0, len_bias_C1_CNN); // propagate delta to previous layer for (int inc = 0; inc < num_map_input_CNN; inc++) { for (int outc = 0; outc < num_map_C1_CNN; outc++) { int addr1 = get_index(0, 0, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN); int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN); int addr3 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN); const double* pw = &weight_C1[0] + addr1; const double* pdelta_src = &delta_neuron_C1[0] + addr2; double* pdelta_dst = &delta_neuron_input[0] + addr3; for (int y = 0; y < height_image_C1_CNN; y++) { for (int x = 0; x < width_image_C1_CNN; x++) { const double* ppw = pw; const double ppdelta_src = pdelta_src[y * width_image_C1_CNN + x]; double* ppdelta_dst = pdelta_dst + y * width_image_input_CNN + x; for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { ppdelta_dst[wy * width_image_input_CNN + wx] += *ppw++ * ppdelta_src; } } } } } } for (int i = 0; i < num_neuron_input_CNN; i++) { delta_neuron_input[i] *= activation_function_identity_derivative(data_single_image[i]/*neuron_input[i]*/); } // accumulate dw for (int inc = 0; inc < num_map_input_CNN; inc++) { for (int outc = 0; outc < num_map_C1_CNN; outc++) { for (int wy = 0; wy < height_kernel_conv_CNN; wy++) { for (int wx = 0; wx < width_kernel_conv_CNN; wx++) { int addr1 = get_index(wx, wy, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN); int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN); int addr3 = get_index(wx, wy, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN); double dst = 0.0; const double* prevo = data_single_image + addr1;//&neuron_input[0] const double* delta = &delta_neuron_C1[0] + addr2; for (int y = 0; y < height_image_C1_CNN; y++) { dst += dot_product(prevo + y * width_image_input_CNN, delta + y * width_image_C1_CNN, width_image_C1_CNN); } delta_weight_C1[addr3] += dst; } } } } // accumulate db for (int outc = 0; outc < len_bias_C1_CNN; outc++) { int addr1 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN); const double* delta = &delta_neuron_C1[0] + addr1; for (int y = 0; y < height_image_C1_CNN; y++) { for (int x = 0; x < width_image_C1_CNN; x++) { delta_bias_C1[outc] += delta[y * width_image_C1_CNN + x]; } } } return true; }
5. 更新各層權值、偏置:經過以前計算的各層權值、各層權值偏差。各層偏置、各層偏置偏差以及學習率來更新各層權值和偏置。
代碼段例如如下:
void CNN::update_weights_bias(const double* delta, double* e_weight, double* weight, int len) { for (int i = 0; i < len; i++) { e_weight[i] += delta[i] * delta[i]; weight[i] -= learning_rate_CNN * delta[i] / (std::sqrt(e_weight[i]) + eps_CNN); } } bool CNN::UpdateWeights() { update_weights_bias(delta_weight_C1, E_weight_C1, weight_C1, len_weight_C1_CNN); update_weights_bias(delta_bias_C1, E_bias_C1, bias_C1, len_bias_C1_CNN); update_weights_bias(delta_weight_S2, E_weight_S2, weight_S2, len_weight_S2_CNN); update_weights_bias(delta_bias_S2, E_bias_S2, bias_S2, len_bias_S2_CNN); update_weights_bias(delta_weight_C3, E_weight_C3, weight_C3, len_weight_C3_CNN); update_weights_bias(delta_bias_C3, E_bias_C3, bias_C3, len_bias_C3_CNN); update_weights_bias(delta_weight_S4, E_weight_S4, weight_S4, len_weight_S4_CNN); update_weights_bias(delta_bias_S4, E_bias_S4, bias_S4, len_bias_S4_CNN); update_weights_bias(delta_weight_C5, E_weight_C5, weight_C5, len_weight_C5_CNN); update_weights_bias(delta_bias_C5, E_bias_C5, bias_C5, len_bias_C5_CNN); update_weights_bias(delta_weight_output, E_weight_output, weight_output, len_weight_output_CNN); update_weights_bias(delta_bias_output, E_bias_output, bias_output, len_bias_output_CNN); return true; }
6. 測試準確率是否達到要求或已達到循環次數:依次循環3至5中操做,依據訓練集數量。每循環60000次時,經過計算的權值和偏置。來對10000個測試集進行測試,假設準確率達到0.985或者達到迭代次數上限100次時。保存權值和偏置。
代碼段例如如下:
bool CNN::train() { out2wi_S2.clear(); out2bias_S2.clear(); out2wi_S4.clear(); out2bias_S4.clear(); in2wo_C3.clear(); weight2io_C3.clear(); bias2out_C3.clear(); in2wo_C1.clear(); weight2io_C1.clear(); bias2out_C1.clear(); calc_out2wi(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2wi_S2); calc_out2bias(width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2bias_S2); calc_out2wi(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2wi_S4); calc_out2bias(width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2bias_S4); calc_in2wo(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, in2wo_C3); calc_weight2io(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, weight2io_C3); calc_bias2out(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, bias2out_C3); calc_in2wo(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, in2wo_C1); calc_weight2io(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, weight2io_C1); calc_bias2out(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, bias2out_C1); int iter = 0; for (iter = 0; iter < num_epochs_CNN; iter++) { std::cout << "epoch: " << iter + 1; for (int i = 0; i < num_patterns_train_CNN; i++) { data_single_image = data_input_train + i * num_neuron_input_CNN; data_single_label = data_output_train + i * num_neuron_output_CNN; Forward_C1(); Forward_S2(); Forward_C3(); Forward_S4(); Forward_C5(); Forward_output(); Backward_output(); Backward_C5(); Backward_S4(); Backward_C3(); Backward_S2(); Backward_C1(); Backward_input(); UpdateWeights(); } double accuracyRate = test(); std::cout << ", accuray rate: " << accuracyRate << std::endl; if (accuracyRate > accuracy_rate_CNN) { saveModelFile("E:/GitCode/NN_Test/data/cnn.model"); std::cout << "generate cnn model" << std::endl; break; } } if (iter == num_epochs_CNN) { saveModelFile("E:/GitCode/NN_Test/data/cnn.model"); std::cout << "generate cnn model" << std::endl; } return true; } double CNN::test() { int count_accuracy = 0; for (int num = 0; num < num_patterns_test_CNN; num++) { data_single_image = data_input_test + num * num_neuron_input_CNN; data_single_label = data_output_test + num * num_neuron_output_CNN; Forward_C1(); Forward_S2(); Forward_C3(); Forward_S4(); Forward_C5(); Forward_output(); int pos_t = -1; int pos_y = -2; double max_value_t = -9999.0; double max_value_y = -9999.0; for (int i = 0; i < num_neuron_output_CNN; i++) { if (neuron_output[i] > max_value_y) { max_value_y = neuron_output[i]; pos_y = i; } if (data_single_label[i] > max_value_t) { max_value_t = data_single_label[i]; pos_t = i; } } if (pos_y == pos_t) { ++count_accuracy; } Sleep(1); } return (count_accuracy * 1.0 / num_patterns_test_CNN); }
7. 對輸入的圖像數據進行識別:載入已保存的權值和偏置,對輸入的數據進行識別。過程至關於前向傳播。
代碼段例如如下:
int CNN::predict(const unsigned char* data, int width, int height) { assert(data && width == width_image_input_CNN && height == height_image_input_CNN); const double scale_min = -1; const double scale_max = 1; double tmp[width_image_input_CNN * height_image_input_CNN]; for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { tmp[y * width + x] = (data[y * width + x] / 255.0) * (scale_max - scale_min) + scale_min; } } data_single_image = &tmp[0]; Forward_C1(); Forward_S2(); Forward_C3(); Forward_S4(); Forward_C5(); Forward_output(); int pos = -1; double max_value = -9999.0; for (int i = 0; i < num_neuron_output_CNN; i++) { if (neuron_output[i] > max_value) { max_value = neuron_output[i]; pos = i; } } return pos; }