Pytorch_Part2_數據模塊

時間 2020-05-09

標籤 pytorch part2 數據模塊简体版

原文原文鏈接

VisualPytorch發佈域名+雙服務器以下：
http://nag.visualpytorch.top/static/ (對應114.115.148.27)
http://visualpytorch.top/static/ (對應39.97.209.22)python

1、Dataloader與Dataset

1. DataLoader

Epoch: 全部訓練樣本都已輸入到模型中，稱爲一個Epoch服務器

Iteration：一批樣本輸入到模型中，稱之爲一個Iterationapp

Batchsize：批大小，決定一個Epoch有多少個Iterationdom

樣本總數：87， Batchsize：8函數

1 Epoch = 10 Iteration ？ drop_last = True學習

1 Epoch = 11 Iteration ？ drop_last = False測試

2. Dataset

人民幣二分類爲例：將RMB_data按照8:1:1分爲train, test, valid三組，構建Dataset類優化

class RMBDataset(Dataset):
    def __init__(self, data_dir, transform=None):
        """
        rmb面額分類任務的Dataset
        :param data_dir: str, 數據集所在路徑
        :param transform: torch.transform，數據預處理
        """
        self.label_name = {"1": 0, "100": 1}
        self.data_info = self.get_img_info(data_dir)  # data_info存儲全部圖片路徑和標籤，在DataLoader中經過index讀取樣本
        self.transform = transform

    def __getitem__(self, index):
        path_img, label = self.data_info[index]
        img = Image.open(path_img).convert('RGB')     # 0~255

        if self.transform is not None:
            img = self.transform(img)   # 在這裏作transform，轉爲tensor等等

        return img, label

    def __len__(self):
        return len(self.data_info)

    @staticmethod
    def get_img_info(data_dir):
        data_info = list()
        for root, dirs, _ in os.walk(data_dir):
            # 遍歷類別
            for sub_dir in dirs:
                img_names = os.listdir(os.path.join(root, sub_dir))
                img_names = list(filter(lambda x: x.endswith('.jpg'), img_names))

                # 遍歷圖片
                for i in range(len(img_names)):
                    img_name = img_names[i]
                    path_img = os.path.join(root, sub_dir, img_name)
                    label = rmb_label[sub_dir]
                    data_info.append((path_img, int(label)))

        return data_info

實例化Dataset與DataLoader:
ui

# ============================ step 1/5 數據 ============================

split_dir = os.path.join("..", "..", "rmb_split")
train_dir = os.path.join(split_dir, "train")
valid_dir = os.path.join(split_dir, "valid")

norm_mean = [0.485, 0.456, 0.406]
norm_std = [0.229, 0.224, 0.225]

train_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.RandomCrop(32, padding=4),	
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])	

valid_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])

# 構建MyDataset實例
train_data = RMBDataset(data_dir=train_dir, transform=train_transform)
valid_data = RMBDataset(data_dir=valid_dir, transform=valid_transform)

# 構建DataLoder
train_loader = DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True)	# 每個epoch樣本順序不一樣
valid_loader = DataLoader(dataset=valid_data, batch_size=BATCH_SIZE)

2、圖像預處理——transform

torchvision.transforms : 經常使用的圖像預處理方法spa

數據中心化
數據標準化
縮放
裁剪
旋轉
翻轉
填充
噪聲添加
灰度變換
線性變換
仿射變換
亮度、飽和度及對比度變換

圖像加強：豐富數據集，提升模型的泛化能力

# 對圖像進行有序的組合與包裝
train_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.RandomCrop(32, padding=4),	# 隨機裁剪
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),	# 標準化
])	

valid_transform = transforms.Compose([
    transforms.Resize((32, 32)),	# 驗證時不須要進行圖像加強
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])

進行標準化能加快模型的收斂！

3、transform圖像加強（一）

數據加強又稱爲數據增廣，數據擴增，它是對訓練集進行變換，使訓練集更豐富，從而讓模型更具泛化能力.

1. 裁剪

padding_mode：填充模式，有4種模式
- 一、constant：像素值由fill設定
- 二、edge：像素值由圖像邊緣像素決定
- 三、reflect：鏡像填充，最後一個像素不鏡像，eg：[1,2,3,4] → [3,2,1,2,3,4,3,2]
- 四、symmetric：鏡像填充，最後一個像素鏡像，eg：[1,2,3,4] → [2,1,1,2,3,4,4,3]
- fill：constant時，設置填充的像素值

2. 翻轉和旋轉

center=(0,0) # 左上角旋轉

expand僅針對center，沒辦法找回左上角旋轉丟失的信息

4、transform圖像加強（二）

1. 圖像變換

padding_mode爲鏡像時，fill不起做用

直接在tensor上進行操做：

transforms.ToTensor()
transforms.RandomErasing(p=1, scale=(0.02,0.33), ratio=(0.5,1), value=(254/255,0,0))

其中value只要是字符串，隨機色彩

2. transform操做

3. 自定義transform

class AddPepperNoise(object):
    """增長椒鹽噪聲
    Args:
        snr （float）: Signal Noise Rate
        p (float): 機率值，依機率執行該操做
    """

    def __init__(self, snr, p=0.9):
        assert isinstance(snr, float) or (isinstance(p, float))
        self.snr = snr
        self.p = p

    def __call__(self, img):
        """
        Args:
            img (PIL Image): PIL Image
        Returns:
            PIL Image: PIL image.
        """
        if random.uniform(0, 1) < self.p:
            img_ = np.array(img).copy()
            h, w, c = img_.shape
            signal_pct = self.snr
            noise_pct = (1 - self.snr)
            mask = np.random.choice((0, 1, 2), size=(h, w, 1), p=[signal_pct, noise_pct/2., noise_pct/2.])
            mask = np.repeat(mask, c, axis=2)
            img_[mask == 1] = 255   # 鹽噪聲：白
            img_[mask == 2] = 0     # 椒噪聲：黑
            return Image.fromarray(img_.astype('uint8')).convert('RGB')
        else:
            return img

4. 實戰

原則：讓訓練集與測試集更接近

空間位置：平移
色彩：灰度圖，色彩抖動
形狀：仿射變換
上下文場景：遮擋，填充
......

分析原圖，主要是色相的問題，將輸入圖片直接轉爲灰度圖

train_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.RandomCrop(32, padding=4),
    transforms.RandomGrayscale(p=1),
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])

源代碼：

# -*- coding: utf-8 -*-
"""
# @file name  : RMB_data_augmentation.py
# @author     : tingsongyu
# @date       : 2019-09-16 10:08:00
# @brief      : 人民幣分類模型數據加強實驗
"""
import os
import random
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
import torch.optim as optim
from matplotlib import pyplot as plt
from model.lenet import LeNet
from tools.my_dataset import RMBDataset
from tools.common_tools import transform_invert


def set_seed(seed=1):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)


set_seed()  # 設置隨機種子
rmb_label = {"1": 0, "100": 1}

# 參數設置
MAX_EPOCH = 10
BATCH_SIZE = 16
LR = 0.01
log_interval = 10
val_interval = 1

# ============================ step 1/5 數據 ============================

split_dir = os.path.join("..", "rmb_split")
train_dir = os.path.join(split_dir, "train")
valid_dir = os.path.join(split_dir, "valid")

norm_mean = [0.485, 0.456, 0.406]
norm_std = [0.229, 0.224, 0.225]

train_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.RandomCrop(32, padding=4),
    transforms.RandomGrayscale(p=1),
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])


valid_transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.RandomGrayscale(p=1),
    transforms.ToTensor(),
    transforms.Normalize(norm_mean, norm_std),
])

# 構建MyDataset實例
train_data = RMBDataset(data_dir=train_dir, transform=train_transform)
valid_data = RMBDataset(data_dir=valid_dir, transform=valid_transform)

# 構建DataLoder
train_loader = DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True)
valid_loader = DataLoader(dataset=valid_data, batch_size=BATCH_SIZE)

# ============================ step 2/5 模型 ============================

net = LeNet(classes=2)
net.initialize_weights()

# ============================ step 3/5 損失函數 ============================
criterion = nn.CrossEntropyLoss()                                                   # 選擇損失函數

# ============================ step 4/5 優化器 ============================
optimizer = optim.SGD(net.parameters(), lr=LR, momentum=0.9)                        # 選擇優化器
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)     # 設置學習率降低策略

# ============================ step 5/5 訓練 ============================
train_curve = list()
valid_curve = list()

for epoch in range(MAX_EPOCH):

    loss_mean = 0.
    correct = 0.
    total = 0.

    net.train()
    for i, data in enumerate(train_loader):

        # forward
        inputs, labels = data
        outputs = net(inputs)

        # backward
        optimizer.zero_grad()
        loss = criterion(outputs, labels)
        loss.backward()

        # update weights
        optimizer.step()

        # 統計分類狀況
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).squeeze().sum().numpy()

        # 打印訓練信息
        loss_mean += loss.item()
        train_curve.append(loss.item())
        if (i+1) % log_interval == 0:
            loss_mean = loss_mean / log_interval
            print("Training:Epoch[{:0>3}/{:0>3}] Iteration[{:0>3}/{:0>3}] Loss: {:.4f} Acc:{:.2%}".format(
                epoch, MAX_EPOCH, i+1, len(train_loader), loss_mean, correct / total))
            loss_mean = 0.

    scheduler.step()  # 更新學習率

    # validate the model
    if (epoch+1) % val_interval == 0:

        correct_val = 0.
        total_val = 0.
        loss_val = 0.
        net.eval()
        with torch.no_grad():
            for j, data in enumerate(valid_loader):
                inputs, labels = data
                outputs = net(inputs)
                loss = criterion(outputs, labels)

                _, predicted = torch.max(outputs.data, 1)
                total_val += labels.size(0)
                correct_val += (predicted == labels).squeeze().sum().numpy()

                loss_val += loss.item()

            valid_curve.append(loss_val)
            print("Valid:\t Epoch[{:0>3}/{:0>3}] Iteration[{:0>3}/{:0>3}] Loss: {:.4f} Acc:{:.2%}".format(
                epoch, MAX_EPOCH, j+1, len(valid_loader), loss_val, correct / total))


train_x = range(len(train_curve))
train_y = train_curve

train_iters = len(train_loader)
valid_x = np.arange(1, len(valid_curve)+1) * train_iters*val_interval # 因爲valid中記錄的是epochloss，須要對記錄點進行轉換到iterations
valid_y = valid_curve

plt.plot(train_x, train_y, label='Train')
plt.plot(valid_x, valid_y, label='Valid')

plt.legend(loc='upper right')
plt.ylabel('loss value')
plt.xlabel('Iteration')
plt.show()

# ============================ inference ============================

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
test_dir = os.path.join(BASE_DIR, "test_data")

test_data = RMBDataset(data_dir=test_dir, transform=valid_transform)
valid_loader = DataLoader(dataset=test_data, batch_size=1)

for i, data in enumerate(valid_loader):
    # forward
    inputs, labels = data
    outputs = net(inputs)
    _, predicted = torch.max(outputs.data, 1)

    rmb = 1 if predicted.numpy()[0] == 0 else 100

    img_tensor = inputs[0, ...]  # C H W
    img = transform_invert(img_tensor, train_transform)
    plt.imshow(img)
    plt.title("LeNet got {} Yuan".format(rmb))
    plt.show()
    plt.pause(0.5)
    plt.close()

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。