Multipath 多路徑配置實踐心得

前言

配置存儲時必定會遇到 multipath 多路徑的問題，不一樣的廠商好比 EMC PowerPath，Veritas VxDMP 等都有獨立的多路徑軟件，而多路徑軟件的功能也很清晰主要用於IO流量負載均衡和故障切換恢復等。在 Linux 環境中 device-mapper-multipath 是一個免費的通用型多路徑管理軟件，其配置文件也很是簡單，主要經過修改 /etc/multipath.conf 來調整。文章提供了《HPE 3PAR Red Hat Enterprise Linux和 Oracle Linux 實施指南》，也分享了本身配置 multipath 的實踐過程，但願對你們有參考價值。

記錄 3PAR8400 存儲 Multipath 多路徑配置

更新歷史

2017年05月04日 - 初稿

閱讀原文 - https://wsgzao.github.io/post...

擴展閱讀

multipath - https://access.redhat.com/lab...
[原]紅旗上使用multipath複合多條路徑 - http://www.linuxfly.org/post/...

---

官方最佳實踐手冊

HPE 3PAR Red Hat Enterprise Linux和 Oracle Linux 實施指南

<iframe src="https://www.slideshare.net/sl... width="479" height="511" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> </div>

什麼是multipath

普通的電腦主機都是一個硬盤掛接到一個總線上，這裏是一對一的關係。而到了有光纖組成的SAN環境，因爲主機和存儲經過了光纖交換機鏈接，這樣的話，就構成了多對多的關係。也就是說，主機到存儲能夠有多條路徑能夠選擇。主機到存儲之間的IO由多條路徑能夠選擇。

既然，每一個主機到所對應的存儲能夠通過幾條不一樣的路徑，若是是同時使用的話，I/O流量如何分配？其中一條路徑壞掉了，如何處理？還有在操做系統的角度來看，每條路徑，操做系統會認爲是一個實際存在的物理盤，但實際上只是通向同一個物理盤的不一樣路徑而已，這樣是在使用的時候，就給用戶帶來了困惑。多路徑軟件就是爲了解決上面的問題應運而生的，多路徑的主要功能就是和存儲設備一塊兒配合實現以下功能：

1. 故障的切換和恢復

2. IO流量的負載均衡

3. 磁盤的虛擬化

爲何使用multipath

因爲多路徑軟件是須要和存儲在一塊兒配合使用的，不一樣的廠商基於不一樣的操做系統，都提供了不一樣的版本。而且有的廠商，軟件和硬件也不是一塊兒賣的，若是要使用多路徑軟件的話，可能還須要向廠商購買license才行。好比EMC公司基於Linux下的多路徑軟件，就須要單獨的購買license。其中，EMC提供的就是PowerPath，HDS提供的就是HDLM，Veritas提供的就是VxDMP。固然，使用系統自帶的免費多路徑軟件包，同時也是一個比較通用的包，能夠支持大多數存儲廠商的設備，即便是一些不是出名的廠商，經過對配置文件進行稍做修改，也是能夠支持並運行的很好。

比較重要的一點仍是遵從原廠工程師的建議根據實際的業務和存儲策略使用合適的多路徑軟件。

配置multipath

原理看了一堆，實際配置仍是比較簡單的，配置文件只有一個：/etc/multipath.conf

#查看主機或者存儲交換機上的WWN號，在存儲上將LUN映射給須要的主機
cat /sys/class/fc_host/host*/port_name
0x2002d0431efb7f5d
0x2001d0431efb7f5d

#在系統內執行掃盤命令，沒有命令先安裝sg3_utils
yum install sg3_utils
rescan-scsi-bus.sh

#查看是否映射到對應的
fdisk -l
lsblk

#查看是否安裝了multipath
yum install device-mapper-multipath

rpm -qa | grep device-mapper
device-mapper-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-persistent-data-0.5.5-1.el7.x86_64
device-mapper-multipath-0.4.9-85.el7_2.4.x86_64
device-mapper-1.02.107-5.el7_2.2.x86_64
device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-event-1.02.107-5.el7_2.2.x86_64
device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64


#拷貝默認的multipath.conf到/etc目錄下，也可使用mpathconf命令建立默認模板
cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
mpathconf --enable --with_multipathd y

#查看本地存儲wwid
/lib/udev/scsi_id --whitelisted --device=/dev/sda
36141877030bcca001d9c4d52106b9d90

#查看存儲參數
cat /sys/block/sdb/device/vendor 
3PARdata
cat /sys/block/sdb/device/model
VV

#添加本地盤到blacklist內,不一樣的存儲和系統參考官方的最佳實踐
vim /etc/multipath.conf

defaults {
polling_interval 10
user_friendly_names no
find_multipaths yes
}
blacklist {
wwid "<wwid_of_the_local_disk>"
}
devices {
device {
vendor "3PARdata"
product "VV"
path_grouping_policy group_by_prio
path_selector "round-robin 0"
path_checker tur
features "0"
hardware_handler "1 alua"
prio alua
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io_rq 1
detect_prio yes
# fast_io_fail_tmo 10
# dev_loss_tmo 14
}
}

#固定設備的別名
cat /etc/multipath/bindings 
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpatha 360002ac000000000000000260001a945
mpathb 360002ac000000000000000270001a945
mpathc 360002ac000000000000000280001a945
mpathd 360002ac000000000000000290001a945
mpathe 360002ac0000000000000002a0001a945

#編輯multipath.conf，增長如下字段
vim /etc/multipath.conf

multipaths {
  multipath {
  wwid 360002ac000000000000000260001a945
  alias 3PAR8400_DEV_LUN26
  }
  multipath {
  wwid 360002ac000000000000000270001a945
  alias 3PAR8400_DEV_LUN27
  }
  multipath {
  wwid 360002ac000000000000000280001a945
  alias 3PAR8400_DEV_LUN28
  }
  multipath {
  wwid 360002ac000000000000000290001a945
  alias 3PAR8400_DEV_LUN29
  }
  multipath {
  wwid 360002ac0000000000000002a0001a945
  alias 3PAR8400_DEV_LUN30
  }
}

#清空已有的multipath記錄
multipath -F
#打印診斷信息
multipath -v3
#啓用多路徑守護程序以在引導時啓動
mpathconf --enable
#啓動多路徑服務
service multipathd start
#若是在啓動 multipath 守護程序後更改多路徑配置文件，請運行如下命令以使更改生效。
service multipathd reload 
#重啓系統測試
init 6
#查看多路徑當前狀態
multipath -ll

#Enable extended logging on the qla2xxx driver
chmod u+x /sys/module/qla2xxx/parameters/ql2xextended_error_logging
echo "1" > /sys/module/qla2xxx/parameters/ql2xextended_error_logging
cat /sys/module/qla2xxx/parameters/ql2xextended_error_logging

#重啓lvm服務是識別共享存儲vg信息
systemctl restart lvm2-lvmetad.service
pvs

#屏蔽WARNING: duplicate提示信息
vim /etc/lvm/lvm.conf

    # Configuration option devices/filter.
    # Limit the block devices that are used by LVM commands.
    # This is a list of regular expressions used to accept or reject block
    # device path names. Each regex is delimited by a vertical bar '|'
    # (or any character) and is preceded by 'a' to accept the path, or
    # by 'r' to reject the path. The first regex in the list to match the
    # path is used, producing the 'a' or 'r' result for the device.
    # When multiple path names exist for a block device, if any path name
    # matches an 'a' pattern before an 'r' pattern, then the device is
    # accepted. If all the path names match an 'r' pattern first, then the
    # device is rejected. Unmatching path names do not affect the accept
    # or reject decision. If no path names for a device match a pattern,
    # then the device is accepted. Be careful mixing 'a' and 'r' patterns,
    # as the combination might produce unexpected results (test changes.)
    # Run vgscan after changing the filter to regenerate the cache.
    # See the use_lvmetad comment for a special case regarding filters.
    # 
    # Example
    # Accept every block device:
    # filter = [ "a|.*/|" ]
    # Reject the cdrom drive:
    # filter = [ "r|/dev/cdrom|" ]
    # Work with just loopback devices, e.g. for testing:
    # filter = [ "a|loop|", "r|.*|" ]
    # Accept all loop devices and ide drives except hdc:
    # filter = [ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.*|" ]
    # Use anchors to be very specific:
    # filter = [ "a|^/dev/hda8$|", "r|.*/|" ]
    # 
    # This configuration option has an automatic default value.
    # filter = [ "a|.*/|" ]
    filter = ["a/sda/","a/mapper/3PAR8400_DEV_LUN26p1/","r/.*/"]
#filter = ["a/sda/","r/.*/"]

    # Configuration option devices/global_filter.
    # Limit the block devices that are used by LVM system components.
    # Because devices/filter may be overridden from the command line, it is
    # not suitable for system-wide device filtering, e.g. udev and lvmetad.
    # Use global_filter to hide devices from these LVM system components.
    # The syntax is the same as devices/filter. Devices rejected by
    # global_filter are not opened by LVM.
    # This configuration option has an automatic default value.
    # global_filter = [ "a|.*/|" ]
    global_filter = ["a/sda/","a/mapper/3PAR8400_DEV_LUN26p1/","r/.*/"]


#一個簡單的multipath.conf配置文件
[root@controller01 ~]# cat /etc/multipath.conf
# This is a basic configuration file with some examples, for device mapper
# multipath.
#
# For a complete list of the default configuration values, run either
# multipath -t
# or
# multipathd show config
#
# For a list of configuration options with descriptions, see the multipath.conf
# man page

## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
#    device {
#        vendor    "IBM"
#        product    "S/390.*"
#    }
#}

## Use user friendly names, instead of using WWIDs as names.
defaults {
    user_friendly_names yes
    find_multipaths yes
}
##
## Here is an example of how to configure some standard options.
##
#
#defaults {
#    polling_interval     10
#    path_selector        "round-robin 0"
#    path_grouping_policy    multibus
#    uid_attribute        ID_SERIAL
#    prio            alua
#    path_checker        readsector0
#    rr_min_io        100
#    max_fds            8192
#    rr_weight        priorities
#    failback        immediate
#    no_path_retry        fail
#    user_friendly_names    yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid.  The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line.  Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
#blacklist {
#       wwid 26353900f02796769
#    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
#    devnode "^hd[a-z]"
#}
#multipaths {
#    multipath {
#        wwid            3600508b4000156d700012000000b0000
#        alias            yellow
#        path_grouping_policy    multibus
#        path_selector        "round-robin 0"
#        failback        manual
#        rr_weight        priorities
#        no_path_retry        5
#    }
#    multipath {
#        wwid            1DEC_____321816758474
#        alias            red
#    }
#}
#devices {
#    device {
#        vendor            "COMPAQ  "
#        product            "HSV110 (C)COMPAQ"
#        path_grouping_policy    multibus
#        path_checker        readsector0
#        path_selector        "round-robin 0"
#        hardware_handler    "0"
#        failback        15
#        rr_weight        priorities
#        no_path_retry        queue
#    }
#    device {
#        vendor            "COMPAQ  "
#        product            "MSA1000         "
#        path_grouping_policy    multibus
#    }
#}
multipaths {
  multipath {
  wwid 360002ac000000000000000260001a945
  alias 3PAR8400_DEV_LUN26
  }
  multipath {
  wwid 360002ac000000000000000270001a945
  alias 3PAR8400_DEV_LUN27
  }
  multipath {
  wwid 360002ac000000000000000280001a945
  alias 3PAR8400_DEV_LUN28
  }
  multipath {
  wwid 360002ac000000000000000290001a945
  alias 3PAR8400_DEV_LUN29
  }
  multipath {
  wwid 360002ac0000000000000002a0001a945
  alias 3PAR8400_DEV_LUN30
  }
  multipath {
  wwid 360002ac0000000000000002b0001a945
  alias 3PAR8400_DEV_LUN31
  }
  multipath {
  wwid 360002ac0000000000000002c0001a945
  alias 3PAR8400_DEV_LUN32
  }
  multipath {
  wwid 360002ac0000000000000002d0001a945
  alias 3PAR8400_DEV_LUN33
  }
  multipath {
  wwid 360002ac0000000000000002e0001a945
  alias 3PAR8400_DEV_LUN34
  }
  multipath {
  wwid 360002ac0000000000000002f0001a945
  alias 3PAR8400_DEV_LUN35
  }
  multipath {
  wwid 360002ac000000000000000320001a945
  alias 3PAR8400_DEV_LUN36
  }
  multipath {
  wwid 360002ac000000000000000330001a945
  alias 3PAR8400_DEV_LUN37
  }
  multipath {
  wwid 360002ac000000000000000340001a945
  alias 3PAR8400_DEV_LUN38
  }
  multipath {
  wwid 360002ac000000000000000350001a945
  alias 3PAR8400_DEV_LUN39
  }
  multipath {
  wwid 360002ac000000000000000360001a945
  alias 3PAR8400_DEV_LUN40
  }
  multipath {
  wwid 360002ac000000000000000370001a945
  alias 3PAR8400_DEV_LUN41
  }
  multipath {
  wwid 360002ac000000000000000380001a945
  alias 3PAR8400_DEV_LUN42
  }
  multipath {
  wwid 360002ac000000000000000390001a945
  alias 3PAR8400_DEV_LUN43
  }
  multipath {
  wwid 360002ac0000000000000003a0001a945
  alias 3PAR8400_DEV_LUN44
  }
  multipath {
  wwid 360002ac0000000000000003b0001a945
  alias 3PAR8400_DEV_LUN45
  }
  multipath {
  wwid 360002ac0000000000000003d0001a945
  alias 3PAR8400_DEV_LUN46
  }
  multipath {
  wwid 360002ac0000000000000003e0001a945
  alias 3PAR8400_DEV_LUN47
  }
  multipath {
  wwid 360002ac0000000000000003f0001a945
  alias 3PAR8400_DEV_LUN48
  }
  multipath {
  wwid 360002ac000000000000000400001a945
  alias 3PAR8400_DEV_LUN49
  }
  multipath {
  wwid 360002ac000000000000000410001a945
  alias 3PAR8400_DEV_LUN50
  }
}