常見的存儲技術(存儲架構)
node
顧名思義,直接附加存儲就是將存儲直接附加在應用服務器中來實現數據的存儲。存儲設備是經過接口類型於應用程序服務器鏈接的。web
IDE (並行):數據可能有干擾 理論:133MB/s。早期是叫作ATA數據庫
SATA (串行)II: 3Gbps III: 6Gbpsvim
IDE速度慢,出現了SCSI(Smal System System Interface)。後端
SCSI接口:bash
UltraSCSI-320:320MB/s服務器
UltraSCSI-640:640MB/s網絡
SAS: 6Gbpssession
USB,sSATA,1394架構
如今直連式存儲與服務器主機之間的鏈接通道一般採用SCSI協議鏈接
通常說來,NAS是文件服務器。像NFS,Samba等。提供文件級別的共享。
SCSI之間通訊是一種C/S架構。
叫作Initator(發起者 C)
這裏稱之爲Target:(目標 S),每一個Target可包含多個LUN(Logical Unit Number 邏輯單元)。真正提供服務的是LUN。
下圖是SCSI協議之間通訊的過程:
內核空間:系統調用 --》 虛擬文件系統 --》 文件系統 --》 buffer cache --》 驅動 --》 SCSI接口
用戶空間:經過GNU c library 調用內核空間的系統調用
以下圖:
所謂iSCSI是利用tcp/ip協議來傳輸SCSI報文的一種機制。
1、環境
系統:CentOS 6.4x64最小化安裝
Target:192.168.3.43
Initator:192.168.3.46
2、安裝配置服務端
Target端(192.168.3.43):
#準備好要共享的設備 [root@target ~]# fdisk -l |grep dev/sdc Disk /dev/sdc: 322.1 GB, 322122547200 bytes /dev/sdc1 1 13055 104864256 83 Linux #大概100G /dev/sdc2 13056 39162 209704477+ 83 Linux #大概200G #安裝iscsi軟件 [root@target ~]# yum install scsi-target-utils -y [root@target ~]# chkconfig tgtd on [root@target ~]# service tgtd start Starting SCSI target daemon: [ OK ] [root@target ~]# netstat -tunlp |grep tgtd #默認端口監聽在3260上 tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 1536/tgtd tcp 0 0 :::3260 :::* LISTEN 1536/tgtd #在iptables中放行3260 [root@target ~]# iptables -I INPUT -p tcp --dport 3260 -j ACCEPT [root@target ~]# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:3260 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination
使用tgtadm命令方式配置scsi服務,命令tgtadm的介紹:
tgtadm 是一個模式化的命令 使用格式:tgtadm --lld [driver] --op [operation] --mode [mode] [OPTION]... 常見的模式(mode):target logicalunit account ##################################################### target模式的管理命令:管理target new:建立一個新的 target show:顯示target delete:刪除target update:更新(修改)target bind:基於IP地址控制 unbind:解除綁定 ##################################################### logicalunit模式的管理命令:關聯tid和後端磁盤 new:建立一個新的LUN deltete:刪除一個已有的LUN ##################################################### account模式的管理命令:管理account CHAP(挑戰式握手應用) 單向認證 new deltete bind unbid ##################################################### target和Initator之間是經過iqn名稱來通訊的 iqn格式名稱: iqn.yyyy-mm.reverse-domaim-name:string[.sunstring] 例如:iscsi-iname -p iqn.2014-09.com.magedu.target 也可經過命令的生成的方式:iscsi-iname -p iqn.2014-09.com.magedu ##################################################### 示例: # (1)、添加一個新的 target 且其ID爲 [id], 名字爲 [name]. --lld [driver] --op new --mode target --tid=[id] --targetname [name] # (2)、顯示全部或某個特定的target: --lld [driver] --op show --mode target [--tid=[id]] # (3)、向某ID爲[id]的設備上添加一個新的LUN,其號碼爲[lun],且此設備提供給initiator使用。 ## [path]是某「塊設備」的路徑,此塊設備也能夠是raid或lvm設備。lun0已經被系統預留。 --lld [driver] --op new --mode=logicalunit --tid=[id] --lun=[lun] --backing-store [path] # (4)、刪除ID爲[id]的target: --lld [driver] --op delete --mode target --tid=[id] # (5)、刪除target [id]中的LUN [lun]: -lld [driver] --op delete --mode=logicalunit --tid=[id] --lun=[lun] # (6)、定義某target的基於主機的訪問控制列表,其中,[address]表示容許訪問此target的 # initiator客戶端的列表: --lld [driver] --op bind --mode=target --tid=[id] --initiator-address=[address] # (7)、解除target [id]的訪問控制列表中[address]的訪問控制權限: --lld [driver] --op unbind --mode=target --tid=[id] --initiator-address=[address]
提供服務,使用tgtadm命令方式:
#步驟以下: #建立一個target: [root@target ~]# tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2015-07.com.weyee:test1 # 建立LUN,號碼爲1: [root@target ~]# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sdc1 # 開放給192.168.3.0/24網絡中的主機訪問 [root@target ~]# tgtadm --lld iscsi --op bind --mode target --tid 1 -I 192.168.3.0/24 其中的-I至關於--initiator-address #查看建立後的結果 [root@target ~]# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2015-07.com.weyee:test1 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 #這個是咱們剛纔建立的target Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 107381 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/sdc1 Backing store flags: Account information: ACL information: #定義哪些網段可以訪問target 192.168.3.0/24
3、配置客戶端
Initiator(192.168.3.46):
#客戶端配置 [root@initator ~]# yum install iscsi-initiator-utils -y [root@initator ~]# chkconfig iscsi on [root@initator ~]# service iscsi start
在客戶端Initator使用命令iscsiadm來發現設備。iscsiadm命令介紹:
iscsiadm是個模式化的工具,其模式可經過-m或--mode選項指定,常見的模式有discovery、 node、fw、session、host、iface幾個。 若是沒有額外指定其它選項,則discovery和node會顯示其相關的全部記錄; session用於顯示全部的活動會話和鏈接 fw顯示全部的啓動固件值 host顯示全部的iSCSI主機 iface顯示/var/lib/iscsi/ifaces目錄中的全部ifaces設定。 # 用法以下: iscsiadm -m discovery [ -d debug_level ] [ -P printlevel ] \ [ -I iface -t type -p ip:port [ -l ] ] iscsiadm -m node [ -d debug_level ] [ -P printlevel ] [ -L all,manual,automatic ] \ [ -U all,manual,automatic ] [ [ -T tar-getname -p ip:port -I iface ] \ [ -l | -u | -R | -s] ] [ [ -o operation ] # 參數說明: -d, --debug=debug_level 顯示debug信息,級別爲0-8; -l, --login -t, --type=type 這裏可使用的類型爲sendtargets(可簡寫爲st)、slp、fw和 isns, 此選項僅用於discovery模式,且目前僅支持st、fw和isns;其中st表示容許每一個iSCSI target 發送一個可用target列表給initiator; -p, --portal=ip[:port] 指定target服務的IP和端口; -m, --mode op 可用的mode有discovery, node, fw, host iface 和 session -T, --targetname=targetname 用於指定target的名字 -u, --logout -o, --op=OPEARTION:指定針對discoverydb數據庫的操做,其僅能爲new、delete、update、show 和nonpersistent其中之一; -I, --interface=[iface]:指定執行操做的iSCSI接口,這些接口定義在/var/lib/iscsi/ifaces中;
經過iscsiadm命令發現target
#查看默認的iqn信息 [root@initator ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:a999167c973 #執行iscsiadm發現target命令 [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 Starting iscsid: [ OK ] 192.168.3.43:3260,1 iqn.2015-07.com.weyee:test1 #成功發現target #登錄target #先看看本機的磁盤信息 [root@initator ~]# fdisk -l |grep dev Disk /dev/sda: 21.5 GB, 21474836480 bytes /dev/sda1 * 1 26 204800 83 Linux /dev/sda2 26 281 2048000 82 Linux swap / Solaris /dev/sda3 281 2611 18717696 83 Linux #從上面的結果咱們能看到本機只有一個/dev/sda磁盤 #執行登錄iscsi命令 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] (multiple) Login to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] successful. #有successful表示登錄成功 #再次查看本地磁盤信息 [root@initator ~]# fdisk -l |grep dev Disk /dev/sda: 21.5 GB, 21474836480 bytes /dev/sda1 * 1 26 204800 83 Linux /dev/sda2 26 281 2048000 82 Linux swap / Solaris /dev/sda3 281 2611 18717696 83 Linux Disk /dev/sdb: 107.4 GB, 107380998144 bytes #這裏顯示多出來一塊磁盤 #將多出來的分區進行格式化並掛載 [root@initator ~]# mkfs.ext4 /dev/sdb [root@initator ~]# tune2fs -c -1 /dev/sdb tune2fs 1.41.12 (17-May-2010) Setting maximal mount count to -1 [root@initator ~]# mount /dev/sdb /mnt [root@initator ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 18G 1.2G 16G 8% / tmpfs 495M 0 495M 0% /dev/shm /dev/sda1 194M 28M 156M 16% /boot /dev/sdb 99G 188M 94G 1% /mnt [root@initator ~]# touch /mnt/target.txt [root@initator ~]# ls /mnt lost+found target.txt #若是客戶端不想使用了,能夠退出 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -u Logging out of session [sid: 1, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] Logout of [sid: 1, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] successful. [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -o delete # 必要的時候,使用 rm -rf -rf /var/lib/iscsi/send_targets/*
4、實現target單向認證
配置target端:
#建立target認證須要的帳號 [root@target ~]# tgtadm --lld iscsi --op new --mode account --user lyao --password 123456 #將用戶和target進行綁定 [root@target ~]# tgtadm --lld iscsi --op bind --mode account --tid 1 --user lyao #查看結果 [root@target ~]# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2015-07.com.weyee:test1 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 107381 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/sdc1 Backing store flags: Account information: lyao #這裏能看到添加的認證用戶 ACL information: 192.168.3.0/24
配置Initiator端:
#先退出target [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -u [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -o delete [root@initator ~]# rm -rf /var/lib/iscsi/send_targets/* #重啓服務 [root@initator ~]# service iscsid restart #再次執行發現命令 [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 192.168.3.43:3260,1 iqn.2015-07.com.weyee:test1 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] (multiple) iscsiadm: Could not login to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260]. #下面顯示認證失敗,咱們在target端啓用的身份認證 iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure) iscsiadm: Could not log into all portals #上面的問題是Initiator沒有提供身份信息致使的 #編輯/etc/iscsi/iscsid.conf [root@initator ~]# vim /etc/iscsi/iscsid.conf node.session.auth.authmethod = CHAP node.session.auth.username = lyao node.session.auth.password = 123456 [root@initator ~]# service iscsid restart [root@initator ~]# rm -rf /var/lib/iscsi/send_targets/192.168.3.43,3260/ #從新執行發現命令 [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 192.168.3.43:3260,1 iqn.2015-07.com.weyee:test1 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] (multiple) Login to [iface: default, target: iqn.2015-07.com.weyee:test1, portal: 192.168.3.43,3260] successful. [root@initator ~]# fdisk -l |grep dev Disk /dev/sda: 21.5 GB, 21474836480 bytes /dev/sda1 * 1 26 204800 83 Linux /dev/sda2 26 281 2048000 82 Linux swap / Solaris /dev/sda3 281 2611 18717696 83 Linux Disk /dev/sdb: 107.4 GB, 107380998144 bytes #iscsi的單向認證配置成功 #說明若是initiator端已經登陸過此target,此時還須要先註銷登陸後重啓iscsid服務,並在刪除此 前生成的database後從新發現target,並從新登入,過程以下: # iscsiadm -m session -r sid -u # iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43:3260 -u # iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43:3260 -o delete # rm -rf -rf /var/lib/iscsi/send_targets/192.168.3.43:3260 # service iscsid restart # iscsiadm -m discovery -t sendtargets -p 192.168.3.43 # iscsiadm -m node -T iqn.2015-07.com.weyee:test1 -p 192.168.3.43:3260 -l
5、經過配置文件配置target
在上文中咱們經過命令配置的target,若是重啓tgtd服務後,全部配置信息都會丟失
[root@target ~]# service tgtd restart Stopping SCSI target daemon: [ OK ] Starting SCSI target daemon: [ OK ] [root@target ~]# tgtadm --lld iscsi --op show --mode target
在這種狀況下,咱們只能經過配置文件來管理target
#編輯配置文件 [root@target ~]# vim /etc/tgt/targets.conf #添加以下內容 <target iqn.2015-07.com.weyee:web> <backing-store /dev/sdc1> lun 7 </backing-store> initiator-address 192.168.3.0/24 </target> [root@target ~]# service tgtd restart Stopping SCSI target daemon: [ OK ] Starting SCSI target daemon: [ OK ] [root@target ~]# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2015-07.com.weyee:web System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 7 Type: disk SCSI ID: IET 00010007 SCSI SN: beaf17 Size: 107381 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/sdc1 Backing store flags: Account information: ACL information: 192.168.3.0/24 #在客戶端進行以下操做 [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 192.168.3.43:3260,1 iqn.2015-07.com.weyee:web [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:web -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] (multiple) Login to [iface: default, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] successful. #登錄成功 #再將/dev/sdc2添加進來並要求身份認證 #編輯配置文件 [root@target ~]# vim /etc/tgt/targets.conf <target iqn.2015-07.com.weyee:nfs> <backing-store /dev/sdc2> lun 4 </backing-store> incominguser weyee weyee2014 initiator-address 192.168.3.0/24 </target> #重載tgtd服務 [root@target ~]# service tgtd reload Updating SCSI target daemon configuration: [ OK ] [root@target ~]# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2015-07.com.weyee:web System information: Driver: iscsi State: ready I_T nexus information: I_T nexus: 1 Initiator: iqn.1994-05.com.redhat:a999167c973 Connection: 0 IP Address: 192.168.3.46 LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 7 Type: disk SCSI ID: IET 00010007 SCSI SN: beaf17 Size: 107381 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/sdc1 Backing store flags: Account information: ACL information: 192.168.3.0/24 Target 2: iqn.2015-07.com.weyee:nfs System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00020000 SCSI SN: beaf20 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 4 Type: disk SCSI ID: IET 00020004 SCSI SN: beaf24 Size: 214737 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/sdc2 Backing store flags: Account information: weyee ACL information: 192.168.3.0/24 #在客戶端執行discovery [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 192.168.3.43:3260,1 iqn.2015-07.com.weyee:web 192.168.3.43:3260,1 iqn.2015-07.com.weyee:nfs #登錄到iqn.2015-07.com.weyee:nfs [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:nfs -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:nfs, portal: 192.168.3.43,3260] (multiple) iscsiadm: Could not login to [iface: default, target: iqn.2015-07.com.weyee:nfs, portal: 192.168.3.43,3260]. #認證失敗 iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure) iscsiadm: Could not log into all portals #編輯客戶端配置文件 [root@initator ~]# vim /etc/iscsi/iscsid.conf node.session.auth.authmethod = CHAP node.session.auth.username = weyee node.session.auth.password = weyee2014 #中止全部已登錄的target [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:web -p 192.168.3.43 -u Logging out of session [sid: 8, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] Logout of [sid: 8, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] successful. [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:web -p 192.168.3.43 -o delete [root@initator ~]# rm -rf /var/lib/iscsi/send_targets/* #重啓客戶端的iscsi服務 [root@initator ~]# service iscsi restart #再次執行discovery [root@initator ~]# iscsiadm -m discovery -t sendtargets -p 192.168.3.43 192.168.3.43:3260,1 iqn.2015-07.com.weyee:web 192.168.3.43:3260,1 iqn.2015-07.com.weyee:nfs #登錄到iqn.2015-07.com.weyee:nfs,這個須要身份認證 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:nfs -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:nfs, portal: 192.168.3.43,3260] (multiple) Login to [iface: default, target: iqn.2015-07.com.weyee:nfs, portal: 192.168.3.43,3260] successful. #下面的不須要身份認證 [root@initator ~]# iscsiadm -m node -T iqn.2015-07.com.weyee:web -p 192.168.3.43 -l Logging in to [iface: default, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] (multiple) Login to [iface: default, target: iqn.2015-07.com.weyee:web, portal: 192.168.3.43,3260] successful. #查看磁盤信息 [root@initator ~]# fdisk -l |grep dev Disk /dev/sda: 21.5 GB, 21474836480 bytes /dev/sda1 * 1 26 204800 83 Linux /dev/sda2 26 281 2048000 82 Linux swap / Solaris /dev/sda3 281 2611 18717696 83 Linux Disk /dev/sdb: 214.7 GB, 214737384960 bytes Disk /dev/sdc: 107.4 GB, 107380998144 bytes #這裏能看到2個iscsi的共享盤能正常顯示