Swift 架構、原理及功能: http://www.cnblogs.com/sammyliu/p/4955241.htmlhtml
總結的很詳細也很全面,受益不淺,感謝分享。node
vmware 11.0
系統:rhel7.2
openstack版本:openstack-Mitakapython
swift-controller: 192.168.0.11
swift-object1: 192.168.0.51
swift-object2: 192.168.0.52mysql
注意:這裏的第一張網卡nat做爲內網網卡,第二張網卡橋接做爲外網網卡(使用一張橋接網卡可實現,這裏是爲了ip規範使用nat網絡)c++
/etc/hosts 192.168.0.11 controller 192.168.0.51 object1 192.168.0.52 object2
時間同步,三節點都執行:git
# ntpdate tiger.sina.com.cn
swift-controller配置:github
# yum install python-openstackclient mariadb mariadb-server python2-PyMySQL rabbitmq-server memcached python-memcached -y
配置數據庫sql
# vim /etc/my.cnf.d/openstack.cnf [mysqld] bind-address = 192.168.0.11 default-storage-engine = innodb innodb_file_per_table max_connections = 4096 collation-server = utf8_general_ci character-set-server = utf8
啓動及初始化數據庫數據庫
# systemctl enable mariadb ; systemctl start mariadb # mysql_secure_installation # 密碼 123456
啓動消息隊列 rabbitmq服務apache
# systemctl enable rabbitmq-server ; systemctl start rabbitmq-server
添加用戶openstack的用戶並賦權
# rabbitmqctl add_user openstack openstack # rabbitmqctl set_permissions openstack ".*" ".*" ".*"
啓動memcache服務
# systemctl enable memcached ; systemctl start memcached
查看服務啓動狀況
3306: mariadb 11211:memcache 2567:rabbitmq # netstat -ntplu | egrep "3306|11211|2567" tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 20546/beam.smp tcp 0 0 192.168.0.11:3306 0.0.0.0:* LISTEN 20412/mysqld tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 21084/memcached tcp6 0 0 ::1:11211 :::* LISTEN 21084/memcached udp 0 0 127.0.0.1:11211 0.0.0.0:* 21084/memcached udp6 0 0 ::1:11211 :::* 21084/memcached
建立keystone數據庫並受權
# mysql -p123456 MariaDB [(none)]> create database keystone; MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'keystone'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystone';
安裝程序包
# yum install openstack-keystone httpd mod_wsgi -y
配置keystone.conf
# vim /etc/keystone/keystone.conf [DEFAULT] ... admin_token = 2b64e54cdce5900a22f8 ... [database] ... connection = mysql+pymysql://keystone:keystone@controller/keystone ... [token] ... provider = fernet ...
初始化 keystone 數據庫
# su -s /bin/sh -c "keystone-manage db_sync" keystone
初始化Fernet keys:
# keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
配置apache
# vim /etc/httpd/conf/httpd.conf
...
ServerName controller
...
配置openstack使用的虛擬主機:
# vim /etc/httpd/conf.d/wsgi-keystone.conf Listen 5000 Listen 35357 <VirtualHost *:5000> WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} WSGIProcessGroup keystone-public WSGIScriptAlias / /usr/bin/keystone-wsgi-public WSGIApplicationGroup %{GLOBAL} WSGIPassAuthorization On ErrorLogFormat "%{cu}t %M" ErrorLog /var/log/httpd/keystone-error.log CustomLog /var/log/httpd/keystone-access.log combined <Directory /usr/bin> Require all granted </Directory> </VirtualHost> <VirtualHost *:35357> WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} WSGIProcessGroup keystone-admin WSGIScriptAlias / /usr/bin/keystone-wsgi-admin WSGIApplicationGroup %{GLOBAL} WSGIPassAuthorization On ErrorLogFormat "%{cu}t %M" ErrorLog /var/log/httpd/keystone-error.log CustomLog /var/log/httpd/keystone-access.log combined <Directory /usr/bin> Require all granted </Directory> </VirtualHost>
配置完成,啓動服務
# systemctl enable httpd ; systemctl start httpd
配置認證令牌、端點URL、api版本
# export OS_TOKEN=2b64e54cdce5900a22f8 # export OS_URL=http://controller:35357/v3 # export OS_IDENTITY_API_VERSION=3
(1)建立keystone服務
# openstack service create --name keystone --description "OpenStack Identity" identity
(2)建立api端點
# openstack endpoint create --region RegionOne identity public http://controller:5000/v3 # openstack endpoint create --region RegionOne identity internal http://controller:5000/v3 # openstack endpoint create --region RegionOne identity admin http://controller:35357/v3
(3)建立域
# openstack domain create --description "Default Domain" default
(4)建立項目
# openstack project create --domain default --description "Admin Project" admin
(5)建立用戶(admin密碼:admin)
# openstack user create --domain default --password-prompt admin
(6)建立角色
# openstack role create admin
(7)添加角色到項目和用戶上
# openstack role add --project admin --user admin admin
(8)建立 service 項目
# openstack project create --domain default --description "Service Project" service
驗證:
# unset OS_TOKEN OS_URL # openstack --os-auth-url http://controller:35357/v3 \ --os-project-domain-name default --os-user-domain-name default \ --os-project-name admin --os-username admin token issue # 密碼上面設置: admin
建立認證腳本
# vim admin-openrc export OS_PROJECT_DOMAIN_NAME=default export OS_USER_DOMAIN_NAME=default export OS_PROJECT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin export OS_AUTH_URL=http://controller:35357/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2
測試認證
# . admin-openrc
# openstack token issue
# . admin-openrc
(1)建立 swift用戶,給swift用戶添加admin角色
# openstack user create --domain default --password-prompt swift
密碼:swift
# openstack role add --project service --user swift admin
(2)建立swift服務
# openstack service create --name swift --description "OpenStack Object Storage" object-store
(3)建立對象存儲服務 API 端點
# openstack endpoint create --region RegionOne object-store public http://controller:8080/v1/AUTH_%\(tenant_id\)s # openstack endpoint create --region RegionOne object-store internal http://controller:8080/v1/AUTH_%\(tenant_id\)s # openstack endpoint create --region RegionOne object-store admin http://controller:8080/v1
安裝 swift 程序包
yum install openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached -y
獲取代理服務的配置文件:
# curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample
修改以下內容:
# vim /etc/swift/proxy-server.conf [DEFAULT] bind_port = 8080 user = swift swift_dir = /etc/swift ... [pipeline:main] pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server ... [app:proxy-server] use = egg:swift#proxy account_autocreate = True ... [filter:keystoneauth] use = egg:swift#keystoneauth operator_roles = admin,user ... [filter:authtoken] paste.filter_factory = keystonemiddleware.auth_token:filter_factory auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = 127.0.0.1:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = swift password = swift delay_auth_decision = True ... [filter:cache] use = egg:swift#memcache memcache_servers = 127.0.0.1:11211
如下操做在 object1 和 object2 上執行
swift-object1: 192.168.0.51 swift-object2: 192.168.0.52
(1)安裝支持工具包
# yum install xfsprogs rsync openstack-swift-account openstack-swift-container openstack-swift-object -y
(2)使用 xfs 格式化磁盤
# mkfs.xfs /dev/sdb # mkfs.xfs /dev/sdc # mkfs.xfs /dev/sdd
(3)建立掛載點
# mkdir -pv /srv/node/sd{b,c,d}
(4)編輯 /etc/fstab 添加掛載內容
# vim /etc/fstab ... /dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2 /dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2 /dev/sdd /srv/node/sdd xfs noatime,nodiratime,nobarrier,logbufs=8 0 2 # mount -a
(5)建立並編寫 /etc/rsyncd.conf 文件
# vim /etc/rsyncd.conf uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 192.168.0.51 # object1爲 192.168.0.51 object2爲 192.168.0.52 [account] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = False lock file = /var/lock/object.lock
(6)啓動 rsyncd 服務並開機啓動
# systemctl enable rsyncd.service ; systemctl start rsyncd.service
從倉庫下載account、container、object、swift配置文件
# curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample # curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample # curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample # curl -o /etc/swift/swift.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample
(1)配置 /etc/swift/account-server.conf
# vim /etc/swift/account-server.conf [DEFAULT] bind_ip = 192.168.0.51 # object1爲 192.168.0.51 object2爲 192.168.0.52 bind_port = 6002 user = swift swift_dir = /etc/swift devices = /srv/node mount_check = True ... [pipeline:main] pipeline = healthcheck recon account-server ... [filter:recon] use = egg:swift#recon ... recon_cache_path = /var/cache/swift
(2)配置 /etc/swift/container-server.conf
# vim /etc/swift/container-server.conf [DEFAULT] bind_ip = 192.168.0.51 # object1爲 192.168.0.51 object2爲 192.168.0.52 bind_port = 6001 user = swift swift_dir = /etc/swift devices = /srv/node mount_check = True ... [pipeline:main] pipeline = healthcheck recon container-server ... [filter:recon] use = egg:swift#recon ... recon_cache_path = /var/cache/swift
(3)配置 /etc/swift/object-server.conf
# vim /etc/swift/object-server.conf [DEFAULT] bind_ip = 192.168.0.51 # object1爲 192.168.0.51 object2爲 192.168.0.52 bind_port = 6000 user = swift swift_dir = /etc/swift devices = /srv/node mount_check = True ... [pipeline:main] pipeline = healthcheck recon object-server ... [filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift recon_lock_path = /var/lock
確認掛載點目錄結構是否有合適的全部權
# chown -R swift:swift /srv/node # mkdir -p /var/cache/swift # chown -R root:swift /var/cache/swift # chmod -R 775 /var/cache/swift
如下操做在 swift-container 上執行
swift-controller: 192.168.0.11
建立帳戶ring
切換到 /etc/swift 目錄。
# cd /etc/swift/
建立account ring 文件
(1)建立基本 account.builder 文件
# swift-ring-builder account.builder create 10 3 1
(2)添加每一個節點到 ring 中:
object1: 192.168.0.51 # swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6002 --device sdb --weight 100 # swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6002 --device sdc --weight 100 # swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6002 --device sdd --weight 100 object2: 192.168.0.52 # swift-ring-builder account.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6002 --device sdb --weight 100 # swift-ring-builder account.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6002 --device sdc --weight 100 # swift-ring-builder account.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6002 --device sdd --weight 100
(3)驗證 ring 內容
# swift-ring-builder account.builder
(4)平衡 ring
# swift-ring-builder account.builder rebalance
建立container ring 文件
(1)建立基本 container.builder 文件
# swift-ring-builder container.builder create 10 3 1
(2)添加每一個節點到 ring 中:
object1: 192.168.0.51 # swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6001 --device sdb --weight 100 # swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6001 --device sdc --weight 100 # swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6001 --device sdd --weight 100 object2: 192.168.0.52 # swift-ring-builder container.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6001 --device sdb --weight 100 # swift-ring-builder container.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6001 --device sdc --weight 100 # swift-ring-builder container.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6001 --device sdd --weight 100
(3)驗證 ring 內容
# swift-ring-builder container.builder
(4)平衡 ring
# swift-ring-builder container.builder rebalance
建立 object ring 文件
(1)建立基本 container.builder 文件
# swift-ring-builder object.builder create 10 3 1
(2)添加每一個節點到 ring 中:
# swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6000 --device sdb --weight 100 # swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6000 --device sdc --weight 100 # swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.0.51 --port 6000 --device sdd --weight 100 object2: 192.168.0.52 # swift-ring-builder object.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6000 --device sdb --weight 100 # swift-ring-builder object.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6000 --device sdc --weight 100 # swift-ring-builder object.builder add --region 1 --zone 2 --ip 192.168.0.52 --port 6000 --device sdd --weight 100
(3)驗證 ring 內容
# swift-ring-builder object.builder
(4)平衡 ring
# swift-ring-builder object.builder rebalance
將ring配置文件拷貝到每一個存儲節點,這裏也就是 object1 和 object2
# scp *.ring.gz object1:/etc/swift/ # scp *.ring.gz object2:/etc/swift/
配置 /etc/swift/swift.conf 文件
# vim /etc/swift/swift.conf [swift-hash] ... swift_hash_path_suffix = mickey swift_hash_path_prefix = minnie ...
複製 /etc/swift/swift.conf 到其餘object節點和代理節點,這裏也就是 object1 和 object2
# scp swift.conf object1:/etc/swift/ # scp swift.conf object2:/etc/swift/
確認三個節點配置文件權限,在全部節點執行;
# chown -R root:swift /etc/swift
(1)swift-controller 節點啓動服務
# systemctl enable openstack-swift-proxy.service memcached.service; systemctl start openstack-swift-proxy.service memcached.service
問題1:在啓動 openstack-swift-proxy.service 服務後,查看啓動信息
# systemctl status -l openstack-swift-proxy.service Jan 22 22:08:30 controller liberasurecode[15717]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Jan 22 22:08:30 controller liberasurecode[15717]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
缺乏庫文件,須要編譯安裝:
# yum install gcc gcc-c++ make automake autoconf libtool yasm -y # cd /usr/local/src/ # wget https://codeload.github.com/01org/isa-l/zip/master # unzip isa-l-master.zip # ./autogen.sh # ./configure --prefix=/usr --libdir=/usr/lib64 # make -j 2 && make install # systemctl restart openstack-swift-proxy.service # systemctl status -l openstack-swift-proxy.service Jan 22 22:16:55 controller liberasurecode[23647]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Jan 22 22:16:55 controller liberasurecode[23647]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
liberasurecode-1.1.0.tar.gz下載地址: https://bitbucket.org/tsg-/liberasurecode/downloads/
升級liberasurecode版本
# tar xf liberasurecode-1.1.0.tar.gz # ./autogen.sh # ./configure # make -j 2 && make install 強制卸載低版本 # rpm -e --nodeps liberasurecode # systemctl restart openstack-swift-proxy.service # systemctl status -l openstack-swift-proxy.service
再無報錯信息
(2)swift-object 節點啓動服務
swift-object1: 192.168.0.51 swift-object2: 192.168.0.52
升級liberasurecode版本,安裝libisal
python-six 包必須安裝,不然服務啓動失敗
python-six 包必須安裝,不然服務啓動失敗 # yum install gcc gcc-c++ make automake autoconf libtool yasm python-six -y # cd /usr/local/src/ # wget https://codeload.github.com/01org/isa-l/zip/master # unzip isa-l-master.zip # cd isa-l-master # ./autogen.sh # ./configure --prefix=/usr --libdir=/usr/lib64 # make -j 2 && make install # tar xf liberasurecode-1.1.0.tar.gz # cd liberasurecode-1.1.0 # ./autogen.sh # ./configure # make -j 2 && make install 強制卸載低版本 # rpm -e --nodeps liberasurecode
啓動服務
# systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service \ openstack-swift-account-reaper.service openstack-swift-account-replicator.service # systemctl start openstack-swift-account.service openstack-swift-account-auditor.service \ openstack-swift-account-reaper.service openstack-swift-account-replicator.service # systemctl enable openstack-swift-container.service \ openstack-swift-container-auditor.service openstack-swift-container-replicator.service \ openstack-swift-container-updater.service # systemctl start openstack-swift-container.service \ openstack-swift-container-auditor.service openstack-swift-container-replicator.service \ openstack-swift-container-updater.service # systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service \ openstack-swift-object-replicator.service openstack-swift-object-updater.service # systemctl start openstack-swift-object.service openstack-swift-object-auditor.service \ openstack-swift-object-replicator.service openstack-swift-object-updater.service
問題2:啓動 openstack-swift-object-replicator.service 報錯
Jan 22 22:40:09 object2 object-replicator[25374]: ERROR creating /srv/node/sdb/objects: #012Traceback (most recent call last):#012... Jan 22 22:40:09 object2 object-replicator[25374]: ERROR creating /srv/node/sdc/objects: #012Traceback (most recent call last):#012... Jan 22 22:40:09 object2 object-replicator[25374]: ERROR creating /srv/node/sdd/objects: #012Traceback (most recent call last):#012...
檢查 /srv/node/ 權限
# ll -d /srv/node/ drwxr-xr-x 5 root root 36 Jan 22 21:24 /srv/node/ # chown -R swift:swift /srv/node # mkdir -p /var/cache/swift # chown -R root:swift /var/cache/swift # chmod -R 775 /var/cache/swift
重啓 openstack-swift-object-replicator.service 恢復正常
在 swift-controller 節點執行:
# . admin-openrc # swift stat Account: AUTH_3b6f963488db4af49e4e0c0d095dd6cf Containers: 0 Objects: 0 Bytes: 0 X-Put-Timestamp: 1516632330.04818 X-Timestamp: 1516632330.04818 X-Trans-Id: tx5fa7f5e817714cbca5cca-005a65f909 Content-Type: text/plain; charset=utf-
建立一個測試文件,上傳該測試文件到 container1 容器中
建立一個測試文件 # touch testfile 上傳該測試文件到 container1 容器中 # swift upload container1 testfile testfile 查看該文件 # swift list container1 testfile 下載該文件 # swift download container1 testfile testfile [auth 0.343s, headers 0.625s, total 0.626s, 0.000 MB/s]
swift 服務測試成功
在 swift-controller 節點執行:
swift-controller: 192.168.0.11
# yum install openstack-dashboard -y
編輯配置文件 /etc/openstack-dashboard/local_settings
# vim /etc/openstack-dashboard/local_settings 修改如下內容 ... OPENSTACK_HOST = "controller" ... ALLOWED_HOSTS = ['*', ] ... SESSION_ENGINE = 'django.contrib.sessions.backends.cache' CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', 'LOCATION': '127.0.0.1:11211', } } ... OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST ... OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True ... OPENSTACK_API_VERSIONS = { "identity": 3, "image": 2, "volume": 2, } ... OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = "default" ... OPENSTACK_KEYSTONE_DEFAULT_ROLE = "user" ... TIME_ZONE = "Asia/Shanghai"
配置完成,重啓服務
# systemctl restart httpd.service memcached.service
瀏覽器登陸:
域: default
用戶名: admin
密碼: admin
dashboard 配置完成。
測試1:swift 可否支持大於5G的文件上傳?
生成一個6G的文件,上傳單個6G的文件
# dd if=/dev/zero of=bigfile-6G bs=1M count=6144 # . admin-openrc # swift upload container1 bigfile-6G Object PUT failed: http://controller:8080/v1/AUTH_3b6f963488db4af49e4e0c0d095dd6cf/container1/bigfile-6G 413 Request Entity Too Large Your request is too large.
報錯信息:413 請求實體太大,你的請求太大,看來沒法普通上傳大於5G的文件
swift 客戶端版本:python-swiftclient-3.0.0
採用分段上傳的方式:
-S 分段大小 (單位:bytes)
# swift upload container1 -S 1000000000 bigfile-6G bigfile-6G segment 6 bigfile-6G segment 1 bigfile-6G segment 5 bigfile-6G segment 0 bigfile-6G segment 2 bigfile-6G # swift list container1 bigfile-6G testfile
上傳成功。
總結:
(1)Swift 對於小的文件,是不分段直接存放的;
(2)對於大文件(超過5G),須要指定大小分段來存放文件,好比6G文件,設置分段爲1G,則會被分爲 6 段上傳到object集羣
測試2:新加object節點
這裏因爲電腦配置緣由再也不測試,由於咱們在建立時,副本爲三份,因此在擴展object節點時候,分區要是 3 的倍數。