openstack運維實戰系列(十三)之glance更改路徑引起的"血案"
1. 背景說明node
glance在openstack中負責鏡像相關的服務，支持將運行的虛擬機轉換爲快照，鏡像和快照都存儲在glance中，glance的後端支持多種存儲方式，包括本地的文件系統，http，glusterfs，ceph，swift等等。
python
默認狀況下，glance採用本地文件系統的方式存儲p_w_picpath，存儲的路徑爲/var/lib/glance/p_w_picpaths,隨着時間的推移，當鏡像愈來愈多的時候，根目錄的空間將會愈來愈大，因此對於glance的路徑來講，須要提早作好規劃和準備，如劃分一個單獨的空間存儲p_w_picpath，或者存放在分佈式的文件系統，如ceph，swift上等。我所在的環境中，剛上線的時候，因爲缺少對glance的規劃，採用默認的路徑/var/lib/glance/p_w_picpaths，後來由於空間的不夠的問題，而採起更改路徑，在更改的過程當中，引起了"血案".
mysql
2. 血案現場sql
建立一臺虛擬機數據庫
#獲取鏡像id [root@controller ~]# glance p_w_picpath-list +--------------------------------------+---------------+-------------+------------------+-------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------+-------------+------------------+-------------+--------+ | 37aaedc7-6fe6-4fc8-b110-408d166b8e51 | cirrors | qcow2 | bare | 13200896 | active | #獲取網絡的id號 [root@controller ~]# neutron net-list +--------------------------------------+---------------+-------------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------------+-------------------------------------------------------+ | 99c68a93-336a-4605-aa78-343d41ca1206 | vmTest | 79cb82a1-eac1-4311-8e6d-badcabd22e44 192.168.100.0/24 | +--------------------------------------+---------------+-------------------------------------------------------+ #獲取flavor的id號碼 [root@controller ~]# nova flavor-list +--------------------------------------+------------------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +--------------------------------------+------------------+-----------+------+-----------+------+-------+-------------+-----------+ | 1 | m1.large | 8192 | 100 | 10 | | 4 | 1.0 | True |
2. 建立instanceswift
[root@controller ~]# nova boot --flavor m1.large --p_w_picpath 37aaedc7-6fe6-4fc8-b110-408d166b8e51 --nic net-id=99c68a93-336a-4605-aa78-343d41ca1206 glance_p_w_picpath_error_test
+--------------------------------------+------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-000001ff |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | X39vzn4RKwrL |
| config_drive | |
| created | 2016-01-27T11:14:46Z |
| flavor | m1.large (1) |
| hostId | |
| id | b143fd7d-b1b7-49b4-ba20-7968777460bc |
| p_w_picpath | cirrors (37aaedc7-6fe6-4fc8-b110-408d166b8e51) |
| key_name | - |
| metadata | {} |
| name | glance_p_w_picpath_error_test |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 842ab3268a2c47e6a4b0d8774de805ae |
| updated | 2016-01-27T11:14:46Z |
| user_id | bc5e46fc4204497185ae3ca6f8b7affb |
+--------------------------------------+------------------------------------------------+
3. 建立失敗後端
[root@controller ~]# nova list |grep b143fd7d-b1b7-49b4-ba20-7968777460bc | b143fd7d-b1b7-49b4-ba20-7968777460bc | glance_p_w_picpath_error_test | ERROR | - | NOSTATE | | ChuangYiYuan_10_16_2_21 |
3.尋根究底api
查看glance日誌，包括glance-api和glance-registry緩存
[root@controller ~]# tail -n 2 /var/log/glance/api.log 2016-01-27 19:15:22.917 2664 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): controller 2016-01-27 19:15:22.948 2664 INFO glance.wsgi.server [89d3f8c3-9d66-4d75-b88c-eafe746f9a6b bc5e46fc4204497185ae3ca6f8b7affb 842ab3268a2c47e6a4b0d8774de805ae - - -] 10.16.2.8 - - [27/Jan/2016 19:15:22] "HEAD /v1/p_w_picpaths/37aaedc7-6fe6-4fc8-b110-408d166b8e51 HTTP/1.1" 200 856 0.031628 [root@controller ~]# tail -n 2 /var/log/glance/registry.log 2016-01-27 19:15:22.946 2763 INFO glance.registry.api.v1.p_w_picpaths [cca31ae2-f412-4605-a5db-0cc0a507955b bc5e46fc4204497185ae3ca6f8b7affb 842ab3268a2c47e6a4b0d8774de805ae - - -] Successfully retrieved p_w_picpath 37aaedc7-6fe6-4fc8-b110-408d166b8e51 2016-01-27 19:15:22.946 2763 INFO glance.wsgi.server [cca31ae2-f412-4605-a5db-0cc0a507955b bc5e46fc4204497185ae3ca6f8b7affb 842ab3268a2c47e6a4b0d8774de805ae - - -] 127.0.0.1 - - [27/Jan/2016 19:15:22] "GET /p_w_picpaths/37aaedc7-6fe6-4fc8-b110-408d166b8e51 HTTP/1.1" 200 847 0.017350 #!!未發現有異常!!
2. 查看nova的日誌，包括nova-api,nova-scheduler,nova-conductor和nova-compute節點日誌網絡
2016-01-09 17:42:09.653 2872 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 9.578928 sec 2016-01-09 17:47:25.755 2872 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 5.842983 sec 2016-01-27 19:14:49.762 2872 ERROR nova.scheduler.filter_scheduler [req-46235a89-6ed4-47e5-ac06-85f6dedc8985 bc5e46fc4204497185ae3ca6f8b7affb 842ab3268a2c47e6a4b0d8774de805ae] [instance: b143fd7d-b1b7-49b4-ba20-7968777460bc] Error from last host: ChuangYiYuan_10_16_2_22 (node ChuangYiYuan_10_16_2_22): [u'Traceback (most recent call last):\n', u' Fil e "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1328, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.6/site-packages/nov a/compute/manager.py", line 393, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager. py", line 1740, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.p y", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1737, in _spawn\n bl ock_device_info)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2287, in spawn\n admin_pass=admin_password)\n', u' File "/usr/lib/python2 .6/site-packages/nova/virt/libvirt/driver.py", line 2656, in _create_p_w_picpath\n project_id=instance[\'project_id\'])\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/li bvirt/p_w_picpathbackend.py", line 192, in cache\n *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/p_w_picpathbackend.py", line 383, in create_p_w_picpath\n prepare_template(target=base, max_size=size, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n ret urn f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/p_w_picpathbackend.py", line 182, in fetch_func_sync\n fetch_func(target=target, *args, **k wargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 653, in fetch_p_w_picpath\n max_size=max_size)\n', u' File "/usr/lib/python2.6/site-packag es/nova/virt/p_w_picpaths.py", line 78, in fetch_to_raw\n max_size=max_size)\n', u' File "/usr/lib/python2.6/site-packages/nova/virt/p_w_picpaths.py", line 72, in fetch\n p_w_picpath_serv ice.download(context, p_w_picpath_id, dst_path=path)\n', u' File "/usr/lib/python2.6/site-packages/nova/p_w_picpath/glance.py", line 331, in download\n _reraise_translated_p_w_picpath_except ion(p_w_picpath_id)\n', u' File "/usr/lib/python2.6/site-packages/nova/p_w_picpath/glance.py", line 329, in download\n p_w_picpath_chunks = self._client.call(context, 1, \'data\', p_w_picpath_id)\ n', u' File "/usr/lib/python2.6/site-packages/nova/p_w_picpath/glance.py", line 209, in call\n return getattr(client.p_w_picpaths, method)(*args, **kwargs)\n', u' File "/usr/lib/pytho n2.6/site-packages/glanceclient/v1/p_w_picpaths.py", line 127, in data\n % urllib.quote(str(p_w_picpath_id)))\n', u' File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py" , line 289, in raw_request\n return self._http_request(url, method, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 249, in _http _request\n raise exc.from_response(resp, body_str)\n', u'ImageNotFound: Image 37aaedc7-6fe6-4fc8-b110-408d166b8e51 could not be found.\n'] #在nova-scheduler和nova-compute的日誌中查看到"ImageNotFound: Image 37aaedc7-6fe6-4fc8-b110-408d166b8e51 could not be found"的報錯信息!
3.查看glance的服務狀態
[root@controller ~]# /etc/init.d/openstack-glance-api status openstack-glance-api (pid 2222) is running... [root@controller ~]# /etc/init.d/openstack-glance-registry status openstack-glance-registry (pid 2694) is running... #狀態正常 [root@controller ~]# glance p_w_picpath-list +--------------------------------------+---------------+-------------+------------------+-------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------+-------------+------------------+-------------+--------+ | 37aaedc7-6fe6-4fc8-b110-408d166b8e51 | cirrors | qcow2 | bare | 13200896 | active | +--------------------------------------+---------------+-------------+------------------+-------------+--------+ #正常工做，嘗試upload一個鏡像，也可以正常工做，緣由何在呢？？
4.抓住元兇
由於在運維過程當中，修改過glance的默認路徑由/var/lib/glance/p_w_picpaths修改成/data1/glance,而且將/var/lib/glance/p_w_picpaths下的鏡像都mv至/data1/glance下了，而此時儘管數據已經前已過去了，可是p_w_picpath的元數據信息卻緊緊的記錄在glance的p_w_picpath_locations表中，查看得知:
mysql> select * from glance.p_w_picpath_locations where p_w_picpath_id='37aaedc7-6fe6-4fc8-b110-408d166b8e51'\G;
*************************** 1. row ***************************
id: 37
p_w_picpath_id: 37aaedc7-6fe6-4fc8-b110-408d166b8e51
value: file:///var/lib/glance/p_w_picpaths/37aaedc7-6fe6-4fc8-b110-408d166b8e51 #元兇
created_at: 2015-12-21 06:10:24
updated_at: 2015-12-21 06:10:24
deleted_at: NULL
deleted: 0
meta_data: {}
status: active
1 row in set (0.00 sec)
真像:原來原有目錄/var/lib/glance/p_w_picpaths目錄下的鏡像都已經mv至/data1/glance下，而數據庫中卻依舊記錄着就的路徑內容，從而，衍生的一個問題：當nova嘗試啓動一臺instance的時候，nova會到instance鏡像緩存路徑，默認/var/lib/nova/_base下查找是否有該鏡像，若是沒有則向glance發起result api請求，請求下載指定p_w_picpath的鏡像到本地，glance則根據數據庫中p_w_picpath_locations所定義的值去查找鏡像，從而致使失敗！
解決方法:更新glance的元數據信息
mysql> update glance.p_w_picpath_locations set value='file:///data1/glance/37aaedc7-6fe6-4fc8-b110-408d166b8e51' where p_w_picpath_id='37aaedc7-6fe6-4fc8-b110-408d166b8e51'\G; Query OK, 1 row affected (0.05 sec) Rows matched: 1 Changed: 1 Warnings: 0 #重建虛擬機，故障解決！！！
5. 進一步探索
glance中，主要有兩張表很重要:p_w_picpaths和p_w_picpath_locations,其中p_w_picpath負責存儲鏡像相關的信息，而p_w_picpath_locations記錄鏡像的存儲url路徑。
p_w_picpaths數據表
mysql> select * from glance.p_w_picpaths limit 2\G; *************************** 1. row *************************** id: 0267dcbf-9f72-4ce8-9976-7106e38ee948 name: cirror1 size: 6899532 status: deleted is_public: 1 created_at: 2015-12-02 01:45:13 updated_at: 2015-12-02 01:46:41 deleted_at: 2015-12-02 01:46:41 deleted: 1 disk_format: qcow2 container_format: bare checksum: 7c607794659403b970a5d0a00fb2c311 owner: 842ab3268a2c47e6a4b0d8774de805ae min_disk: 0 min_ram: 0 protected: 0 virtual_size: NULL *************************** 2. row *************************** id: 2437cede-d03a-4680-b704-6d27c4d7198e name: test1 size: 0 status: deleted is_public: 0 created_at: 2015-12-21 09:02:41 updated_at: 2015-12-21 09:06:02 deleted_at: 2015-12-21 09:06:02 deleted: 1 disk_format: qcow2 container_format: bare checksum: d41d8cd98f00b204e9800998ecf8427e owner: 842ab3268a2c47e6a4b0d8774de805ae min_disk: 0 min_ram: 0 protected: 0 virtual_size: NULL 2 rows in set (0.00 sec) #即記錄着建立時候相關信息，還記得deleted字段的做用麼？哈哈，刪除鏡像的原理？？額
2. p_w_picpath_locations表
mysql> select * from p_w_picpath_locations;
+----+--------------------------------------+--------------------------------------------------------------------+---------------------+---------------------+---------------------+---------+-----------+--------+
| id | p_w_picpath_id | value | created_at | updated_at | deleted_at | deleted | meta_data | status |
+----+--------------------------------------+--------------------------------------------------------------------+---------------------+---------------------+---------------------+---------+-----------+--------+
| 1 | 437d860f-1c9f-4bb2-a3ca-8ec062441909 | file:///var/lib/glance/p_w_picpaths/437d860f-1c9f-4bb2-a3ca-8ec062441909 | 2015-06-24 10:40:39 | 2015-12-01 11:52:20 | 2015-12-01 11:52:20 | 1 | {} | active |
| 2 | 5ce414b0-660a-46e1-ad0a-b842b2afc0b7 | file:///var/lib/glance/p_w_picpaths/5ce414b0-660a-46e1-ad0a-b842b2afc0b7 | 2015-06-25 02:49:33 | 2015-06-25 02:49:33 | NULL
6. 附錄
p_w_picpaths表的結構:
mysql> desc glance.p_w_picpaths; +------------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------------+--------------+------+-----+---------+-------+ | id | varchar(36) | NO | PRI | NULL | | | name | varchar(255) | YES | | NULL | | | size | bigint(20) | YES | | NULL | | | status | varchar(30) | NO | | NULL | | | is_public | tinyint(1) | NO | MUL | NULL | | | created_at | datetime | NO | | NULL | | | updated_at | datetime | YES | | NULL | | | deleted_at | datetime | YES | | NULL | | | deleted | tinyint(1) | NO | MUL | NULL | | | disk_format | varchar(20) | YES | | NULL | | | container_format | varchar(20) | YES | | NULL | | | checksum | varchar(32) | YES | MUL | NULL | | | owner | varchar(255) | YES | MUL | NULL | | | min_disk | int(11) | NO | | NULL | | | min_ram | int(11) | NO | | NULL | | | protected | tinyint(1) | YES | | NULL | | | virtual_size | bigint(20) | YES | | NULL | | +------------------+--------------+------+-----+---------+-------+ 17 rows in set (0.00 sec)
2. p_w_picpath_locations表結構
mysql> desc p_w_picpath_locations; +------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+-------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | p_w_picpath_id | varchar(36) | NO | MUL | NULL | | | value | text | NO | | NULL | | | created_at | datetime | NO | | NULL | | | updated_at | datetime | YES | | NULL | | | deleted_at | datetime | YES | | NULL | | | deleted | tinyint(1) | NO | MUL | NULL | | | meta_data | text | YES | | NULL | | | status | varchar(30) | NO | | active | | +------------+-------------+------+-----+---------+----------------+ 9 rows in set (0.00 sec)
相關文章
- 1. openstack運維實戰系列(十四)之neutron agent重啓引起「血案」
- 2. openstack運維實戰系列(十七)之glance與ceph結合
- 3. openstack運維實戰系列(八)之glance鏡像上傳
- 4. openstack運維實戰系列(十五)之nova虛擬機目錄變動引起"血案"
- 5. 一個路徑引起的血案require('')
- 6. mybatis事務路徑引起的「血案」
- 7. openstack運維實戰系列(十一)之neutron替換instance的IP
- 8. openstack運維實戰系列(十九)之cinder與ceph結合
- 9. openstack運維實戰系列(十二)之nova aggregate資源分組
- 10. openstack運維實戰系列(十六)之ceph存儲安裝
- 更多相關文章...
- • Eclipse Java 構建路徑 - Eclipse 教程
- • Spring實例化Bean的三種方法 - Spring教程
- • Docker容器實戰(七) - 容器眼光下的文件系統
- • Java Agent入門實戰（三）-JVM Attach原理與使用
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. openstack運維實戰系列(十四)之neutron agent重啓引起「血案」
- 2. openstack運維實戰系列(十七)之glance與ceph結合
- 3. openstack運維實戰系列(八)之glance鏡像上傳
- 4. openstack運維實戰系列(十五)之nova虛擬機目錄變動引起"血案"
- 5. 一個路徑引起的血案require('')
- 6. mybatis事務路徑引起的「血案」
- 7. openstack運維實戰系列(十一)之neutron替換instance的IP
- 8. openstack運維實戰系列(十九)之cinder與ceph結合
- 9. openstack運維實戰系列(十二)之nova aggregate資源分組
- 10. openstack運維實戰系列(十六)之ceph存儲安裝