RHEV VM has been paused due to lack of storage spa

RHEV VM has been paused due to lack of storage space after RHEV-H upgrade to version 20140112.0.el6

問題

環境
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140112.0.el6)
vdsm 4.13.2-0.6

問題
VM has been paused due to lack of storage space.
VM has been paused due to no Storage space error.
Error message received in audit_log:

2014-02-07 13:39:42.781 | Failed to start VM vmname, because exist snapshot for stateless state. Snapshot will be deleted.  | c1102bb

Since upgrading to RHEVH-6.5 v.20140112.0.el6 /var/log/messages are being flooded by these messages:
因爲兩個rhevh升級至6.5 v.20140112.0.el6 /是/日誌/信息是被flooded這些信息：

Jan 29 05:17:39 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
#012Traceback (most recent call last):#012  
File "/usr/share/vdsm/sampling.py", line 351, in collect#012  
File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  
File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  
File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012  
File "/usr/share/vdsm/vm.py", line 842, in f#012  
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012  
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

/var/log/vdsm/vdsm.log contains these logs:
/var/log/vdsm/vdsm.log包含如下日誌：

Thread-5949::DEBUG::2014-01-29 05:14:33,027::sampling::285::vm.Vm::(start) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Start statistics collection
Thread-5969::DEBUG::2014-01-29 05:14:33,027::sampling::314::vm.Vm::(run) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats thread started
Thread-5969::DEBUG::2014-01-29 05:14:33,033::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 8 edom: 10 level: 2 message: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
Thread-5969::ERROR::2014-01-29 05:14:33,034::sampling::355::vm.Vm::(collect) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
Traceback (most recent call last):
  File "/usr/share/vdsm/sampling.py", line 351, in collect
  File "/usr/share/vdsm/sampling.py", line 226, in __call__
  File "/usr/share/vdsm/vm.py", line 529, in _highWrite
  File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded
  File "/usr/share/vdsm/vm.py", line 842, in f
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

決議

If this error is received, an error relating to the VM being paused due to lack of storage space. For information on the advisory, and where to find the updated files, follow the link: vdsm 3.3.0-3 bug fix update
若是收到此錯誤，則與因爲缺乏存儲空間而暫停的虛擬機相關的錯誤。有關建議的信息以及在何處查找更新的文件，請訪問如下連接：vdsm 3.3.0-3 bug fix update

For more troubleshooting, for upgrades and roll-back issues, see the following guide: RHEV 3.2 to 3.3 Upgrade Guide: Tips, Considerations and Roll-back Issues
有關更多故障排除，有關升級和回滾問題，請參閱如下指南：RHEV 3.2到3.3升級指南：提示、注意事項和回滾問題

根源

This seems to occur if a VM is migrated to a hypervisor running vdsm 4.13.2-0.6 from a hypervisor running an earlier version of vdsm.
若是從運行早期版本vdsm的管理程序遷移到運行vdsm 4.13.2-0.6的管理程序，則彷佛會發生這種狀況。

After the VM is migrated libvirt and vdsm will end up having different path information about the VMs disks.
在遷移VM以後，libvirt和vdsm最終將擁有關於vms磁盤的不一樣路徑信息。

During the ExtendDrivesIfNeeded method vdsm asks libvirt for block info about the the disk and passes in a path to the disk, however libvirt has a different path to the disk and reports the invalid path error.
Note that if the VM is started on the vdsm 4.13.2-0.6 hypervisor then the error doesn't occur.
在extenddrivesifrequired方法期間，vdsm要求libvirt提供關於磁盤的塊信息，並將路徑傳遞到磁盤，可是libvirt有不一樣的磁盤路徑，並報告無效的路徑錯誤。
注意，若是VM是在vdsm 4.13.2-0.6 hypervisor上啓動的，則不會發生錯誤。

診斷步驟

After migrating the VM to hypervisor running vdsm 4.13.2-0.6, one can see it maintains path information in the format /rhev/data-center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid], like so:
將VM遷移到運行vdsm 4.13.2-0.6的hypervisor後，能夠看到它以/rhev/data center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid]的格式維護路徑信息，以下:

# vdsClient -s 0 list

ed1061e9-a91d-45f2-a62e-e90a57bcd32a
    Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...

But looking at the info libvirt maintains about the VM, the path is specified as /rhev/data-center/[spuuid]/[sduuid]/images/[imguuid]/[voluuid], like so:
但查看libvirt維護的有關虛擬機的信息，路徑被指定爲/rhev/data center/[spuid]/[sduuid]/images/[imguuid]/[voluuid]，以下所示：

# virsh -r domblklist testvm
Target     Source
------------------------------------------------
hdc        -
vda        /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5

And then the /var/log/messages is flooded with these errors:
而後' /var/log/messages '被這些錯誤淹沒:

Jan 29 07:15:36 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012  File "/usr/share/vdsm/vm.py", line 842, in f#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

But if the VM is started directly on the vdsm 4.13.2-0.6 hypervisor then vsdm and libvirt maintain the same path information:
可是，若是虛擬機直接在vdsm 4.13.2-0.6管理程序上啓動，那麼vsdm和libvirt會維護相同的路徑信息：

# vdsClient -s 0 list

ed1061e9-a91d-45f2-a62e-e90a57bcd32a
    Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...

# virsh -r domblklist testvm
Target     Source
------------------------------------------------
hdc        -
vda        /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5