Hello
When I try to live-migrate a xen vm through cloudmin the vm looses write access to it's disk (LVM on iscsi). First everything works like expected. Cloudmin does move the .cfg file to the new host, creates the iscsi connection and I can ping and ssh to the VM on the new host after migration. Xentop shows the machine running on the new host. But if I ssh to the vm there's no more write access to the disk. I have to destroy the vm and then I'm able to reboot the vm. Sometimes with, sometimes without the need to first fsck the vm's disk.
If I shutdown the vm before moving, everthing's fine.
BUT: If I have a running vm on host-1 and do a live migration on console directly on the xen host the vm is moved to host-2, still accessible and if I ssh to the moved vm there's still write access to it's disk. I'm not sure yet (would have to test again properly) but I think that I'm only able to live-migrate the vm like this after the new host has an already connected iscsi connection to the iscsi server.
So also here (like in my post some minutes ago) I'm not sure If I have an missconfiguration of the kind the xen-hosts are connecting to the iscsi server. But I do not really know how the iscsi passwords which cloudmin is setting are distributed to the different vm-hosts (if they really are).
Kind regards
Fix
Comments
Submitted by JamieCameron on Tue, 09/03/2013 - 16:52 Comment #1
Could you post the output from the move process as shown in Cloudmin when this happens? The code path uses for live migration when disks are on iSCSI is different from the method used for non-live migration, and I'd like to see what order operations are happening in.
Submitted by fix on Tue, 09/03/2013 - 17:15 Comment #2
Hi Jamie
Here's the output of cloudmin when I live-migrate the vm test0123
Host 1: a.xy.server Host 2: b.xy.server iscsi: i.xy.server
----- Quote moving test0123 from a.xy.server to b.xy.server -------------
test0123 Moving Xen system from a.xy.server to b.xy.server .. Checking if live migration is possible .. .. live migration appears to be possible Adding iSCSI connection to i.xy.server target test0123_1_img on new host system .. .. done
Adding iSCSI connection to i.xy.server target test0123_0_img on new host system .. .. done
Live migrating virtual system .. .. done
Fixing and copying Xen config file /xen/test0123.cfg .. .. copied to /xen/test0123.cfg
Configuring Xen instance for VNC console access .. .. added on dynamic port
Deleting 1 config files from original host .. .. done
Removing iSCSI connection to i.xy.server target test0123_1_img on old host system .. .. done
Removing iSCSI connection to i.xy.server target test0123_0_img on old host system .. .. done
Re-fetching system status .. .. done. New status is : Webmin
Refreshing status of host system a.xy.server .. .. done. New status is : Webmin
Refreshing status of host system b.xy.server .. .. done. New status is : Webmin
.. done
------ End quote moving server ------------
And here's a output of 'xm console test01234' on new host after moving
------ quote output on xm console test01234 after moving ---------
end_request: I/O error, dev sda1, sector 2518800 Buffer I/O error on device sda1, logical block 314850 lost page write due to I/O error on sda1 Aborting journal on device sda1. ext3_abort called. EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only end_request: I/O error, dev sda1, sector 5088 Buffer I/O error on device sda1, logical block 636 lost page write due to I/O error on sda1 journal commit I/O error
------- End quote output on xm console test01234 after moving ---------
Regards
Fix
Submitted by JamieCameron on Tue, 09/03/2013 - 17:46 Comment #3
It looks like maybe even though Cloudmin makes the iSCSI connection on the new system, it isn't actually usable right away.
What gets logged to /var/log/messages on the new host when you do a move?