Problem with autostart of VM's, hosted on iscsi server, at dom0 reboot

Submitted by fix on Tue, 09/03/2013 - 04:57

Hello

I'm still testing my iscsi-server-cloudmin-xen installation and now have another problem. When I install new VMs through cloudmin on a XEN host which store is hosted on LVM on a iscsi server only the last created VM will restart at a reboot of the xen-host. Although the symbolic links for the .cfg files from /xen/vm1.cfg to /etc/xen/auto are created properly only the last created VM will start. If I afterwards tell cloudmin to boot the VM the iscsi connection is created and the VM boots.

I think it could something to do with my iscsi-target settings, but I tried different authentication settings and none of them worked. It seems the xen-host can only reconnect one of the iscsi connections at bootup.

Any help would be kindly appreciated.

Kind regards

Fix

Status: 
Active

Comments

Submitted by JamieCameron on Tue, 09/03/2013 - 16:36

Do you have access to the host system console in order to see the error messages from the Xen startup script? That may reveal why it is failing to start.

You may want to look into changing the boot order so that starting Xen VMs happens after making iSCSI connections.

Submitted by fix on Tue, 09/03/2013 - 16:54

Hi Jamie

Yes, I have access to the xen-hosts console. I will do some testing and have a look at the console output while I reboot the server.

About the boot order. You mean I should try to first start the iscsi-client and then start the VM's. For this I should know which task does start the xen-vm's. Could you give me a tip on these tasks? But against a problem with the boot order is, that the server is able to boot the last created VM.

Regards

Fix

Submitted by JamieCameron on Tue, 09/03/2013 - 17:49

I think the boot action that starts Xen VMs is called xendomains

Submitted by fix on Tue, 09/03/2013 - 19:11

Hi Jamie

I'm sorry, but I'm a little bit confused right now (here it's 1:45am now ;-). But I guess all of my actual problems are based on the iscsi "Authentication Options" in Webmin.

The Console on either the xen-host as well as the iscsi-host is full of authentication problems. 'Xendomains' only reports "Cannot create domain xy". But iscsid reports "Login failed to authenticate with target..."

After had a look into some different files and logs involved, I think some files have probably wrong login information in it. The main problem seems to be in the default "Authentication Options" in the iscsi-target and iscsi-client Webmin-Modul of the iscsi-server and the xen-hosts. During my testing with cloudmin and iscsi I never was sure how to set them correctly. So after trying different settings I ended up to setup the iscsi-host and the xen-hosts to use 'chap' and fix setted username/login for either "Authentication method" as well as "Discovery authentication method". With this setting I was able to create and boot the vm's correctly, but now it seems that exactly those login settings prevent the xen-host to restore all vm's on bootup.

Can you probably tell me which settings you recommend to use als default login settings (for iscsi-target and iscsi-client in Webmin) and I clean out all my vm's again to have another try then. As noted before I need a brake. I can get you further logs or screenshots tomorrow.

Thank you for your help & regards

Fix

Submitted by JamieCameron on Tue, 09/03/2013 - 23:31

For authentication, you shouldn't need to change anything - Cloudmin will for each VM create an iSCSI user with permissions to access just that VM's disks, and configure the client host systems to login as that user. In fact, changing the username or password for an iSCSI export after creating a VM will probably break it.

Perhaps try creating a new VM on iSCSI, verify that it starts up and its disks work properly, and then see if it gets re-starting on reboot.

Submitted by fix on Wed, 09/04/2013 - 01:20

Hello Jamie

Cloudmin will for each VM create an iSCSI user with permissions to access just that VM's disks

I know that. What I do not know ist where on the client this information is stored. Is this somewhere in /var/lib/iscsi?

I newer had authentication Problems when a vm is created, startet or moved out of cloudmin. Only while live-migrate or reboot.

Now I went back to start, cleaned out all testmachines on this server and created 2 new ones. These Machines are bootable from cloudmin, but if I reboot the xen-host still only one iscsi-connection ist restored. If I go to cloudmin and tell the vm to boot the iscsi connection is created. Where should I further check? Where exactly should the iscsi logins be stored on the client?

Regards

Fix

EDIT: ISCSI Client and ISCSI Target are both now set to "No authentication needed" now. No change!

Submitted by fix on Wed, 09/04/2013 - 11:05

Hi Jamie

Found something interesting. I know now about the node-files in /var/lib/iscsi/nodes where (IMHO) the iscsi client get's it's nodes from to connect at bootup.

When I create a VM on the xen-host there is a file created with all the needed data for the iscsi client. In this file also are the login data for the share. When I create a second machine now, this node-file for the first VM changes and after the creation of the second VM there aren't any login informations in the node-file anymore. Please compare those two attached textfiles. One is a copy after creation of the first vm, the second file ist a copy of the same file after creating a second vm.

If I change back this file after the creation of the second VM and do a reboot of the xen-host both vm start correctly at bootup.

Any clou how this can happen? I'm running Centos 5.9 with ietd as iscsi-target.

Regards

Fix

Submitted by JamieCameron on Wed, 09/04/2013 - 18:12

Interesting, the lack of authentication information in that file could certainly cause problems.

However, Cloudmin/Webmin doesn't actually update that file directly - instead it uses the iscsiadm command to make new connections on the client side.

On your iSCSI server, do the targets for both your VMs have usernames and passwords set? And if so, are they the same?

Submitted by fix on Thu, 09/05/2013 - 00:28

Yes, both targets have different User/PW set.

When I made some testing with iscsiadm yesterday I found out, that when I run 'iscsiadm -m discovery -t sendtargets -p ServerIP,Port' on the Client iscsiadm doesn't read out the login information.

But if I set a fixed discovery User/PW on the iscsi-server (through cloudmin GUI) and set this password also on all the targets it worked. I think I have a Problem between the iscsiadm and ietd. Will change back from ietd to iscsid as target and check if it works then. Keep you updated.

Submitted by Franco Nogarin on Fri, 09/20/2013 - 15:25 Pro Licensee

Just a suggestion, I was seeing similar oddness until I edited the target and client module config and found that some commands listed as "command" needed to be "/usr/sbin/command" so that they would actually get used by cloudmin. after doing this on each and every host and customizing it to the actual locations my iSCSI would not work as expected. Its very hard for cloudmins iSCSI config to work on ANY distribution let alone most, so I would make sure that all of it is correct (in my case both target and client needed more than one modification) Hope this helps.

Submitted by fix on Sat, 09/28/2013 - 15:01

Hello Franco

Thanks for your suggestion. Double-checked all my paths and settings. They're seems to be ok. All commands are correct entered with /sbin/...

Thank you & regards

Fix