Hi All,
I'm a longtime user of webmin and a recent user of cloudmin. I have purchased a cloudmin pro license and I am trying to setup KVM hosts using an LVM ISCSI target as storage. The end result (hopefully) is to setup a host failover group that shares storage. If I understand this correctly, cloudmin will create a new logical volume on the volume group shared by the ISCSI server when I create a new VM, share that logical volume via ISCSI and deploy the VM. My setup is as follows:
I have three servers named host1, host2 and store. They are all interconnected on a private network and each has a public interface. I want to share an LVM (called storage) over ISCSI. I have setup the iscsitarget server by following the instructions here: http://www.virtualmin.com/documentation/cloudmin/iscsi, however, i keep getting the following error when trying to make a new KVM machine:
Copying 437.79 MB image file to host system host2.example.com .. .. already in host cache
Creating virtual system with KVM .. .. creation failed : iSCSI disk creation failed : iSCSI server is not running on store.example.com
I have checked the config files and everything is enabled/working. I have tested the system by manually mounting a shared logical volume (called test). The setup works manually but gives the above error when creating a system from cloudmin.
After much googling, I decided to ask because the documentation on the subject (with reference to cloudmin) is sparse. Another thought I had after looking at the cloudmin ISCSI documentation page was that it was looking for the wrong type of ISCSI server (ie. ISCSI Server & ISCSI Target modules in webmin). I might be wrong here as I have no previous experience with ISCSI.
Any help would be greatly appreciated. Thanks.
Bump +1 same issue here, did you ever solve it, I installed everything, set it up in webmin, but still cant put anything on the iscsi host/target with same error message
If you login to Webmin on the iSCSI server system and go to Hardware -> iSCSI Target (on Debian / Ubuntu) or iSCSI Server (on CentOS / Redhat) , does it show that the iscsi server is running?
If not, I would be interested to know which Linux distribution and version you are running on the server system.
''
Hi Jaimie,
No it does not indicate that it is running, but when i check via SSH it says its running, see both attachments.
It looks like Webmin is looking in the wrong place for the iscsitarget server PID file. The expected path is
/var/run/iscsi_trgt.pid, which is configured in/etc/webmin/iscsi-target/config. You should check where the PID file is actually located on your system.''
All my HOST systems are Ubuntu Server 64 12.04.3 LTS
3 are running kernel: 3.2.0-52-generic #78-Ubuntu SMP Fri Jul 26 16:21:44 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
1 (SMVMCLOUD) is running 3.5.0-39-generic #60~precise1-Ubuntu SMP Wed Aug 14 15:38:41 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
here is what I did on smvmcloud (my soon to be shared iscsi share)
sudo apt-get update;sudo apt-get upgrade
sudo apt-get update;sudo apt-get dist-upgrade
sudo reboot # to ensure i am on the latest kernel
sudo apt-get install open-iscsi iscsitarget iscsitarget-dkms
in the webmin panel for this server I refreshed the modules so I could see
iscsi client
iscsi target
I them configured each module to indicate the correct paths for binaries:
eg: client
/sbin/iscsiadm
/sbin/iscsi-iname
/etc/iscsi/ifaces
target
/usr/sbin/ietadm
i then ensured that he services were running by entering a shell and running
sudo service iscsitarget status
response was:
* iSCSI enterprise target is running
then:
sudo service iscsi-network-interface status
response:
iscsi-network-interface start/running
then: sudo service open-iscsi status
response:
* iscsid is running
* Current active iSCSI sessions:
iscsiadm: No active sessions.
next in my cloudmin master, i choose "system storage"> "iscsi servers"
my chosen server SMVMCLOUD was already listed so I opened it and saved it accepting the LVM group listed.
Next I went to my cloumin master again and chose HOST SYSTEMS -> KVM HOST SYSTEMS
I opened my cloudmin master SMVMHOST and set Create disk images in: to Devices on iSCSI server
, then chose SMVMCLOUD as my iScsi server, and hit save.
Next I created a new VM (empty system from ubuntu server install cd) on SMVMHOST (hoping to have its drive on the iSCSI server)
for KVM hosting system chose: SMVMHOST
for Location for disk image i chose: Host system default
2gm ram, 10gm disk, default network and hit create system
it copies the install CD to the HOST, then
when it got to the create harddisk stage:
Copying 638.66 MB CD image to system smvmhost ..
.. already in host cache
Creating virtual system with KVM ..
.. creation failed : iSCSI disk creation failed : iSCSI server is not running on smvmcloud
The next test was to use the new move hardisk feature and I grbbed a logical volume for a VM that was shutdown hosted on SMVMHOST and tried to move it using new feature Move Disk:
From inside the Edit Disk page
Move Disk to: Storage location: iSCSI Server smvmcloud and clicked "Move Now"
Moving disk for /dev/vda to storage iSCSI server smvmcloud ..
.. move failed : Failed to create new disk : iSCSI server is not running on smvmcloud
Next test: I entered webmin on SMVMCLOUD -> Hardware-> iSCSI Client->iSCSI connections
got:
No iSCSI connections are currently active.
Tried to connect to SMVMCLOUD and it said:
Failed to add iSCSI connection : No iSCSI targets were found on smvmcloud
Next I went to iSCSI Target and setup a target-> Add a new iSCSI target.
set target name to: target_test (iqn.2013-08.smvmcloud:target_test)
Device to share 1: Logical volume (chose one I had made for this 400GB formatted ext4)
chose block based and hit create
restarted iscsitarget from shell
back to iSCSI Client connections to connect to SMVMCLOUD
iSCSI server hostname or IP: smvmcloud
iSCSI server port: Default
iSCSI interface: Choose automatically
Connect to target: target_test on 192.168.80.236
Login to server: Use default authentication method
and got: Failed to add iSCSI connection : iscsiadm: no records found!
tried again using IP instead of hostname smvmcloud and it worked.
I can see the connection in the list and syslog shows:
Aug 27 15:31:28 smvmcloud kernel: [ 3641.858449] iSCSI Enterprise Target Software - version 1.4.20.3
Aug 27 15:31:28 smvmcloud kernel: [ 3641.858499] iscsi_trgt: Registered io type fileio
Aug 27 15:31:28 smvmcloud kernel: [ 3641.858502] iscsi_trgt: Registered io type blockio
Aug 27 15:31:28 smvmcloud kernel: [ 3641.858504] iscsi_trgt: Registered io type nullio
Aug 27 15:34:32 smvmcloud kernel: [ 3825.540333] scsi4 : iSCSI Initiator over TCP/IP
Aug 27 15:34:33 smvmcloud kernel: [ 3825.798288] scsi 4:0:0:1: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Aug 27 15:34:33 smvmcloud kernel: [ 3825.798709] sd 4:0:0:1: Attached scsi generic sg3 type 0
Aug 27 15:34:33 smvmcloud kernel: [ 3825.799147] sd 4:0:0:1: [sdc] 838860800 512-byte logical blocks: (429 GB/400 GiB)
Aug 27 15:34:33 smvmcloud kernel: [ 3825.799314] sd 4:0:0:1: [sdc] Write Protect is off
Aug 27 15:34:33 smvmcloud kernel: [ 3825.799319] sd 4:0:0:1: [sdc] Mode Sense: 77 00 00 08
Aug 27 15:34:33 smvmcloud kernel: [ 3825.799707] sd 4:0:0:1: [sdc] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
Aug 27 15:34:33 smvmcloud kernel: [ 3825.801312] sdc: unknown partition table
Aug 27 15:34:33 smvmcloud kernel: [ 3825.801791] sd 4:0:0:1: [sdc] Attached SCSI disk
Aug 27 15:34:33 smvmcloud kernel: [ 3825.852648] iscsi_trgt: scsi_cmnd_start(1059) Unsupported 85
Aug 27 15:34:33 smvmcloud kernel: [ 3825.853050] iscsi_trgt: cmnd_skip_pdu(472) 74000000 1c 85 0
Aug 27 15:34:33 smvmcloud kernel: [ 3825.853790] iscsi_trgt: scsi_cmnd_start(1059) Unsupported 85
Aug 27 15:34:33 smvmcloud kernel: [ 3825.854184] iscsi_trgt: cmnd_skip_pdu(472) 75000000 1c 85 0
Aug 27 15:34:33 smvmcloud iscsid: connection2:0 is operational now
tried the move again:
Moving disk for /dev/vda to storage iSCSI server smvmcloud ..
.. move failed : Failed to create new disk : iSCSI server is not running on smvmcloud
So from what I can tell I have the client and server working but not the cloudmin components. Any help at this point would be super....
Franco
AHA!!!! recent version of iscsitarget on ubuntu DO NOT USE the old pidfile, as that was a part of a bug, it now looks for and uses /var/run/ietd.pid
I will reassess everything with this new information, I set this in the module config and immediately got buttons and options i never had before. Good call and thanks. I will post back my working steps later today!
Franco
Ok, great - I will make Webmin look for that PID file as well in the next release. Till then, you can easily change the path Webmin uses by editing
/etc/webmin/iscsi-target/config.''
Hmm, this is still proving troublesome,
there is no way to attach an existing iSCSI target as a drive in the Cloumin Manage Disks GUI, only to create one. From the comman line there is vague references in the modify-disk docs but no example of how to attch an iSCSI drive to a vm from command line either.
I have done it the hard way (using webmin, create a target on server, using webmin create a connection on client, then mount the resulting local device file, but this is not cloudmin aware/compatible.
Franco
i discovered that adding the device by path eg: /dev/disk/by-path/ip-192.168.80.236:3260-iscsi-iqn.2013-08.smvmcloud:sparcsgit.sparcsonline.com_0_img-lun-1
in the GUI makes it iSCSI..... :D
Yes, there isn't really a nice UI for this yet, sorry. But I plan to add one in the next release.
Till them, the steps to take are :
/dev/disk/by-path/ip-somethingOnce the disk is added, it should appear exactly like a iSCSI disk that was created by Cloudmin itself.
''
I have been unable to move a disk from lv to iscsi using cloudmin, but I have done it manually.
I am unable to create a VM from a KVM image on iSCSI host
I am unable to create an image of a VM using iSCSI
Sorry Jaime, just trying to document where I am having sucess and failure. I realize that some of this functionality may not yet be ready......
Moving disk for /dev/vda to storage iSCSI server smvmcloud ... .. move failed : Failed to create new disk : LVM logical volume creation failed : Failed to read from fastrpc.cgi :
Make sure that TCP ports 10000 to 10010 are open on the iSCSI server system smvmcloud - failure to do this can cause RPC errors.
''
They are open in the sense that the linux firewall is allowing all traffic on everything and port 10000 appears to be listening according to netsat.
is this a bug? When viewing the VMs in the list, iSCSI hosted drives say unlimited for size, even if they are defined as 15gb?
That does sound like a bug. Can you attach a screenshot, or sent it to me at jcameron@virtualmin.com ?
Also, was this iSCSI drive created entirely within Cloudmin? Or is it an existing drive that was attached?
''
Hi Jaimie, the three iSCSI VMS i have exhibit this behaviour:
if there is a CD attached it shows that size instead of the iSCSI drive. (the top one in my attachment)
If not it shows unlimited. (bottom two in screenshot)
2 of the three were made by creating an empty system using an ISO directly on the iSCSI server host. the third was created by cloning one of the first two.
Franco
If you select one of those problem VMs from the left menu, and then go to Resources -> Manage Disks, does it still show the size as zero or unlimited?
''
Neither, it says 15GB
Franco
This may be due to the total disk size just being incorrectly cached then. If you click on one of these VMs and then click "Refresh fully" on the left menu, does the size now appear correctly in the VM list?
''
No Jamie, it stays the same, and the last VM I built on iSCSI from scratch did not do this (confusing). I think it might be related to cloning or creating from images on iscsi based vms. I will have to run some more tests but I will try cloning the vm thats reporting size correctly and see if that messes it up ;-) Franco
Actually, I just found a bug in Cloudmin that can cause the total disk size display to be incorrect in some cases when using iSCSI. It will be fixed in the next release though.
''
Super. I can confim that cloning a running iSCSI vm without the unlimited issue creates a clone with the unlimited issue.
Changing the HOST to be iSCSI also breaks its normal LVG/LV functions.
Here is a simple clone on a HOST with iSCSI enabled(Not an issue when iSCSI is off and set to LV): Creating clone of KVM system sparcs-fmd.localhost named sparcsfmd2 .. Allocating new IP addresses .. .. allocated 192.168.80.8 Copying virtual disks .. Copying disk /dev/lv_storage/sparcs-fmd_localhost_img .. .. copy failed : "/dev//sparcs-fmd_localhost_img": Invalid path for Logical Volume The origin name should include the volume group. Run `lvcreate --help' for more information. Refreshing status of original system .. .. done. New status is : Down
.. failed : Copy of critical disk failed : "/dev//sparcs-fmd_localhost_img": Invalid path for Logical Volume The origin name should include the volume group. Run `lvcreate --help' for more information.
same job on same host but I set it to LVG instead of iSCSI:
Creating clone of KVM system sparcs-fmd.localhost named sparcsfmd .. Allocating new IP addresses .. .. allocated 192.168.80.7 Copying virtual disks .. Copying disk /dev/lv_storage/sparcs-fmd_localhost_img .. .. snapshotted to /dev/lv_storage/sparcsfmd_sparcsonline_com_img .. done Adding DNS entry sparcsfmd.sparcsonline.com. for IP address 192.168.80.7 .. .. done
Mounting new instance's filesystem .. Refreshing status of original system .. .. done. New status is : Down
Refreshing status of new system .. .. done. New status is : Down
Re-fetching current status of host system smvmbackup64 .. .. done. New status is : Webmin
.. clone sparcsfmd.sparcsonline.com was partially created : No Linux partitions found in disk image /dev/lv_storage/sparcsfmd_sparcsonline_com_img
Sorry for continually adding things here, but it just makes sense to put it all in one place related to iSCSI.
I have been making my VMs like this:
Create Empty System via Ubuntu ISO
Force system on my iSCSI hosts
force vm to use my chosen IP
configure static networking in ubuntu
manual drive config of LVM on iSCSi on LVM
By this last line I mean that the host is serving up an LV via iSCSI and my VM disk is created on that, and then inside the ubuntu installer I am creating a LVG and then an LV which is mounted as /
Is this the ideal approach or is there a better one?
Thanks
Franco
I would recommend not setting up LVM inside the VM - instead, just create a single regular partition that occupies the whole disk, and mount it on /
The reason for this is that Cloudmin cannot access the filesystem within the VM if it is inside an LV. Also, you can expand the VM disk and filesystem using Cloudmin, which negates one of the benefits of using LVM (the ability to expand a filesystem onto another disk).
''
ok, I could not find an exisiting bug for this, but...
on my system I tried to move LVM disk to iSCSI and got this, looks familiar:
Moving disk for
/dev/vdb
to storage iSCSI server smvmcloud .....................
..
move failed :
Failed to create new disk :
iSCSI connection failed :
Failed to read from fastrpc.cgi :
And of course it actually did make the LV on iSCSI and even created the target and connection too.
but the disk move did not complete (I am finishing it manually using dd)
Franco
Does the system that is being moved have multiple disks?
''
no this one only had one 11 GB LV on the cloudmin master which is I think is currently configured for for iSCSI (I cant be sure since I cannot get in as the fiber is down right now).
Confirmed, this VM only has a single LV disk and the host (Cloudmin master) SMVMHOST is set for iSCSI on SMVMCLOUD.
also a subsequent attempt to move it will fail stating that the disk already exists, whereby you must then manually delete the iSCSI connection, target, and the LV itself before you can try again.
Are you still seeing this after I applied the fix to your system for the iSCSI server addition error?
''
yes, sadly I am, but I am all for finding and killing each and every bug we can ;-)
After you fixed the bug, I added the master as an iscsi server so I could move lvms to iscsi on the master.
my latest test was to take zoneminder.virtual a 10GB LVM with a single disk, on the cloudmin master (SMVMHOST) and tried to move the disk from local LVM to local iSCSI and got this almost immediately:
Moving disk for /dev/vda to storage iSCSI server smvmhost .....
..
move failed :
Failed to create new disk :
iSCSI connection failed :
Failed to read from fastrpc.cgi :
when I examined the iSCIS connection via webmin for the master I found a valid connection mounted to a valid target and a new iSCSI lv under that:
Connection:
216.108.146.10 3260 default iqn.2013-09.smvmhost zoneminder.virtual_0_img None /dev/sdd
Target:
iqn.2013-09.smvmhost:zoneminder.virtual_0_img LVM VG lg_storage, LV zoneminder.virtual_0_img 10 GB zoneminder.virtual
Logical Volume
zoneminder.virtual_0_img lg_storage 10 GB
iSCSI shared device iqn.2013-09.smvmhost:zoneminder.virtual_0_img
I have a suspicion as to what could cause this - on your iSCSI server, are disks stored in regular files, or in LVM logical volumes?
''
all my iscsi serve out on LVG yes.
Hi Jamie, We followed the documentation on http://www.virtualmin.com/documentation/cloudmin/iscsi but so far getting the following error. We are using cloudmin pro.
creation failed : iSCSI disk creation failed : iSCSI server is not running on
and we are not able to see the manage disks option under Resources. We have also added the iscsi connection to the server using cloudmins Iscsi client option under Webmin>hardware but so far no success. Please guide us.
your help would be greatly appreciated. thank you
Can you file a support ticket for this? It will be easier to track the issue there than on the forums..
''