Cannot remove images from LVM partition if VM contained a swap partition without rebooting the entire cloud server

Submitted by dlynes on Fri, 06/01/2018 - 05:45 Pro Licensee

The title pretty much sums it up.

Create a vm under KVM. It doesn't matter how you do it...whether it's using an image template or create using empty system.

Create two disks in the raw LVM. One will be the data disk the other the swap disk.

Install the OS.

Start the OS up. Make sure the swap partition is active and being used.

Shut the OS down.

Tell cloudmin to delete the instance (and delete any images).

Cloudmin will error out.

Tell it to delete the instance again (and delete any images).

This time, it won't display any errors and the vm will be gone from the vm list.

Go to webmin -> hardware -> manage LVM

Try to delete the image that's still there.

Cloudmin won't let you delete it.

Reboot the cloudmin machine.

Go back into webmin -> hardware -> manage LVM

Try to delete the image that you tried deleting previously.

Now it works.

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Sat, 06/02/2018 - 13:41

Interesting - it sounds like the host system automatically picked up and maybe started using the swap file.

When you tried to delete the VM, what exact error message did you get?

Submitted by george.asenov on Thu, 01/17/2019 - 03:42

We faced similar issue: While I tested after VM creation script (this is not related probably) I try to delete (with selected to delete all disk images) with no errors and recreate a VM with the same name. Surprisingly the create process came up with error that lvm with the same name already exists and not only the swap partition but all of them. And in webmin LVM management it was truly there. I tried to delete it but the error sayed that it is used by other device ?!?!?.

Unlike the OP I'm unable to reboot the Host so I started researching. The lvm in question is "test_h5" .

lvs
  LV                           VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_root                      vg_h5_host -wi-ao---- 179.68g
  lv_swap                      vg_h5_host -wi-ao----  15.62g
  cent7_h5_domain_com_img  vg_h5_kvm  -wi-ao----  50.00g
  cent7_h5_domain_com_swap vg_h5_kvm  -wi-ao----   2.00g
  image_h5_domain_com_img  vg_h5_kvm  -wi-ao----  10.00g
  image_h5_domain_com_swap vg_h5_kvm  -wi-ao----   4.00g
  proxy_h5_domain_com_img  vg_h5_kvm  -wi-ao---- 500.00g
  proxy_h5_domain_com_swap vg_h5_kvm  -wi-ao----   2.00g
  test_h5_domain_com_img   vg_h5_kvm  -wi-ao----   3.00g
  test_h5_domain_com_swap  vg_h5_kvm  -wi-------   2.00g
  v17_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v17_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   4.00g
  v18_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v18_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   4.00g
  v19_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v19_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   4.00g
  v20_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v20_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   4.00g
  v21_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v21_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   3.00g
  v22_h5_domain_com_img    vg_h5_kvm  -wi-ao---- 200.00g
  v22_h5_domain_com_swap   vg_h5_kvm  -wi-ao----   4.00g lvremove -f /dev/mapper/vg_h5_kvm-test_h5_domain_com_img
  Logical volume vg_h5_kvm/test_h5_domain_com_img is used by another device.

in this case the swap lvm do not exist in /dev/mapper but

lvscan
  ACTIVE            '/dev/vg_h5_kvm/v17_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v17_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/proxy_h5_domain_com_swap' [2.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/proxy_h5_domain_com_img' [500.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v19_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v19_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v20_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v20_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/cent7_h5_domain_com_swap' [2.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/cent7_h5_domain_com_img' [50.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v18_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v18_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v21_h5_domain_com_swap' [3.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v21_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/image_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/image_h5_domain_com_img' [10.00 GiB] inherit
  inactive          '/dev/vg_h5_kvm/test_h5_domain_com_swap' [2.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/test_h5_domain_com_img' [3.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v22_h5_domain_com_swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_kvm/v22_h5_domain_com_img' [200.00 GiB] inherit
  ACTIVE            '/dev/vg_h5_host/lv_root' [179.68 GiB] inherit
  ACTIVE            '/dev/vg_h5_host/lv_swap' [15.62 GiB] inherit dmsetup ls|sort
vg_h5_host-lv_root      (253:0)
vg_h5_host-lv_swap      (253:1)
vg_h5_kvm-cent7_h5_domain_com_img   (253:25)
vg_h5_kvm-cent7_h5_domain_com_imgp1 (253:27)
vg_h5_kvm-cent7_h5_domain_com_swap  (253:24)
vg_h5_kvm-cent7_h5_domain_com_swapp1        (253:26)
vg_h5_kvm-image_h5_domain_com_img   (253:31)
vg_h5_kvm-image_h5_domain_com_imgp1 (253:33)
vg_h5_kvm-image_h5_domain_com_swap  (253:30)
vg_h5_kvm-image_h5_domain_com_swapp1        (253:32)
vg_h5_kvm-proxy_h5_domain_com_img   (253:7)
vg_h5_kvm-proxy_h5_domain_com_imgp1 (253:9)
vg_h5_kvm-proxy_h5_domain_com_swap  (253:6)
vg_h5_kvm-proxy_h5_domain_com_swapp1        (253:8)
vg_h5_kvm-test_h5_domain_com_img    (253:35)
vg_h5_kvm-test_h5_domain_com_imgp1  (253:37)
vg_h5_kvm-v17_h5_domain_com_img     (253:3)
vg_h5_kvm-v17_h5_domain_com_imgp1   (253:5)
vg_h5_kvm-v17_h5_domain_com_swap    (253:2)
vg_h5_kvm-v17_h5_domain_com_swapp1  (253:4)
vg_h5_kvm-v18_h5_domain_com_img     (253:11)
vg_h5_kvm-v18_h5_domain_com_imgp1   (253:13)
vg_h5_kvm-v18_h5_domain_com_swap    (253:10)
vg_h5_kvm-v18_h5_domain_com_swapp1  (253:12)
vg_h5_kvm-v19_h5_domain_com_img     (253:17)
vg_h5_kvm-v19_h5_domain_com_imgp1   (253:19)
vg_h5_kvm-v19_h5_domain_com_swap    (253:16)
vg_h5_kvm-v19_h5_domain_com_swapp1  (253:18)
vg_h5_kvm-v20_h5_domain_com_img     (253:21)
vg_h5_kvm-v20_h5_domain_com_imgp1   (253:23)
vg_h5_kvm-v20_h5_domain_com_swap    (253:20)
vg_h5_kvm-v20_h5_domain_com_swapp1  (253:22)
vg_h5_kvm-v21_h5_domain_com_img     (253:15)
vg_h5_kvm-v21_h5_domain_com_imgp1   (253:29)
vg_h5_kvm-v21_h5_domain_com_swap    (253:14)
vg_h5_kvm-v21_h5_domain_com_swapp1  (253:28)
vg_h5_kvm-v22_h5_domain_com_img     (253:39)
vg_h5_kvm-v22_h5_domain_com_imgp1   (253:41)
vg_h5_kvm-v22_h5_domain_com_swap    (253:38)
vg_h5_kvm-v22_h5_domain_com_swapp1  (253:40)

with this command I found that there are lvm volumes that still exists even after they are deleted years ago.

Looks like that every LVM is listed 2 times in the device map one for the lvm and again with the underlying partition (*p1)

The solution I found after some googling is: first delete the partition

dmsetup remove vg_h5_kvm-test_h5_domain_com_imgp1

note the p1 at the end!!!

then

lvremove /dev/mapper/vg_h5_kvm-test_h5_domain_com_img
Do you really want to remove active logical volume test_h5_domain_com_img? [y/n]: y
  Logical volume "test_h5_domain_com_img" successfully removed

then the non existing regarding "lvs" swap partition:

lvremove /dev/mapper/vg_h5_kvm-test_h5_domain_com_swap
  Logical volume "test_h5_domain_com_swap" successfully removed

To summarise cloudmin does NOT successfully remove Logical volumes during VM delete and did not throw error. Actually it is unable to remove them because the underlying partition actually use them (do not know if this is cloudmin or lvm/centos issue) and you need to first remove the partition and then you are able to remove the logical voll.

As I said this issue is here for years because some logical volumes still exists for a VM's deleted more than year ago but we didn't notice because we didn't try to create VM with the same name like some previously deleted one.

Submitted by JamieCameron on Fri, 01/18/2019 - 00:49

That's odd, because Cloudmin already does remove those p1 mapper entries as part of the disk deletion process.

Does the file /dev/mapper/vg_h5_kvm-test_h5_domain_com_imgp1 actually exist on your system before running dmsetup remove ?

Submitted by george.asenov on Fri, 01/18/2019 - 03:16

The odd think was that /dev/mapper/vg_h5_kvm-test_h5_domain_com_imgp1 wasn't but /dev/mapper/vg_h5_kvm-test_h5_domain_com_img was still there

and /dev/mapper/vg_h5_kvm-test_h5_domain_com_swap and /dev/mapper/vg_h5_kvm-test_h5_domain_com_swapp1 ware also there

probably cloudmin deleted /dev/mapper/vg_h5_kvm-test_h5_domain_com_imgp1 but failed to delete /dev/mapper/vg_h5_kvm-test_h5_domain_com_img /dev/mapper/vg_h5_kvm-test_h5_domain_com_swap /dev/mapper/vg_h5_kvm-test_h5_domain_com_swapp1

Submitted by JamieCameron on Mon, 01/21/2019 - 18:11

Ok, looks like /dev/mapper entries aren't always in sync with "dmsetup ls" . In the next release I'll have Cloudmin check both to determine which mappers to remove.

Submitted by IssueBot on Thu, 02/28/2019 - 09:07

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.