I took delivery of a new server today, and decided to try out the new Xen 4 packages from the guys at CentOS. Install went smoothly per instructions at http://wiki.centos.org/HowTos/Xen/Xen4QuickStart. All that was left to do is set up some network bridges which I did manually and I could then add the host to Cloudmin.
Next I tried creating a new Xen VM via Cloudmin. The creation worked OK, but there was one odd glitch. The creation script failed to start up the new VM. The output from the creation process is below.
I logged in to SSH on the host and checked out the logical volume, config file etc and all looked, OK, so I can 'xm create dave.cfg -c' and watched the new VM boot up just fine. So I am wondering, why is the Cloudmin creation script failing to boot up the system, when I can boot it fine on the command line?
Interesting now the VM is created I can stop and start it using the System State -> Startup System and System State -> Shutdown System functions in Cloudmin. It was just the automated script that wasn't working.
Here's the output from the creation process:
Copying 4.91 GB image file to host system ams2-xen-3-2.localdomain .................................................................................. ...................................................... .. done, and added to host cache
Creating virtual system with Xen .. .. creation started.
Waiting for creation to complete .................................................................................. ........................ .. creation has completed successfully.
Creating swap file of 1024 MB .. .. done
Removing missing disks from fstab file .. .. done
Mounting new instance's filesystem .. .. mounted on /mnt/xen-dave
Configuring kernel for Xen instance .. .. setup to boot Xen system's kernel with Pv-Grub
Fixing GRUB configuration file .. .. done
Setting root password .. .. done
Updating configuration files with hostname and IP address .. .. done
Updating network configuration files .. .. done
Modifying /etc/fstab file .. .. done
Allowing SSH logins by root .. .. already allowed
Setting SSH server port .. .. done
Un-mounting instance's filesystem .. .. done
Adding DNS entry dave.ams2-cloudmin.anu.net. for IP address 83.96.163.8 .. .. done
Configuring Xen instance for VNC console access .. .. added on dynamic port
Starting up new Xen instance .. .. failed to start :
PTY PID: 4525 Using to parse /boot/grub/grub.conf (B)0[1;24r[m[?7h[?1h=[H[J[?1h= pyGRUB version 0.6 [0mlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk [0mx [0;7mCentOS (2.6.32-279.19.1.el6.x86_64) [m [0mx [0mx[72C[0mx [0mx[72C[0mx [0mx[72C[0mx [0mx[72C[0mx [0mx[72C[0mx [0mx[72C[0mx [0mx[72C[0mx [0mmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj [70D[0mUse the ^ and v keys to select which entry is highlighted. [58DPress enter to boot the selected OS, 'e' to edit the [52Dcommands before booting, 'a' to modify the kernel arguments [59Dbefore booting, or 'c' for a command line.[12A[26C[17B[68DWill boot selected entry in 10 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 9 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 8 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 7 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 6 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 5 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 4 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 3 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 2 seconds[?1h= [J[17A[73C[17B[68DWill boot selected entry in 1 seconds[?1l>[24;1H [?1l>Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-279.19.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Wed Dec 19 07:05:20 UTC 2012 Command line: console=hvc0 xencons=tty0 root=/dev/xvda1 ro crashkernel=no KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls ACPI in unprivileged domain disabled released 0 pages of unused memory BIOS-provided physical RAM map: Xen: 0000000000000000 - 00000000000a0000 (usable) Xen: 00000000000a0000 - 0000000000100000 (reserved) Xen: 0000000000100000 - 0000000280000000 (usable) DMI not present or invalid. last_pfn = 0x280000 max_arch_pfn = 0x400000000 last_pfn = 0x100000 max_arch_pfn = 0x400000000 init_memory_mapping: 0000000000000000-0000000100000000 init_memory_mapping: 0000000100000000-0000000280000000 RAMDISK: 02031000 - 04238000 No NUMA configuration found Faking a node at 0000000000000000-0000000280000000 Bootmem setup node 0 0000000000000000-0000000280000000 NODE_DATA [0000000000008000 - 000000000003bfff] bootmap [000000000003c000 - 000000000008bfff] pages 50 (9 early reservations) ==> bootmem [0000000000 - 0280000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [000443b000 - 0004462000] XEN PAGETABLES ==> [000443b000 - 0004462000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] #3 [0001000000 - 0002010024] TEXT DATA BSS ==> [0001000000 - 0002010024] #4 [0002031000 - 0004238000] RAMDISK ==> [0002031000 - 0004238000] #5 [0004238000 - 000443b000] XEN START INFO ==> [0004238000 - 000443b000] #6 [0040000000 - 0280000000] XEN EXTRA ==> [0040000000 - 0280000000] #7 [0000100000 - 00008db000] PGTABLE ==> [0000100000 - 00008db000] #8 [0004462000 - 0005068000] PGTABLE ==> [0004462000 - 0005068000] crashkernel: memory value expected Zone PFN ranges: DMA 0x00000001 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00280000 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000001 -> 0x000000a0 0: 0x00000100 -> 0x00280000 SFI: Simple Firmware Interface v0.7 http://simplefirmware.org SMP: Allowing 1 CPUs, 0 hotplug CPUs No local APIC present APIC: disable apic facility PM: Registered nosave memory: 00000000000a0000 - 0000000000100000 PCI: Warning: Cannot find a gap in the 32bit address range PCI: Unassigned devices with 32bit resource registers may break! Allocating PCI resources starting at 280100000 (gap: 280100000:400000) Booting paravirtualized kernel on Xen Xen version: 4.2.2-22.el6 (preserve-AD) NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 PERCPU: Embedded 31 pages/cpu @ffff88002804f000 s94424 r8192 d24360 u126976 pcpu-alloc: s94424 r8192 d24360 u126976 alloc=31*4096 pcpu-alloc: [0] 0 Xen: using vcpu_info placement Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2583489 Policy zone: Normal Kernel command line: console=hvc0 xencons=tty0 root=/dev/xvda1 ro crashkernel=no PID hash table entries: 4096 (order: 3, 32768 bytes) Checking aperture... No AGP bridge found AMD-Vi disabled by default: pass amd_iommu=on to enable PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 software IO TLB at phys 0x20000000 - 0x24000000 Memory: 764004k/10485760k available (5072k kernel code, 388k absent, 9721368k reserved, 7246k data, 1252k init) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:256 Console: colour dummy device 80x25 console [tty0] enabled console [hvc0] enabled allocated 83886080 bytes of page_cgroup please try 'cgroup_disable=memory' option if you don't want memory cgroups installing Xen timer for CPU 0 Detected 3100.064 MHz processor. Calibrating delay loop (skipped), value calculated using timer frequency.. 6200.12 BogoMIPS (lpj=3100064) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio Initializing cgroup subsys perf_event Initializing cgroup subsys net_prio CPU: Physical Processor ID: 0 CPU: Processor Core ID: 2 alternatives: switching to unfair spinlock SMP alternatives: switching to UP code Freeing SMP alternatives: 34k freed ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 20987 entries in 83 pages Performance Events: no APIC, boot with the "lapic" boot parameter to force-enable it. no hardware sampling interrupt available. Broken PMU hardware detected, using software events only. NMI watchdog disabled (cpu0): hardware events not enabled Brought up 1 CPUs devtmpfs: initialized Grant table initialized regulator: core version 0.5 NET: Registered protocol family 16 PCI: Fatal: No config space access function found bio: create slab at 0 ACPI: Interpreter disabled. xen_balloon: Initialising balloon driver. last_pfn = 0x280000 max_arch_pfn = 0x400000000 vgaarb: loaded SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default Switching to clocksource xen pnp: PnP ACPI: disabled NET: Registered protocol family 2 IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) TCP established hash table entries: 524288 (order: 11, 8388608 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 524288 bind 65536) TCP reno registered NET: Registered protocol family 1 Trying to unpack rootfs image as initramfs... Freeing initrd memory: 34844k freed platform rtc_cmos: registered platform RTC device (no PNP device found) audit: initializing netlink socket (disabled) type=2000 audit(1372357929.890:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 1560 alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring - Added public key 822A73AE165296BF - User ID: CentOS (Kernel Module GPG key) Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Non-volatile memory driver v1.3 Linux agpgart interface v0.103 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input0 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver PNP: No PS/2 controller found. Probing ports directly. mice: PS/2 mouse device common for all mice rtc_cmos: probe of rtc_cmos failed with error -16 cpuidle: using governor ladder cpuidle: using governor menu EFI Variables Facility v0.08 2004-May-17 usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 XENBUS: Device with no driver: device/vbd/2049 XENBUS: Device with no driver: device/vbd/2050 XENBUS: Device with no driver: device/vif/0 XENBUS: Device with no driver: device/console/0 drivers/rtc/hctosys.c: unable to open rtc device (rtc0) Initalizing network drop monitor service Freeing unused kernel memory: 1252k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 1052k freed Freeing unused kernel memory: 1732k freed dracut: dracut-004-284.el6_3.1 udev: starting version 147 dracut: Starting plymouth daemon xlblk_init: register_blkdev major: 202 blkfront: xvde1: barriers disabled blkfront: xvde2: barriers disabled dracut Warning: No root device "block:/dev/xvda1" found
Fetching current status .. .. status successfully retrieved (Down)
Re-fetching current status of host system ams2-xen-3-2.localdomain .. .. status successfully retrieved (Webmin)
Enabling system at host boot time .. .. done
Creating console access user .. .. done
Perhaps the device names within the VM are different under Xen 4?
Did you create this VM using an image provided by Cloudmin? Also, did you create it before or after upgrading to Xen 4?
''
Didn't think about that. It was from a custom image. I just tried downloading the CentOS 6 x86_64 image provided by Cloudmin, and that works fine.
Any ideas what might be causing this or should I just create new images?
I checked the xen config files for both and can't see any differences in the disk, vif, kernel or extra settings.
Our own images are based off the Cloudmin ones with some additional packages and configuration, so I'm drawing a blank as to what might be different.
The problem is almost certainly that the device file paths in /etc/fstab in that image are incorrect - I seem to recall that these changes in Xen 4.0. I'd recommend creating a new image, perhaps based on the Cloudmin-provided one.
''