Using Shared Storage for High Availability (NFS or FC)

Submitted by itinfra on Wed, 06/20/2018 - 05:33 Pro Licensee

Hello Support,

Please i need your help setting my environment up. I have provisioned two Webmin/Virtualmin Instances with the same content on both of them and Cloudmin Pro on a seperate machine. One is supposed to act as a primary and the other as a secondary. They are both Virtual Machines provisioned on a VMware Esxi and they are installed on a FC SAN. These two machines use their own storage currently and considering their sizes, it will be a waste of storage space having them act independently, since they already exist on a SAN. How do I configure these machines to use the storage of the Primary alone for both VMs? I already created an NFS Share of the root partition on the primary and added it to the secondary successfully. Do i boot the secondary from this nfs share of the root directory (this seems not to work as it indicates another machine is using the storage) or am I supposed to share just the home directory between them so that Cloudmin replication can work well? If i'm sharing the home directory, how do i set the secondary server to use the home directory share as its default directory? Also, i'm having issues accessing the NFS share from the Secondary while FirewallD is running. I need to turn off FirewallD before i can access it and i've added the NFS, mountd and rpcinfo services and ports, yet it isn't accessible until i turn off FirewallD. Is there something else i need to do?

Thanks

Status: 
Active

Comments

Submitted by JamieCameron on Wed, 06/20/2018 - 22:10

You could export /home from the primary via NFS, and configure the secondary to mount the same directory - then use Virtualmin's replication feature to sync up other domain data.

However, if the primary machine goes down, the NFS export would also be unavailable. So this wouldn't really protect you against downtime ..

Submitted by itinfra on Fri, 06/22/2018 - 06:11 Pro Licensee

Hi JamieCameron,

Thanks for the response and suggestion. I've created the NFS Share on the SAN and successfully mounted the /home directory to it on both the primary and secondary Virtualmin Servers.

Regards

Submitted by JamieCameron on Sat, 06/23/2018 - 21:36

Ok .. and do you have any MySQL databases? If so, they should also be stored on the SAN for reliability.

Submitted by itinfra on Mon, 06/25/2018 - 04:18 Pro Licensee

Hi jamie,

No we don't yet. I'll however change the location to the SAN as well as you have suggested.

Thanks.

Submitted by itinfra on Thu, 06/28/2018 - 04:24 Pro Licensee

Hi Jamie,

Is there a way to achieve an Active/Passive failover system with my setup as opposed to using Active/Active option that DNS Roundrobin load balancing provides? We want the Primary system to serve requests and if anything happens to it, it should failover automatically to the Secondary.

I await your response on this.

Thanks

Submitted by JamieCameron on Sat, 06/30/2018 - 18:23

Yes, you can use Cloudmin's DNS Roundrobin feature to have a DNS entry that points to whichever of the primary or secondary is up.

Submitted by itinfra on Fri, 07/06/2018 - 06:39 Pro Licensee

Okay Jamie,

So how do I set up the DNS Roundrobin feature on Cloudmin to make the DNS entry resolve only to the Primary when both Systems are up and then failover to the Secondary when the Primary is down.

Thanks.

Submitted by JamieCameron on Sat, 07/07/2018 - 14:19

You need to select the primary and secondary in the "Selected systems" field, and in the "Address records to include" field select "At most" and enter 1 in the adjacent text box.

Submitted by itinfra on Mon, 07/09/2018 - 10:15 Pro Licensee

Hi,

Following your direction, i noticed that the Status From Check tab showed the Primary Server as "IP Address Included" and the other one as "IP Address not included".

However, i simulated a failover by shutting down the Primary and the Status Check changed. The Primary showed "Excluded due to Status being down" but the Secondary Server still remains at "IP Address not included".

Submitted by JamieCameron on Wed, 07/11/2018 - 19:54

Did you wait a few minutes for Cloudmin to detect the downtime and update the IP address?

Submitted by itinfra on Thu, 07/12/2018 - 06:32 Pro Licensee

Yes i did... waited a longtime. It detects that the Primary host is down and changes the the status to "Excluded due to Status being down" but the secondary remains at "IP Address not included" all the while i was waiting and the IP address is not included in DNS as well.

Submitted by JamieCameron on Sun, 07/15/2018 - 16:52

I had a look at the code behind this, and noticed that the "IP address not included" message can be shown incorrectly when only one server of the two is up. In the case where the primary is down, which IPs are actually included in the DNS record?

Submitted by itinfra on Mon, 07/16/2018 - 08:56 Pro Licensee

Hello Jamie,

Thanks for your help. I checked as you said and realized that though Roundrobin still shows "IP address not included", the BIND DNS record has been changed to the Secondary Servers IP, which shows its actually working. But i noticed that the DNS record wasn't still resolving to the secondary servers IP when i tried to browse to it and did an nslookup until i cleared the cache of BIND DNS on the Cloudmin server. I also had to do the same thing when i brought up the primary server to have the record change back to its IP. My TTL for the A record is set to 60 secs on cloudmin and i'm wondering if i have to clear the cache every time for this record to be updated or is there something else i can tweak.

Submitted by JamieCameron on Wed, 07/18/2018 - 00:01

What did you do to clear the cache exactly? Restart BIND, or something else?

Submitted by itinfra on Wed, 07/18/2018 - 06:43 Pro Licensee

I ran the commands;

rndc flush

rndc reload

It was afterwards nslookup for the record was then pointing to the primary server.

Submitted by JamieCameron on Fri, 07/20/2018 - 00:27

I'm curious as to why you also needed to run rndc flush ?

Submitted by itinfra on Fri, 07/20/2018 - 05:40 Pro Licensee

I ran it to flush the cache.... after the Primary was down and the secondary's IP address was included/showing in the DNS, the record was still resolving to the Primary's address. It didn't change until i did the flush.

The same thing happened when the Primary came back up and it was its IP that was included in DNS.

Submitted by JamieCameron on Fri, 07/20/2018 - 13:39

Ok, so it was the local DNS cache. What's the TTL set to on that DNS record?

Submitted by itinfra on Tue, 07/24/2018 - 04:52 Pro Licensee

TTL on the record is set to 60 seconds.

So i simulated this again and left it for over 12 hours now..... BIND DNS shows the record pointing to another IP but opening the page or doing an nslookup of the record points to another. This doesn't change until one does a DNS flush.