Hi all,
I just came across a customer's server that did not have SPF records after a migration from cPanel (it didn't have them under cPanel...but if you're gonna upgrade to Virtualmin, you might as well go all out and upgrade behavior in all kinda ways!). So, I needed to add SPF records to all of the domains on the server. It's easy to use the BIND module to do so for one or two domains, but for more than that, I always like to test my mad bash skills and script it with a one-liner. So, here's how to do it quick-like. First change into the directory where your named hosts files live. It's probably /var/named/chroot/var/named on Red Hat based systems (you can add the cd to your one-liner, if you like, but I like to get my bearings before doing any mass-editing).
files=ls *.hosts | sed "s/.hosts//"; for i in $files; do echo $i. IN TXT "v=spf1 a mx ?all"]> $i.hosts; done
The first bit feeds the list of files ending in .hosts, minus the .hosts extension, into a variable called files. The for loop just operates on each entry in the variable. The command in the loop simply prints out the new SPF record line and the]> appends it to the end of the respective hosts file. Whee! bash is the bee's knees.
this will add the spf even if a dom already has it, doubling it up, is that ok?
Hey Tom,
You're quite correct. I was assuming a system without any SPF records. I don't know if multiple SPF records is a problem, or not. Better just to avoid it--if only for clarity and maintenance sake.
But it could be modified with a "grep -l -v spf1 *.hosts" somewhere in there to skip hosts files that already have a line containing "spf1". Actually, I think just replacing the ls *.hosts will do the trick:
<i>files=`grep -l -v spf1 *.hosts | sed "s/.hosts//"`; for i in $files; do echo $i. IN TXT "v=spf1 a mx ?all"]> $i.hosts; done</i>
Hope this helps.
--
Check out the forum guidelines!
Just FYI - the one-liner did not exclude the records that already included SPF records, but duplicated them.
Hey Harald,
Are you sure you used the second one (in the third post in this thread)? I'm pretty sure it doesn't double up existing records.
--
Check out the forum guidelines!
yeah it does add it again :(
Hey Tom,
That's bizarre. I tested the second example right after Harald's post and it didn't double them for me...I must be missing something. I'll look closer.
--
Check out the forum guidelines!
Oops. You're absolutely right. It totally does add it again. Silly Joe, thinking that grep is acting per-file and not per-line. Not sure how I missed that it doubled up in my hosts files that already had it (I must have checked one that didn't already have one to start with, but I thought it did).
So, here's the way to go about it:
<blockquote><i>files=`ls *.hosts | sed "s/.hosts//"`; for i in $files; do if grep -l spf1 $i.hosts]/dev/null; then echo "Skipping $i.hosts"; else echo "Modifying $i.hosts"; echo $i. IN TXT "v=spf1 a mx ?all"]> $i.hosts; fi; done</i></blockquote>
--
Check out the forum guidelines!
Or, in a form that'll safely cut and paste:
<blockquote><i>files=`ls *.hosts | sed "s/.hosts//"`; for i in $files;
do if grep -l spf1 $i.hosts]/dev/null; then echo "Skipping $i.hosts";
else echo "Modifying $i.hosts"; echo $i. IN TXT "v=spf1 a mx ?all"]> $i.hosts;
fi; done</i></blockquote>
Not so much a one-liner after all of that!
--
Check out the forum guidelines!
Hey all,
Just to update this one-liner with the new format of SPF records we're using:
<i>
ip=192.168.1.1
files=`ls *.hosts | sed "s/.hosts//"`; for i in $files;
do if grep -l spf1 $i.hosts]/dev/null; then echo "Skipping $i.hosts";
else echo "Modifying $i.hosts"; echo $i. IN TXT "v=spf1 a mx a:$i ip4:$ip ?all"]> $i.hosts;
fi; done
</i>
Note that you now need to enter the primary IP address of your server in the first line, as it is used later in the directive.
--
Check out the forum guidelines!
Thanks Joe,
This look like fun.. !
What directory to I issue his command from? Will it work for FC4?
Cheers,
Ryan
Hey Ryan,
Yes, it'll work on every OS. The BIND hosts file format is identical.
You'll execute it in the directory where your hosts files live (domain.com.hosts). On Fedora Core 4, and most other Red Hat based systems, I believe that'll be /var/named/chroot/var/named
--
Check out the forum guidelines!
Hmmm...
I just don't get how to use this.
I created a bash script with this code in it, ran it, and it like deleted all the text in my .hosts files.
So I'm sitting here asking myself why didn't I backup in a test directory and run it...
bummer.
<div class='quote'>I created a bash script with this code in it, ran it, and it like deleted all the text in my .hosts files.
So I'm sitting here asking myself why didn't I backup in a test directory and run it...</div>
Oh no! I'm not sure how that could happen, but this post does pre-date the migration to Joomla, and so the formatting may have made something go terribly amiss...and, of course, there's now a command line tool to accomplish this same task. So, it's no longer necessary.
Here's the way to do it now (the path to the Webmin dir is different on different operating systems...this is RHEL/CentOS/Fedora):
[code:1]
cd /usr/libexec/webmin/virtual-server
sudo ./modify-dns.pl --all-domains --spf
[/code:1]
You can also add other hosts, if you want using the "spf-add hostname" option.
As for your problem of losing all of your zones...I think one could do (I'm going to ask Jamie to check my work on this one, so wait until he confirms that this is non-dangerous):
[code:1]
sudo ./disable-feature.pl --all-domains --dns
sudo ./enable-feature.pl --all-domains --dns
[/code:1]
Which I think will recreate the DNS zones for all of your domains. Of course, if you have domains that you <i>don't want</i> to have DNS enabled, you'll probably need to get a bit more clever (or manually specify domains and run the command several times).
Again, on this last one, let's wait for Jamie to chime in with a sanity check to be sure I'm not telling you to do something stupid.
--
Check out the forum guidelines!
You can safely re-create all your DNS records using the commands Joe gave, although this will create them with just the standard records.
Also, this assumes that you have DNS enabled for all your domains. If not, a better command would be like :
[code:1]sudo bash
cd /usr/libexec/webmin/virtual-server
domains=`./list-domains.pl --name-only --with-feature dns`
for d in $domains; do
./disable-feature.pl --domain $d --dns
./enable-feature.pl --domain $d --dns
done
[/code:1]
--
Check out the forum guidelines!
Thanks guys, I did something similar through the "list virtual servers" interface.
The problem is, not postfix has the infamous "User unknown in virtual alias table;" and much of the mail is not being accepted.
I've tried to fix it and read a few threads - (i.e. don't have "myorigin" set and so on...)
Don't know where to go with it at this point.
<div class='quote'>I've tried to fix it and read a few threads - (i.e. don't have "myorigin" set and so on...)</div>
If that's the case, your hostname is broken in some way.
There are three possible causes of the error (that I've ever seen, anyway):
1. myorigin set incorrectly (pretty much whatever you set it to is gonna be incorrect, because nobody knows what it's for...so, leaving it out is the only sane choice) ;-)
2. User actually does not exist in the virtual alias table. You can look to see...either it's there, or it isn't.
3. Hostname of the box is broken. Either it has a DNS entry that points somewhere else, there is an MX record for it pointing somewhere else (though I think we'd get a relaying error instead), the hosts file entry is wrong somehow, or you've explicitly told Postfix that your hostname is something that it isn't. Somehow Postfix doesn't think your box is the hostname it thinks it's running on. Fix your hostname, restart postfix, problem solved.
--
Check out the forum guidelines!
It was the DNS issue.
The IP addresses had changed on the server (one went down with a different IP, the other came up with another IP, reverse DNS and nameserver glue records were all tweaked....)
Once DNS was sorted out - it's faring well.
Thanks again!
Yeah...
Well - running this command
[code:1]./disable-feature.pl --all-domains --dns[/code:1]
brought my server - with 151 domains - to a grinding, swapping, miserable halt today. I can't even log in to a ssh terminal. Last thing I saw that I could control was "top" showing over 15 instances of lookup-domain.pl and those instances taking 1-6% CPU each.
It's a dual opteron(250) x86_64 server with 2 CPUS and 4gb of ram running the 64bit version of CentOS 5.2
This is surprising and disappointing...but I still hope it works, because I have to reconfigure my entire DNS setup today because I was careless and had a misconfigured slave server delete almost my zones on my primary name server....(sigh)....
<div class='quote'>brought my server - with 151 domains - to a grinding, swapping, miserable halt today. I can't even log in to a ssh terminal. Last thing I saw that I could control was "top" showing over 15 instances of lookup-domain.pl and those instances taking 1-6% CPU each. </div>
That sounds bug-like. It might be a bug that is corrected by Webmin 1.470 and Virtualmin 3.67, though, as those introduce a new module system that dramatically reduces memory usage (by about 2/3). On 64 bit systems, in particular, this is a huge difference.
That said, I suspect you were seeing a side effect of the command...not the command itself, which wouldn't have spawned lookup-domain at all, much less 15 of them. lookup-domain is called during mail processing, and isn't actually related to DNS at all (and wouldn't be called during this command run). What I suspect is happening is that mail delivery is being broken by the lack of DNS service for those domains, and lookup-domain is getting hammered because of it. I'm betting if Postfix were stopped before running the command, this wouldn't have happened (I'm not suggesting you should have known to stop Postfix before running the command--it wouldn't have even crossed my mind that this could happen--just saying that I believe the issue is mail processing related rather than the command itself).
Anyway, in a Webmin 1.470 and Virtualmin 3.67 environment, memory usage of those 15 lookup-domain.pl processes will be smaller, they will run faster, etc.
You might want to be sure you're using the daemonized versions of clam and spamassassin, though, since you have a pretty hard-working system there.
--
Check out the forum guidelines!
How do I install these two versions?
I have run "yum update" and according to my records, the pro subscription of our licenses is not up either...and yet, neither webmin nor virtualmin is being upgraded...
What am I missing? Are these updates in the "bleeding edge" repository?
Just wondering.
Thanks,
-Jeremy
They don't exist yet...should be released in another day or so. And no, "bleeding edge" is for stuff like PHP (and never recommended for anyone).
Actually, I think Jamie just gave me the thumbs up on Webmin, so it'll probably go out first thing in the morning. Virtualmin will come in another couple days.
--
Check out the forum guidelines!
Hmmm.
On my system, which you guys set up a year+ ago, it's just /var/named