[Ilugc] Learning on backup systems

Arun Khan knura9 at gmail.com
Sat Dec 26 01:27:32 IST 2015

On Wed, Dec 23, 2015 at 1:34 PM, Shrinivasan T <tshrinivasan at gmail.com> wrote:
> Last weekend, I had a great learning about backups from one of our
> customers.
> We had few servers attacked by rootkits.
> Few binaries were installed in 5 servers and they started to consume
> network bandwidth highly. A new mail server was installed recently and it
> started to send spam mails. We missed to harden the mail server as it was a
> test mail server and we thought of keep it alive for a week only.
> These issues were reported by network monitoring team.
> After trying to clean the rootkit attacked server, we realised that it is
> better to reinstall the entire os, hoping to restore the data from backup.
> We reinstalled all the servers. Then we checked for the backups to restore.
> But found that the disks for backup server were full 1 month ago.
> We missed to notice that. There was no monitoring client on that backup
> server. :-(
> Restrored  the available data and trying to collect the lost data from
> users machine and other servers.
> Lesson learned :
> 1. don't destroy existing servers without checking the backups
> 2. Don't missout any server from the eyes of monitoring system
> 3. Run mock runs for restoring data from backup often.
> 4. Do the hardening of any server as first task after installation, even
> though the server is for one day use.
> 5. Setup intrusion detection systems for critical servers. I thought it was
> boring and not an essential one. But understood the importance of them.
> 6. Having multiple backup server is really good. It is not waste of money
> or effort. It can help on hard times.
> Though we know about these already, unless we suffer, we don't realise the
> effects.
> Requesting all the sys admins to make sure about their backup systems and
> security.
> Good backup and security systems will give you peace of mind and reduce
> high tensions.

All great points and they should be the "due diligence" list for any
sys admin worth his/her weight in salt.

Frankly, I am bit surprised that many items listed in your post fell
through the cracks by your sys admins.

You missed one critical point -- keep your production servers (at a
minimum open to the WAN) up to date with security and bug fixes.

Beside IDS/IPS  like aide, fail2ban etc, 'logwatch' is a wonderful
tool.  Install it and have the reports emailed to you; more
importantly read them and you might get a head start on impending
problems in your servers.

-- Arun Khan

More information about the ilugc mailing list