[Ilugc] Learning on backup systems
scorpionking.sahil at gmail.com
Mon Dec 28 09:09:05 IST 2015
On Thu, Dec 24, 2015 at 1:04 AM, Shrinivasan T <tshrinivasan at gmail.com>
> Last weekend, I had a great learning about backups from one of our
> We had few servers attacked by rootkits.
> Few binaries were installed in 5 servers and they started to consume
> network bandwidth highly. A new mail server was installed recently and it
> started to send spam mails. We missed to harden the mail server as it was a
> test mail server and we thought of keep it alive for a week only.
> These issues were reported by network monitoring team.
> After trying to clean the rootkit attacked server, we realised that it is
> better to reinstall the entire os, hoping to restore the data from backup.
> We reinstalled all the servers. Then we checked for the backups to restore.
> But found that the disks for backup server were full 1 month ago.
> We missed to notice that. There was no monitoring client on that backup
> server. :-(
> Restrored the available data and trying to collect the lost data from
> users machine and other servers.
> Lesson learned :
> 1. don't destroy existing servers without checking the backups
> 2. Don't missout any server from the eyes of monitoring system
> 3. Run mock runs for restoring data from backup often.
> 4. Do the hardening of any server as first task after installation, even
> though the server is for one day use.
> 5. Setup intrusion detection systems for critical servers. I thought it was
> boring and not an essential one. But understood the importance of them.
> 6. Having multiple backup server is really good. It is not waste of money
> or effort. It can help on hard times.
> Though we know about these already, unless we suffer, we don't realise the
> Requesting all the sys admins to make sure about their backup systems and
> Good backup and security systems will give you peace of mind and reduce
> high tensions.
Thanks sir for this write-up and your insights on real-world scenario. I am
sure everyone has something to learn in this post.
And, I request everyone in this forum and to those who deals day-in and
day-out on real world IT issues to write-up about their experiences so that
everyone will get a chance to learn new things. This could help new-comers
and those who want to learn. :)
I know this may be time-consuming for them but a weekend post may work. If
we get 5posts like this then eventually we will have a good knowledge base
with the passage of time.
email has been sent from a virus-free computer protected by Avast.
More information about the ilugc