Business Continuity and How to Achieve Up Time

Whether you are running your own IT firm or have a hand in running an IT department for a larger organization, at some point or another you will need to plan for business continuity in the event of some major interruption in business operations.  Regardless of whether that interruption comes in the form of natural disaster, economic disaster, hardware or application failure, security attack, or some other traumatic event, it is your responsibility to ensure that the business can continue, at least from a technological perspective.  While I won’t go into great detail on the intricate development of a continuity plan (there are already many resources on this subject floating around the web), I will mention a few things here that I consider important considerations.

Backups, Backups, and more Backups!

Yes, yes, you’ve heard many, many times the importance of having good backups.  Even when continuity planning isn’t being discussed formally, the importance of backups seems to be emphasized fairly regularly. It is kind of an obvious aspect of an IT administrator’s responsibilities, but it is also one that is still overlooked all too often.  I am still surprised at some of the businesses I have come across that have either a poorly designed backup strategy, or a non-existent one.  This won’t be the first step in developing a good continuity plan, as you will first need to assess what services would need to be re-established in the event of a disaster, in what order, how it would be done, etc., but it is one of the more important aspects of it.  After all, without good backups, continuity is nearly impossible.

Also, make sure you go beyond simply backing up common files.  Don’t forget your databases and system configuration files as well.  It might be a good idea to backup entire hard drives as well, possibly making good use of LVM snapshots to do so.  With as cheap as hard disk space is becoming these days, don’t be stingy.  It is better to back up too much than too little.  You will also want to consider keeping your backups off-site as well, and preferably in a different state or geographic area if at all possible.

Make Good Use of Virtualization for Effective Business Continuity Planning

Virtualized servers not only make good use of hardware resources, but also make backups much easier.  After all, what could be better than backing up entire servers and redeploying them on new hardware completely intact?  Virtualized servers can save you many hours or days of work that would be spent rebuilding your various systems in the event of a disaster, by allowing you to simply power on a virtual machine copy and make some minor configuration changes.  Since virtual hosts are not immune to hardware failures, it would be a good idea to maintain multiple live virtual servers.  Some technologies, such as KVM (Kernel-based Virtual Machine) allow for live migration, and might be a good choice.  In this way, you could potentially move virtual machine guests to your secondary or tertiary virtual server with little to no downtime at the first sign of hardware failure.  Also be sure to plan well when deploying your virtual machines.  If your backup database server is on the same virtual host as the primary database server, you may run into problems when (notice I didn’t use “if”) the hardware for that virtual host begins to fail.

While we are at it, many technologies such as database and email servers contain features that can make backups much easier, such as replication and database clustering.  So, if possible, why not setup backup servers for your network files, database servers, and email servers on virtual machines running at another locations (Or ideally, at least two alternative locations separate from your primary), so that in the event of some disaster your secondary infrastructure is already established.

Consult Upper Management

When planning for disaster recover and business continuity, it is easy to only consider things from a technical perspective.  This, however, can be a huge mistake.  By all means, include management in the planning process.  Often times, managers from other departments and executive directors will have insights that we as IT folks simply won’t consider, such as legal and operational issues as well as obligations your organization might have to clients that will need to be met regardless of whatever disaster may occur.  Management from outside the IT department can be crucial in this process (even if they give inapplicable advice from time to time).

Project Management and Disaster Recovery Planning

It can be natural to treat continuity planning as a sort of a “Plan B” plan to be implemented in the event disaster strikes.  This sort of contingency-type planning can quickly lead you down the wrong path or cause you to overlook some critical steps.  It is better to treat it as sort of detailed migration plan and create a complex project plan to help carry it out.  Proper project management involves a detailed analysis and planning process that should cover every single facet of moving from point A to point B, and also provides us with the opportunity plan for nearly every possible pitfall we may come across along the way.  Taking this approach with regards to your continuity planning can contribute far more than you might think to the success of your disaster recovery preparation.  In fact, in my humble opinion, I would venture to say that employing a traditional project management approach could be the most crucial aspect of such a plan.  If you know little about project management, Project Management for Dummies might be a good place to start.

Get Started Now With You Business Continuity Plan!

It has happened to me more than once that I have built some new server for “testing” purposes that eventually found its way to production and suffered some disaster before I had a chance to include it in my backup plan.  It really is worth the effort to be a little obsessive about those sorts of things.  And, if you haven’t bothered to start a contingency/backup plan yet, get started now!  Few things bring into question an IT person’s worth like an inability to recover lost data in the event of a disaster because time to design a proper backup plan was never devised.  Words like “Backup? Umm…yeah, I forgot.” should never come from our mouths.