What I Inherited
My first job after I earned my degree in electronics was for A Pineywoods Home Health, Inc. On my first day, I was introduced to their “backup” system. In a nutshell, the way it worked is every day I would eject a tape from a server and put it in a bag and hand it to the office manager. She would then, in turn, give me another tape to insert into the machine. I did that religiously for about 5 months. Finally, I was caught up enough on emergencies to take a look at the backup tapes. After 5 or so months if something goes wrong they’re going to expect *ME* to fix it.
I gathered up the tapes and inserted the first one in the server and proceeded to take a look at what was on it. HORROR as I realized the tape was useless. The system was configured all wrong and there was nothing useful on the tape. If we’d have a server crash I would not have been able to get the company back up and running any time soon. I quickly reconfigured the system and implemented a schedule to verify the tapes.
What I learned
The result of that experience is now one of the things I include in all our managed services agreements is “Disaster Recovery Planning”. Basically, I work with my client to develop a system to get them back up and running in case their server crashes. I had the opportunity to see just how good my backup plan was last week during those storms that wreaked havoc in our area. In the mornings I look at all the networks I manage and look for issues. Sure enough, the storms caused quite a few issues for me to handle. One of my doctor’s offices was completely off-line.
That in and of itself isn’t much to be concerned with during or after a storm. It happens. I assumed I would just go to the office and check to make sure they had the Internet and if needed reboot the server. I showed up at the office before they opened and went to check everything. Sure enough, the server was failing to boot up. I tried a couple of times with no success.
Putting What I Learned Into Practice
Guess it’s time to see how good my backup is.
Here’s how I have that particular office set up:
- The primary server is a VM (virtual machine) on a VMware ESXi server.
- Anytime a major change to the operating system (OS) on the server occurs, a VM copy is made to a second physical host.
- Daily the database is saved locally to a virtual machine file server on the second host.
- Backed up the database is uploaded daily to a cloud server off-site.
I fired up the backup VM (virtual machine) on the other host. Then I copied the most current database from the backup file server. The process took less than 3 hours from the time I saw there was a problem with the network from my house. The doctor literally had no idea his server had crashed until I showed it to him unplugged on a counter.