Network Guru question

[Bob Thomas]

This might be for John or anyone that might be able to offer some insight.

I'm the "IT Liason" for a small Police Department in Central Mass. I know very very little about computers but could type so I got defaulted to the position.

We have 10 workstations, a domain controller, an Exchange server and another larger server for our records management system. This particular server is about 5 years old, IBM and is fairly large, from what I can recall. At the time, we had IBM install a DLT tape drive for our backups. Now, we're about to or are, in fact, exceeding our tape sizes. Second tapes are not an option as it currently takes 10 hours for a backup and really not feasible. Purchasing a new server (which we intended to do last year) is definitely out of the question...we're having enough problems trying to prevent a layoff.

What are my options as far as having a new backup installed? Can I get another sort of tape backup installed in the existing server or is there a better option?

I've heard about "online backups" but we're concerned with the security aspect and court cases.

Thanks for any and all info!!

Happy New Year

[The Dad Fisherman]

How big are your backups? You could get a second tape Drive and then split them onto 2 different tapes or you could replace the one you have with a larger capacity one.. You could also backup to disk...get an external hard drive and run backups as a file (not the greatest but would work).

Do you run a 7 day rotation on tapes or a 5 day. If a 5 day then you could run incremental backups....a full on friday and then incrementals on mon-thurs.

Depends how much you want to backup and how fast you would need to recover in a crisis.

Online backups are good....but pricey. I use to use a company called Amerivault to do those....it was awesome and I could restore files onto the fileserver in the UK before the chuckleheads over there could figure out what tape they needed to put in.

[Bob Thomas]

TDF,

We're backing up about 49gb each night. I run a 5 tape system with weekly, monthly and yearly backups on separate tapes.

I am hesitant doing incremental backups. If we have a crash right now, we'd need everything from yesterday due to the records management. I.e. if we arrested Joe Smith yesterday and crash today, we'd need those records.

Are tapes still the way to go now or should we be looking at CD/DVD's? We replace daily tapes each year by rotating the weekly and monthly up.

[striperman36]

I would look at a hosted option for Outlook.
Given the cost to replace the hardware/expand it hosting it makes more sense.
We hosted out our parish exchange at rackspace, about the same size
for less money per year that the hosted site than the 3 year amortization cost of a new server
We don't have have the archive retention requirements you may have, however that can be addressed with the vendor.

If you still wish to have it locally hosted, TDF's hosted archive solution is the way to go.

[RI Plugger]

The cheap way: Use your current backup system and use 2 tapes or delete some data that is getting backed up and is not needed.

The fast way : Buy a SNAP server and connect it to your network for RAID 5 Disk to Disk backup, Then the SNAP can archive to tape for you. (This is what I use.) If you can connect with Gig Ethernet then your backup should run faster and restore faster.

The expensive way: Buy a new tape library system from IBM.

[kenyee]

Originally Posted by Bob Thomas

Tcould type so I got defaulted to the position

LOL. Now that's funny

I think we need to know more about what your data is composed of.
49GB is a lot of stuff to back up daily. Is most of it Outlook/Exchange mail files? If so, one quick way to cut the size down is getting mail policy tools that purge or back up mail and forces users to keep their mail files smaller.
I'm assuming your arrest DB/app is no in Outlook/Exchange (or I hope it's not).

Simplest thing to do is seriously reconsider incrementals. Most people don't do full backups every day. It's usually weekly fulls and daily incrementals w/ rotating tapes in/out to a backup location.

Nowadays, the thing most people do is to use a network storage devices to do backups to. It'll go faster, but the main issue is not being able to move it offsite easily (you can get two and swap them weekly though).

It'd help if we knew what your budget is as well...

[striperman36]

budget for IT? Its a police dept!!

[The Dad Fisherman]

E-mail backups are an IT guys worst nightmare.....backing up gigs of "Xmas Pics at Aunt Pegs House" and "10 reasons why Woman are Better than Men Jokes".

If most of that data is Email data I would set up an e-mail policy where each e-mail box is only 100 meg in size and delete anything over 90 days old. your monthly backup tape should archive everything.

also set up some rules to block certain attahments (i.e. .mpg, .wmv, .mp3, and so on) get rid of the files that are obviously NOT work related. you may even want to go to text only e-mails if all you are supposed to be using it for is sending messages and not anything else.

I find that if you want to run a smooth e-mail back up....you need to be a dink...because people keep friggin everything.

If you take some of these steps you may not even need to get a new backup system....you may find your good with the current one for years.

What types of data are you backing up? any files that you an put on a tape and then archive the tape....then delete them from the server or move them to a location thats not part of the nightly backup for quick access.

[JohnR]

Option 1: Clean out some data. Find out WHAT is s#^&#^&#^&#^&#^&g up all of your information, and trim it. This will reduce your backup times and keep your existing system going longer. BUTTTTTT!!!!!! There is a finite timeline on how long tapes will last and tape drives. They will fail. 5 years is in that fail range. Price: Cheap - time to rumage through to see what can be eliminated or trimmed by a system professional

Option 2: A 5 year old DLT will take a LONG time to back all that stuff up. A new LTO-2 will backup your 50GB in 1.5-2 hours with verification. The tapes capacity is in the couple hundred gig range. This is the traditional way, and for ease, I would do a full nightly with Monday through Thursday tapes, Friday 2-5, and Friday 1 Odd Month, Friday 1 Even Month. This will give you some individual file recovery based on date over extended periods of time. Even better is having the ability to retire a monthly tape but that can get costly. Price: Couple grand

Option 2, look into a DATTO. I am installing these at several clients in the next month or two. http://www.dattobackup.com/ This will allow a full data backup onsite AND offsite. Because we know someone religiously takes a tape offsite for disaster recovery

Price $300-800 initial outlay, $200 - 400 per year. ASSUMES you have decent internet speeds.


Ohhh and next time they are looking to do servers, you guys should get a Small Business Server - perfect for your size organization....

[Bob Thomas]

Unfortunately, our budget for the IT was HACKED to bits by the town. It was in effort to save manpower. We do have "ways" of playing with numbers but those ways are going fast.

John - I think this IBM server is a "small business" server. I just dug out the specs:

IBM E Server X Series 225
2.5 gb RAM
Server 2003
2 Mirror RAID Drives
CD Rom
Backup: DLT VS80
Veritas 10

It's this bad boy
http://reviews.cnet.com/soho-servers...-30066685.html

Probably need to do some budget figuring here to see what we have to deal with.

Thanks for the info. Going to hit the boss up on Weds

[JohnR]

Bob - then you need to see how you can whittle down the files getting backed up. With less stuff getting backed up - by getting rid of stuff you don't need - the better you will be. Keep in mind I have clients with younger DLT VS80 drives that have either failed, or the tapes are no good (which tells me you have 40GB tapes that are "advertised" to get 80 but as you can see really don't).

Are those the original tapes? Your backups are probably no good if they are the original tapes copied over repeatedly for 5 years....

The "SBS" part is a version of windows Server 2003 that comes with its own Exchange and some other stuff - easier to manage for the non-geek

[Bob Thomas]

Not original tapes. We cycle them out each year.

Going to have to see what else can be whittled down. I think we're pretty much "whittled out" but will have to see.

Thanks for the help. Probably time to upgrade, as much as it hurts

[gf2020]

Why not just do incremental backups?

What backup software are you using? Was that mentioned above?

[Bob Thomas]

We're using Veritas Backup Exec. 9.0. I thought about incremental backups but the problem is getting people, other than just myself, to keep changing tapes. I can't tell you how many times I get sidetracked and forget to change the tape out.

Hit the Chief up with the bad news today. He said to start getting information and, since it's a vital necessity, we will find the funding somewhere.

[kenyee]

Bob:
did you ever get a chance to break down what that 49GB is composed of? That'll be an important part of figuring out what your "upgrade" should consist of (and possibly include spanking employees for sending buttloads of useless attachments around ;-)

[Bob Thomas]

Not yet, Ken. I really have no idea as to how to look for that stuff. I'm back to work tomorrow night and will be starting a search both on the backup and on finding a new way to backup

[JohnR]

If you were closer I'd take a swing in and look for you. That said, if it is OK with the boss (err chief), I'd be willing to make a remote connection while you are there and spend a few minutes looking at what is being backed up. Odds are there has been file creep which has expanded this. Second. Of vital importance, and I cannot stress this enough, you need to periodically TEST your backups by RETRIEVING a test block of files to the server to verify that what is being backed up can actually be restored. While there are certainly other things I would recommend - like setting up to do close a bare-metal-restore or better, at minimum, every 6 months you need to test your ability to restore that data so that you know your backups are working.

[The Dad Fisherman]

Originally Posted by Bob Thomas

I thought about incremental backups but the problem is getting people, other than just myself, to keep changing tapes. I can't tell you how many times I get sidetracked and forget to change the tape out.

That right there could be a good reason to look at a tape library....swaps out the tapes automatically.

Or you can also look into a tapeless system to get rid of that problem all together.

Look into offsite companies too....Like I mentioned earlier we used a company called Amerivault. Backups are offsite and secure. They also do offsite of your backups to another vault in another state.

So if the Station should blow up tomorrow you can always get your data back.

Restoring was a breeze too....as I mentioned too I could restore a file quicher in our UK office faster than they could get the right tape in. You manage everything right from you desk

[kenyee]

Bob:
Start up Backup Exec with the tape in. There should be some sort of catalog or "show me what's on the tape" function in the menu (haven't used it in a while so I don't remember what the name is). It'll probably list it into a window which you can then copy/paste into an Excel document or notepad. The thing to look for is filenames and filesizes for clues as to what's so big now.
What part of central MA are you in? If you list your location, one of the closer folks might be able to stop by to take a peek.

[Bob Thomas]

Folks,

Thanks so much for all the suggestions, replies and offers. I spoke with the Chief this evening and he gave me the "go ahead" to have our IT company remote in and look at it.

When they did so, they were confident they could buy us a little more time and deleted Patch uninstalls, Veritas Catalogs (whatever those are) and
\\*\*\*\*\Deleted Items

\\*\*\*\*\Junk E-mail

\\*\*\*\*\Journal

\\*\*\*\*\Sync Issues
(quoted from their email)

However, they hit me with yet more bad news. This particular server is about 5 years old and they believe IBM will not allow us to buy another year of the service contract. Basically, it goes, we need to buy a new server. Now the problem comes finding money to replace this server to the tune of $15k (we looked into replacement but did not have the money) Not sure how we're going to swing this one Going to start researching any grants out there for this or try to hit the town up for money (that should go well)

Just when you thing you dodged a bullet, it swings around like a boomerang and bites you in the backside

I told the Chief about this thread and he was thankful for all the insight!

Hope everyone survives this storm! I'll keep you all updated

[Bob Thomas]

Originally Posted by JohnR

If you were closer I'd take a swing in and look for you. That said, if it is OK with the boss (err chief), I'd be willing to make a remote connection while you are there and spend a few minutes looking at what is being backed up. Odds are there has been file creep which has expanded this. Second. Of vital importance, and I cannot stress this enough, you need to periodically TEST your backups by RETRIEVING a test block of files to the server to verify that what is being backed up can actually be restored. While there are certainly other things I would recommend - like setting up to do close a bare-metal-restore or better, at minimum, every 6 months you need to test your ability to restore that data so that you know your backups are working.

John,

As part of our yearly service (probably should be more often) our IT company comes in (at premium cost) and checks all the servers and verifies the backups. They said it was alright in 2008. We'll see what goes on now...going to be a bumpy ride!

Thanks again!

Bob

[JohnR]

Bob - how new is your other server? You have two right? Or just one? How is the other server backed up?

For $15000? I'll do it for $10K There has GOT to be more stuff there for it to be that expensive.

On the other hand, your department has GOT to understand that you can't go and run this stuff and NOT have an ANNUAL realistic budget for this stuff AND a 4 year (5 MAX) replacement schedule. I know a little bit about town budgets and how some bleep up year after year is almost comical.

[kenyee]

I agree w/ JohnR...what in the world are you buying for $15K?? You can get something (mirrored systems) that you don't need to worry about backups for a while with if you're dropping that much cash :-)
Let us know how big the next backup is. If they kept all the deleted mail and patches, that'd explain a lot...

[JohnR]

Oh, I hope they just moved or deselected the patches / other important-but-not-needed-in-a-backupfiles from backup the patches instead of deleting them

[The Dad Fisherman]

$15k is way too much....

Go with a Dell....here is one i priced out for you to look at....its got everything you would need and plenty of room to grow into it for the next 3-4 years.....even includes a 20 pack of tapes for the drive....for less than 1/2 the price

PowerEdge 2900 III

Starting Price $6,963

Instant Savings $200

Subtotal $6,763



My Selections All Options

Catalog Number / Description Product Code Qty

PowerEdge 2900 III:
Quad Core Intel® Xeon® X5460, 2x6MB Cache, 3.16GHz, 1333MHz FSB 29H316 1

Additional Processor:
Discounted Upgrade! Quad Core Intel® Xeon® X5460, 2x6MB Cache

Memory:
8GB 667MHz (4x2GB), Dual Ranked DIMMs 8G4D6D 1

Operating System:
No Operating System NOOS 1

Chassis Configuration:
Tower Chassis Orientation TOWER3 1

Primary Controller:
PERC 6/i Integrated Controller Card PERC6I 1

Hard Drive Configuration:
Integrated SAS/SATA RAID 1, PERC 6/i Integrated/SAS6/iR 6SR1 1

Primary Hard Drive:
500GB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive 500SU 1

2nd Hard Drive:
500GB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive 500SU 1

Power Cords:
Power Cord, NEMA 5-15P to C14, 15 amp, wall plug, 10 feet / 3 meter

Bezel:
Tower Bezel Included TBEZEL 1

Power Supply:
Redundant Power Supply with Y-Cord RPSWY 1

Network Adapter:
Dual Embedded Broadcom® NetXtreme II 5708 Gigabit Ethernet NIC OBNIC 1

Optional Feature Upgrades for Integrated NIC Ports:
LOM NICs are TOE Ready TOE 1 [430-2968] 6

Removable Disk and Tape Drives:
PowerVault 110T, LTO2-L Tape Backup, 200/400GB, w/Controller, Internal LTO2LC 1

Media for Removable Disk (RD1000) and Tape Backup:
LTO2, Tape media cartridge, 200/400GB, 20PK LTO2M20 1

CD/DVD Drive:
16X DVD-ROM 16XSDVD 1

Floppy Drive:
No Floppy Drive NOFD 1

Keyboards, Mice, Displays and Related Devices:
No Keyboard or Mouse Selected NONE 1

Documentation:
Electronic Documentation and OpenManage DVD Kit EDOCS 1

Data Protection Offers:
3Yr BASIC SUPPORT: 5x10 HW-Only, 5x10 NBD Onsite U3OS 1

Installation Services:
No Installation Assessment NOINSTL 1

[JohnR]

I would go so far as to tweaking that and virtualizing the other one

[The Dad Fisherman]

I gotta mess around with Virtualization.....Don't have the opportunity where I'm at now. been on my "To Do" list.

Right now I'm all Citrix....all of the time.

[Bob Thomas]

Well, when it rains, it becomes a monsoon!!

Ok....

1. We run a D/C, Exchange and a seperate server for the in-house records management. The $15k was to put the D/C and In-house into 1 server with some larger backup systems. I'll have to get the original quote to see what was involved/suggested but we thought it was pretty high!!!
2. Got in today and was informed the internet/email was down. Tracked and found that it appears our firewall took some sort of hit and is toast. Typing now on a seperate computer attached directly to the modem. Fantastic news!
3. We had some great figures for IT and, about 5 years ago, converted our wiring (eaten by mice) to Gigabit wiring and moved the servers to a dedicated server room. That's when we purchased all the servers and made some great updates. Since then, the only places to cut was the IT or we'd lose bodies. Pretty bad situation.
4. If you think WE have it bad, the town is even worse. They could not find their server (you read that right) for about 2 weeks. Someone finally tracked the wires back and found it in a closet. It is probably as old as me

Talked to the IT company today. They were able to get in before the firewall crashed. It was backing up some very unnecessary items and they believe it will be cut down by about 15 gig. When we install a new version of the in-house software, the vendor makes a seperate backup of the records. Apparently, there were about 4 of these redundant backups. Confirmed with them and it is unnecessary to back them up again as long as the primary information makes the tapes...which it does.

They also said it appears very little is being missed during the failed backups, namely the exchange server. Personally, I could care less if my email does not make it....less responsibility for me

I just can't believe everything is crashing down at once. We're wondering if the ice storms (previous and current) had anything to do with it all. We were on generator for 3 days and they've been changing cable & power lines ever since. We'll see. I need a break...this is ridiculous!!

Thanks again...I'll let you know what happens...if we ever get back up and running!!

[Raven]

chewing thru wires...

i recently bought a tin cat mouse trap...

i put half a chunk of bird suet cake inside of it...

and thus far in two weeks have caught ten mice...

there 17 bucks but worth the price...

[kenyee]

Originally Posted by Bob Thomas

The $15k was to put the D/C and In-house into 1 server with some larger backup systems.
...
firewall took some sort of hit and is toast.
...
They could not find their server (you read that right) for about 2 weeks. Someone finally tracked the wires back and found it in a closet.

$15K still sounds a bit high for one new server and the work to migrate two servers into one and to add a backup system. My guess would be around $10K (roughly 1-1.5 days of labor) with around $6-7K for the hardware (and the servers should be virtualized so future migrations are easy). If you get the chance, you should ask that IT company if there are any email purge policies in place...I'm still suspecting you have a lot of old email in people's mailboxes.

Do you guys have UPS's at all? If not, I'm really surprised. That should be part of any server setup and should include the firewall. Power outages/spikes will kill equipment; if not immediately, they damage the power convertors inside.

That server story reminds me of an AS/400 story...a company a while back had drywalled the room the AS/400 was in shut and no one knew where it was...they didn't notice is was missing for 5 years and dug it out only because they wanted to know where it was. Those systems are ridiculously priced and slow compared to PC's but they'll run forever without crashing or needing reboots