Thanks Mac, I'll take you up on that!  
I'm going to build a jig for the hand drill tonight similar to what you showed me... 
 
I can drop by after work any day next week (mon-wed), if any of those days works ok let me know...
I was so pleased with the way the jig turned out.  I have a fence that is attached via slots in the drill press table.  When I push it all the way back it is 2 1/8 from dead center.   So I made my jig 4 1/4 wide, so all I have to do to line up the jig is push the fence back as far as it goes and I should be centered.  To make sure its centered I put in a 1/4 bit and with the drill off I can lower the drill and if it goes into the hole for the dowel center,  I know I am centered.   
 
The jig is reversable in that I can flip it around to the other side and its still centered.   That way I can always hold the jig w/ the left hand and lower the drill with the right...