Ya got the right idea Jigman. Try roughing out the opening w/ a big drill before you use that router bit - save wear and tear on the tool and I know its another step/operation as well.
Some De-Sta-Co clamps (toggle style)would be ideal if you have a lot to do.
They make several different kinds - not cheap, but once you have it set up you flick the lever or two and pop in the next piece of stock.
Most drill presses have too much slop in the spindle - a manual milling machine is better way to go.
Lots more $ as well.
I'm not a plug builder, but I have lots of experience fixturing.
Jigman - pm me a mailing address and I'll send ya out a couple of clamps to try out. I have some used ones in a box.