do you have a cheap Chinese drill chuck? Could be the taper is not very precision, thus a little slop. No burrs on the taper of the drill chuck or in the female tapered bore? Tang on the drill chuck not lining up right?
"A beach is a place where a man can feel he's the only soul in the world that's real"