I often shut down when I think it will affect the bite or save gas. Long drifts while fluking to find concentrations of fish, in a safe and open area - I'd probably shut down.
I've found the bite better when live lining shallow rock piles and reefs for Bass with the engine off. That might just be my imagination though.
Even though most of my drifts are short and are in fairly shallow water I shut it down so not to spook fish. If I ran a four stroke I leave it running.