I hurt my shoulder a few yrs ago. Incredible pain, mri, all kinds of checks, they couldnt find anything wrong,said just a strain and i didn't even know how I did it. With physical therapy took months to feel ok enough to fish.
Listen to the Doctor.. and don't hate him.. "Don't shoot the messenger".. as it were.. and do what he says.. or.. there might be MORE down time, in your future...
Trust me on this.. I know of what I speak.. first hand.