My guess would be that it is all dependent on how long you actually hold the fih vertical. Quick lip grab, quick unhook and fast release must do less damage than hanging him by the lip while your buddy sets up the camera and takes a shot.
I'd say probably not, but at 20 pounds its kind of marginal. Why not just leave it in the water and take out the hook? Then you know you haven't done any damage.