Well I have fished near them before and if there a lot of fish around they might leave you alone.
But sooner or later they are going to ruin your day and grab your fish!!
Most of the time like you I just move and hope they don't follow!
I think we should have a seal hunt!!