Not every surfer is a surfcaster & doesnt understand the rules of the shoreline when it comes to fishing. They don't know when to pack it up & move somewhere else. I think they simply just don't understand & feel they have just as much of a right to be there as the fishermen (which they do).
There's a blitz among 15 surfers, youve got all your gear on & you walk down to the beach. You gonna cast, or turn around and go somewhere else?
Equivalent: The surf is up, you got your wetsuit & board, its one of your favorite waves to ride. You gonna go home cause someone is fishing?