I live here year round and while I wish the island had different thinking when it came to trash collection, it is not a dirty place. They (the towns) refuse to put enough trash cans around (because then they have to empty them)and when they do they become overflowed with a lot of household trash that is dumped late at night because they charge per bag to get rid of trash here and many (esp visitors) just find a dumpster to sneak their trash in there. It tends to overflow and blow around in the vicinity of the trash can. All trash is taken off the island and dumped at a landfill someplace in central mass I think.
My philosophy is they they should increase taxes on everyone enough to pay for this (trash collection and proper disposal), thus discouraging this dumping behavior and encouraging people to use trash cans because it is "free". But the mere mention of this brings hated scowls from across the room But they do not realize that it will cost more in terms of clean ups and dirty's up the place.
All that said...it is a clean place.
As for fishing...it appears the breach is nearly closed, you may be able to drive across to chappy in the next couple years!!
Breach Narrows At Norton Point, May Close This Year - 1/20/12 - Vineyard Gazette Online