Someone's going to need to be rescued by the Coast Guard soon, it's just a matter of time.
Seriously though, recognize your limitations. I don't care how good a swimmer you are (I used to run triathalons, although swimming wasn't my strong suit) it's a different game toting along all that gear. I had brief moment of concern tonight when there was a little more current then I expected, missed the middle bar I usually rest on, and had to go ALL the way to my final spot in one shot. I was in Newport and I thought my next stop was Cuttyhunk. Remember you float if/when you get tired. Once I realized I was off course and got a little tired I just "idled" in place for a while and then continued on my way. When you start breathing hard that's when you're vulnerable to s#^&#^&#^&#^&#^&g in some unexpected sea water, take a break if you can.
As Goose said 3mm is a short season. I'm already back in my 5mm on most nights. If only one it should be a 5mm and it is much more bouyant if you want to just float. That said, those of us who have more "natural" insulation don't chill off as easily. For me when I'm in a 3mm on a 70 degree dry night I really struggle to stay warm enough after evaporative cooling takes place.
The new Korkers shoes with interchangable soles are THE ideal boot for this sort of activity. They aren't just better, more importantly they are safer. You can swim much better in them then a dive boot and Korkers as they are dramatically lighter. Just get appropriate neoprene socks to go underneath.
Some sort of gloves are highly advisable, Aquaskinz or similar. Pulling yourself up on rocks (or grabbing them as you get knocked off) hurts without them.
Be safe out there...