I did public relations for The Florida Keys & Key West for three years, and was in Key West for about a week total each month, sometimes more when I was also down fishing or there was a good event going on.
There are always events going on in The Keys and Keys West - this coming week there is a music festival through Sunday
Key West Songwriters Festival May 1-5 - that website is the official site of The Florida Keys and Key West.
As for restaurants - El Sibony is a great restaurant - reasonably priced and great food, also on Malory Square is El Mason De Pepe's a higher end Cuban restaurant and awesome too, and you can watch the activities taking place on Mallory Square if you get a good seat.
Pepe's Cafe on Caroline Street, as highlighted above, has awesome breakfasts ... it's an old-time restaurant and great.
There are also sunset cruises and variety of that stuff if you want to check out the harbor and surrounding area.
For fishing, Compass Rose Charters - for offshore and reef, tarpon and shark in the shallows - Captain Mike Weinhoffer is worth the trip - he's docked at Charter Boat Row - and will catch you fish, lots of them. Mike has been running charters out of Key West for more than 20 years - and is very, very active in the local community both from a fishing and activities stand point - assured to show you a fishy time.
Hybrid Mode
