Billy,
hope you are feeling better.
It all comes down to what you want to do with it an where. Your boat, and Pauls, is IDEAL for where and when you guys fish, but too small for where I like to fish... If the bait changes distribution in the bay, and stays outside, you guys will feel more undersized
If you wanted to fish Brenton reef daily, it would be undersized by 4 feet IMHO. You also have to think about flexibility; If you are locked in to 2 days a week you can go out, bigger gives you more options those days if the weather is bad. When I had a free dock, I had my 13ft whaler in the water, and you're right I could fish the bay almost every day, but could I always go where I wanted? not even close.
I know when I upgrade someday it will be as big as I can afford to run!