Personally, I don't think it's ever warm enough to swim no matter how much the sun is burning us up--and believe it or not, some places in Alaska run 90 degrees F
Dressing warm is only required for the winter though so, yeah.