My response -
As long as you don't swim too close to large ships or get caught in a trawler net. I'd stay in shore rather than 50 miles out. Try the first 20 yards off the beach and don't get out of your depth. Seriously, rip tides are a terror, I've been caught in a couple. Even I always swim with a kicker float on a piece of string ... something to grab onto in case I get cramp or caught in a current. Oh, and I might even wear a wetsuit and a woollen hat.
I'm sailing the Atlantic at the end of the year. I don't intend ever to be in the water unless there is an emergency. I won't fall in as I'll always be attached to a harness. I'm asthmatic so falling into cold water is dangerous.