I agree you'll probably be fine, but let me tell you two similar stories:
Growing up on Whidbey we would watch boats come and go off my bluff in our backyard (looking towards Camano Island). During breakfast one pleasant spring morning my little sister said she saw a whale. We pulled out the binoculars to check it out. The whale was a small sailboat (like a laser) tipped over with the dagger board looking like a dorsal fin. There was a man trying to stay on top of the hull waving to the occasional boat for help. No one saw him he was very low on the horizon and there was little boat traffic. We called SAR right away and my dad had me go out to the edge of our bluff and shoot a shotgun three times to try to signal to the man that help was on the way (he was probably 1/2 a mile from shore). Unfortunately, he went in to the water one last time, partially righted his boat, tipped over, and was not found until the helicopter came to assist the Zodaic rib in finding him (no life jacket or wet suit). CPR and heart massages, proved too little too late. It took SAR over 40 minutes to arrive with the boat and more than an hour for the helicopter. Rescuers said he was most likely so hypothermic that he could not properly right his vessel or get back on once he fell in the last time.
Fast forward 20 years to a nice sunny afternoon on Whidbey Island driving by Penn Cove on the way back to the house I grew up in (Penn Cove is across from Coupville on Whidbey) . I noticed some people looking out across the water and I stopped to ask them what they were looking at. They said there was a small boat tipped over and a guy in the water (with a life jacket). I asked if they had called SAR, bystanders said "no" so I had my wife immediately call. I knew a man with a small aluminum boat/ motor he used to tend his personal scallop farm, so I ran to his house and got him to come down and put the boat in, we motored out quickly, had great difficulty pulling the man into our boat (he couldn't help at all due to being so cold), and by the time we got to shore an ambulance had arrived at the public boat launch to treat him for hypothermia (still no boat for rescue). The paramedics told me we probably saved his life, they took over and treated him for hypothermia, the rescue boat arrived about 30 minutes later. I acted quickly and decisively because of my previous experience.
The man did have on a 2mm shorty wetsuit which probably helped a little, but using a similar suit for kitesurfing has shown me how inadequate this suit would be for open water (I mostly kitesurf in shallow bays that get quite warm during the summer). When his boat tipped over, the dagger board fell out and was carried away, he tried to swim to get it, gave up, and then swam back to his boat (we went back out and rescued his boat for him later). He was quite close to shore (about 1/4 mile or maybe closer).
The lessons I have taken from this:
1 Life Jacket!
2-Being "close" to shore can still be too far from shore.
3. Wear a wet suit if tipping over is a good possibility (4-5mm thickness does me well most of the time).
4. Don't rely on rescue getting there quickly enough and have a way to call for your own rescue.
5. Puget Sound water is cold.