While at the boat school, IIRC, I would estimate that approximately 80% of black midshipmen struggled with the basic swim test. They kicked a** in everything else, (especially a classmate who cleaned my clock in boxing-I woke up staring at the ceiling in the boxing gym wondering WTH?!?) Anecdotally, I knew that for a few, when they entered the pool at USNA, it was the first time in their life. My plebe year roommate, who was black, struggled as well, but he was just a lousy swimmer like me
The one exception that I knew personally of was on the diving team. Oh yeah, he became a SEAL.