Now, he's determined to avoid it.
Only weeks after retiring as the most decorated Olympian in history - 22 swimming medals, including 18 gold - Phelps is ready for his next challenge: becoming a scratch golfer.
Lofty goal, to say the least. But why golf?
"I ask myself that a lot. There are a lot of frustrating days on the course," said Phelps, who spent his afternoon at the Omega tent as one of its brand ambassadors. "This is something I do want to improve on. This is something that my friends can still beat me on the course. I want to give them a challenge once in awhile.
"At this point, when you have that one, two or three shots a round, for me, I use that as motivation to come back. It's pretty much like learning how to ride a bike all over again."
And instructor Hank Haney is going to try and help him ride that bike in a new way with the next rendition of the Golf Channel's "Haney Project," set to start airing in February. Phelps says that Haney is "cutting his swing in half."
"I see a lot of similarities between (Hank) and Bob (Bowman), my (swimming) coach," Phelps said. "Sometimes I ask myself, 'Man, I am going to put myself through this all over again?'
"There are goals that I want to achieve. I asked Hank what it would take to achieve my goals and he said five days a week and a couple hours a day. It's going to be a long, frustrating road to achieving those goals."
His main goal is to be able to compete with his buddies, many of whom are scratch golfers themselves. "I don't like to lose at anything I do," says Phelps.
But Phelps knows there is a process that he must adhere to. Creating a great golf swing doesn't happen overnight, and the amount of suggestions and coaching thrown at him every day has become overwhelming.
One of the marquee names at the Past Captains/Celebrity Scramble on Tuesday at the Ryder Cup, he says that he had 20 or 30 people giving him tips.
"It made me think too much," said Phelps.
The Baltimore native says that not thinking was a big reason why he was so successful in the pool, as he was able to clear his head before a race without much of an issue.
"I've tried that a coupe of times on the golf course, and it doesn't work out (well)," said Phelps.
For a man that has faced immense pressure on the biggest of athletic stages in four Olympics, Tuesday's appearance at Medinah Country Club tops his list of nerve-wracking moments.
"(It was) the most nervous I have ever been in my life," said Phelps. "When I was on the range, I could barely put the ball on the tee. My hands were shaking so much."
The nerves didn't stop there, as he "was like a kid caught in a candy store" when he was introduced to one of his childhood heroes, Michael Jordan, who is an advisor for the U.S. squad.
Could Phelps see himself fill a similar role for the U.S. Olympic golf team in 2016?
"I would love to, it would be a great opportunity for me," said Phelps.
For now, he'll simply take in this weekend in Chicago, settling to be a fan in a worldwide event on American soil.
"Our country is behind us in everything we do," said Phelps. "The support will be crazy."
He didn't have to think twice about that.