Imagine you just inherited some money -- what would you do with it? Invest it in the stock market? Make a big purchase? Pay off some debts? What if your boss said you could trust him to invest it for you?

Decades ago, Harry Hanon and his wife, Elizabeth, did what many might do in that situation: They said "no, thanks" to investing their money with Elizabeth's boss.

The hitch: Her boss was Warren Buffett.

Passing up the Oracle of Omaha's offer turned out to be one of Harry Hanon's great regrets, according to Kent Hanon, his son. Harry Hanon passed away in 2004 at the age of 91, and though he may have missed out on the investment opportunity of a lifetime, he certainly didn't die a poor man, having built a few houses across the country and selling them at a profit. In addition, Hanon also took up casual investing later in life, which didn't make him a huge amount of money but did become a "valuable and interesting hobby," according to his son.

"Until the day he died, I reminded him of his thing with Warren Buffett," Kent Hanon, a 72-year-old retired school teacher living in Bellevue, Neb., told The Huffington Post. "We could have been well-known millionaires along with him.”

Instead of turning his inheritance over to the future billionaire, Harry Hanon put the money into a pyramid scheme, losing the whole sum. Once Buffett became the people's billionaire -- recognizable in photos eating Dairy Queen ice cream, drinking Coca-Cola and hanging out with Jay-Z -- all Hanon's family could do was laugh, Kent Hanon said. They couldn't believe their mom's boss, whom they knew as "just a guy," had achieved such success.

"Everybody gave my Dad a hard time, deservedly so," Hanon said.

Hanon's father had access to Buffett's investing genius early on, thanks to his wife, who was one of the Oracle of Omaha's first employees, working as a "secretary/whatever," as Hanon described it.

One day Hanon's father came into the office to talk to Elizabeth about how to invest some money he had recently inherited.

"He was up there talking to my mother, and Warren Buffett joined in and he said, 'Give it to me and I'll put it into my company,'" Hanon said. "Of course, my dad didn't tell him to his face that he didn't think he was going to go anywhere."

For his part, Buffett told HuffPost that Elizabeth Hanon was one of his first two employees, starting in 1962. He couldn't confirm the rest of Hanon's tale, however: "Elizabeth Hanon was employed by my partnership, and also by my dad when we officed together in the early to mid-1960s (we split her salary)," Buffett wrote in an email.

Buffett made his way from those humble beginnings, when he shared an office with his dad, to become the second-richest man in the country, now worth about $46 billion, according to Forbes.

But Buffett isn't the only success story with a trail of would-be millionaires who passed up the chance to get in early on the next big thing. Joe Green, a college friend of Facebook founder Mark Zuckerberg, reportedly missed out on the opportunity to invest in the then-nascent social network, thanks to the advice of his father.

Actor John Cusack told Jimmy Kimmel how he passed up an offer from Apple CEO Steve Jobs to star in a commercial for iTunes and the iPod back before anyone knew how those products would revolutionize music. Cusack's response: "I just didn't feel like doing a commercial."

Also on HuffPost:

Loading Slideshow...
  • Coca-Cola

    <strong>Holding Value:</strong> $15.75 billion <strong>Shares:</strong> 400,000,000 (split adjusted) <strong>Stake in company:</strong> 8.9 percent <a href="http://www.cnbc.com/id/33608379?slide=16">Read more at CNBC.</a>

  • Wells Fargo

    <strong>Holding Value:</strong> $13.96 billion <strong>Shares: </strong>411,045,245 <strong>Stake in company:</strong> 7.8 percent

  • IBM

    <strong>Holding Value:</strong> $13.22 billion <strong>Shares:</strong> 6,645,396 <strong>Stake in company:</strong> 5.8 percent <a href="http://www.cnbc.com/id/33608379?slide=14">Read more at CNBC</a>.

  • American Express

    <strong>Holding Value:</strong> $8.50 billion <strong>Shares:</strong> 151,610,700 <strong>Stake in company: </strong>13.3 percent <a href="http://www.cnbc.com/id/33608379?slide=13">Read more at CNBC. </a>

  • Proctor And Gamble

    <strong>Holding Value:</strong> $3.98 billion <strong>Shares:</strong> 59,602,203 <strong>Stake in company:</strong> 2.2 percent <a href="http://www.cnbc.com/id/33608379?slide=12">Read more at CNBC.</a>