Giving it All to an Ex and Starting Over in Your Forties - How to Build Wealth from a Late Start

06/18/2015 10:15 am ET | Updated Jun 17, 2016

Many men have complained about getting fleeced by an ex-wife and trying to begin a new life dead broke. But Dennis Miller voluntarily gave his wife virtually everything when their marriage dissolved when he was 45. Now 74, Miller enjoys driving his Chevy SSR roadster and having the freedom to do what he wants.  Learn how he turned things around so he could retire successfully.

Starting Over in Your Forties
Miller and his ex-wife were married at the age of nineteen with no financial assets and had three children in the next three years. Miller worked three jobs and went to night school for ten years so he could better support his family. Although Miller and his ex-wife were a great parenting team, after 26 years of marriage and ten years of couples counseling, they decided to get divorced.

In the settlement, Miller gave his soon-to-be-ex everything except one car and his own IRA. Their attorney told Miller that he was a fool for giving up so much, but Miller says now, "We didn't want to nuke our family just because the two of us didn't want to live together anymore. It was only money and I knew that I would figure it out." To this day, he and his ex are great friends.

A year after getting divorced, Miller met his current wife. He was in his forties and broke. He had to play catch up, but he believed, "If you want something you have to work for it."

This motivated Miller and his wife to sit down with a retirement planner". In what he calls his "epiphany moment", Miller suddenly realized that he would have to get serious about accumulating wealth, or "live in a double-wide for the rest of my life."

The Lessons Miller Learned Along the Way
Miller was an international lecturer for forty years and wrote many books on sales management. He served as an adjunct professor at Northwestern University and learned about investing through study and engaging with investment analysts. Among his lessons learned during his journey to retirement:

Start your planning as early as possible.
Know how much you need to retire with a comfortable lifestyle. "It is vital to set your financial goals," Miller stresses.

In his fifties, Miller watched his peers live lifestyles based on instant gratification -- a "charge it, figure it out later mindset", but he knew that planning for retirement requires a different mindset: "I'm going to get out of debt and put away money today, so that I have options when I'm older to do what I want to do.  So my financial goal in my fifties was to hit the magic number."

Dennis Miller, 74, retired at 64 and acts as a Retiree Mentor for MarketWatch

Aim to shoot past your ideal dollar value by about 20%. You won't necessarily need that money, but Miller points out that having it means that "you won't be in a panic every time the stock market has a correction, and you will be able to sleep better at night."

Once you have set up your nest egg, it gets much simpler: "If you live on the interest and never touch the principal, the principal will grow at least by the amount of inflation."

Both spouses must be in sync to reach your goals.

You might each be following your own career paths now, but once you retire, you will have to work together.

Miller and his wife still do an annual budget. At their age, Miller confides that preservation of capital is their main priority. "Here's how we tell whether we had a good year: If we had a lot of fun, our health was good and as long as the balance in our brokerage account grows ahead of inflation (we don't care if it came from stock appreciation, dividends, or interest), we had a good year."  He often asks himself, "What is the least amount of money we can put at risk to achieve our goals?"
Buy yourself time. Miller points out that when you are working, you trade your time for money. When you are retired, however, you trade your money for time.

Do as much as you can to reduce the expenses you will have in retirement. Pay off your house and your car as soon as possible. Avoid the cost of taxes in retirement by paying them now instead: "I took my retirement plan and rolled it into a Roth IRA so that any future withdrawals from our savings would be tax-free," Miller advises. This will make your retirement income more steady and predictable.

Education is continuous.
Miller describes his path to financial freedom as a "stream of choices" largely based on the simple lessons of his elders "It was 100 minus your age," he explains. "If you were seventy years old, you would put 70% of your net worth into certificates of deposit or high-quality bonds that all pay 6-7%. The other 30% would go into the market to keep you ahead of inflation."

These age-old guidelines all went out the window, however, when the good interest rates went off the table with the Great Recession. "I realized that I have no clue what is going on," Miller admits. "You have to learn the new market to survive."

Many of the Millers' friends lost 40-50% after following bad advice to put lump sums from their 401(k)s into the market. "Find a financial advisor who is in sync with your goals," advises Miller.

You do not have to retire as soon as you hit your goal amount.

"Staying young is a matter of attitude," says Miller, who at 74 is moving with his wife from Florida to Arizona, because one of the items on his bucket list is to live in the desert.

According to Miller, having a sense of purpose in life is essential, speaking with admiration of the grey-haired workers he sees at McDonald's and Wal-Mart. "Many of them have accumulated a great bit of wealth, but they still want to work a few days a week to feel like they are making a contribution."

Miller, who retired at 64, has just re-entered the workforce to write a financial newsletter for his peers and says of this experience, "I never thought my working life could be fun, but it actually is when you are enjoying your work instead of doing it to pay the bills." He has also written a full account of his story in a free eBook called Retirement Reboot. He is a retiree mentor for Marketwatch because he loves learning and helping others.

The Modest Millionaire Lifestyle
Despite being a millionaire now, Miller estimates his lifestyle would cost you $50,000-$200,000 per year, depending on your level of debt and your tax situation. Because he has no debt, his cost structure has changed dramatically. He defines successful retirement as "the freedom to enjoy ourselves, and pick and choose what we want to do".

Family is still important to Miller. Besides his genuine friendship with his ex-wife, he is serious about mentoring his grandchildren.

He and his wife eat out regularly and their hobbies change from time to time. I find Miller's humility refreshing as he tells me of the year they spent driving around the country in a motorhome.

Miller compares retiring to driving his motorhome into a campground: "I would look out the window at a family in a camper and think that, even though we were in a luxury motorhome, we were hardly having any more fun than they were. Now we have friends that retired to a beautiful home in Fort Myers and visit the country club regularly, and other friends who live in a double-wide and eat at family-style restaurants. Neither of them has more nor less worries than the other. The secret is that they live within their means and are enjoying life. It's not the money that you've accumulated, but knowing that your money is going to last and allow you to do the things that you enjoy."

Miller believes that your bucket list should not be finite. "As you scratch things off, you will add others," he says, "There are too many cool things out there to enjoy. I want to live till I die."

Miller is among the finance and insurance professionals whose journey to retirement is featured in the free eBook, "The Retiree Next Door:  Financial Professionals Reveal Their Retirement Secrets," published by While some finance experts neglect the advice they give their own clients, this group of successful retirees scrupulously followed what they preached until it paid off.