Why $5 Million Is The Tax “Danger Zone”

In chess, the transition from the middlegame to the endgame is the most dangerous part of the match. It is the moment when the board has opened up, the major pieces are still in play, and one wrong move can undo everything you spent the entire game building. Interestingly, most games are not lost because of a bad opening. They are lost because someone misplayed the transition from the middlegame to the endgame.

Retirement works the same way. You may have heard that $5 million is the number that signifies you have truly made it, when your portfolio can sustain you on interest alone without ever touching the principal.

Maybe you’re not there yet. Most people aren’t. But the game I’m about to describe is one that every serious retirement saver is playing whether they realize it or not, because the rules do not change based on the size of your portfolio. The order you pull money out of your accounts can quietly cost you hundreds of thousands of dollars over a retirement that lasts twenty or thirty years, and that is just as true at $800,000 as it is at $5 million.

There is nothing magical about the $5 million number, but something tends to change once you cross that threshold. You have multiple accounts pulling in different directions. Your portfolio alone can push you into a higher tax bracket. A large portion of your wealth is probably sitting in pre-tax accounts, and required minimum distributions are starting to loom. At this level, your net worth is not what determines your tax bill; it’s how you pull the money out.

Most people think about taxes once a year, usually in April, usually with a groan. But the retirees who end up keeping the most of what they earned are the ones who stopped thinking about this year’s tax bill a long time ago. Instead, they started thinking about their lifetime tax bill. That is a completely different question, and it leads to different decisions.

Here is what I mean. When you are working, your paycheck determines your tax bracket. You do not get much say in the matter. But in retirement, if you have a mix of brokerage accounts, pre-tax accounts like IRAs and 401(k)s, Roth accounts, and maybe an HSA, you suddenly have something most people never realize: You have the ability to choose. You can choose how much income to create in a given year. You can fill up the lower tax brackets on purpose, pull from the right accounts at the right time, and avoid the brackets that would take the biggest bite. Think of it like choosing which piece to move in a chess game and when. A rook is powerful, but if you move it at the wrong moment, you lose the advantage.

During retirement, the goal is to pay taxes deliberately at the lowest possible rate, spread across the years where it costs you the least.

Roth conversions are a good example of how this can go right or wrong. You have probably heard the general advice to convert your traditional IRA to a Roth. It has become one of those things people repeat to their friends as if it’s always a good idea. But it’s not always a good idea. There are situations where a Roth conversion makes a lot of sense, but the decision involves more than just comparing your tax rate today to your tax rate later. How much you convert, when you do it, and how it interacts with the rest of your portfolio and financial picture matters considerably. If you’re still earning a high income and you aggressively convert a large sum without thinking through the variables, you could end up paying far more in taxes than you would have had you taken a more measured approach. I’ve seen projections where over-converting cost a couple nearly a million dollars compared to a more measured approach. Same assets, same people, completely different outcome, just because of timing. In chess terms, they sacrificed their queen when they did not have to.

Then there is something called asset location, which is different from asset allocation. Asset allocation is about what you own, stocks versus bonds versus alternatives. Asset location is about where you own it, meaning which account holds which investment. At higher portfolio values, this distinction matters greatly. Investments that throw off a lot of taxable income, such as high-dividend stocks or bonds, often belong inside tax-advantaged accounts where that income is not taxed every year. More tax-efficient investments can sit comfortably in a regular brokerage account. The goal is to keep more of the returns you are already getting.

And if charitable giving is part of your life, and for many in our community, it is a significant part, the way you give matters just as much as how much you give. Writing a check is simple, but it is often the least tax-efficient way to be generous. Donating appreciated stock, using a donor-advised fund, or making qualified charitable distributions from an IRA can reduce your tax bill while actually increasing the impact of your donation. You end up giving the same amount, or less, while making a bigger impact, and the difference can fund a lot of other goals.

You are going to pay taxes in retirement, that much is certain. What separates the people who keep their wealth from the people who quietly lose chunks of it is whether they pay at the wrong time, at the wrong rate, and from the wrong account. When withdrawal planning, Roth conversions, asset location, and charitable strategies all work together, you gain control over your financial life that most retirees never know is possible. You spent decades saving and getting your pieces into position. It would be a shame to lose the game in the endgame. n

Jack Strulowitz is a Financial Advisor at Bernath & Rosenberg in Cedarhurst, NY, where he helps high–net worth individuals and families manage their investments and build comprehensive strategies for retirement, tax, and estate planning. For questions or to schedule a consultation, please contact [email protected] or 847-962-3352.

The opinions voiced in this material are for general information only and are not intended to provide specific advice or recommendations for any individual. 

Securities and advisory services offered through LPL Financial, a Registered Investment Advisor. Member FINRA/SIPC.