Should I Pay Off My Mortgage?

Having a paid off house has always been a goal. Living mortgage-free sounds awesome, but it can also put the brakes on financial growth. Any money put toward home equity is robbing investment accounts of hard working capital. After years of investing, we finally took some of our investment gains and paid off our house. This decision was a struggle for me, and I still think investing is better than paying off a mortgage in many cases.

Paying down debt isn’t just about the numbers

Compounding interest is an incredible force. Increasing returns by a little over 2% means having twice as much money after 30 years. Since I’d rather be a beach bum and let my dollars do all the hard work, I try to find the optimal way to deploy them.

In discussing whether I’d pay off my student loan debt (I’m not) I characterized my financial decisions as a struggle between two economic models. I call them Homo Sharkinus and Homo Chickenus. Both of these characters want to get the most out of my dollars, but handle risk differently. Sharkinus is always looking to maximize returns and doesn’t flinch when it comes to taking on risk. Chickenus on the other hand, can’t stomach much risk, and prefers the security of guaranteed returns.

We all need to find our own balance of risk

Each of us has our own mix of these characters, and when making financial decisions we need to find the right compromise to make them both happy. My wife doesn’t stress much about money and has a little more Sharkinus to her, meanwhile as someone who is a bit more OCD and prone to stress, I’m more Chickenus.

Even though these characters have opposing views on risk, there are some financial decisions where they can both come to an agreement.

When you shouldn’t pay off your mortgage

While working, our marginal tax rate was around 30%. That meant for every $10 that I could put in my pre-tax retirement account, I could only put $7 after taxes toward the mortgage. That’s 43% more that I could put in my 401k.

Of course, I’ll have to pay taxes on that 401k money eventually, but I plan on being in a much lower tax bracket when I withdraw it. In fact there is a good chance that I won’t have to pay any taxes on my 401k withdrawals. Not only is it important to take advantage of this tax arbitrage opportunity, but there are also ways to withdraw money from your 401k before retirement age. So even if you plan on retiring early, you can never have too much money in pre-tax accounts.

Since I stood to benefit by as much as 43%, both my Chickenus and Sharkinus agreed to first max out pre-tax contributions. And there’s one more low risk decision they agreed on.

Can you refinance?

Before even thinking about whether to pay down debt or invest, it’s worth considering a refinance. If you can reduce your mortgage interest rate by 1%, you should probably refinance. But depending on the size of the mortgage, even reducing the rate by as little as .5% could be worthwhile.

The key is to determine how much you will be saving in interest payments, then compare that with the refinancing fees. Last time I refinanced, I went from 5% to 4% rate on my mortgage and paid around $3,600 in fees. That 1% reduction on a $200,000 mortgage meant saving around $2,000 in interest each year. Well worth the fee I paid, so long as I held my mortgage for more than a couple of years.

Of course, if you refinance into another 30 year mortgage, you will have reduced payments, but the clock starts over again for another 30 years. This isn’t a problem since you’re saving on interest, and could just pay some additional principal each month to pay it off sooner.

But should you pay of your mortgage sooner, or invest?

Eventually I had extra cash to deploy even after maxing out my 401k and refinancing the mortgage. Now the decision to invest or pay down debt was getting harder.

My Chickenus loathed seeing how little of my mortgage payment was going toward the principal, and how much I was paying in interest. “Just a couple hundred extra dollars a month could set us forward years in paying off the mortgage, and those extra payments stored up as equity will keep saving us 4% a year guaranteed!”

My Sharkinus on the other hand was indifferent to the interest payments. “Who cares about all that interest, we can make up for it by investing in the market.” Over the stock markets history, the S&P500 has returned nearly 10% on average.

Generally Sharkinus wants to invest in the market before paying off any debt under 6%, and even wants to take out more debt to keep investing. The thing is, the market can be volatile, and there is no guarantee future returns will mirror what we’ve seen in the past.

While I did go back and forth, sometimes making additional principal payments, I mostly followed my Sharkinus and stuck with investing. Even though my Chickenus loathed the additional risk, I wanted to speed up my journey to financial independence as much as possible… Besides, I was just starting my career and would have plenty of time to recover if the investments didn’t work out.

The benefit of investing vs. paying down mortgage

Over a decade down the road, I took a look at how the Sharkinus route of investing vs. the Chickenus route of paying off the mortgage played out. Using historical data, I examined the effects of investing $1k a month into an S&P500 index fund vs. putting that money toward a $200k mortgage with 4% interest.

Had I followed my Chickenus tendencies and made additional mortgage payments of $1k a month, I would have just paid off the entire $200k mortgage and saved $28k in interest.

Sharkinus on the other hand, would have directed that additional $1k a month into an S&P500 index fund. This kind of investment over the last 12 years would have yielded just over $300k today with approximately $157k in gains. Not only does that beat the mortgage interest savings of $28k by a long shot, but I’d be able to pay off the house with the investment gains alone, which is kinda what we just did.

First, I started selling bonds…

As a lazy investor, all of my money has been going into an efficient three fund portfolio – a portion of which is allocated to bonds. These bonds weren’t making more money than my mortgage was costing, but I figured they also served as an emergency fund. The bond interest was being taxed, but my mortgage interest was tax deductible, so in a way they cancelled each other out.

Last year however, there were some pretty big changes in the tax code. With the standard deduction increasing dramatically – now $24k for married filing jointly – our mortgage interest is no longer worth keeping as a deduction. That makes the difference between our bonds (which are taxed as income) and the mortgage interest we are paying, much larger.

At the beginning of the year, I started selling bonds from our brokerage accounts. The proceeds went towards our mortgage, and we and managed to pay off a bit more than half of our six figure balance.

Then I sold some stock…

With our money working hard for us, we got to a point where we could start taking it easy. Two years ago, after reaching financial independence, I quit my job. Two years before that, my wife quit her corporate job to teach at our local community college. Our income is now about 1/4 of what it had been, putting us in the first two tax brackets.

Being in the first two tax brackets has the benefit of 0% capital gains tax. To take advantage of this, we’ve been harvesting capital gains each year by selling stock and re-investing. But, as I’ve written earlier this year, I do believe the market has been getting high. Whether it ends up crashing, couch-locked, or getting even higher, many economists forecast that it will slightly under perform in the future.

The market might be having a bit too much fun

In addition, we already left our corporate jobs and are more in the preservation phase of our financial journey. With my Chickenus screaming for stability, we went against the policy we’d followed for over a decade. After selling some stocks, we paid off the last chunk of our mortgage.

But was it a good idea to pay off the mortgage?

While I’m stoked that we don’t have any more mortgage payments, under different circumstances, I probably wouldn’t have paid it off…

For example, if we were in a higher tax bracket, I’d still be maxing out any pre-tax contributions (HSA, 401k, 403b, 457, etc.) before anything else. I also wouldn’t be selling any stock investments if I was going to pay a 15% tax on the capital gains. However, I probably still would have sold the bonds that were in a brokerage account where it’s taxed as income, and put that toward the mortgage with a higher interest rate.

While there may be a market crash looming, it still pays to invest for the long term. Twelve years ago, we were heading into the great recession when I started investing instead of paying down the mortgage, and I’m glad I did.

Even though it might not be the financially optimal route that my Sharkinus wanted, paying off the mortgage was the right balance for us at this point in our lives. It’s one less thing my Chickenus has to worry about, and it feels pretty awesome to have a paid off house.