In the more than a decade that I’ve been actively investing in the stock market, I’ve never directly owned gold, or a gold miner. I’ve owned ETFs in the past that have held positions in gold companies, but I’ve never made a direct play on the yellow metal itself.

This has everything to do with my own personal preference, and I agree with the experts who say that investors should have at least a small position in gold. From 2009-2012, the sector would have been one of the best performers in any investor’s portfolio. Sure, the last 18 months haven’t been great, but many companies aren’t in bad shape over a five-year basis.

Why don’t I own gold? My thought process is that gold’s main purpose would be to serve as an inflation hedge. However, I already own too much real estate, which is also a natural inflation hedge. Why would I add another asset to my portfolio that accomplishes the same thing? Plus, real estate gives me cash flow for owning it, while gold doesn’t.

Let’s look at a few other reasons why investors would avoid the yellow metal, and whether these ideas might make sense for your portfolio.

1. Operational risk

Just about every gold company followed the same pattern between 2010-2013. They saw share prices go up, and started making money from existing operations. Because gold was a hot sector, bankers lined up to give these companies cash, so they grabbed it, borrowing extensively to bolster reserves and increase production.

We all know what happened next.

Barrick Gold Corp. (TSX: ABX)(NYSE: ABX) was perhaps the worst offender. Between 2010 and 2012, it borrowed nearly $7 billion, more than doubling its existing debt load. It invested heavily in a few different projects, the largest being the Pascua Lama project, located in South America. After spending more than $5 billion on the project, the company officially mothballed it in late 2013.

Gold mines take billions of dollars worth of investments over years to get going. There’s a huge operational risk factored in with every gold producer. Most spend freely when times are good, and often get hurt when the market moves against them.

2. Fear-mongering

I think I’m an optimist, which is why I put little faith in the doom-and-gloom scenarios that often go hand-in-hand with being ultra-bullish on gold.

Even if the Canadian dollar collapses for some reason, that’s why I have investments in the U.S. and in other countries. If the entire world’s economy collapses, I don’t think a gold bug’s bars of metal are going to be much help. Instead, people will trade things like food and water, the very basics for survival.

Besides, I don’t want to think that my country’s economy is about to collapse. If I saw upcoming disaster for Canada or the United States, I’d move.

3. Little actual utility

During his illustrious career, billionaire Warren Buffett has never invested in gold either. Here’s what he has to say about it.

“Gold gets dug out of the ground in Africa, or someplace. Then we melt it down, dig another hole, bury it again and pay people to stand around guarding it. It has no utility. Anyone watching from Mars would be scratching their head.”

The Oracle of Omaha is oversimplifying a little here, since gold does have a few industrial applications, and continues to be in demand from consumers in the form of jewelry. For the most part, though, he’s right. Holding gold will never give an owner cash flow. Most of the world’s gold just sits in vaults, and even jewelry has little practical use. It looks nice, and that’s about it.

There are a bunch of positive features to holding gold. It serves as an inflation hedge, and will generally zig when the market zags. There are also plenty of reasons to avoid the yellow metal. In the end, it all comes down to investor preference.