The long-term statistics of being a Canadian bank shareholder—and especially a Toronto-Dominion Bank (TSX:TD)(NYSE:TD) shareholder—are nothing short of staggering. Canadians mostly know this, and this is why Canadian banks are some of the most widely held stocks.

Canadian banks in general have grown their earnings at an impressive 12% annual growth rate over the past 20 years. At the end of 2015, TD’s five-year total shareholder return (price appreciation plus reinvested dividends) has grown at a compound annual growth rate (CAGR) of 11.8%. However, 2015 was a bad year, but when you factor it out, TD’s five-year CAGR for total shareholder return is actually 16.5% (above its Canadian peer group, which is at 14.2%).

With double-digit long-term earnings growth and returns (not to mention double-digit dividend hikes on an almost-annual basis over the past 15 years), Canadians may be mistaken to believe this type of return can continue. While TD will almost certainly do well going forward, double-digit returns may be too optimistic.

There is plenty of top-line pressure

There are basically three factors that drive total shareholder returns—earnings growth, dividend growth, and changes in the valuation (or earnings multiple) of the stock.

Going forward, all three of these could come under pressure. TD, for example, has published a medium-term growth rate of 7-10% annually. Most analysts, however, are skeptical of the bank’s ability to consistently hit the high end of this range (and, in some cases, even the low end). RBC analysts, for example, see TD growing earnings by 3% in 2016 and 7% in 2017. Weaker earnings growth means weaker dividend growth (assuming payout ratios stay steady), which means a lower total shareholder return.

A large portion of this weakened earnings growth comes from top-line pressure. About 62% of TD’s revenue comes from Canada, and there are numerous Canadian revenue headwinds. Canadian household debt-to-disposable income, for example, grew to a record 165.4% recently. This is bad news for TD, since it means that the general trend going forward will likely be one of de-leveraging for Canadians. High debt levels ultimately mean less loan growth for TD and less consumer borrowing in general, which could hit TD particularly hard since they are a retail-dominated bank.

This type of debt level could also make Canadians especially vulnerable in an interest rate-increase situation, which could add more pressure to disposable income and slow the rate of loan growth even more. In addition, Canadian real estate continues to soar, and any pullback in the Canadian housing market will be a headwind for Canadian GDP growth and consumer spending.

To make matters worse, interest rates are at historically low levels, which means the margin TD will earn on a slower-growing loan base is under pressure. TD’s net interest margin in the last quarter was 2.06% excluding wealth, which has been on a steady decline since 2009.

These factors combined with greater competition from non-traditional financial players (like Apple or new robo-advising firms like WealthSimple) and more regulatory pressure will pressure TD’s earnings growth over the next several years.

Non-interest expenses are also under pressure

Slowing revenue growth is only an issue if expense growth is unable to remain within the revenue growth, which would cause profit margins to erode. There are normal growth pressures on non-interest expense growth (such as wage growth), but over the next several years, large amounts of investments in technology will be needed to stay competitive, which could add additional pressure to expense growth.

TD is meeting this challenge well. The bank took a series of restructuring charges that are estimated to save $600 million, an undisclosed portion of which will go towards investment in TD’s digital and mobile capabilities. Rising investment requirements are nonetheless a headwind, as they apply upward pressure to costs at a time when the bank is least able to absorb them.

TD is currently trading at a forward price-to-earnings ratio of 11.6, which is in line with its long-term average, meaning that while TD can be expected to do well for investors going forward, those looking for mid- to double-digit returns are best to look elsewhere.

If you're focused on dividends and are looking for secure growth, consider these three picks

These three top stocks have delivered dividends for shareholders for decades (and in some cases, centuries). Check out our special free report: "3 Dividend Stocks to Buy and Hold Forever". Click here now to get the full story.

NEW! This Stock Could Be Like Buying Amazon In 1997

For only the 5th time in over 14 years, Motley Fool co-founder David Gardner just issued a Buy Recommendation on this recent Canadian IPO.

Stock Advisor Canada’s Chief Investment Adviser, Iain Butler, also recommended this company back in March – and it’s already up a whopping 57%!

Enter your email address below to claim your copy of this brand new report, “Breakthrough IPO Receives Rare Endorsement.”

Fool contributor Adam Mancini has no position in any stocks mentioned. David Gardner owns shares of Apple. The Motley Fool owns shares of Apple.