1) People that spend their entire lives in Toronto are strange, period. They live indoors, literally. For them, woods and rivers are fearsome, best seen out of an airplane window, and those who go into the outdoors are freaks. They honestly believe that you are almost surely BREAKING A LAW, harassing an endangered species of clam, or lying in wait for small children. It is inexplicable to them that anybody with an internet connection would ever want to go fishing. This is becoming a common attitude as we turn into an urban culture. Congratulations! By simply wielding a fishing rod, and enjoying it, you have become... a rebel!
2) City folk have heard so much propaganda about how polluted our waters are that they don't go anywhere near them and are STUNNED to see somebody using the resource. They literally assume that urban water is DEAD, it is all acid leavened with PCB's, and contact with it will result in instant cancer, or at the least, hideous skin lesions. This especially applies to old folks, who will stand there remembering the old days when they used to take bushels of fish out of that river, before the Dow Chemical plant went up. Or was it that other river, where cousin Eddy got the big one, back in '54, before the commies took over and everything went to hell.
3) EVERYBODY assumes an angler fishes because he wants to eat fish. This incredibly common and bizarre assumption, combined with #2, produces complete and utter befuddlement! They wonder why you would fish for something you cannot eat. They assume you must be soft in the noggin, and are waiting for you to do something else funny, like have a seizure, or chase your tail. My grandmother was utterly unable to comprehend that I do not fish because I like to kill and eat fish... every time I came back from a trip she asked where the fish were. For five years she did this.
4) PETA has perverted the minds of many. When urbanites are not busy spending $5,000 for a new inbred and short-lived "purebred" from a puppy mill, they spend their time sniffing the wind for any hint that somebody might be abusing an animal. They watch you with grim intensity, fully expecting to catch you in the act of viciously torturing a fish, strangling a baby duck, or molesting an endangered species, if you get no fish. They look forward to recording your imminent misdeeds with their Google Glass or their $600 iPhoney, testifying at your trial, and having you locked up; which will make them a hero, and earn a merit badge from PETA.
5) Other obvious possibilities: Were you not the correct shade of pale white? Were you wearing a hoodie? Did your clothing not have the correct brand names? Are you guilty of being a teenager?
You should ignore the idiots. Keep fishing. When asked if you plan to eat that fish, say "No, why would I? Do you eat the ball after a round of golf?"
Trick is to be there at the crack of dawn, when it's just you and the dead hookers.
Or get away from the braindead masses via cheap inflatable or kayak, which will also give you about a million more fishing opportunities.
Happy trails. Do not let them win. Keep fishing.
And if you see a giant fat bald white guy smoking a little glass pipe under a bridge somewhere in the GTA... get a video, you could sell it for a couple hundred thousand.
PS - The above is meant as humorous satire. I do not mean to offend the good folk of Toronto. I know many Torontonians, several of whom are quite normal.