You wouldn't need to know where someone got the drug from. If you were going to allow home growing of recreational drugs, you'd control and tax the seeds / seedlings, so you're still getting tax income. It doesn't matter whether someone's taking home-grown drugs / drinking home-brewed beer or they're buying it from a licensed supplier.
I don't see how you can possibly say growing a couple of plants of a legal intoxicating substance is different from brewing a large barrel of a different legal intoxicating substance. What's the difference? Growing cannabis successfully requires specialist equipment / lighting / etc. (in the same way that brewing beer successfully requires a certain amount of specialist equipment, hops, brewers yeast etc.) and you still need to acquire seeds or seedlings, so the government will still be able to control those. If growing viable cannabis was as easy as putting a pot on a windowsill and waiting, don't you think more people would do it, rather than buying off some random dealer?
Most recreational drugs are much more complex to produce than cannabis / magic mushrooms anyway, and aren't really practical for home production, but again it would be up to the government to legislate and enforce growing rules as it saw fit. I'm sure that the vast majority of potential users of those substances would rather buy over the counter than invest time and effort in growing their own - in much the same way as the vast majority of alcohol users prefer to buy their alcohol over the counter rather than invest time and effort in brewing their own.