I am not sure of long term effects, but there have been a few athletes who died of heart attacks (or something similar) in the short term due to cocaine.
The upper drugs like cocaine, PCP, meth, et cetera cause strain on your system. In time they will shorten your life. You burn out sooner than normal. Much sooner for some.
The downer drugs like morphine, heroin, grass, mushrooms, et cetera probably do not do as much damage as alcohol if you can afford both the drugs and good food, medical care, a warm apartment & good clothes in the winter, et cetera.
There have been documented cases of doctors addicted to opiates (typically morphine) who showed very little physical deterioration. If you do not drive a car, practice medicine, et cetera while on downers, they might not cause much (if any) trouble.
Some substances (notably alcohol) put a strain on your body chemistry, usually affecting the liver, pancreas, or other organs required to metabolize the foreign substances. Extreme use of any drug is known to have really bad effects, although the bad effects of the downers seem to be due to driving a car or various other activities while under the influence.
My alcoholic son ruined his pancreas & liver by the time he was about 45 and will probably not outlive me even though he is 30 years younger. I do not think it is a coincidence that three of my cocaine using college buddies died 20 years or more ago and I am still going strong.
My father was born about 1880 or a few years earlier. He was about 35 when substance abuse laws were enacted (about 1913). When I talked to him about drugs, his observations of drug users prior to the enactment of those laws indicated that uppers caused health & behavioral problems, while downers usually did not create problems, except for really excessive use and/or doing certain activities while influenced by the drugs: Riding a bicycle, driving a car (less common than a bike back then), using power tools, practicing medicine or law, et cetera.
It is my guess that downer drugs might have bad long term effects if used in formative years (prior to being 18-25 years old). A lot of your knowledge of how to cope with life is developed before you are 20-25. If drug use interferes with that early learning, you might end up being dysfunctional or functional at an impaired level.