Every week, when Ian Hanley sits down with his therapist, he goes through a list of depression treatments he’s been researching online. The best-known treatments at the top of the list–half a dozen antidepressants and known combinations of those drugs–are all crossed out.

“My therapist says he’s never had this much difficulty with somebody,†says Hanley, “which is sort of a dubious honor.â€

Hanley is only 21 years old, but he’s already six years into his search for something, anything, that can help him feel better for more than a few weeks at a time. “I’ve heard people describe it as sadness, and that’s not specific enough,†he says. Numbness is closer, but it’s not like depression inures you to suffering. “It’s like not quite being alive,†he says, “but still having to go through all the crappy parts of being alive.â€

When he was in the 10th grade, Hanley basically lost all desire to get out of bed in the morning. He started seeing a psychiatrist and a therapist–the same one he sees today–and went on Zoloft. “I wasn’t catatonic anymore,†he says. But the positive effects soon disappeared, like they would with every medication he’s tried since.

This year, Hanley quit college and put on ice his ambitions to become a screenwriter. Now he spends most of his time waiting to start a new treatment, trying that new treatment or waiting to see if he feels better.

Most diagnoses do not come with 20-plus medicines approved by the Food and Drug Administration to treat it–and yet with depression, more options don’t always mean better outcomes. Ever since the first antidepressants were introduced 60 years ago, doctors have had patients like Hanley–people who don’t seem to get better even after they’ve worked their way through the lengthy list of available drugs. About 30% of all people with depression don’t respond adequately to the available treatments. That’s a dismal failure rate for a class of drugs designed to improve a person’s basic ability to function.

Not that it has hurt the market. At last count, about 12% of Americans took antidepressants. Global revenue for antidepressants was about $14.5 billion in 2014 and is projected to grow to nearly $17 billion over the next three years. Clinical depression affects 6.7% of U.S. adults, or about 16 million people, and a growing number of children and teenagers too. It’s the leading cause of disability in the world, costing the U.S. economy alone $210 billion a year in lost productivity, missed days of work and care for the many physical and mental illnesses related to depression, like anxiety, posttraumatic stress disorder, migraines and sleep disorders.

There hasn’t been a major depression-drug breakthrough in nearly three decades, but a number of factors are conspiring to change that. Scientists are gaining a more nuanced picture of what depression is–not a monolithic disease, but probably dozens of distinct maladies–and they’re getting closer to learning what works for which kind of ailment. With suicides in the U.S. at their highest number in 30 years, experts agree that patients need faster ways to feel better, without waiting the typical four to eight weeks it takes for antidepressants to kick in. And as old drugs have gone off patent–making them less lucrative for drugmakers–companies are eager to find new revenue streams.

The biggest development has been the rediscovery of a promising, yet fraught, drug called ketamine. It’s best known as a psychedelic club drug that makes people hallucinate, but it may also have the ability to ease depression–and fast. In a race to shape the next generation of antidepressants, Johnson & Johnson and Allergan are fast-tracking new medicines inspired by ketamine. The FDA could be reviewing new drug submissions by as early as next year. Researchers, too, are exploring how to harness big data and even genetic testing to come up with new ways of treating depression for the 30% of people who don’t respond to the current standard-of-care treatment options.

All of this has psychiatrists, long frustrated with their menu of available treatments, hopeful for the first time in years. Finally, they say, their field is on the cusp of a much-needed breakthrough. “At this point, any new depression treatment that makes it to the finish line is a huge win,†says Dr. George Papakostas, director for treatment-resistant depression studies at Massachusetts General Hospital. “It’s going to have a major impact.†The question is, which method will prove to help the most people in the safest way possible?

Doctors have always seen depression as something that’s distinct from ordinary sadness, but what causes it and how best to treat it has changed wildly over the years. In the 5th century B.C., Hippocrates believed the body was made up of four humors and that too much “black bile,†the humor secreted by the spleen, resulted in melancholia. Melancholia as described by the Greeks looked a lot like depression today: persistent dark moods with a deep, lasting fear or sadness that isn’t based on reason. The Greeks prescribed lifestyle treatments like diet, exercise, sleep, bathing, and massage as well as rougher approaches like vomiting and bloodletting. In later ancient times, sex was also considered a helpful remedy.

By the Middle Ages, depressive-like behavior was believed to be a disease of the spirit–the result of demonic possession. Many depressed women were thought to be witches, and the cure for them was to be burned alive.

The idea that depression is rooted in the brain–and not a bodily fluid or possessed spirit–didn’t take hold until the 17th century, when a neurologist named Thomas Willis decreed that melancholia was “a complicated Distemper of the Brain and Heart.†Although he had little else to offer patients besides the lifestyle remedies of the Greeks, melancholia was finally thought to be at least partly biological. In subsequent years, everything from herbal remedies to opium to music therapy to spinning stools designed to make people too dizzy to feel pain fell in and out of vogue.

Beginning in 1938, electroconvulsive therapy was thought to be the only effective modern treatment for depression, but the procedure sometimes caused memory loss, among other side effects.

It wasn’t until the 1950s that doctors hit upon the idea that certain chemical cocktails could be used to treat depression. The first, called iproniazid, was found by accident when it was being tested as a treatment for tuberculosis. Doctors noticed that the TB patients taking the drug transformed from miserable and near death to euphoric, energetic, and social. Newspaper articles of people “dancing in the halls tho’ there were holes in their lungs†captured the popular imagination, and in 1957 scientists decided to try it on a small group of psychiatric patients.

According to that study, 70% of them became happier and more social. By the end of the decade, about 400,000 people with depression were on the drug. The high was short-lived, however: scientists soon discovered that the pills caused liver damage, and it was pulled from the market in 1961.

Another experimental drug, imipramine, was supposedly an antipsychotic, but scientists discovered in 1957 that it worked much better on people with depression. It became the first in a class of what’s called a tricyclic antidepressant, so named for its three-ringed chemical structure.

By studying how those early drugs worked, scientists were able to hit on a new idea about what caused depression in the first place: depleted levels of the brain’s neurotransmitters, namely serotonin, norepinephrine, and dopamine. Those early drugs may have come with nasty side effects, but they ultimately gave rise to selective serotonin reuptake inhibitors, the drugs most widely prescribed today.

Eli Lilly released the first one, Prozac, in 1987. It was an immediate hit; in just three years, 2 million people around the world were taking it, and pharmaceutical companies began churning out their own only slightly different versions of the drug soon after. SSRIs are still the field’s proudest and most profitable achievement, but they’re far from perfect. They can make people feel worse before they start feeling better several weeks after treatment begins. When the drugs work, they’re life-changing. But they don’t work for everyone. On SSRIs, about 35% of people taking them experience full remission.

“We have no idea, after many decades of studying these drugs, why some people get better and some people don’t,†says Dr. Roy Perlis, director of the Center for Experimental Drugs and Diagnostics at Massachusetts General Hospital. “We desperately need truly new interventions.â€

The largest, longest study conducted on depression treatments, called the STAR*D trial, found that after people tried four antidepressants over the course of five years, about 70% of them were free of depressive symptoms. That’s a lot of trial and error to end at a place where 30% of patients don’t experience remission at all.

The STAR*D trial ended in 2006, and despite a clear hole in the market–and a clear patient need–a golden age of new depression treatments still hasn’t arrived. Drug companies do not want to spend their money developing yet another SSRI–there are already a dozen approved by the FDA. “The drug companies basically shut down a lot of their brain research when it comes to psychiatry,†says Dr. Richard Friedman, director of the psychopharmacology clinic at Weill Cornell Medical College.

“In my opinion, it wasn’t necessarily a lack of interest–it was just that people had felt that they’d gone as far as they could with things like Prozac, and they weren’t sure what the next targets were going to be,†says Husseini Manji, global head for neuroscience at Janssen, the pharmaceutical arm of Johnson & Johnson.

Now, many experts–and drug companies–believe that target is ketamine hydrochloride, the only legally available psychedelic drug in the U.S.

In large doses, anesthesiologists use ketamine to put people under before surgery. In smaller doses, clubgoers use “Special K†to trip and hallucinate; it’s one of the top drugs of abuse in Asia. However, its newest application discovered serendipitously in the late 1990s, opened ketamine up to a whole new audience: those looking for a fast-acting drug for stubborn depression.

There are two drugs, one from Allergan and one from Johnson & Johnson, that work similarly to ketamine and are in late-stage clinical trials. As of now, ketamine is not FDA-approved for depression. So far, clinical trials on ketamine for depression have been small, and there aren’t many of them. Only about 400 people have participated in such studies, and many had fewer than 100 people. (By comparison, the STAR*D trial had almost 3,000 participants.) Still, the results are promising enough to excite a number of prominent researchers in the field.

“In the past 20 years, I’ve not seen anything like this,†says Dr. Cristina Cusin, a clinician and researcher who runs the ketamine clinic at Massachusetts General Hospital. Studies have shown that 60% to 70% of people with treatment-resistant depression respond to ketamine.

Ketamine has also shown promise in putting an end to suicidal thoughts. “We have patients saying, ‘I’m exactly as depressed as I was before, I just don’t want to kill myself anymore,’†says Cusin. “This was very surprising. We can’t explain it.â€

Barbara Reiger, who’s 59 and lives in San Diego, says she tried nearly everything to lift her out of the depression that had plagued her since childhood and sometimes rendered her suicidal. When she learned about ketamine in 2015, she decided to try it at a ketamine clinic in her hometown. Since ketamine is FDA-approved as an anesthetic, physicians can legally prescribe it off-label for any condition they believe it may help, including depression.

Since then, every six weeks or so, she shows up at a private ketamine clinic, where she’ll put on an eye mask in a dim room, sit back in a recliner, and have a needle full of ketamine plunged into her deltoid. It doesn’t make her feel high, exactly, but she remembers that the first time she tried it, a grin and happy tears spread across her face. “I felt I was putting things in order, moving pieces of a puzzle around to make it all make sense,†she says.

Ketamine trips are known to help people disconnect from their bodies and their thoughts, and it worked that way for Reiger. After her first trip, she remembers feeling better than she had for years. “I immediately felt relief … a lightness of the depression kind of lifting,†says Reiger. And it all happened in less than an hour.

“It’s been a paradigm shift, that now we can achieve rapid antidepressant effects,†says Dr. Carlos Zarate, chief of the experimental therapeutics and pathophysiology branch at the National Institute of Mental Health and one of the foremost researchers of ketamine. “Now we know there’s something radically different.â€

Experts aren’t sure exactly how ketamine works, although they agree it doesn’t target the standard antidepressant route of serotonin, norepinephrine, and dopamine. Instead, they think it stimulates a series of receptors in the brain, kick-starting something called synaptic plasticity–the ability of parts of the brain to grow and change. While antidepressants have been shown to also increase plasticity, ketamine seems to do it much more quickly–and much more powerfully.

How it works is one of the drug’s many unknowns. Doctors also don’t know the consequences and potential side effects of taking tiny doses of ketamine over and over again. People can become dependent on it, and long-term use of the drug has been linked to bladder toxicity and cognitive problems in those who abuse it recreationally. And although ketamine has been shown to be safe as an anesthetic, “you get anesthesia once,†Cusin says. “You don’t get anesthesia for months or a year. We have no idea what it does.â€

Ketamine’s effects are temporary; depression reliably creeps back, so people need infusions every few weeks or more to keep their symptoms at bay. It’s also costly. The treatments are rarely covered by insurance and run from $400 to $800 per infusion. On a biweekly schedule at the high end of going rates, that adds up to more than $15,000 out of pocket per year.

When Reiger first started taking ketamine, she quickly drained the funds she had set aside for treatments. She was on an every-six-weeks schedule, not including doctor’s visits, but at $500 a pop it was adding up fast. “It was cutting too much into our budget,†she says.

She thought she’d have to stop treatment, but she got lucky: another of her doctor’s ketamine patients heard about her plight and volunteered to pay for her treatments. “I’m grateful,†Reiger says. “But a donor can only be so generous, and I know someday I’ll probably have to face these demons again.â€

As many as 5 million people in the U.S. have treatment-resistant depression, and to capture such a huge potential market, private ketamine clinics run by psychiatrists or anesthesiologists have popped up across the country in recent years. These clinics have pitched themselves to doctors as get-rich-quick side businesses and to patients as a ray of light in the darkness. But there are no rules governing how these clinics administer ketamine or whether they monitor blood pressure, heart rate, psychological changes, or substance-abuse risk in patients who are taking it. “There is pretty large variability in what one clinic is doing compared to another,†says Dr. Gerard Sanacora, director of the Yale Depression Research Program at Yale School of Medicine and a longtime ketamine researcher, who surveyed 57 of these clinics in a July paper.

“I still think it’s the most exciting treatment to come in mood disorders, probably of the last 50 years,†says Sanacora, who worked on the APA statement. “But the reality is, there’s still a lot we don’t know about the treatment.â€

Other scientists are skeptical that ketamine is a good long-term solution to depression. “I think a lot of the hope on ketamine may prove to be misplaced,†says Friedman, of Weill Cornell Medical College. “People may be disappointed.†One problem, he says, is that in addition to enhancing neuroplasticity, it may also stimulate an opioid receptor in the brain. “If you stimulate that receptor–which is what Percocet and oxycodone and morphine and heroin do–you’ll relieve pain and make people feel high and euphoric,†says Friedman.

Until recently, ketamine proved to be a tough drug to imitate. Because it has widespread effects in the brain and doesn’t act neatly in just one spot, many of the drugs developed in its likeness–including those from pharma giants like AstraZeneca and Roche–were simply not as effective as the original. But now a few new drugs are showing promise.

Last year, Zarate at the National Institute of Mental Health and his colleagues keyed in on a part of ketamine’s chemical structure that they think is responsible for the drug’s antidepressant effects. Now they’re working on developing it as a treatment for depression. Early research in mice suggests that, unlike ketamine, the compound they’ve identified does not appear to be addictive.

Janssen, part of Johnson & Johnson, has developed and patented a version of ketamine called esketamine that doctors would be able to give their patients through the nose like a nasal decongestant, making it a more convenient alternative to an IV. The FDA has designated esketamine a “breakthrough therapy,†which means it can speed through the typically lengthy drug development process and get to market more quickly. If the ongoing efficacy trials prove successful, Janssen could file esketamine for FDA approval in 2018.

Rapastinel, an intravenous depression drug under development by Allergan that acts on the same receptor as ketamine, can be given in just 30 seconds and doesn’t seem to cause the hallucinogenic effects ketamine is known for. “So far in our clinical trials, the incidence of anything like hallucinations or psychosis has been incredibly low,†says David Nicholson, chief R&D officer at Allergan. That drug also received breakthrough status by the FDA and is in the final phase of clinical trials, which are expected to finish in 2019. Once those trials are done, Allergan will petition the FDA for approval to market the drug for major depression. They’re also working on an oral version of rapastinel.

Other new approaches to treating depression are on the horizon. For instance, researchers are testing drugs that target altogether different potential biological roots of depression–including chronic inflammation.

And at Perlis’ lab at Massachusetts General Hospital, a team of scientists is trying to use big data to uncover whether any other existing FDA-approved medications–including drugs for completely different issues, like high cholesterol, pain, and even acne–could have any mood-lifting side effects. Finding a medication that can be repurposed for depression would have huge advantages over traditional drug development: they are already proven to be safe, they don’t cost a fortune to create from scratch and they’re available for people to use now.

A revolution in genetics also offers promise for treating depression. “After lots of years of really not going anywhere, suddenly there’s a lot to be excited about,†says Perlis. In a landmark 2016 study, he collaborated with the genetics company 23andMe and Pfizer Pharmaceuticals to analyze the genetic profiles of nearly 460,000 people. For the first time, they discovered 15 different regions of the human genome associated with the development of depression.

Understanding what these depression genes do in the body makes it easier for scientists to find new drugs and interventions that have the opposite effect, which will make the search for treatments ever more precise. “It essentially gives us a target to aim at,†Perlis says.

For those who are suffering without hope, these new innovations, whether they stem from ketamine or whatever compound scientists will stumble upon next, can’t come soon enough.

For his part, Ian Hanley is waiting to see if ketamine, which he recently crossed off the list of treatments he has tried, will be a success or just another failure. His first IV infusion left him dazed and, after a long nap, uncharacteristically energetic. On the night of his first trip, he shocked his parents by cleaning his room and going for a walk for the first time in ages. But about half of his subsequent treatments sent him on bad trips. He was consumed with worries that he’ll stay depressed, that ketamine won’t work for him and that, if it doesn’t, he won’t know what to do next.

By MANDY OAKLANDER