Merck said the experimental medicine had significantly raised the incidence of some types of nonfatal but serious side effects in the study, which followed more than 25,000 patients for almost four years.

Merck shares fell 2.5 percent to $42.56 in midday trading on the failure of one of its most closely watched experimental drugs.

Tredaptive combines an extended release form of niacin, a nutrient that has been used for decades to raise HDL cholesterol, with a drug called laropiprant, which is meant to reduce the incidence of facial flushing that is a side effect of niacin therapy.

Merck said it no longer planned to seek regulatory approval for the drug in the United States and is advising doctors in other countries against starting new patients on it. The medicine was approved in the European Union in 2008, but U.S. regulators were unwilling to approve it until Merck conducted the costly long-term study to better assess its safety and effectiveness.

The U.S. Food and Drug Administration has been especially concerned about potential heart risks from laropiprant.

The failed study, called HPS2-THRIVE, is the latest high-profile trial that has called the heart-protective value of niacin into question.

Merck said Tredaptive had sales of about $13 million for the first three quarters of 2012 in the 40 countries where it is already sold. Had the drug been successful in the HPS2-THRIVE study, brokerage Cowen and Co predicted its annual global sales would have jumped to $300 million by 2016.

Sanford Bernstein analyst Tim Anderson had forecast Tredaptive sales of $1.1 billion by 2020, had it been approved in the United States.

"It seems conceivable that the drug could be removed from the market" overseas, he said in a research note.

Merck is developing another drug, anacetrapib, which raises HDL cholesterol by a greater magnitude than niacin, to see if it can reduce risks of heart attacks and stroke. If that trial is successful, many industry analysts believe anacetrapib could become a huge seller.

Anderson said data from late-stage testing of anacetrapib should become available in 2017, while results from trials of evacetrapib, a similar drug from Eli Lilly, should be unveiled in late 2015.

Garret FitzGerald, chairman of pharmacology at the University of Pennsylvania, said negative findings for Tredaptive should not lessen enthusiasm for anacetrapib and evacetrapib because they raise HDL through a different mechanism from niacin. Instead, they block a protein called CETP.

(Reporting by Caroline Humer and Ransdell Pierson; Editing by Gerald E. McCormick and Lisa Von Ahn)

• A Permanent Economic Emergency