The dramatic twist threw into disarray attempts to head off $600 billion worth of indiscriminate tax hikes and spending cuts that could push the U.S. economy into recession next year.

It also cast doubt over Boehner's future as speaker after failing to control unruly conservatives in his caucus.

With only 11 days left for bickering politicians to prevent automatic tax hikes and spending cuts, U.S. stock futures fell sharply on the news of the rebuke to Boehner.

The Ohio congressman had hoped to demonstrate Republican unity by passing a bill through the House, known as "Plan B," that would limit income-tax increases to the wealthiest sliver of the population - those earning $1 million and more, a far smaller slice of taxpayers than Obama wants to pay higher taxes.

But Boehner canceled the vote after failing to round up enough support from his party because many conservative Republicans are opposed to tax hikes on even the richest wage-earning Americans.

"The House did not take up the tax measure today because it did not have sufficient support from our members to pass," Boehner said in a statement after huddling with other Republican leaders.

The White House pledged to work with Congress to reach a deal as quickly as possible.

"We are hopeful that we will be able to find a bipartisan solution quickly that protects the middle class and our economy," White House spokesman Jay Carney said in a statement.

The bill, had it passed, would have put Republicans on record as supporting a tax increase on those who earn more than $1 million per year, breaking with decades of orthodoxy. It won the blessing of influential anti-tax activist Grover Norquist, but other conservative groups fiercely opposed it and many rank-and-file members said they would not support it.

Obama wants to raise taxes on families earning more than $400,000, a much lower threshold.

RECESSION THREAT

Obama and Boehner aim to reach a deal before the New Year, when taxes will automatically rise for nearly all Americans and the government will have to scale back spending on domestic and military programs. Economists say the combined $600 billion hit to the economy could push the U.S. economy into recession.

Boehner said Obama now must first pass a bill through the Democratic-controlled Senate before he holds another vote in the House.

Democrats said Boehner should first hammer out a deal with Obama. "The only way to avoid the cliff altogether is for Speaker Boehner to return to negotiations," said Adam Jentleson, a spokesman for Senate Democratic Leader Harry Reid.

With Republicans in chaos, Boehner will almost certainly need support from House Democrats to pass a deal before the end of the year. But he will have to keep an eye on his right flank before he stands for re-election as the top House lawmaker on January 3.

Alternatively, Boehner could wait until the new year to hold a vote. At that point, tax cuts passed in 2001 and 2003 will have expired for all Americans, and it presumably would be easier to pass a bill that would restore tax cuts for most.

Opinion polls show that more Americans would blame Republicans rather than Obama if they don't reach a deal before then.

So far, negotiations appear to be following the dysfunctional pattern set by the 2011 battle over the debt ceiling: fitful progress alternating with public posturing. Boehner also struggled during that showdown to corral the most conservative members of his own party.

Washington narrowly avoided defaulting on the U.S. government's debt in August 2011, but the down-to-the-wire nature of the effort prompted a first-ever debt rating downgrade and spooked investors and consumers.

This time around, concern over the fiscal cliff has weighed on markets but analysts say that investors appear to be assuming that the two sides will avert disaster.

"The markets are likely to interpret this as signaling even tougher negotiations in coming days," Mohamed El-Erian, chief executive of bond giant PIMCO, told Reuters.

S&P 500 stock futures fell 1.6 percent while Dow Jones stock futures and Nasdaq futures both lost 1.5 percent. At one point S&P 500 e-mini futures were down as much as 3.6 percent.

Lawmakers had hoped to wrap up work before the year-end Christmas break, but leaders in both the House and the Senate have indicated that they may call members back to work next week.

"The brinkmanship will continue," said a senior Republican aide. "This isn't the end of the story. More drama to come."

(Additional reporting by Thomas Ferraro, Mark Felsenthal, Richard Cowan, Jennifer Ablan, Dominic Lau and Kim Dixon; Writing by Andy Sullivan; Editing by Philip Barbara)

• A Permanent Economic Emergency