RBS is under investigation by U.S. and UK authorities over its part in the interest rate rigging scandal and is expected to be one of the next banks to settle after its UK rival Barclays was fined $450 million in June.

"The group expects to enter into negotiations to settle some of these investigations in the near term and believes the probable outcome is that it will incur financial penalties," RBS said as it reported quarterly results on Friday.

The part-nationalized bank said it had dismissed a number of employees for misconduct after its own investigations into interest rate setting.

RBS Chief Executive Stephen Hester said the timing of a settlement is in the hands of regulators.

"We have to dance to the tune of the relevant regulators," Hester told reporters on a conference call. "We are up for settling with all and every one as soon as they are ready."

He said it was difficult to know if RBS faces a bigger fine than Barclays, which is the only bank to so far settle.

Even if the fine was smaller than Barclays' it would still be a "miserable day" for RBS, Hester said. "It is a deeply regrettable thing....this is the sort of thing the industry has to put behind it," he said.

That and other past mistakes are threatening to overshadow Hester's attempts to turn around the bank, which he said would be complete in the next 15-18 months.

LEANER INVESTMENT BANK

RBS made a third-quarter operating profit of 1.05 billion pounds ($1.69 billion), up from 2 million pounds in the same period the previous year, as losses from bad debts dropped.

By 0922 GMT, RBS shares were down 1.2 percent at 284 pence, compared with a flat European bank index.

The part-nationalized bank set aside another 400 million pounds to compensate customers mis-sold loan insurance, bringing its total provision to 1.7 billion pounds.

It said it was changing the way it pays staff particularly within its UK retail arm. "This is a move away from the sales-based approach of the past," RBS said.

Bad debts were 1.18 billion in the third quarter, down 12 percent from the previous three months.

Staff costs fell 5 percent. The bank has cut 9,900 staff in the past year, or 7 percent of its workforce, most notably at its investment bank.

That business saw a 2 percent dip in revenue from the previous quarter. But operating profit jumped 18 percent to 295 million pounds due to lower costs.

Taxpayers are sitting on a loss of over 20 billion pounds after Britain pumped 45 billion pounds into the bank to keep it afloat during the 2008 financial crisis.

But despite that loss, there has been speculation that Britain could start selling some of its 81 percent holding before the next election in 2015.

Compensation payouts for payment protection insurance (PPI), the investigations into interest rate rigging and possible breaches of sanctions on Iran still hang over the bank's recovery.

RBS last month exited a government insurance plan to cover its riskiest loans and sold a first tranche of shares in its insurance arm Direct Line.

But a 1.65 billion pound deal to sell 316 branches to Spain's Santander demanded by European regulators was scrapped after delays. RBS has restarted the sale.

(Reporting by Matt Scuffham and Steve Slater; Editing by Helen Massy-Beresford)

• A Permanent Economic Emergency