Worried about diminishing market share and increasing competition, EBS, at one point the world's top foreign exchange dealing system, took a gamble and decided to let clients trade out to a fifth decimal point - that's one-thousandth of a cent.

The move backfired. It accelerated a decline in the firm's market share in the nearly $5 trillion forex market and contributed to a management shake-up that caused three top executives to leave earlier this year.

Stocks get the lion's share of popular attention, but the currency market is the largest and most liquid in the world, facilitating global trade and transactions among corporations and banks. Participants range from central banks and hedge funds to corporations and mom-and-pop retail investors.

But in the past few years, activity has declined due to the global economic downturn and the euro zone debt crisis, while trading has shifted to other venues. Banks are using proprietary platforms, and other institutions are launching new FX dealing businesses rather than use exchanges like EBS, which is owned by London-based interdealer broker ICAP, or rival Thomson Reuters.

Major banks are increasingly handling trading and counterparty risk in-house, which has contributed to dwindling activity on EBS. The average daily cash forex volume on the EBS platform was $95.5 billion in August, down 49 percent from a year earlier, according to ICAP, and far from the daily average of $250 billion in February 2008. Thomson Reuters data showed its volume in August fell 30 percent from a year earlier to $115 billion.

As a result of the drop in volume, EBS' share of the cash forex market has fallen to 7 percent this year from more than 15 percent in 2007, according to Richard Perrot, analyst at German private banking firm Berenberg Bank.

Perrot has no comparable figures for Thomson Reuters. Jas Singh, director of Marketplaces at Thomson Reuters in London, said the recent purchase by the company of foreign exchange trading platform FX Alliance should boost its share of the market, though he declined to provide specific figures.

EBS Chief Executive Gil Mandelzis, who came to the company six months ago, said weakness in EBS' volume was in line with the rest of the market. "The reality is global trade has decreased significantly."

Fearing for its future, EBS tried to cement its hold on the smaller banks that had started to trade in small fractions and were being targeted by competitors. These smaller banks were key EBS customers that balanced the presence of big banks and "algorithmic" computer-driven traders on its platform, said a source at EBS familiar with the dealing system.

That, in part, led EBS to quote currency prices at five decimals - say, $1.30323 for the euro against the dollar instead of $1.3032. This is known as "tenth pricing," or "decimalization."

In general, tenth pricing is beneficial to customers and traders because it reduces the cost of the trade. It also represents what many traders believe to be the most precise price in the market.

But that fifth decimal attracted super-fast computer traders, often disrupting the flow of liquidity on the EBS platform. It angered banks that have slower technology, as they had difficulty executing large transactions because high-speed traders sliced them into smaller chunks, market participants said.

EBS has returned to pricing many exchange rates in four decimals, essentially suggesting tenth pricing has failed and appeasing larger banks, a major revenue generator for EBS.

"There was some dissatisfaction with tenths that's widely recognized," Mandelzis said.

TRADING IN-HOUSE

With banks investing in their own platforms, they are acting as a counterparty to their customer orders - known as internalizing - rather than going through big platforms like EBS.

In 2007, less than 25 percent of trades were done within the major banks, Berenberg's Perrot said. By 2010, banks - which have invested heavily in trading platforms - were matching 80 percent or more of customer trades and avoiding the fees charged by EBS and Thomson Reuters.

Mandelzis said the shift to banks doing their own trades has already happened and will have a less severe impact on EBS in the future. This year, it's all about the decline in overall forex trading, he added.

STAYING AHEAD OF THE CROWD

EBS introduced tenth pricing at a time when bank platforms such as those run by Barclays and Deutsche Bank already provided tenth pricing to their customers.

A source familiar with the launch of tenth pricing said EBS had to quickly adopt decimalization "because it was fast losing volume and clients," specifically the small non-dealer banks.

But the company's major banking clients, the big dealers that execute huge forex trades, were not happy. Many big banks aren't fully committed to decimalization because it would mean costly technology upgrades.

"We didn't think decimalization would enhance our liquidity and we expressed those views to the previous EBS management," said a top executive from a major bank.

Two sources familiar with the launch of tenth pricing said EBS didn't think the scheme would anger its major banking customers because top executives from these top dealer banks, which are part of EBS' advisory board, unanimously approved it at a meeting in 2010.

EBS' advisory board is a private group that meets regularly and its minutes are not open to the public.

The sources declined to name the banking executives who approved the tenth pricing plan at the EBS meeting. Calls to major dealer banks to verify their stance on tenth pricing were not returned.

Mandelzis said he cannot comment on the decision-making process that led to tenth pricing because at the time, he was chief executive of Traiana, an ICAP subsidiary that handles post-trade services.

After initially holding steady following the introduction of tenth pricing, EBS volumes subsequently dropped. Trading volume declined by more than 20 percent year-over-year in seven of the first eight months of 2012.

EXECUTIVE DEPARTURES

The source familiar with the EBS system said the decline in volume and complaints from big banks hastened the ouster of Dave Rutter, the EBS chief executive who oversaw decimalization. Rutter declined to be interviewed for this article.

ICAP Chief Executive Michael Spencer said at the company's annual meeting in July he fired Rutter because he "believed he was losing touch with what the customer base wanted."

A source close to Rutter said the two differed about EBS' future direction. Spencer wanted to reinforce EBS' relationship with the big banks, the source said, while Rutter believed growth in the forex world would not come from the bank-dominated dealer sector, but the mid-level market: the smaller banks and financial institutions.

An ICAP spokesperson said it disputes this, stressing that EBS is focused on providing liquidity to all market participants, not just the banks.

After Rutter left, Brian Andreyko, global head of FX product and development, and Chip DeFilippo, head of FX product management, resigned. Andreyko declined to be interviewed for this article.

DeFilippo, who is now doing consulting work, said he does not agree with EBS' new approach. "It seems to me EBS wants to get closer to the banks and I do not agree that approach would work."

EBS has gone back to quoting four decimals for many currency pairs. On major currencies, it has adopted "half-pip pricing," with the fifth decimal just showing increments of five and zero.

Mandelzis said he believes EBS could lose some high-speed trading volume with the changes but that increased trading from both banks and buy-side firms will offset the loss over time.

"We have made some decisions that could cost us over the short term, but I believe the world is ready for a material change and a new direction," he said.

(Editing by David Gaffen, Martin Howell and Steve Orlofsky)

• A Permanent Economic Emergency