The boat, belonging to Hongkong Electric Co, controlled by billionaire Li Ka-shing, was taking passengers to watch fireworks in the city's Victoria Harbour on Monday when the two vessels collided near the picturesque outlying island of Lamma.

Five children were among the dead. More than 100 people were taken to hospital, with nine suffering serious injuries or in critical condition, the government said in a statement.

"We suspect that somebody did not fulfil their responsibility, that's why we made the arrests," Police Commissioner Andy Tsang said. "We do not rule out the possibility that further arrests will be made."

The arrests involved crew of both vessels.

The collision sparked a major rescue involving dive teams, helicopters and boats that saw scores plucked from the sea. A large crane on a barge was connected to the stricken boat.

"Within 10 minutes, the ship had sunk. We had to wait at least 20 minutes before we were rescued," said one male survivor, wrapped in a blanket.

Survivors said people had to break windows to swim to the surface. "We thought we were going to die. Everyone was trapped inside," said a middle-aged woman.

The fireworks marked the mid-autumn festival, when the moon is full, and China's National Day. Hong Kong returned to Chinese from British rule in 1997.

Hongkong Electric, a unit of Power Assets Holdings (0006.HK) which is controlled by Asia's richest man Li, said the boat had capacity to hold up to 200 people.

The tragedy was the worst to hit Hong Kong since 1996 when more than 40 people died in a fire in a commercial building.

"OUR CAPTAIN IS NOT WELL"

The ferry, owned by Hong Kong and Kowloon Ferry Holdings, suffered a badly damaged bow in the collision but made it safely to the pier on Lamma, an island popular with tourists and expatriates about a half-hour away from downtown Hong Kong.

Several of its roughly 100 passengers and crew were injured.

"After the accident, it was all chaos and people were crying. Then water began seeping in and the vessel began to tilt to one side and people were all told to stand on the other side and everyone started putting on life jackets," a passenger said.

Hong Kong is home to one of the world's busiest shipping lanes, but serious accidents are rare. The city is known for its high-quality public services and advanced infrastructure.

A spokeswoman for Hong Kong and Kowloon Ferry said the company was trying to assess what happened.

"Our captain is not well and we have not been able to talk to him so far," the spokeswoman told local television.

A Hong Kong Fire Services official said the search was hampered by the vessel being partly sunken, poor visibility and too much clutter. The search for survivors was continuing on Tuesday.

Teams of men in white coats, green rubber gloves and yellow helmets carried corpses off a police launch in body bags.

At one of the city's public mortuaries, around 50 grieving relatives gathered, some crying, while others were called in to identify the dead.

Hong Kong leader Leung Chun-ying visited survivors and pledged a thorough investigation. He declared three days of mourning starting on Thursday.

Thousands of Hong Kong residents live on outlying islands such as Lamma, which lies about three km (two miles) southwest of Hong Kong Island. (Additional reporting by Stefanie McIntyre, Donny Kwok, Farah Master, Venus Wu and Tyrone Siu; Writing by Anne Marie Roantree; Editing by Nick Macfie)

• A Permanent Economic Emergency