MOGADISHU, Somalia — Islamist fighters who officials feared would attack famine victims pulled out from many bases in the capital overnight, a move that could speed up the delivery of humanitarian services, the prime minister said Saturday. However, a spokesman for the African Union peacekeeping force said the militants had blended into the civilian population, making them even harder to fight. And the al-Qaida-linked group insisted it was merely a tactical withdrawal to launch a counterattack. "We shall fight the enemy wherever they are," al-Shabab spokesman Ali Mohamed Rage told a local radio station. Somali Prime Minister Abdiweli Mohamed Ali estimated the militants now have vacated 90 percent of the capital. The AU had said last week that militants had left 60 percent of Mogadishu. Ali said the government wants to send security forces into the new areas vacated by the al-Qaida-linked militants, describing the withdrawal as the "first phase of the new war." Al-Shabab fighters have blocked many aid organizations from the south and have complicated efforts to help those in Mogadishu. More than 29,000 children under the age of 5 have died in the last 90 days in the country's south alone, according to U.S. estimates. Residents reported al-Shabab militia leaving their positions overnight but it was not clear if they had left the city. Since it was born from the ruins of another radical Islamist group in 2007, al-Shabab has never abandoned Mogadishu entirely. Mohammed Ali said he saw about 150 al-Shabab fighters leaving the northwest part of the capital. He said they may have left town due to a lack of finances and disagreements between top leaders, explaining that he had an insight into the militia because "our brothers are that side." Lt. Col. Paddy Ankunda, a spokesman for the 9,000 African Union peacekeeping forces in Mogadishu, said the al-Shabab has melted into the population and will become more difficult to deal with. "We need more troops now than ever before the area has become too big for the force to cover," Ankunda said. Sodio Omar Hassan, who was seeking treatment for her child's malaria at a hospital set up by African Union peacekeepers, said people are angry at al-Shabab's response to the relief effort. She said militia groups declined to grant the U.N. permission to distribute maize and cooking oil in a territory it controls. "People are angry now they are dying," she said. "They (al-Shabab) don't bring us anything." More than 12 million people in the Horn of Africa are in need of immediate food aid but the crisis has been exacerbated by al-Shabab's unwillingness to allow many aid organizations into the areas it controls, saying it is better to starve than accept help from Western countries. The U.N. says 640,000 children are acutely malnourished in Somalia, where the U.N. has declared five famine zones, including the refugee camps of Mogadishu. Somalis who have fled the famine zones and reached Mogadishu told The Associated Press that militants are threatening refugees who leave the south and often stopping — and sometimes killing — the men, leading to a disproportionate number of women and children at camps for displaced people in the capital. Somalia has been mired in war and anarchy for two decades, and piracy flourishes off its coastline. In a sign of how desperate the famine has become, many Somalis have fled from rural areas to Mogadishu, a war zone where AU peacekeepers have been battling the al-Shabab militants daily.