Buckling under the US pressure, Pakistan's Inter-Services Intelligence agency asked the Taliban to withdraw from Buner on Friday, according to a media report here on Sunday.
The Sunday Times said that the US got tough with Pakistan as the terrorists moved to within 100 km from Islamabad.
America made clear that it would attack Taliban forces in their Swat valley stronghold unless the Pakistan government stopped the militants' advances towards Islamabad, the report said.
Quoting a senior Pakistani official, it said the Obama administration intervened after Taliban forces expanded from Swat into the adjacent district of Buner.
The Pakistani Taliban's inroads raised international concern, particularly in Washington, where officials feared that the nuclear-armed country which is pivotal to the US war against the Taliban in Afghanistan and against al-Qaeda, was rapidly succumbing to Islamist extremists.
"The implicit threat -- if you don't do it, we may have to -- was always there," a Pakistani official said. He said that under American pressure, Pakistan's ISI agency told the Taliban to withdraw from Buner on Friday.
However, reports on Saturday indicated that the Taliban withdrawal was less than total. Thousands of people in the district were still at the mercy of armed militants and their restrictive interpretation of Islamic law.
According to the report, the Taliban stole livestock, took vehicles belonging to government officials and ransacked the offices of some local non-government organisations. In a phone call, Hardline commander, Maulvi Khalil denied that the Taliban were terrorists.
He said: "We've raised the arms to spread the message of Allah. This is the responsibility of each and every Muslim." But residents fear it is just a matter of time before their daughters are forced to marry Taliban commanders, a process that has already begun in Swat along with public floggings, the report said.
On Friday, in a much publicised agreement with the government, Khalil agreed to withdraw. Local residents said the withdrawal was incomplete. He had left men behind to supplement local armed Taliban groups and newly recruited sympathisers.
"There is a collective holding of breath," said Sam Zarifi, Amnesty International Asia director from Islamabad. "The Taliban edicts are still in force and the dismantling of the civilian infrastructure is still very much in effect, so a lot of doctors, midwives, civil servants have left and people are hunkering down because they fear an army operation."
The government sent a few hundred paramilitaries to Buner last week but they kept a low profile. It has not sent any troops. Americans want the government to shift troops from the Indo-Pak border to meet the Taliban threat.
The residents of Buner, however, fear civilian casualties in an event of army operation against the militants.