The Islamic cleric behind plans to build a mosque near Ground Zero in New York warned today that retreating on the project would only strengthen the hand of Muslim extremists around the world.

Despite the warning, Imam Feisal Abdul Rauf dodged the question when asked if he intended to keep the Islamic cultural center at its current site, two blocks from where Al-Qaeda crashed planes into the World Trade Center.

"The decisions that I will make -- that we will make -- will be predicated on what is best for everybody," he told ABC's "This Week" program.

On Saturday's ninth anniversary of the 9/11 attacks, thousands marched in dueling protests over the row, which was stoked by a Florida pastor's threat to burn the Koran to mark the occasion.

Terry Jones used his threat, which triggered demonstrations across the Muslim world, as a bargaining chip to try and get the mosque moved and flew to New York to meet the imam.

Abdul Rauf has so far refused to meet him and his initial response in a statement to CNN was: "We are not going to toy with our religion or any other. Nor are we going to barter."

The imam told ABC in the interview on Sunday that the "discourse has been, to a certain extent, hijacked by the radicals," and said this made his decision on the mosque "very difficult and very challenging.

"The radicals on both sides, the radicals in the United States and the radicals in the Muslim world, feed off each other. And to a certain extent, the attention that they've been able to get by the media has even aggravated the problem."

The mosque, to be built on the site of a derelict clothing store two blocks from Ground Zero -- the name given to the site of the downed Twin Towers -- was proposed by Rauf as a way of giving Islam a new face in the United States.

"My major concern with moving it is that the headline in the Muslim world will be Islam is under attack in America," the imam told ABC.

"This will strengthen the radicals in the Muslim world, help their recruitment," he continued. "This will put our people -- our soldiers, our troops, our embassies, our citizens -- under attack in the Muslim world and we have expanded and given and fueled terrorism."

New York Mayor Michael Bloomberg has warmly endorsed the mosque idea and President Barack Obama has repeatedly said that religious freedoms in America allow anyone to build a holy site where they please.

Polls show a majority of Americans, and some 70 percent of New Yorkers, think the mosque should be moved further from Ground Zero, not least because of the sensitivities of those who lost loved-ones in the 9/11 attacks.

Opposition to the mosque has developed into a cause celebre for ultra-conservative radio hosts and Tea Party activists dedicated to battling Obama and what they denounce as his liberal, "socialist" agenda.

The row was further inflamed by pastor Terry Jones and the threat of his Dove World Outreach Center -- a tiny church in the town of Gainesville with a congregation of less than 50 -- to burn 200 Korans on Saturday.

Abdul Rauf said that if the Koran burning plan had been implemented, it would have been "a disaster" in the Muslim world.

"It would have strengthened the radicals," he said. "It would have enhanced the possibility of terrorist acts against America and American interests."

The imam described a "growing Islamophobia" in America, but said that contrary to the claims of radical Islam, Muslims in the United States were free to observe their religion, happy and thriving.

Abdul Rauf said the way America was treating its Muslims was being watched by over a billion Muslims worldwide and that the main ideological battleground today was not between Islam and the West.

"The battleground has been: moderates of all faith traditions in all the countries of the world against the radicals of all faith traditions in all parts of the world."

Moving ceremonies were held Saturday to honor the nearly 3,000 people killed when Al-Qaeda extremists slammed airliners into New York's World Trade Center, the Pentagon outside Washington and a field in Pennsylvania

Obama told a deeply polarized America that Islam is not the enemy as somber ceremonies marked an unusually tense ninth anniversary of the September 11 attacks.

"As Americans, we will not and never will be at war with Islam. It was not a religion that attacked us that September day. It was Al-Qaeda, a sorry band of men, which perverts religion," he said.