At the height of an investigation into a group of Islamists plotting Al-Qaeda-inspired bomb attacks across Britain in 2004, British spies analysed more than 4,000 telephone contacts to build up a picture of what they were planning and with whom.

The police and intelligence agencies have fallen behind those they are trying to track

The security services say the information was crucial in helping to thwart what could have been one of the deadliest attacks on Britain and to bring the cell to justice.

But a decade on, the police and intelligence agencies warn they have fallen behind those they are trying to track, as advances in technology and the growth of services like Skype and Facebook, increasingly put criminals beyond their reach.

In response, Britain is seeking to bring in what critics say are the West’s most far-reaching surveillance laws that could change the international landscape in this area. The proposals would force communications firms to collect and store vast reams of data about almost every click of British online activity.

By doing so, ministers have provoked the wrath of human rights campaigners, sown division within the coalition government and alarmed major corporations such as Google and Microsoft.

“Nobody wants to live in a tyranny. I certainly don’t and I don’t want people snooping on what I do,” said Gary Beautridge, the lead chief British police officer on the issue.

“This is about maintaining capability. It’s not increasing capability, it’s maintaining it in the face of change in technology,” he said, rejecting talk of an Orwellian scheme.

Beautridge and all those involved in law enforcement say they are now unable to see about 25 per cent of all communications data, hindering the secret war against bomb plotters, drug lords and paedophiles.

Almost everyone, from law-makers to privacy campaigners, accepts something needs to be done. But trying to find a solution that is technically possible, will not cost billions and is not overly intrusive is proving a challenge.

Politicians across the world are grappling with the same problem but privacy campaigners say Britain is going further than any other democratic state.