As the damaged BP oil well continues to spew millions of gallons of crude from the depths of the floor of the Gulf of Mexico, the immediate challenge is how to mitigate an ever-magnifying environmental catastrophe. One can only hope that the spill will be contained soon, and that the ever-darkening worst-case scenarios will not materialise.

The disaster, however, poses a much deeper challenge to how modern societies deal with regulating complex technologies. The accelerating speed of innovation seems to be outstripping government regulators' capacity to deal with risks, much less anticipate them.

The parallels between the oil spill and the recent financial crisis are all too painful: the promise of innovation, unfathomable complexity and lack of transparency (scientists estimate that we know only a very small fraction of what goes on at the oceans' depths.) Wealthy and politically powerful lobbies put enormous pressure on even the most robust governance structures. It is a huge embarrassment for US President Barack Obama that he proposed - admittedly under pressure from the Republican opposition - to expand offshore oil drilling greatly just before the BP catastrophe struck.

The oil technology story, like the one for exotic financial instruments, was very compelling and seductive. Oil executives bragged that they could drill a couple of kilometers down, then a kilometer across, and hit their target within a few meters. Suddenly, instead of a world of "peak oil" with ever-depleting resources, technology offered the promise of extending supplies for another generation.

Western officials were also swayed by concerns about the stability of supplies in the Middle East, which accounts for a large proportion of the world's proven reserves. Some developing countries, most notably Brazil, have discovered huge potential offshore riches.

Now all bets are off. In the United States, offshore drilling seems set to go the way of nuclear power, with new projects being shelved for decades.

And, as is often the case, a crisis in one country may go global, with many other countries radically scaling back off-shore and out-of-bounds projects.

Will Brazil really risk its spectacular coastline for oil, now that everyone has been reminded of what can happen? What about Nigeria, where other risks are amplified by civil strife?

Oil experts argue that offshore drilling never had the potential to amount to more than a small share of global supply. But there now will be greater concerns about deep drilling in any sensitive environment. And the problem is not just with oil. The big news in energy these days is the revolution in technology for tapping shale gas. With important reserves near populated areas, governments will need to temper their enthusiasm and think about the balance between risks and riches.

The basic problem of complexity, technology and regulation extends to many other areas of modern life.

Nanotechnology and innovation in developing artificial organisms offer a huge potential boon to mankind, promising development of new materials, medicines, and treatment techniques. Yet, with all of these exciting technologies, it is extremely difficult to strike a balance between managing "tail risk" - a very small risk of a very large disaster - and supporting innovation.

Financial crises are almost comforting by comparison. Speculative bubbles and banking crises have been a regular feature of the economic landscape for centuries. Awful as they are, societies survive them.

True, people who thought, "This time is different," before the recent Great Recession were proven wrong. But, even if we are not getting any better at dealing with financial crises, things have not necessarily been getting worse, either.

Perhaps the G-20 government leaders have not done quite as brilliant a job plugging the hole in the financial system as they claim. The raging sovereign-debt problems in continental Europe, and the brewing ones in the US, Japan, and elsewhere are proof enough of that. But, compared to British Petroleum's efforts to plug its deep-sea oil hole, the G-20 leaders look omnipotent.

If ever there were a wake-up call for Western society to rethink its dependence on ever-accelerating technological innovation for ever-expanding fuel consumption, surely the BP oil spill should be it. Even China, with its "boom now, deal with the environment later" strategy should be taking a hard look at the Gulf of Mexico.

Economics teaches us that when there is huge uncertainty about catastrophic risks, it is dangerous to rely too much on the price mechanism to get incentives right. Unfortunately, economists know much less about how to adapt regulation over time to complex systems with constantly evolving risks, much less how to design regulatory resilient institutions.

Until these problems are better understood, we may be doomed to a world of regulation that perpetually overshoots or undershoots its goals.

The finance industry already is warning that new regulation may overshoot - that is, have the unintended effect of sharply impeding growth. Now, we may soon face the same concerns over energy policy, and not just for oil.

Given the huge financial stakes involved, achieving global consensus will be difficult, as the Copenhagen climate-change fiasco proved.

The advanced countries, which can best afford to restrain long-term growth, must lead by example.

The balance of technology, complexity, and regulation is without doubt one of the greatest challenges that the world must face in the 21st century.

We can ill afford to keep getting it wrong.

The author is Professor of Economics and Public Policy at Harvard University, and was formerly chief economist at the IMF.

© Project Syndicate, 2010, www.project-syndicate.org.