Thirty minutes into the crippling outage that hobbled the Nasdaq stock market on Thursday afternoon, stopping all trading in $5.9 trillion worth of US equities, exchange officials had the problem fixed.
It was another two-and-a-half hours, however, before they were ready to flip the switch and turn the all-electronic market back on.
Most of the 191 minutes that the exchange was dark was spent in sometimes frantic conversation with scores of banks, brokers, investment companies and rival exchanges who wanted Nasdaq’s assurance that a restoration of trading would be orderly and would not lead to panic.
“We had to make sure all the exchanges were connected to us successfully and if the firms on the outside could get in,” said one Nasdaq official who spoke on condition of anonymity.
Meanwhile, banks’ trading desks were cautioning Nasdaq, operated by Nasdaq OMX Group Inc, not to rush to reopen, fearing that a restart full of technical errors would only sap more confidence from rattled markets, according to three sources at brokerages and banks who declined to be identified.
In the end, the reopening of trading did go relatively well. Transactions first restarted at 3pm EDT and 25 minutes later the rest of the market opened, and, according to a Nasdaq statement “the trading day finished in normal course”.
Shares of Nasdaq itself, which initially fell by more than five per cent when trading resumed, recovered some of their lost ground to close the day 3.4 per cent lower. The widely tracked Nasdaq Composite Index gained nearly 1.1 per cent.
While the worst case scenarios may have been averted, the outage still stands as among the most serious in a series of recent technological failures to hit the US securities business, including a software issue at the Chicago Board Options Exchange this spring that delayed the start of trading there for half a day.
It was also the latest black eye for Nasdaq, which in May agreed to pay $10 million, the largest penalty ever against a stock exchange, to settle SEC civil charges over its mishandling of Facebook’s initial public offering in 2012.
Late Thursday Nasdaq identified the problem as a “connectivity issue between an exchange participant and the SIP,” or Securities Information Processor -– essentially the system that receives all traffic on quotes and orders for stocks on the exchange.
This problem “led to degradation in the ability of the SIP to disseminate consolidated quotes and trades,” Nasdaq said in a statement. “The cause of the issue has been identified and addressed.”
Whether it has been addressed to the satisfaction of regulators is another question. US Securities and Exchange Commission chair Mary Jo White called for a meeting of Wall Street leaders to help insure the “continuous and orderly” functioning of securities markets.
The incident “should reinforce our collective commitment to addressing technological vulnerabilities of exchanges and other market participants,” she said.
Thursday’s outage could well give White fodder to press ahead with new rules, proposed in March, that would hold exchanges, clearing agencies and certain “dark pool” trading venues more accountable for taking steps to prevent potential systems disruptions.
The rules, if adopted, would replace the current regulatory model in which exchanges rely on voluntary guidance known as “Automation Review Policies” to address security and stability issues with their systems.
The SEC rushed to roll out its proposal as a direct response to several high-profile software problems last year, including Nasdaq’s debacle with the Facebook IPO and a near-collapse at Knight Capital, a major Nasdaq market maker, as well as the two-day shutdown of the US equity market due to Superstorm Sandy.
Exchanges, including Nasdaq and rival NYSE Euronext, parent of the New York Stock Exchange, pushed back, citing a number of concerns, including costs and an “unduly broad” requirement to disseminate information to member firms about certain incidents.