August 1, 2012. 8:01 a.m.
An internal system at Knight Capital starts sending emails. Each one says "Power Peg disabled."
By the time the market opens at 9:30, 97 of them have arrived. Nobody acts on them.
Knight was based in Jersey City. That's where I'm writing this from.
THE SETUP
Knight handled about 10% of all US stock trading.
NYSE was starting a new program that morning, so Knight updated the software that sends its orders to the market. The new code went onto 8 servers over several days. A technician missed one.
Nobody checked. Nothing in Knight's process said anyone had to.
The new code also reused an old switch. On the missed server, that switch turned on Power Peg, a feature Knight stopped using in 2003 but never deleted. That's what the emails were about.
45 MINUTES
At 9:30, the eighth server started sending orders and didn't stop.
212 small customer orders turned into more than 4 million trades in 154 stocks.
The team tried a fix. They removed the new code from the 7 servers that were working. That made it worse.
It took about 45 minutes to stop. The loss was more than $460 million.
Five days later, six firms rescued Knight with $400 million and took over 70% of the company.
WHAT ACTUALLY BROKE
The code did exactly what it was told. What failed was the process around it.
The SEC's report lists what was missing. No second person checked the update. Nobody owned the warning emails. Nobody had a plan for stopping the system.
MY OWN 97 EMAILS
I receive so much cold outbound from other companies that I've had to start using Superhuman to tag all my incoming mail. Now I have a Pitches folder with over 1,000 cold emails.
We have manual steps too. In our Success team, customers on custom plans get a usage and billing report every month. For one customer, we've had to send it 3 years straight.
What we got right: our automated alerts go to a Slack channel with an owner. Their job is to find out why an alert was triggered. Was it a false alarm? If it was genuine, what was the root cause?
Knight had 97 warnings and nobody whose job it was to read them.
ONE ASK
What's one alert your team gets every day that nobody reads?
Hit reply. One line is plenty.
Rushabh