August 1st, 2013. Knight Capital Group opened its trading business. With over 1.400 employees, Knight was the largest trader in U.S. equities, with a market share of 17.3% on NYSE and 16.9% on NASDAQ. For any broker, working in a place like Knight Capital would be a dream.
But today was not like any other day. At 9:00 AM, when the New York stock exchange opened for business, the first retail investor of the day placed an instruction to engage with the market. Just 45 minutes later, Knight Capital’s software had executed over 4 million transactions, losing the company $460 million and leaving it at the edge of bankruptcy.
What happened? Like most catastrophes, it was an unfortunate chain of events, one that had to do with computer code. The day before, the development team had pushed an update to their production environment. On the surface, nothing major, but the unintended bug that got deployed was a ticking time bomb.
Bugs can vary from mildly annoying to downright destructive. Sometimes they are seen as a quirk of a piece of technology, much like how the JavaScript community has accepted that its rather byzantine take on floats is part of its charm.
Other times bugs can be devastating, affecting millions of users worldwide. A case in point is the Log4j debacle that had the tech community up in arms for weeks on end.
For those who are unaware, Log4j is one of the most popular Java-based logging utilities on the market. An exploit was found that allowed third parties to execute code remotely on a target computer, allowing them to steal data, or install malware.
How massive was the issue? Akamai Technologies reported over 10 million attempts to exploit the bug per hour just in the U.S. considering that companies like Apple, Amazon, and Twitter rely on Log4j, you can start to get a picture of how sensitive this was.
But What is a Bug?
A software bug is an error or flaw in computer software that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Contrary to popular belief, a bug isn’t necessarily caused by writing bad code (although, we can’t disagree that it’s one of the main causes).
Take for example NASA’s Climate Orbiter, the $125 million project that crashed and burned on the surface of Mars. The reason? A piece of the software calculated the force that the thrusters needed to exert in pounds of force. While another read the data assuming it was on the metric system.
In isolation, each piece of code was doing what it was intended. The problem was a miscommunication, the engineering consultants from Lockheed Martin Astronautics in Colorado ran the numbers but forgot to convert them to the metric system. NASA on the other hand assumed that the calculations were in newtons per square meter as it was the standard.
Another example, Knight Capital’s bug was due to some legacy code that was never removed from their systems, one of the flags that came with the update triggered the old code, and made the software it was in a testing environment, so it tried to process as many operations as possible.
On a review of the events, the investigation found that Knight Capital didn’t have formal code reviews or a QA department for that matter. In other words, no one was assigned to check for possible errors. They didn’t have enough safeguards.
Unfortunately, QA departments, DevOps engineering, software testing, and code reviews aren’t enough to prevent bugs. At best, sometimes we can only spot bugs as they happen in production and try to patch them up as fast as possible.
The Outside Perspective
Software is developed in a very specific environment, between having to comply with user requirements, working to meet a deadline, aligning your workflow with other developers, and having to answer to last-minute changes, bugs can go unnoticed.
As the saying goes, hindsight is 20/20, reviewing code after the fact is a very different beast than doing it under pressure. Every software developer has reviewed code that they’ve done in the past and realized that they could’ve done it better. That’s easy to say when it’s not 4 AM in the morning and you have 10 hours before going to production.
Users, on the other hand, engage with our products in a different environment. They get to use it at their leisure, on their own platform. As such, it’s not uncommon for users to experience bugs.
For a user, bugs can range from quirky to outright frustrating, but what if there is a way for bugs to benefit users, the coding community, and ourselves?
Enter Bug Bounty Programs
A bug bounty program is a deal offered by many websites, organizations, and software developers by which individuals can receive recognition and compensation for reporting bugs, especially those related to security exploits.
Bug Bounty is extremely popular and is used by some of the biggest tech companies in the business including Twitter and Google. Even the U.S government rewards individuals who report security vulnerabilities on their websites.
Dozens of aggregate sites keep a list of active Bug Bounty programs and whole communities have grown around them. Why are they so popular?
Well, software development is one of those fields where a million minds think better than one. There is only so much testing a developer can do on their own, by asking the community for help, users and programmers alike can test edge cases searching for potential bugs and security exploits.
It incentivizes people with the know-how to help you safeguard your product instead of exploiting potential weaknesses. It’s a democratic effort to build better and safer software.
In fact, some young developers have managed to find positions in companies thanks to bug bounty programs. They’ve managed to show firsthand their talent and prove that they understand programming, security practices, and the software development culture.
For a company, setting up a bug bounty program is relatively easy. Once again, there are dozens of sites listing open programs and it’s just a matter of publishing your own project as well as offering a reward that aligns with the size and scope of your business.
Suffice to say, bug bounty programs aren’t QA or an alternative to something like DevSecOps, it’s a program that exists alongside your own security measures. Its full potential can only be exploited if we are already doing our best to create the best software possible.
