Bug Bounty Programs: Are You Ready? - part 2

In Part 1 of “Bug Bounty Programs: Are You Ready?” we examined the growth of commercial bug bounty programs and what organizations need to do before investing in and launching their own bug bounty. In this part we’ll discuss why an organization needs to launch a bug bounty program, and what limits the value they will likely extract from such an investment.

Down the Bug Bounty Path

As organizations contemplate bug bounty programs, it is important that they understand what can be achieved. Since the creation of commercial bug bounty management platforms, a lot of attention has focused upon the “many eyes” and “global talent pool” pitch – implying that these new platforms enable more security researchers and bug hunters from around the world to disclose their latest findings direct to the organization behind the bug bounty.

Contributions to bug bounty programs are, unfortunately, much more nuanced than that. Over the last three years, largely due to the increasing number of bug bounty programs and growth of commercial bounty platforms, it can be argued that such programs are returning less security value than a decade ago – and that the value extracted from bug bounty programs will continue to fall going forward.

The primary reasons for launching a bug bounty program should be:

Public Relations
Irresponsible public disclosure protection
Economically managing bug submissions

The primary reason an organization should be considering the creation of a bug bounty program is for media and public relations reasons. A bug bounty program is generally perceived as an invitation to the security research community and that the organization is not antagonistic and unlikely to pursue legal reprisal for disclosures of bugs.

This places those researchers at odds with the community if they choose to publicly disclose or sell vulnerabilities that were not initially disclosed through the bug bounty program. Should a researcher ever disclose a vulnerability publicly independent of the program, the affected organization can comfortably address the public that they had followed best security recommendations – and that the discloser is the villain.

Most organizations traditionally struggle to manage the third-party submission of bugs and security vulnerability disclosures. Well planned and operated bug bounty programs facilitate the triaging, remediation, and public disclosure aspects of inbound submissions. The primary value of commercial bug bounty platforms assuredly lies in their ability to workflow the disclosure process, track communications, and coordinate payments to the bug hunter.

Expectations versus Reality

It is likely that many companies approach bug bounty programs with an expectation of reducing the per-bug cost of discovery. In essence, there is a belief that bug bounties are a cheap and effective way of uncovering embarrassing or critical security flaws in an organizations Internet facing systems.

Purveyors of bug bounty platforms tend to extol the virtues of crowdsourced vulnerability discovery. The premise being that there are tens-of-thousands of software engineers, security researchers, and hackers online that have the necessary skills, time, and motivation to probe and investigate an online system or asset and – if appropriately incentivized – report their findings back to the bug bounty program owner.

One perspective on the likelihood of a bug bounty program identifying meaningful security flaws and exploitable bugs for a company is to examine the kind of researcher that may participate in the campaign:

Newbies and Trainees. Companies that offer bug bounties are perceived to be friendly platforms for people new to the security industry to practice and improve their hacking skills – so are actively encouraged by mentors and veterans to participate in bug bounties. These people will predominantly use freely available tools and book-led discovery methodologies. Probability of finding an unknown security flaw: Low to Nil Probability of reporting their findings: 5-10% Pros: Low expectations of bounty reward Cons: Discoveries largely limited to vulnerability scanning results. Poor communications of discoveries and missing security context or proof-of-concept validation mechanism.
Millennial Hacker. Budding and wannabe hackers, uncertain of which color hat they want to wear longer-term, and a desire to test and refine their skills and attack tools – with the expectation of some form of compensation for findings. Most have 1-3 years of web programming experience and have a solid understanding of hacker techniques. Probability of finding an unknown security flaw: Low Probability of reporting their findings: 50-75% Pros: Good representation of average hacker and script-kiddie capabilities Cons: Tests often drift “out of scope”. Enthusiasm for a discovery may trump actual risk. Low tolerance for delays in recognizing a finding.
2nd/3rd World Security Researcher or Consultant. Many experienced security consultants and researchers located in 2nd/3rd world countries have found they can not only supplement their income, but can greatly increase their monthly salary by participating in bug bounty programs. These people tend to have 3-5yrs of industry experience, often use manual testing processes, and may dedicate many days to each bug bounty site. Probability of finding an unknown security flaw: Low Probability of reporting their findings: 75-100% (if financial reward is available) Pros: Experienced with a consulting background. Can be incentivized with lower financial payments. Cons: Not typically interested in bounties without cash rewards.
Professional Penetration Tester. Many professional penetration testers find that they can supplement their consultancy income and refine their skills against large sites that offer interesting or sizable bounties. Participation is bug bounty programs is often limited in time – often 2-4 hours, two to three days per week. Many participants have developed specialized skills or discovery tools and seek to apply that focused bug hunting technique to multiple sites. Probability of finding an unknown security flaw: Low to Medium Probability of reporting their findings: 10-25% (if the bug is serious enough) Pros: Experienced with a consulting background. Easy to communicate and validate bug findings. Cons: Part-time at-will participation. Only bugs deemed to be “interesting” or profitable will be disclosed. May have conflicts of interest with full-time employer.
Professional Bug Hunter. A growing number of professional, full-time, bug hunters from around the world pursue bug bounties as their primary source of income. Their experience and depth of discovery vary considerably, but will often expend many days of continuous effort to uncover and submit bugs. For many bug hunting is also a competitive sport – and are often motivated to achieve high status on bug bounty leaderboards. Probability of finding an unknown security flaw: High Probability of reporting their findings: 90-100% Pros: Competitive bug bounty reporters. Will often report every bug or flaw they uncover – no matter the criticality of the finding Cons: Often impatient to be rewarded or recognized for their contributions. High volume of unverified bug findings.

Organizations that offer bug bounty programs typically prefer the attention of the last three categories of bug contributor. Since a financial reward is often a key element for these workforces, any offered bounties need to be both competitive and likely to yield findings.

Paying the most for bugs does not necessarily mean attracting the best bug hunters. Any experienced security researcher will know that the probability of finding new bugs is likely higher in a company or web site that has not previously participated in a bug bounty program – so they may “skip” the harder (highest paying) sites for other softer targets that will net more financially or facilitate leaderboard advancement.

Depth of bug pursuit

Understanding the differences between bug bounty rewards and the probability of a bug hunter reaping rewards is an important ingredient in tuning a bounty program and enticing the necessary hunting expertise.

Organizations should also consider the technical depth underpinning bug discovery and the effort being expended to enumerate those bugs. Having a large pool of “many eyes” hacking away at an online site or product does not confer depth or breadth of security coverage.

The vast majority of bug bounty participants will initially use automated vulnerability scanners – and many will not undertake investigations far removed from the capabilities of good commercial vulnerability scanning products. As such, it is highly recommended that any organization contemplating launching a bug bounty program regularly scan their assets with multiple vulnerability scanners prior to (and during) a bug bounty program. The use of such scanning technology will typically uncover 95-99% of all the vulnerabilities most bug hunters are likely to discover when they use their own customized tools.

It is important that all known vulnerabilities deemed to be “acceptable risk” differences to commonly promoted best security practices are documented in advance and noted for exclusion from the bug bounty program. For example, it may be acceptable that the application serves a robots.txt file that disallows web crawlers from indexing an administrative directory. Every bug hunter using a vulnerability scanner will identify the existence of the file and likely file a security bug related to the enumeration of the admin directory – with some hunters then demanding a bounty on it. Do you want to handle the response a hundred submissions of this single “bug”?

In Part 3 we’ll look at the crystal ball. Managed vulnerability scanning and regular penetration testing form the basis of vulnerability management and certification today. Can bug bounty programs and platform providers close the gap on vulnerability management and usurp the commercial penetration testing market, or is this all just a flash in the pan?

-- Gunter Ollmann, Founder/Principal @ Ablative Security