Bug Bounty Programs: Are You Ready? - part 3

The Bug Bounty movement grew out a desire to recognize independent security researcher efforts in finding and disclosing bugs to the vendor. Over time the movement split into those that demanded to be compensated for the bugs they found and third-party organizations that sought to capitalize on intercepting knowledge of bugs before alerting the vulnerable vendor. Today, on a different front, new businesses have sprouted to manage bug bounties on behalf of a growing number of organizations new to the vulnerability disclosure space.

Looking forward, given the history and limitations of bug bounty operations described in parts 1 and 2 of this blog series, what does the future hold?

The Penetration Testing Business

The paid bug bounty movement has been, and continues to be, a friction point with the commercial penetration testing business model.

Since penetration testing is mostly a consultant-led exercise (excluding managed vulnerability scanning programs from the discussion for now), consumers of penetration testing services effectively pay for time and materials – and what’s inside the consultants heads. Meanwhile, contributors to bug bounty programs are paid per discovery – independent of how much time and effort the researcher expended to find the bug.

Initially many commercial penetration testing companies saw bug bounty programs as a threat to their business model. Some organizations tied to adapt, offering their own bug bounty programs to their clients, using “bench time” (i.e. non-billable consultancy hours) to participate in third-party bug bounties and generate revenue that way, or sought collaboration with the commercial bug bounty operators by picking-up the costly bug triaging work.

Most of the early fears by penetration testing companies were ill founded. The demand for compliance validation and system certification has grown faster than any “erosion” of business due to bug bounties, and clients have largely increased their security spend to fund bug bounty programs rather than siphon from an existing penetration testing budget.

While the penetration testing market continues to grow, it is perhaps important to understand the future effect on the talent pool from which both that and bug bounty industry must pull from.

There are several constraints that will influence the future of bug bounty and penetration testing businesses. These include:

The global pool of good and great bug hunters is finite (likely limited to less than 5,000 people worldwide in 2017). Both industries need to tap this pool in order to be successful in finding bugs and security vulnerabilities that cannot be found via automated tools.
Advances in automated code checking, adherence and enforcement of the SDL (Secure Development Lifecycle), adoption of DevOps and SecDevOps automation, and more secure software development frameworks, are resulting in less bugs making it to public release – and those bugs that do make it tend to be more complex and require more effort to uncover.
The growing adoption and advancement of managed vulnerability scanning services. Most tools used by bug hunters are already enveloped in the scanning platforms used by managed services providers – meaning that up to 95% of commonly reported bugs in web applications are easily discovered through automated scanning tools. As security researchers identify and publish new attack and exploitation vectors, tools are improved to identify these new vectors and added to the scanning platforms. Over time the gap between automated tool and bug hunter is closing – requiring bug hunters to become ever more specialized.

It is possible to argue that the growth and popularity of bug bounty programs is a direct response to often poorly scoped, negligently executed, and over-priced penetration testing. As many penetration testing service lines (and levels) became commercialized and competition subsequently drove down day-rates, providers were apt to use lesser-qualified and inexperienced consultants on client engagements. This resulted in reductions in breadth and depth of the bug hunting – resulting in a higher proportion of embarrassing bugs being discovered by independent third-parties.

Penetration testing companies are still well placed to combat day-rate erosion by bug bounty programs in the future if they:

Work closely with managed vulnerability scanning providers to automatically uncover the “low hanging fruit” and focus consulting efforts on hunting for bugs and logic flaws not easily uncovered through automated tools.
Provide a warrantee their work – ensuring confidence in penetration testing coverage with a verifiable and provable methodology. By offering a warrantee against each major threat category tested and documenting the agreed scope of testing, the penetration testing company warrants that it fully tested the areas within scope. If, at some later time, a bug is uncovered out of the scope of the testing, it would be the client’s omission.
Enable clients to easily view and vet the credentials, skills, and expertise of the consultants working on the penetration testing engagement. Clients gain visibility over who is actually conducting the penetration testing and their capabilities in hunting for bugs in a particular technology – removing the prospect of overpaying for inexperienced or unqualified consultant substitution.
Embrace the “one throat to choke” CISO strategy. As corporate security officers face greater public scrutiny, they must increasingly be able to point to suppliers of security services and hold them accountable for lapses in service quality. It is impossible for bug bounty programs (and providers) to be held accountable for missed bugs.

Managed Vulnerability Scanning

Managed vulnerability scanning represents the largest threat to the traditional penetration testing and newer bug bounty program industries.

Over the last few years commercial vulnerability scanning platforms (leveraging an increasingly wide variety of automated discovery and monitoring tools) have closed many of the major gaps in the bug hunting world. More to the point, as programs have shifted from irregular one-off scans into continual scanning platforms supplemented by third-line experts tuning tests for individual environments, the cost of uncovering new bugs has fallen substantially.

A half-decade ago, a typical penetration test of a substantial e-commerce site would cost $25k-$100k and yield a few hundred vulnerabilities (with a typical spectrum of findings being classified as 50% low risk, 25% medium risk, 20% high risk, and 5% critical risk). Using a current generation managed service-delivery vulnerability scan would likely identify 95% of all these vulnerabilities – the missing bugs likely to be tied to application logic flaws in the medium or high risk threat range. Continual scanning of such a site today may cost as little as $200 per month (repetitively scanned every few days).

As more organizations have purchased managed vulnerability scanning services they have simultaneously noted the reduced number of bugs and vulnerabilities reported through bug bounty and penetration testing programs.

Pressures on Bug Bounty Platforms

Bug bounty providers will continue to facing growing pressures from penetration testing companies and managed vulnerability scanning service providers, and will be forced to evolve in order to provide value. Today’s model of merely building and running a platform for managing the coordination of bugs and payments is not sufficient as the novelty wears off and the operational economics become more defined.

Key pressures include:

As bugs get harder to find, bug bounty contributors with sufficient skills will likely switch to “time and materials” compensation models – and be enticed to (re)join security consulting companies. Thereby reducing the pool of experienced bug hunters the platform can tap.
As more companies adopt bug bounty programs, skilled bug hunters find themselves thinly applied across a broader range of bug bounties. This is likely to result in less bugs per bounty being uncovered and a resultant pressure on participant companies to increase bounty rewards to gather the most bug hunter eyes. This in turn will make the economics of bug bounties much harder to justify.
Triaging of submitted bugs has proven to be costlier than expected. The time and effort required to review each bug submission has resulted in costly overheads to the bug bounty business. As the pool of expert bug hunters decrease (assuming replacement with less skilled or experienced bug hunters) the number of false positive, poorly documented, and unverifiable bug submissions will increase – likely requiring more efforts and expenditure on the triaging front.
As penetration testing companies and managed vulnerability scanning service providers adopt and promote warrantees of bug discovery coverage, bug bounty programs will increasingly be relegated to marketing and PR projects by clients – reducing funding for bug bounty payments further.

Some bug bounty platform providers have already begun a shift away from discovery payments and into a “time and materials” model.

Having identified the most prolific and skilled bug bounty researchers submitting to their platforms, they offer clients access to a verifiable or accredited subset of their best bug hunters. For their best bug hunters, they offer “first look” programs and higher bounties on some client programs, or provide subcontracting options as they seek to redefine “penetration testing”.

The Future of Bug Bounties

Bug bounties and bug bounty platform providers face different futures.

The changes initiated by bug bounties were predominantly brought about by poor penetration testing practices and execution, and an immaturity in the managed vulnerability scanning market. Both respective security offerings have responded to close many of the gaps – with plans on closing more in the future.

While bug bounties will remain popular for the immediate future – driven increasingly by their RP and marketing repercussions rather than overall contributions to corporate security – the payments for bug discovery are unlikely to affect the penetration testing or managed security services markets in the future.

Bug bounty platform providers are in a fragile position and will be forced to either reinvent their platforms (e.g. as a triage management and bug tracking system for developers), or extend their platforms in to embrace the service offerings of competitor spaces (i.e. managed vulnerability scanning and penetration testing) to survive. The easiest transition will be towards consultant-driven services (e.g. refactoring bug bounty researchers as billable consultants); but existing penetration testing companies are well positioned to combat this threat to day-rate erosion.

Since many of the bug bounty platform providers were recently conceived in a SaaS world, they have a natural head start in cloud service delivery and are likely better positioned to compete against the older managed vulnerability scanning service providers – especially those that add consultants to provide the tier-two and tier-three technical support to customers.

If the bug bounty platform providers are able to entice their best bug hunters in to co-developing (or licensing) the specialized scripts and custom tooling they use to uncover cross-system bugs, and bring to market a platform that automatically discovers the remaining 1-5% of bugs not uncovered by existing automated vulnerability discovery tooling, they may find new recurring revenue options with their existing customer base.

-- Gunter Ollmann, Founder/Principal @ Ablative Security