Bugcrowd - is crowd-sourced security testing a good idea?

"We use the power of the crowd to find and eliminate security vulnerabilities"

Bugcrowd offers managed "bug bounty" programs for businesses... but is crowd-sourced security testing actually a good idea?

First, let's take a look at the registration screen.

"A steady stream of new targets to hone your skills"

Put another way... " don't focus on one target, try as many as possible while you learn the trade"

Security testing is a serious business. It requires absolute focus by a team of experts; experts who have already "honed their skills" sufficiently to demand the rates which penetration testers charge.

"Cash prizes for bugs"

So how much are we talking? Let's look at Bugcrowd's intro on StartMate.

"Those who come up with the most creative or potentially serious security bugs will win the bounty and share in a pool of prize money — $5,000 for this current competition"
http://www.startmate.com.au/introducing-bugcrowd

A share of $5,000! Sounds good... until you actually think about it logically.

$5000 is peanuts to a real security/penetration tester - and the more people involved in the bounty, the more diluted the share. No security tester worth his/her salt would even begin a pen test for such a small amount, even double that amount. It's far more likely to attract script kiddies and newbies. Of course, it's entirely possible for anyone to stumble on a serious bug - but as is usually the case, these script kiddies know just enough to be dangerous - but understanding the mechanics of what's gone wrong requires a professional and (perhaps more importantly) an accountable penetration tester.

"Build your resume with Bugcrowd Kudos points"

Nobody that works in a professional capacity wants to gain "kudos"... they certainly don't hang their "targets" on a metaphorical wall as proof of their achievements. Why? They don't need to.

"There's no obligation or minimum level of participation"

... no code of conduct or non-disclosure agreement either. Basically - it's a free for all with absolutely no accountability.

"... complete your tester profile. It takes about 30 seconds"

So anyone from any walk of life can become a tester. It doesn't matter if you've been tried & convicted of fraud, deception, extortion or have absolutely no talent whatsoever... just sign up and have a play. Just think... you can do away with your costly HR team and employ everyone on the same basis.

Are the risks even greater WITH crowd-sourced penetration testing? #responsibledisclosure

At first glance - it's sounds like a great idea. Leverage millions of like-minded people with varying degrees of talent to test your infrastructure; but what's the real cost?

Let's say your crowd-sourced "penetration tester" finds a serious problem (database design, bespoke crypto, confidential personal information) - they have absolutely no obligation (as mentioned earlier) to hand over that crucial information. It could be used to compromise your business, or be sold to your nearest competitor... or worse, end up on the black market.

Real penetration testers (the ethical, responsible ones at least!) adhere to a code of conduct, both written and understood. We use safe/responsible disclosure to both raise awareness and allow the issue to be fixed; all without exposing the company to unnecessary risk. Those looking to make quick $$$ are likely to sell to the highest bidder. That $5000 doesn't sound so great now...

You could argue that any system is open to attack - with or without Bugcrowd programs. You'd be right... the only difference being a legal one.

If someone infiltrates your network illegally - through a 0-day attack or failure in your own code - there's legal redress for the company involved. The person/persons involved can be prosecuted, if found.

By crowd-sourcing your penetration testers without a contract/rules, it could be seen as condoning & promoting this type of activity. All well and good if they inform you of the findings, but without a contract - you're exposing yourself (and your clients) to massive risk. That $5000 might be the worse 5k you've ever spent.

Ah yes, the legal issues...

Here's an email from Bugcrowd this morning.

Ignoring the pitiful EUR10,000 bounty for a moment, there's a serious mistake in the email. Have you spotted it?

Oops! This embarrassing mistake really proves my point. It's pretty vital to ensure you get the right target before you start testing; unless you're a fan of being in court.

Will you actually get paid?

Your guess is as good as mine. Again, there's no contract or limit to how many people can participate (to my knowledge). If a vendor claims it's already been reported, you often don't get paid a penny. If you're the only one to find it, your share depends on how many people were involved. Could be $5, who knows.

But crowd-sourced penetration testing works - Google/Facebook prove that!

Yep. The difference... they all have strict rules, minimum payouts and aren't time-limited. If, as a penetration tester, you stumble on an SQLi bug - you're rewarded accordingly. It's not a "job" or a "target". You also report directly to the company responsible, not an unknown 3rd party. Has Bugcrowd's own infrastructure been tested to ensure any attack vectors are stored safely? More importantly, was the penetration test crowd-sourced or carried out by professionals? Point is... we don't know. Assume the worse, hope for the best.

Even so, these bug bounty programs do not replace rigorous & expensive testing by professionals.

"When you're paying for time, people will take their time"

Source: http://www.startmate.com.au/introducing-bugcrowd

But is faster better? It certainly isn't from a security standpoint (cryptography or otherwise). Personally, I'd want a penetration tester to take their time and make sure the job is done properly; not switch to another, more lucrative deal part way through. If your penetration tester drags out the job, that's a failure of the person - not the method. It all boils down to how serious you are about the testing process. Arguably, Bugcrowd serves its purpose - but be aware... it's more likely to catch the (so called) low hanging fruit than anything of significance.

Bugcrowd is one of those rare self-defeating sites - if nobody uses it, it's useless. If thousands use it, the dilution is such that it's not financially beneficial enough to waste any time on. You're better off reporting directly the company involved.