There’s a lot of conversation taking place around the merits of penetration testing vs. bug bounties, pentesting vs. vulnerability assessments, bug bounties or a red team engagement, and the role of trusted advisors in all this.

In my estimation, far too few organisations understand the differences between these enough to decide which one to use, and when. The easiest thing to do would be to blame sales, but the problem is deeper than that.

Let me cover these definitions briefly below:

A vulnerability assessment is designed to find as many flaws as possible in order to make a prioritized list of remediation items

A penetration test is designed to see if a mature defense can stop an attacker from achieving one or more specific goals

A bug bounty is designed to use the benefits of a crowd to discover as many vulnerabilities as possible

A red team engagement is designed to continuously test and improve the effectiveness of a company’s blue team by mimicking real-world attackers

A trusted advisor (in the security context) is someone who should be able to tell a company, based on their maturity and goals, which approach to take at any given point in time

Maturity matching

In my opinion, the maturity of the organization in question is the most important factor in the decision of what type of assessment to use, and when.

What do I mean by maturity?

Is the company patched?

Does the company have a list of everything it owns, who has access to it, and when that access was last reviewed?

Does it know where its data is and how it’s protected?

Does it know how that data moves around the environment during business processes?

If the answer to these questions is no, then they’re likely low-maturity. Meaning, they should stop thinking about hybrid-cloud, crowd-red-team ninja assessments and focus on basics. Make a list of where your stuff is and get patched.

But let’s assume you’re at medium maturity—and you’re ready to start doing some security assessments. Where do you start?

It’s never a bad time to have a trusted advisor. You should always have one. Whether they’re internal or external doesn’t matter that much, but you need to be able to quickly ask questions like these and get answers. So, first priority is to have someone who can help you build and stay on track with a strategy.

The first type of security assessment to do, i.e., the assessment type for lowest-maturity companies, is the vulnerability assessment. This is because it’s not for testing mature defenses, per say, but rather giving you a list of all the stuff you need to fix to have good defenses. It’s a prioritized list of everything you should fix, and in what order.

After you’ve done one or more vulnerability assessments and you’ve fixed everything that’s been found (that’s an important part, otherwise you’re wasting a lot of money) it’s time to move on to penetration testing. This is probably best done with a trusted vendor or even an internal group (if you have one). You probably don’t have one at that point, though, otherwise you wouldn’t be in this position. So a trusted external group is likely best because what is likely to be found at this stage could be pretty bad, and pretty embarrassing.

After you’ve done lots of vulnerability assessment and lots of pentesting, and you’re now coming back with few results whenever you do them, you’re ready for the next stage, which can go in multiple directions. If you want to continue finding more vulnerabilities, and the systems you’re testing are not overly sensitive (source code reviews, private networks, crown jewels, etc.), then you should start thinking about doing a bug bounty. A properly sized and managed crowd can often find additional vulnerabilities that can only be caught by “many eyes”, but you want to be sure that you’re picking the right things for them to look at, and managing it closely.

Finally, either after or at the same time as pentesting, you should start looking into red team options. Red teams are permanent, continuous campaigns designed to emulate real-world attackers. Done right, they’re the highest maturity testing type as they must remain mostly independent from the organization they’re testing, must remain extremely current and sharp, and must constantly evolve their tools and techniques to stay effective. The one key for the red team is that its purpose is to improve the blue team (defense team) , which is a very different goal than that of a vulnerability assessment, a penetration test, or a bounty.

In conclusion;

Have a trusted advisor (a person or company) who can steer your security assessment efforts throughout your maturity levels

Start with vulnerability assessments, and don’t do any of the other types of testing until you have cleaned up your environment through remediation enough that it’s hard to find things

Then move to pentesting with a trusted company

If you’re getting no results anymore from trusted advisor pentesting, consider adding a bounty program to leverage the “many eyes” of a crowd

After the vulnerability assessment stage is complete, when you’re moving into the pentesting phase, you should feel free to start looking into red team options as well, if not just because it will take some time to research and find good options. Red teams are not “better” pen testers; they’re a different kind of assessment altogether, with different goals

Remember to have your trusted advisor(s) guiding you through all of this; there are lots of variables that determine when, and how, to do what. This is just a guide to get you started

Food for thought:

We know some people need a penetration test to prove that they even have security problems, but just know that it’s a really bad situation to be in. If you have that little understanding from management that they need a forest fire before you get budget, you should expect a whole lot more drama as you proceed.

Full Disclosure: Summit Consulting is an elite penetration testing company that falls into the category of “trusted advisor” or “trusted pentesting vendor”. Hopefully you agree that it doesn’t matter here, but it needs to be said in a piece like this.

Pentesting is not “better” vulnerability assessment; it’s a different thing altogether.

Red team is not “better” pentesting; it’s a different thing altogether.

Bug Bounties are most similar to vulnerability assessments in that they are designed to find as many issues as possible. They are a higher maturity version, however, since you wouldn’t want to waste money on a bug bounty when you have a low maturity environment where bugs are easy to find. You also don’t necessarily want to set a crowd lose on low-maturity environments due to potential trust issues, but it may be possible to use vetted, private bounties for this purpose. It’s probably best to use internal or trusted resources to perform low-maturity vulnerability assessments, and then use bounties once maturity is high and you’re having trouble finding additional issues.

The trusted advisor role is so essential because it helps you contextualize findings and prioritize activity based on the goals of the business. Without this you might do lots of work on the wrong things, and not reduce as much risk as you could have. There is always too much to fix for the amount of resources you have; most of the game is deciding where to focus.