The History of Pentest Providers

You have done one or more pentests whether for PCI compliance purposes, for internal policy requirements, or perhaps your clients require it. Now it is time to perform another one, and the question that always seems to arise, should you change your penetration testing provider? One opinion out there is to change vendors every 2–3 years, as it is believed a new vendor would come with a fresh perspective, and hence new bugs.

This advice seems to date back to the 2000's where application security was pretty new and penetration testing providers wouldn’t have any trouble identifying a fair number of issues per test. At that time, security controls were pretty much non-existent, perimeter security ruled as a the method of choice for protecting everything from the network down, to applications, and everything in between. One of the other sources of this advice is a SANS whitepaper from 2010 that outlined changing vendors as good practice.

Applications have changed quite a bit since the 2000's, they are more complex, have modern languages, a native framework protection, and a multitude of other factors suggests that this advice might be obsolete. While there are legitimate reasons for changing your application security vendor, there are a few things that should be kept in mind.

Here are the top reasons for alternating your vendors:

1. Complacency

Application security engineers just like developers can be blind to some aspect of their work. For software developers, it can be very hard to find their own bugs, hence the practice of peer code review and the slew of quality assurance controls that are put in place such as unit tests, manual and automated quality assurance tests.

Application security engineers can face the same problem where they would be blind to any other bugs in the application that they didn’t find.

One of the questions you need to ask your pentest vendor is what do they do to overcome this problem.

2. Quality and Value Intelligence

There's a clear difference between a high-quality and low-quality penetration test. It might be worth to consider other vendors to understand what service you are getting at what price point. Additionally, not all penetration testing providers are created equal, so one vendor could be really good at identifying vulnerabilities but post-report support might not be great, while another vendor really excels at the services provided post-report.

3. Leveraging Different Expertise

It might be worth exploring different vendors for different expertise areas. For example one vendor could be very strong in performing a mobile penetration test, but not as much in social engineering. So consider leveraging different vendors for different areas of strengths.

Here are the top considerations before  deciding to switch vendors:

1 .  Losing Context and Application Knowledge: There is most likely 3 levels of depth to any pentest:

Level 1: The low hanging fruit: which is basically what any scanner can find.

Level 2: The medium range issues: bugs that scanners can’t find but an engineer can, they are still easy to find, it just needs someone looking for them, agile enough with their approach.

Level 3: The "Difficult Bug": these are only found when the engineer gets intimate enough with the application to understand exactly how it works.

For a 2–3 week long penetration test, it is very hard for a new vendor to reach Level 3 in such a short amount of time. Depending on the nature of your application, there might not be a lot of business logic to the application and hence there is no need for Level 3. Another thing to consider is who are your primary threat actors? Are you up mostly against script kiddies or professional hackers and cyber gangs?

2 .  Losing the Partner Relationship:

It takes time for a pentest service to understand your business, application, and the dynamics of your team. Working with a pentest provider should not stop at the delivery of the report. A good pentest provider should be able to identify really good bugs, and more importantly help you mitigate those bugs and further fortify the application. If you are not getting that help from your current vendor, it might be worth it to have a conversation with that vendor or search for a new one.

3 .  Losing Motivation:

For most companies in the professional services space, it is (or at least it should be) their top priority to go the extra mile to keep their clients. Knowing from the get go that it is temporary relationship might not motivate that vendor to go above and beyond for you. Don’t get me wrong, I don’t mean the vendor will offer degraded or less than normal service. It means not going the extra mile for a client who is guaranteed not to come back.

There are several factors that go into choosing a pentest partner: skills, history, price, and processes among others. Building a solid relationship with them and getting the most value requires communication, trust and transparency.