What is the Difference Between a High-quality and Low-quality Penetration Test

Evaluating your security is as important as implementing security. There are various types of security testing such as Vulnerability Scanning, Bug Bounty Program, and Risk Assessment. Our focus in this post is on one of these types - Penetration Testing.

If you search for penetration testing(pentesting) services on the internet, you’ll find exhaustive results. With so many options, how will you decide what’s best for you? How will you decide what’s good enough? That’s what we’ll address in this. We’ll first see how the term penetration testing is misrepresented and then look into the characteristics of a low and high-quality pentest.

Penetration Testing is a Misrepresented Term

Cybersecurity is a huge domain and has several sections under it. With this exhaustive list of sections, concepts, and terminologies, it’s common for one to misunderstand terms. Penetration testing is one such term in the security testing category.

Many people do not know the difference between a vulnerability scan and a penetration test, amongst other security testing concepts. Some people use the terms vulnerability scan and penetration test interchangeably. But that’s far from the truth.

A vulnerability scan is a test performed using automated tools to scan the target to identify vulnerabilities. A penetration test is a simulation of an attack trying to break the target’s security in order to evaluate its security. Vulnerability scans use a database of known vulnerabilities and check if the target contains these vulnerabilities. These tools are only as good as the database. Therefore it’s not possible for these tools to detect zero-day vulnerabilities. Additionally, a penetration test considers the context, includes an attack plan, and focuses on exploiting a discovered vulnerability.  Many companies use the term “penetration testing” incorrectly, just because a human “fact checks” a vulnerability scan, does not mean this is a penetration test.

If that makes the difference clear, we’re not done yet. Another difference one should know is the difference between the qualities of a penetration test. Because penetration testing is so widely used, it can be hard to tell if you are getting a low-quality or a high-quality pentest. So let’s look into how you can differentiate between them.

Characteristics of a Low-Quality Pentest

Cost

We all love a good deal. But when it comes to a pentest, the quality of the pentest is directly proportional to the cost. So, if the deal is too good to be true, it’s likely that the quality is low. “Why?”, you ask. Because a high-quality pentest requires resources for thorough testing and that doesn’t come cheap.

The price of a pentest is also proportional to the test size. For example, let’s say you have a massive application and they give you a quote for $1200, it’s likely that they’re just using automated tools or not being thorough enough, and someone is creating beautified reports for you. Doesn’t look like a high-quality “pentest”, does it? The bottom line is that the cost of a pentest is one of the factors to determine its quality and the cost of a low-quality pentest is on the lower side.  

Process

As mentioned earlier, pentesting is far from merely running automated scans. The process of a “pentest” is a crucial factor. The process of a low-quality pentest is usually along the lines of running an automated tool. A slightly better version is to run the tool, and then have a person (“pentester”) verify the results from the automated tool. This is a tiny bit better but still not good enough. There’s a huge difference between the verification of a scan and a pentest. You will get fewer false positives and they can claim they “manually pentested” the application but they did not.

Even with advanced and sophisticated tools, there’s still a lack of that human element. A human will see the whole picture and have instincts when something doesn’t look right. Tools cover the breadth but a pentester looks at it like a maze and focuses on the depth as well. Understanding the target, context, and logical workflows plays a crucial role in pentest, and tools lack in this aspect.

Reporting

A pentest report is equally important as the test itself. Low-quality pentest will not deliver a detailed, descriptive, and customized report for the findings. You might receive a report autogenerated by a tool, a report written by another 3rd party tool, not the pentesting company or even a human. And to make it look like a high-quality pentest, minor issues are sensationalized. This selling of artificial value reduces the overall value of the pentest report and impacts the outcome of the test.

Remediation

Cheaper options will point you to the references generated by the 3rd party tool and/or tell you to do your own research. They’re not typically inclined to sit down and explain their findings on how to properly fix them. Applying a fix that doesn’t properly fit your system might not completely fix the issue or it might create even more security weaknesses.

Now that we know some indicators of a low-quality pentest and what to avoid, let’s look into the characteristics of a high-quality pentest.

Characteristics of a High-Quality Pentest

Costs

There are several factors that determine the cost of a pentest such as the size of the target, scope, methodology, experience, endpoints, test duration, and providing recommendations for fixing found security weaknesses. Each of these comes at a price.

You would want the test to cover both breadth and depth while testing so that you can get the most out of the test. This takes time, resources, and effort. So it’s natural for the cost of a high-quality pentest to go up. But bare in mind, high-quality pentests come at high costs but that doesn’t mean all high-cost pentests are of high quality.

Customer Service

Pentesting is not a one-sided activity. The pentesting side and the product owner side have to come together before and after the pentest. Before the pentest begins, expectations have to be set, there is exchange of information. And after the pentest, report sharing and helping the teams fix the issues. All of this might need both the aforementioned parties to come together on calls or meetings and communicate several times throughout the process.

High-quality pentests focus on customer success and customer service equally high as the pentest itself. These pentesting teams make communication easy and efficient because they understand its impact on the overall success of the task at hand. The turn around time for meetings, calls, emails, etc., is good. They try to understand your problem, and consider it while pentesting your product.

Process

A high-quality pentest involves thinking and acting like an attacker. There might be different ways to break into a system and might also require someone to think outside the box especially if the target has some security mechanisms in place.

A pentration tester has to gather information about the target, identify potential weak points, think about the tactics, and custom attack cases that target specific goals. Threat modeling is another crucial part of security testing because you consider the context of the target to identify and understand threats.

You can learn more about STRIDE Threat Modeling here.

Without understanding the target, its context, and how it works, the pentest might not be rigorous. And a high-quality pentest should cover all these aspects. You might have to do some sessions with the pentesters and show demos, but all of this will reflect in the outcome. Another factor of a high-quality pentest process is retesting after the team has fixed the security issues to confirm the mitigation.

Reporting

High-quality pentests definitely result in descriptive and informative reports. It’s not just about what a pentester found but also how they found and exploited a security weakness. This again involves the context keeping the target system/application in mind. Different applications/industries have different PII to be determined for each report, and some things should be included and others omitted based on context.

High-quality pentest reports should have in-depth reporting, such as our portal features: how many vulnerabilities were found, comments, mitigations, and guidelines for your vulnerabilities. A high-quality pentest report should also include the severity of the security issue found. The baseline is having CVSS. Such information helps in prioritizing the mitigation.

SLAs are based on severity and recommendation. And this might change from one organization to organization depending on the client's data and context. General SLAs are as follows:

• CRITICAL (5 business days)
• HIGH (asap but within 30 days)
• MEDIUM (90-180 days)
• LOW (around 180 days)

Remediation

High-quality pentest providers will be more inclined to sit down with you and thoroughly explain findings in a language that everyone can understand, especially the ones that will have to work on fixing the issue. Providing the context of the attack, the tactic, and the thought process of an attacker will not just help in fixing the issue but will also help in designing and building more secure systems and applications in the future.

These are some major factors that one should look for in a high-quality pentest.

Conclusion

Pentesting is one of the commonly misrepresented terms. And there’s even more obscurity in the understanding of a high-quality and a low-quality pentest. We addressed what a pentest is and how it is different from its commonly misrepresented term, vulnerability scan. Then we understood how to differentiate a high vs low-quality pentest based on cost, process, reporting, and remediation.

Here’s a quick summary of the factors of a high-quality pentest:

• It is more costly due to resources, time, and effort.
• Considers the context of the target and attackers’ thought processes in attacking the system.
• Reporting is detailed along with helping with prioritizing the fix.
• Help with custom remediation along with an insight into the attack process.

In addition to high quality, there are also some aspects that the company should take care of such as regular audits and reminders to resolve SLA.

If you’re looking for a high-quality pentest, you might want to look at Software Secured’s Penetration Testing as a Service (PTaaS) and Pentest 360.