What to Do When You Don't Agree With Your Penetration Test Report

Read the report

The first thing you should do when receiving a penetration test report is to read it in entirety. You need to understand the key findings, how the testing was performed and get an idea of what remediation work needs to be done.

Most penetration tests come with an executive summary which can be very useful for getting a high level understanding of the test. You can also use this opportunity to evaluate how close your organization is to its final state. For example if the test was related to a compliance standard then the report should let you know if you have the proper controls in place or what controls you are lacking.

Make a plan & implement

Now that you have investigated the findings and have identified what work needs to be done, the next step is to develop a plan and implement. This plan should outline the order in which vulnerabilities should be fixed, who will be responsible for doing the fixes, the overall timeline for completion of all vulnerabilities and final validation.

Assign resources

Once you have an overall understanding of the findings of the penetration test the next step is to assign resources. These resources will be needed to start working on remediating the issues identified in the report. This should be a combination of technical staff as well as management that will be needed to help drive the actions to completion and provide authorization to make required changes.

Investigate findings

Once you have the resources in place you can start investigating the findings outlined in the report. Typically, this should be done in order of severity with the highest severity issues being investigated and remediated first. In addition to finding and fixing issues there should be attention given to identifying the root cause of the vulnerability and addressing that. For example rather than simply fixing a misconfiguration in a system you may want to look at the SOPs associated with configuring those systems and ensuring that the documented instructions specify secure configurations. Doing this extra step helps to reduce the likelihood of future vulnerabilities being introduced to the environment.

Test and retest

The final step is to perform a retest of your environment. This is critical to ensure that the fixes you implemented remediate the vulnerability and don’t leave any workarounds that can be exploited by a hacker. This retest should be done by professional penetration testers, not internal devs or staff.

What to do when you don’t agree with finding(s)

Make sure you understand the finding: If you don’t think a finding is accurate the first thing you should do is make sure you understand it correctly. Read the details of how the finding was discovered in the report and work with your internal technical staff to understand how this finding may be affecting your business. If you still disagree with the finding then you should contact the testers and speak with them to ensure that it is a vulnerability and not a false positive. The goal is to understand the full content of the finding, this includes the technical issue, sensitivity of data involved and potential business impact.

Take it as a learning opportunity: Any negative penetration test can be demoralizing, but any penetration test that finds a vulnerability benefits the organization by allowing it to improve its overall security posture. A negative penetration test should be seen as a learning experience, not a failure.

If you have evaluated the context, severity, sensitivity of data involved and the other measures, you can use the following steps to help further determine the potential impact of any finding in your pen test report:

Talk to your penetration test provider

As mentioned above you shouldn’t hesitate to speak with your penetration testing provider if you have concerns about the report you were given. To do this first you should book a meeting with the penetration testing team. In this meeting you can explain that you do not agree with the finding(s) provided in the report. Typically, penetration testing reports include the exact steps the testers used and how you can reproduce the same results. While this is included in the report, if you have been unable to reproduce the results yourself the testers can walk you through the steps during this meeting and help you troubleshoot why you were unable to reproduce these results yourself during your initial validation. They can also walk through the steps as to why they may have scored this vulnerability as a certain ranking. There are many factors that can affect the severity ranking of a vulnerability, many of which may not be apparent from the client’s point of view. Sometimes with context, a pair of low or medium vulnerabilities can become a high or even critical depending on the scenario and how they are used. This may be a reason why something that appears to be a small risk to the client may be given a high ranking because of how it can be used or combined with other vulnerabilities. If they walk through the entire story/scoring/reproducibility and you still do not agree, there are more steps you can take to verify the impact.

Evaluate the risk rating systems

Rather than focusing on vulnerability itself you may want to look at the company’s risk rating system. Verify if your pentester uses multiple risk rating systems, and how they come up with their scoring, if they are only using one risk rating system this might be an issue. Ideally, the pen testing provider should have more than one rating system and they should have a method of scoring vulnerabilities that is objective and not done by feeling. If they use 2 that are well known in the industry and do not score by feeling, there is more merit to their rating. You should have gotten a good picture of their scoring system already from the report itself, their follow up call, or during the penetration testing process. If you were not privy to this information the entire time, this is a large red flag.

Consult a reputable, independent 3rd party

If you have done these previous two steps, and still are not happy with the outcome, it might be useful to ask a third party to consult on the matter. This does not mean that you need to get a second penetration test, but another security consultant should be able to go through the report along with the pentesters notes, reproducibility, and have your applications context in order to make a determination on whether the penetration test and its findings are credible. Depending on the feedback you get from the third party you can decide whether or not a second penetration test is a good idea or not.

How to get the most out of negative penetration test

• This is a learning experience and not a failure: If you get a penetration test that comes back with multiple vulnerabilities you should look at this as a chance to improve. The reality is those weaknesses were there before the test and now that you are aware of those issues, you have the opportunity to fix them before they could be exploited.
• Look into training your developers on secure coding: A common cause of security vulnerabilities in applications is a lack of security awareness among developers. It’s important for developers to understand secure coding practices so that the amount of vulnerabilities that occur in company applications is reduced.
• Understand the issues found and how they came to be: It’s important to do a root cause analysis to identify the source of these vulnerabilities. Unless you find and resolve the root cause there is a possibility that similar vulnerabilities will surface in the future.
• Create a plan to make sure the same vulnerabilities are not found in future code snippets: Once you have identified the root cause you need to make changes to your standard operating procedures (SOP) to ensure that this doesn’t happen again. For example this may mean modifying your current testing procedures to ensure that you account for these types of vulnerabilities going forward.
• Consider regular/consistent penetration tests to stay on top of potential vulnerabilities: Once you fix your initial issues you should consider doing regular penetration tests to ensure that new vulnerabilities in your environment are detected early and fixed. If you wait a long time between penetration tests then you give time for vulnerabilities to accumulate undetected.
• Proactivity is better than reactivity: Remember that going forward it’s always better to be proactive than reactive. Make a plan to do routine security activities like source code reviews and security awareness training for your employees to ensure that you prevent vulnerabilities rather than trying to fix them after a long time of them going undetected.

Conclusion

If you're not happy with your penetration test report, there are a few things you can do. First, try to understand where the testers were coming from. It's possible that they are considering different contexts and scenarios. Second, reach out to the testing company and ask for clarification. Let them walk you through their process and hear their explanation for their findings and rankings. Finally, if you still don't agree, you can always request a second opinion from a qualified third party.