Penetration Testing ROI: 5 Metrics to Communicate Real Value

Because of how sophisticated security attacks are, penetration testing has now become essential. But why is it so hard to communicate the real tangible value of penetration testing? The return on investment (ROI) of penetration testing is easy to prove, but difficult to see immediately. With preventative measures, the best case scenario is that nothing happens. So, there's no big celebratory moment like when sales hits quota, or product teams launch a new release.

There are various ways to communicate the value of penetration testing, and all have different values to different organizational departments. The IT or Security team will have different measures of ROI than CFO’s or CEO’s. It is important to understand these differences and provide metrics that are valuable to both parties involved in security strategy decisions. When the CFO and CTO work together, organizations are better able to allocate resources and make strategic decisions to improve their cybersecurity posture. We have put together 5 key metrics to communicate tangible value to your CFO, in ways that make sense to them.

5 metrics to communicate penetration testing ROI

There are 5 key metrics to communicate the value of penetration testing. To prove ROI of a cybersecurity investment like penetration testing, measuring these metrics is crucial.

• Impacts of severe risks
• Vulnerability density trends
• Open-to-remediated ratio and triage efficiency
• Remediation effort costs
• Security program maturity

Impacts of severe risk

Tracking the impacts of severe risk for vulnerabilities is essential in helping organizations understand the potential impact of a successful attack on their systems. By identifying and prioritizing vulnerabilities based on their level of risk, organizations can allocate their resources effectively to address the most critical vulnerabilities first. Software Secured uses a combination of CVSS and DREAD scoring systems to accurately measure the impact and likelihood of attack for each vulnerability. For each critical or high vulnerability found on your penetration test report, associate a dollar value to it. You can do this by calculating the estimated legal fees, fines, lost revenue, and wasted employee salaries, for example.

By tracking measures of risk, organizations can calculate the breach risk in monetary terms.

Breach risk ($) is equal to breach likelihood (%) multiplied by breach impact ($).

This calculation can be great for giving a ballpark estimate of the potential breach risk your company could face based on a certain vulnerability.

You can then determine the ROI of a particular security investment. For example, if you had a critical vulnerability, you can determine the reduction of breach likelihood based on a before and after patching score. Critical vulnerabilities are generally patched completely, therefore changing the breach likelihood to a reduced value. You can compare the breach risk of an unpatched critical to a patched/resolved critical to get an accurate picture of the breach risk cost difference. The equation to determine ROI of a security investment generally follows this calculation.

ROI is equal to the potential breach risk ($) divided by the cost to fix the breach OR breach risk after patching. To get ROI in percentage multiple ROI x 100.

By using a combination of the breach risk and ROi equations, you are able to estimate the potential breach damage of a particular vulnerability and ROI of a security investment (such as penetration testing) to help reduce the breach risk.

There are also other monetary factors to consider when talking about potential breach risk based on vulnerability severity. As mentioned above in this section, there are factors such as legal fees, fines, lost revenue, and wasted employee salaries that can all add additional costs alongside breach likelihood and impact.

By calculating the average monetary value of a breach risk for each individual vulnerability and the ROI of patching or investing measures to prevent that vulnerability from being exploited, this can help you identify the ROI of security investments in tangible numbers your CFO will appreciate.

Vulnerability density trends

Vulnerability density is the number of vulnerabilities per unit of code, and tracking this metric over time can help organizations identify trends in their security posture. In order to determine your vulnerability density trends, your team needs to analyze how many vulnerabilities (of all scores) you have discovered in an average year, and track specific security mitigations that could have been put in place to reduce this number overtime. If you conduct more than one penetration test per year, you can change this number to track how many vulnerabilities are found per penetration test. By tracking the density of vulnerabilities found per penetration test, you are able to determine trends, and if your vulnerability count is increasing or decreasing as you are doing more testing.

The vulnerability density is given by:

VD +  V / S

where S is the size of the software and V is the number of vulnerabilities in the system. Following the common practice in software engineering consider 1000 source lines as the unit code size.

For example, if the vulnerability density decreases after a penetration testing engagement, this can be a clear indicator that the organization's security posture has improved. Furthermore, tracking vulnerability density can help organizations identify areas of their codebase that are particularly vulnerable, allowing them to focus their penetration testing efforts where they will be most effective. This not only reduces the number of vulnerabilities, but it also teaches development teams where there are weaknesses within their applications, and will begin to code more securely by practice in the future.

Open-to-remediated ratio and triage efficiency

Companies need to carefully track the open-to-remediated ratio of vulnerabilities to provide evidence of the effectiveness of their penetration testing program. On average, it takes 8 days​ for a vulnerability to be weaponized after it is found by a hacker. It takes the average development team around 202 days to patch​​ these issues.​ This means that for almost 3 months, an organization can be vulnerable to attack before they even realize it. By tracking the open-to-remediated ratio of vulnerabilities, companies can identify gaps in their remediation process and address them promptly to minimize the window of vulnerability.

Quarterly penetration tests reduce staff time spent by 69 minutes of triage per vulnerability, which is an average of 29 hours per pentest.

By tracking your triage efficiency alongside the open-to-remediated vulnerabilities, your organization can determine how effective, how fast their response time is, and how long it takes to remediate a vulnerability.

It can also be useful to track your Security Level Agreement (SLA) adherence rate. A SLA is an agreement between a security team and the business that outlines the security requirements that the team will provide for the organization. The purpose of an SLA for vulnerabilities is to define the expected response time for identifying, reporting, and resolving vulnerabilities in a system or application by the security team. It is a critical component of an organization's vulnerability management program and helps ensure that vulnerabilities are addressed in a timely manner to prevent potential security incidents. Standard SLA timelines can vary from industry to industry, but typically follow these timelines:

CRITICAL: 24 hours - 7 days

HIGH: 8 days

MEDIUM: 30 days

LOW: 60 days

INFORMATIONAL: 90 days

SLA’s should be custom to your organization, and reflect the amount of sensitive data that could be exposed in a cyber attack. If your application(s) collects sensitive or confidential data, there should be an evaluation to see what timelines are both realistic and suitable for your team. In some organizations, the SLAs are purely driven by clients contracts or compliance mandates. The standard SLA timelines will not fit everyone, and it is crucial to set SLA’s based on your particular situation​. In Software Secured's own Vulnerability Management Portal, there is a feature to customize your SLAs based on each project's unique requirements.

By tracking both the open-to-remediated ratio, triage efficiency and SLA adherence, you will be able to get an accurate picture of how efficient your development team is at remediating vulnerabilities in a timely manner. As your team conducts more penetration testing, the ability to remediate and adhere to SLAs should effectively increase as time goes on.

Remediation effort costs

Remediation effort costs are the costs associated with addressing vulnerabilities that are identified during a penetration testing engagement.

Once a vulnerability is identified, a critical metric is quantifying the cost of how much effort is going into remediation. For example, if vulnerabilities are found in code in production, someone has to change it, perform regression testing on the software, test to make sure nothing else is broken, push through a QA environment, do an additional test to ensure it was fixed correctly, and then push the updated code to production. Tracking these cost metrics:

Personnel costs ($) multiplied by the hours (hrs) spent

allows you to determine if you're gaining efficiency in remediation. Effective quarterly penetration testing allows for vulnerabilities to be caught earlier in the development stage when aligned properly with the agile framework.

This image demonstrates that tracking remediation costs can help companies reduce the overall cost of their security program by optimizing their remediation process to find vulnerabilities sooner. Practicing security proactively throughout the year provides a large ROI for your organization. The cost to fix a vulnerability in the production phase is 100 times more costly than fixing a vulnerability in the design phase. Meaning the average cost of $500.00 to repair a vulnerability in the design stage is multiplied by 100. On average, Software Secured finds 26 vulnerabilities per test. The average costs saved when finding vulnerabilities in testing staging versus maintenance is over 1 million dollars per test. If an organization does quarterly tests, they can save over 4.4 million dollars annually. Tracking remediation effort costs can help determine where resources need to be allocated, and reduce overall costs of a security program when used correctly.

Security program maturity

Security program maturity refers to the level of effectiveness of an organization's overall security program, including policies, procedures, and technical controls.

A study by PwC found that companies that achieved higher levels of security program maturity had lower overall security-related costs and better incident response capabilities.

As C-suite executives continue to strive for high collaboration and security maturity, organizations are beginning to close large cybersecurity gaps seen in previous years. Security program maturity requires thoughtful and efficient processes paired with the knowledge of where to implement cybersecurity measures. A mature security program consists of efficient risk management, governance, compliance, consistent training, and continuous improvement. Mature security programs have developed agile implementation methods that are efficient for their organizations and team. For example, an organization with a mature security program will perform penetration tests on a quarterly basis, if not more frequently, to align with their code releases to catch vulnerabilities before they hit production. As shown in the “Remediation effort costs” there are clear monetary savings when comparing fixing vulnerabilities in a staging versus production environment. A mature security program will have determined these efficiencies and built their program custom to their organizational needs.

Conclusion

Communicating the penetration testing ROI is essential due to the increasing sophistication and frequency of cyber threats. Managing security budgets in today's economic climate is more challenging than ever, and organizations are pushing to prove ROI wherever possible. In order to effectively prove the penetration testing value, it is crucial to reflect and record your security program metrics, such as impacts of severe risk, vulnerability density trends, open-to-remediated ratio, remediation effort costs, and security program maturity. Once these metrics are trackable and readily available, you can begin to compare the potential damages and costs associated with severe vulnerabilities. It is important to revisit these metrics every quarter. This should line up with your penetration tests, but depending on your organization you may conduct penetration tests semi-annually or annually. If you or your organization needs further assistance with getting these metrics in place before your penetration test, we are happy to help! Book a meeting with us to learn more about the ROI of penetration testing.