Static code analysis is an essential ingredient in any semi-decent software security assurance program. Let’s get this out of the way, whether commercial or open source, they are just one important component in software security testing these days.

The question is, which one? Which one is better than the others? Why is it better? is it because other people are using it? Is it because the price seems right? Is it because it is free? All valid questions and there is one answer to all the questions, the answer is: “It depends.”

Evaluation Criteria

So as a consumer of static code analysis tools, you need to know what matters when you are evaluating these tools. As an application security professional, my ideal version of a static code analysis tool is the one that finds as many vulnerabilities as possible with the minimum amount of false positives, can be customized to understand the applications I am working on better and provide accurate steps to developers on how to fix the issues. Last it needs to optimize the time spent going through the findings.

Do you know what is your ideal version of a static code analysis tool?

You need to know what would actually matter to your organization? Is it a tool that saves you time or a tool that understands Python? or maybe a tool with the minimum amount of triage process?.

Let me ask you a question. If you are going to buy a car, you would go and research, and compare the MPG the car gets, how much does maintaining this car costs, what is the resale value, you might even go on consumer reports and check it out. And at the end you make an educated decision towards the best car that suits your circumstances, while taking prices, miles to the gallon, resale value…etc in consideration.

The same process applies for acquiring one of these static code analysis tools and here is why…

It is a Huge Investment

Some of these tools are really expensive, and I am talking up to six figures. For some small to medium software companies that are out there this price is just unheard of to spend on one piece of “software”. Specially with a culture that is so accustomed to everything being free and open source. Even with the free static code analysis tools, there is still cost with the setup, maintenance, learning curve and the time is takes to dissect the results of these tools. Bottom line is, even with absolutely free tools, there is a cost associated with implementing any of these tools.

The No Criteria Option

If there is no criteria to base the choice of the tool on, then we are just depending on marketing messages coming from different tool vendors to convince us. The same if you walk into a random dealership and ask them to show you what they got, without proper preparation and without knowing what matters and what are you actually looking for. You will most likely, guaranteed to walk out of their store with one of their most expensive cars, having made a car salesman really happy.

You are going to spend far less time choosing and integrating the tool, if you actually chose tools that perfectly fits your environment, your kind of projects, the skill set and budgets available.

If you understand what matters, you will actually get a tool that actually does help you find vulnerabilities and fix them in the least amount of time possible.

That’s why we have been working on a set of criteria that people could use to evaluate Static Code Analysis tools. The Web Application Security Consortium initiated a project that aims at achieving this very exact goal. The final outcome will be in the form of a reference document. Here are the target audience and scope as described by the project:

Target Audience:

The target audience of this document  are the technical staff of organizations dealing with software security issues. The document will take into consideration those who would be evaluating the tool and those who would actually be using the tool. Most of the time these two groups are the same, but some time they might not.


The purpose of this document is to develop a set of criteria that should be taken into consideration while evaluating Static Code Analysis tools for security testing. Every software organization is unique in their environment. The goal is to help organizations achieve the better software security in their own unique environment, the document will strictly stay away from evaluating or rating tools. However, it will aim to draw attention to the most important aspects of static analysis tools that would help the target audience identified above to choose the best tool for their environment and development needs.

Interested? Easy…just join and pitch in your opinion.