How to know whether you need security

So I went to the turquoise blue beaches of Mexico about a month and a half ago. Hot weather, nice resort, turquoise blue beach and excellent food. I even got free passes to about 5 of the area’s hottest attractions worth about $1000 but this is another story. So what can be wrong and how is related to security, right?

Do you feel something wrong?

A couple of days into my week long vacation, I started to feel tired, fatigue, achy…I thought it was a cold because of the different weather between Canada and Mexico…and it would really suck to stay in my room 3000 miles away from my home, right?

Symptoms: No body talks about security inside your company? No security specific role in your company? Do you think that SSL = security?

Take a small step like ask a friend or search the internet?

My sister is a physician, so I decided to give her a call and ask her about what I feel. Which I did….

Her answer wasn’t the sort of answer I was kind of hoping to hear, something along the line that it was just a “cold” and it is gonna go away. Her answer was different and it meant that I had to see my doctor as soon as possible.

Good Steps: Attend an OWASP\ISSA\COINS meeting, attend a seminar online, take a security awareness course..etc

Try to figure out what’s going on?

My doctor had no idea what’s going on so he kept ordering blood tests after blood tests and after about 3 rounds of them he finally told me…I don’t know. So I said to myself: Dahhhh, you caught an infection and you gotta go to an infection specialist

Takeouts: Stay in this phase until you understand your security posture and where you would like to improve first even if you ended up hiring someone to tell you so.

Take Big Steps: Go to the experts

Finally I got a name of an infection specialist which I am going to see in a couple of a weeks (yes, this is the health care in Canada if you are reading from The States).

Big Steps: Depending on the outcome of the last step, you can now go directly to the source which will help you get where you want, security wise

Finally..

Never go to Mexico without the right vaccines

I have been blessed by working with some of the best software developers out there. Some of them changed jobs recently and I have noticed a pattern that got me thinking:

- They do their best in their current job to the last minute: to the extent that their employers *think* they are so happy doing what they do. Employers don’t understand that great software developers don’t know how to do their jobs other wise not that they like or dislike their jobs

- If these great software developers don’t like their jobs, they try to talk it out in very subtle way. Again, if the employer is not paying attention, unfortunately, this is all the chance they get to rectify the situation.

- If they don’t feel they are heard, they take the decision silently and start to look somewhere else and usually through their network.

The same exact strategy with a few modifications is followed with all the great software developers I know. Did you notice a different pattern? tell us about it…

Ask pen-testers and they will tell you that the attack is as successful as how much information they can gather during the reconnaissance phase. Reconnaissance is the phase where the attacker tries to gather as much information about your application as possible. How is this useful? ……Simple….Information is POWER.

If the attacker knew which operating system you are running on, they can look for vulnerabilities related to this specific operating system. Even better, if they can find out the specific version, you just made it easier for them.

Another instance, if the attacker found a name inside an HTML comment, they can start social engineering this name and through him, or his personal computers, they can get to the application. And this is exactly what happened when Twitter’s admin account was hacked.

So what are the information that can be leaked:

- Detailed Error Messages: As mentioned above, this can help an attacker profile and fingerprint the application. Afterwards, it is just a matter of time until they find a way to exploit your system.

- Confidential Information: Credit card numbers, social security numbers, passwords, usernames, dates of birth…etc. All these are goodies for the attacker.

- Names: As mentioned before, names are very helpful in a social engineering attack.

- Source Code: Yes, source code. That’s an attacker’s dream to get the source code, just a gold mine. All the business logic, passwords, details, names, history, servers…everything he’s ever wished for would be in there.

So where are the places you would look for information leakage in your information?

- Web Services: A lot of developers rest assured that since web services does not have an actual interface, so they think an attacker will not be able to find these services. They are right, attackers will not find them….they will let some tools find them !!. A lot of web services out there are not properly protected or are not protected at all. And there is a lot of information that can be leaked by these web services.

- Printing to the console: I am guilty of this myself. I admit it, I have used the console in the first programming years almost always, it is easy to use, easy to check what went wrong with the application and almost zero maintenance. What’s the problem then? exactly what I just said: It is easy to see what went wrong with the application. I can hear you saying: but the attacker has to be on the machine in order to attack the application? The answer is yes. BUT, remember a lot of the attacks happen by or through internal employees. Here is an article by ITSecurity that claims that internal threat is responsible for up to 85% of the application attacks.

- Log Files: Log files contain a lot of information. A lot of programmers log usernames, passwords and other sensitive data. Heck, I have seen applications that log the whole request in the log files. Now imagine, the following scenario: some problem happens on the production, the programmers ask for the log files in order to know what went wrong. The IT person sends the files to the programmers, and CC a whole bunch of people. One of the programmers decide that it is not their problem and forwards the email to another person and CC a different whole bunch of people. See how many individuals had access to the what’s supposed to be confidential data?….you get the point.

- Web Pages: Confidential data written unmasked to web pages is a serious problem. why? the problem is that these web pages could be cached either by the browser or by the cache server. Or it does not have to be that complicated, a shoulder surfer can see your user’s credit card number while they are enjoying their favorite latte at Starbucks and empty it in a matter of seconds.

There are numerous causes for information leakages. The above is just the tip of the iceberg. So the question is: is your application leaking?

No matter how much security is built in the application, if your configuration is not up to bar, I can safely tell you that it does not really matter how much security is in the application. As the old saying goes: it is as secure as the weakest link, so don’t let the configuration be your weakest link. Let’s have a closer look at web.xml, this is by no means a detailed list of security related settings in web.xml.

1- Session timeout: the amount of inactivity time after which the user will have to relogin. A lot of application now offers the user an extended login time so that the user doesn’t have to relogin.

This is bad.

The reason is that it extends the window in which an attacker can launch session-related attack, for example session hijacking, cross-site request forgery and clickjacking. The amount of session timeout shouldn’t be more than 30 minutes of inactivity, preferably less.

2- Error pages: custom error pages in web.xml in my opinion is the most underutilized safety net in J2EE applications. You need to change how you think about custom error pages. Think of them as your safety net, what keeps you in control even when you screw up as a developer and at the same time they are the most decent thing you can do to your user instead of those ugly stack-trace-exception-jargon-full pages.

The problem is that exceptions happens and when they happen they reveal a lot to the attacker. From the stack trace, the attacker will know what frameworks the application is built upon, which database is being used and maybe a bit about operating system too. Now the attacker can use this information to launch attacks against these system or use weaknesses in these systems to get to your application.

3- Welcome file list: The problem with these files is that they are publicly accessible, if an attacker running his tools and scanning sites, these are the pages that will appear on his radar. Why? because they don’t need authentication. Ofcourse, any other page that does not need authentication in your application will have the same problem, but these pages are easier to discover because your application gladly offer them to the attacker.

Make sure you are not leaking information in there. These pages are publicly accessible and can be leaking information or have any major vulnerabilities in them such as cross-site scripting.

4- Security constraint: This is where you can easily harden your application against HTTP Verb Tampering. You only need to allow the HTTP methods you want to allow and disallow everything else.

Now, there are more to web.xml than the above but starting with these four would help in mitigating a bunch of threats in only a few minutes of implementation.

Software Security is mostly self-taught

I was thinking the other day where does an average software developer or a quality assurance engineer draw their fundamental software knowledge from. The fundamentals and basics are mostly engraved in university and maybe the first couple of years of professional experience, education and learning journey does continue to go on but maybe at a slower pace. But where would a software engineer professional (whether they are a developer, quality assurance…etc) go for more self-education?

Older  software developers generations grew on Donald Knuth’s books, younger generations grew on books like Code Complete and Programming Pearls, generation Y grew on blogs like JoelOnSoftware and Coding Horror. As one of the fundamental problems always debated is that software security is not taught at universities and hence the huge amount of software vulnerabilities nowadays.

How does one learn about software security then?

• Get your hand dirty: This is by the far the best way to learn anything. Get involved in open-source security projects. That was one venue that benefited me the most starting out my own security career. OWASP in specific has an extremely awesome program called Season of Code, where you get to work on security projects and still get paid….awesome! Another guaranteed way to teach yourself web security is playing with WebGoat which is a web application intentionally vulnerable to numerous attacks, it consists of lessons and each lesson targets a specific vulnerability. Taking it up a notch would be to contribute a lesson yourself.

• Fortunately, the good old way of reading still holds: But be aware that the complexity and diversity of attacks nowadays are growing exponentially. We are living in a digital age, today’s attacks are orders of magnitude more complex than yesterday’s attacks. So books provide good basics, somewhere to start and general information regarding software security but you can’t really depend on them for everything. OWASP provides excellent material for education that is updated regularily, you should expect to find guides on web application security development in Java, ASP.NET, RoR…etc. Security code review guide and a bunch of other tools that you can use a long the way.

• Local Security Communities: Your local software security community is one of the best ways to learn about security. They usually provide some of the edgy up-to-date information about security but also you would expect to get some information that will really benefit you as a beginner in the field. Some of the communities you might want to check out are: OWASP, SANS, Security Klatch and ISSA. OWASP in particular focuses more on web application security, more often than not you will find an active chapter near where you are, check out the next event here

• Online Security Community: hmmm, isn’t it the same as the local? Short answer is NO. The local community you get to talk to people, exchange information, more localized view of the challenges that face the local community you belong to. However the online version of the community is more broad and general view of the security industry. For example, I live in Ottawa, Canada. And for those who don’t know, it is a government town, a lot of discussion and interests are driven by the local market which is The Government. We once had a guest in the local OWASP Chapter from Taiwan, their challenges as security professionals are somehow different than Ottawa, Canada.

• Blogs: Blogs are my favorite source of security news and information. But you got to find your perfect source. Here are some of my favorites:

What is your favorite source for teaching yourself software security?

Software security awareness has to start from the bottom up

Jeremiah Grossman wrote an excellent post on Infrastructure vs Application security spending last Friday that got me thinking the whole weekend. Jeremiah was trying to make a point that software companies are over-spending on infrastructure and under-spending on application security and more spending\attention should go towards the application security vs infrastructure security.

I kept thinking in the weekend what would be the first obstacle towards companies realizing the importance of software security and it threw me back to Erik Klien’s comment last week where I was trying to give software companies more reasons to implement an application security program. Erik made the point that software companies don’t see a need.

And they will continue to do so as long as developers are not fully engaged in the problem. I am not saying that developers are the problem at all, I am just saying that the key to the puzzle lies in the developers’ hands, not the execs, not the CIOs and not the managers.

CIOs are not the right approach because a place that needs a CIO is a place already aware of the problem and you don’t really need to do much there, this is not where I see most bang for the buck.

Executives are not the right approach either because they only talk ROI, stock value and dollars and application security was not proven yet to speak this language. Application security lies in the neighborhood of insurance but not ROI.

I think that developers are the right approach to the problem, developers love doing the right thing, they take a lot of pride in what they do and hate others pointing out mistakes in their code, application security provides all that.

Software developers are the most well-positioned parties to see the problem because they are the most ones familiar with the code and know the code upside down.

Unless we get developers to talk to developers about security, I think 5 years from now, the  level of spending on application security will be the same as it is today. More awareness in the developers communities is the way to go, developers will start understanding that little changes in their code can make a lot of difference security-wise. And I like to believe that security awareness is contagious, so many team while organizing OWASP’s Ottawa Chapter meetings, I see the impact and revelation-like impact on software developers when they understand how attacks happen and how attacker go to great length to find ways to exploit code.

From your experience what do you the right approach towards security awareness is?

There are a lot of reasons for software companies to not implement security as part of their SDLC or take it to another level and implement an overall security program. A lot of software developers are just not aware of the problem, and if they are, there managers are not. Sometimes the developers and managers are aware but the decision makers want the product out of the door yesterday and security being one of the nice-to-haves slips out of the door.

Sometimes I feel that the ambiguity the whole software security brings might be a cause. Companies want to do the right thing but it is just too much. Static code analysis, security code reviews, pen testing, forensics, controls…etc.

If you are starting from the first level already exhausted even the second level seems too far.

By the way, if you are a developer trying to convince your manager or a manager trying to convince the decision makers, this will help you a great deal.

Now, let me give you more reasons to take that first step towards a security program.

You don’t want to get sued: In October 2009, a bank got sued because they didn’t implement multi-factor authentication. Not for not implementing authentication, not for having a bug in their authentication system. But for not having a multi-factor authentication.

This story changed the way I look at things. Internet users are more aware of the dangerous web, and now they can do something about it. Are you ready?

Security architecture reviews will add another perspective to your design. From 10,000 ft altitude, security flaws at the end of the day are software flaws, you will have to fix them, and usually they are the most rushed, incomplete and annoying to your customers. Having security architecture\design review meetings at the beginning of your project can add a new and fresh perspective to your overall architecture. Regular design\architecture meetings will result in a fast, reliable, scalable architecture. Maybe user-friendly, easy-to-use as well, but this is not all what can cause bugs or flaws. Absence of security can also add flaws. Data breaches costs up to 200$ per customer record and the whole hack could cost up to a scary 8 figure number.

Security architecture reviews can avoid that.

You can use application security program as an employee retention tool. The most brilliant, smart and efficient programmers are never on the job market. There number one motivation is not salary, not a nice a team and not a foosball table. There number one motivation is an environment where they do great programming they are proud of. Every brilliant programmer I have seen takes a lot of pride in their work. Adding an application security program would allow your developers to get that feeling.

Did I say you should implement a security program to avoid being sued? Attackers might not be interested in your customer data in particular, they might be interested in just using your infrastructure to distribute malware or even more.

What is your reason for implementing an application security program or NOT implementing for what it is worth?

Custom error pages are one of the things that I have always seen in the nice to have requirements document when I was a software developer. You know, when we are done with the “real work”, you will start putting those in web.xml or web.config and start testing them out.

It almost never happened.

There were so many excuses, no time for starters. Sometimes, it was hard to get the log files from the production server so error pages, as bad as they were, were being looked at as a faster way to see the bug and get it fixed quickly to get that manager off our backs.

Plus, why implement something that will never be used, don’t we create perfect code anyways? all the time?

My experience in software security and my deep interest in graphic design taught me differently.

Neglecting custom error pages can add a HUGE security hole in your application. 75% of the attacks that happen on the internet today are targeted towards the application level. This includes your application but more importantly the underlying systems that support your application, like IIS, SQL Server, mySQL….etc. There are a lot of vulnerabilities with every single underlying system you might imagine. Windows, SQL Server, .NET, Oracle….etc. The attacker just wants to know which system you are using and then it is just a matter of time until they are able to penetrate it specially for non-patched systems.

Custom error pages keeps you in control: Although, we know that it is pretty much a bug and something went wrong with the application but a nicely designed error page allows you to fail with grace. Users wouldn’t think of this as a screwed up application, they will feel like being informed of what’s going and that’s it. It gives you a chance to say: I screwed … I am sorry

Your customers will appreciate it.

Custom error pages are user friendly. You know the ugly stack traces FREAK people out? Don’t you? The first thing I (and I am a software developer) think is: What did I do wrong?. You don’t want your customer to feel guilty or bad using your application. Custom error pages communicates the message to your users that it is the application’s fault, we are sorry and please come back again.

Users will understand. I do every single time LinkedIn gives me the screen above.

Custom error pages can add traffic to your web application: Just a last thought, if you changed your website and a customer bookmarked one of the old pages, he will get a 404 with the new website and he might not come back again, specially with traffic shaping\steeling ISPs. With the custom error pages, it is a way to redirect this user back to the new website by providing  a nicely designed 404 page with a link to the home page. This can add a lot of lost traffic back to you mainstream.

How did custom error pages help you?

I am interested in watching the job market trends and what employers are looking for. As I have recently explained, I was laid-off twice in 3 months so it got me thinking about the job market, companies’s hiring trends…etc. Something started catching my eyes recently which in my opinion is very important if you want to stay employable the next 5 years. Companies are more and more interested in hiring people that have skills that cross domains. And I don’t mean something like:  ”Java Developer with Great Communication Skills”.

If you remember late 90s, there was very clear separation between the skills you were deemed to have as far as employers were looking for. For example, if you were a C++ developer, you were a C++ developer, you wouldn’t be asked or expected to know much more, maybe some libraries related to C++ for example: MFC or ATL but no more. If you were a web developer, you were a web developer with no expectations to know C++ or any of that hardcore stuff. Same with databases.

This hard lines started to dissolve years after, and developers were expected to know more than one area in programming. It wasn’t too weird to see a post looking for a Software Engineer with

• 5 years of experience in Java, .Net and/or C++.
• Familiar with SQL Server 2000 and MySQL
• ….etc

That was alright, companies wants more bang for their buck. But starting maybe two or three years back when Web 2.0 started to really kick off and social media started to be the center of attention, but more importantly when more people found a way to monetize them.

Rules has changed.

Companies are no longer looking for: AN ENGINEER or A DEVELOPER. Companies are looking  for people that have skills that cross multiple domains.

I mean Java Developers who know how to design using Photoshop or Graphic Designer who knows how to write HTML, CSS and PHP or Programmer who knows who to setup a network or the .NET engineer who writes secure code. Imagine if you are a software engineer who can write patents! or if you are a quality engineer who can perform web penetration testing.

Companies want even more banks for their buck. Rules have changes, technologies have changed and so does the skills required. I can hardly see a product that would use one or two technologies only now, companies wants one person to do them all.

Having a slash in your title from an active perspective: Going after something that is not in your current core field opens up a ton of possibilities. Few years back, I got really interested in security I was a Software Developer and a Team Leader with next to zero interest in security in my company. The CEO of the company kind of encouraged me to go this route to bring extra expertise into the company and to get the word out there for the company I was working for and it worked. One thing led to the other and when I hit the wall during my career in development, I found the security road wide open with tons of possibilities. I once heard a wise man say: Prime yourself so when the opportunity comes you are ready.

Having a slash in your title from a passive perspective: Help me out here. If you are the guy who is responsible for laying off people and you have two candidates both are on the same level of expertise in C++ and get paid the same, however, one of them is really good in SVN installation and configuration. Which one would you keep? Think of each slash in your title as one more layer of defense from being dispensable.

You get the point.

I feel I am not stuck. When I started out my career I was a classic ASPs programmer, I was really good at it. However, I knew nothing else except some C++. The recession came and I was trying to find a job in Canada and there was NOTHING out there.

I felt stuck.

That’s when I learned the lesson of having a “slash” in my title, I quickly started learning Java and .NET which opened up more possibilities. Few years later, I wanted to break out of the programming\project management area to have a real “slash” in my title so I headed towards software security, I had no idea what I am going to be but I thought experience comes first and then the jobs. I remembered: Prime yourself and when the opportunity comes you are ready.

Bottom line is, I did NOT feel stuck.

Every time I think about software security and its  adoption inside software companies, I remember my habits regarding recycling. Let me give you a bit of recap, the City of Ottawa decided to introduce something called the green bin which is your regular garbage bin only green in color and should only contain the organic waste. This is ofcorse different than the black box which should have all the paper waste, blue box for the glass and plastic waste and finally the regular bin for everything else.

I like recycling because I feel I am doing something good to my environment. However, I hate the fact that I have to have 4 bins in my kitchen (or even in my small garage) just for garbage, I mean this way I won’t even have a space for my car for God’s sake.

Software companies must think of security the same way, too many bins out there ! Security code review, dynamic scans assessments, web app firewalls, intrusion detection, forensics…..etc

I just want to feel good and have one bin only to deal with!

I drive my wife crazy trying to optimize everything. She might be right though. We live together, no kids for now in a 3 bedroom house and I work from home. I am the kind of dude who sets the thermostat at 10:00 PM – 7:00 AM to 15 Celsius (60 F) because we are usually asleep during this time and don’t need the whole house heated. I have an oil heater in the bedroom that’s set automatically to start at 9:30 PM at 21 Celsius (70 F) to heat up our room and another oil heater that starts automatically at my office 8:00 AM till 5:30 PM because this is where I usually spend the whole day. The house thermostat is set to 21 Celcius from 7-8 AM in the morning as this is time we wake up and use most of the house and from 5-10 PM every day as this is again dinner and wind time and we usually use most of the house. I might be paying more for the electricity and gas combined but I feel that I am optimizing my resources and heating only the room I need.

Software security is the same thing, it is about optimizing your resources. Recent statistics reveal that as many as 70% of websites have vulnerabilities and according to Gartner and the U.S. Computer Emergency Response Team (U.S. CERT), 75% of new attacks specifically target the application layer. This means that there is a 75% chance your application will get hacked. How much time and money will be spent fixing the problem? And from my experience these problems never come at a good time. If you are still not convinced, check out the recent attacks against Google, Adobe and 30 other companies.

I rest my case.

They both will need some change. So I want to recycle, feel good and do the right thing but to think every time I am throwing something out?!! Ok, should this go to the blue, green or yellow bins? Oh, there is no yellow !! Let me get back to the chart to find out where does yogurt containers go!! What just happened here?? Something that I used to do without thinking twice about and was handled previously using the left side of brain, now it has to move to the right side of the brain, I have to think before doing it and we humans don’t like to think. At least not garbage related decisions.

Involving software security in the software development life cycle needs some changes, it needs changing the mindset of senior management, needs changing the mindset and habits of the developers, changing the way developers write code and introducing some controls. And again, we humans do not like change.

They both have no immediate monetary reward: While I am throwing the plastic-inside of the cereal box and trying to remember which bin it should go to, I think to myself, what’s in it for me? Yes, I will feel good, I have done something good to my environment and I can now stand proudly next to my green bin as being the “green” neighbor on the block. But what’s really in it for me? I mean I am paying for this every year to have my green bin picked up, can’t I just pay and not do the work involved?

Software companies think of it the same way. I was once talking to a CEO about the importance of security in the life cycle of their software service and he told me: I can’t introduce any security controls because this would mean extra cost to our business model and our customers won’t pay for it because they should have their software already secured 100%. I asked: Why do you add extra cost for quality assurance then?

Software companies don’t think that there is any return of investments for software security. I always ask this: Why do you do quality assurance then? why do you always put cost, estimate, time and engineers into quality assurance and not security testing?