A Little Backstory

Your company secrets are safe, right?  Well, you can’t assume just because you enforce company wide 2-factor authentication on Github that you aren’t vulnerable.

A while back when I was working at a young silicon valley startup, we found out why you shouldn’t embed credentials in source code, the hard way.

We had been on a hiring blitz, which is not uncommon in a start-up, and had been expanding the QA team. Then, one fine day we discovered our mailing service had blocked our account and raised a security ticket with OPS.

It turns out that one of the new hires had “accidentally” forked the main server source repo and made it available publicly.  Within minutes, bots started scouring the code on github for anything that looked like a key or credential, and with our code it hit the jackpot.  In one json config file, could be found all the amazon AWS credentials, mail service credentials, azure, crashlytics keys and more.

The OPS team performed their duties with heroic effort in order to help stop the leak quickly, but after the dust settled, hackers had been using the mail service for spam, our domain was blacklisted on several lists and our mail reputation took a tremendous dive.

Prevention

This could have been avoided in several ways.  First off, developers *should* know better than to put credentials in source code, but that is a culture and training issue which can’t always be solved immediately. The second barrier should have been a static analysis tool incorporated into the build that would detect suspicious hard coded strings.

CORPORATE APPLICATION SECURITY OPTIONS

The Fix

If you only have a few credentials, a simple solution is to use environment variables on the server.  This has the advantage of being configurable per-machine (ie. developer settings can be separate from server settings), and that exposing the values usually requires significant privilege escalation.  The challenge is to ensure the scripts used for setting up the environment (often a dockerfile) also do not contain the keys.

A better solution is to use a password vault to load keys at run time, such as AWS Secrets Manager.   Using the API call GetSecretValue it is possible to retrieve keys from the password manager at runtime.  It is recommended to use an existing SDK to ensure that accessing the password vault is fully secure, as it is easy to miss important configuration that might cause the GetSecretValue request to leak sensitive information.  This method comes with the additional benefit that it is easy to roll passwords without requiring any redeployment.

Resources:

• https://www.veracode.com/blog/managing-appsec/hardcoded-credentials-why-so-hard-prevent
• https://stackoverflow.com/questions/13820640/whats-the-most-secure-way-to-embed-a-password-inside-java-code
• https://www.technewsworld.com/story/71369.html