DevOps is eating the software development world, providing a much quicker way for development teams to push code to production and get feedback immediately.

At the beginning of 2021, a survey of 3,200 enterprises determined that 74% of organizations had already adopted DevOps practices in some form. From this group, 63% agree that it has increased their productivity, even with remote working in the long-term.

What makes DevOps work so well for organizations is that teams can move at a higher velocity, ensure reliability, scale wider, collaborate more, and yes, even improve security. Increasing the frequency and quality of releases can go a long way for saving time and budget.

The Importance of Scaling DevSecOps

Adopting DevOps also means adopting more automated processes. This decreases manual work and increases consistency in production. Complex systems can be managed more easily and less risk is imposed.

An example of this is in Infrastructure as Code (IaC), where the infrastructure is provisioned and managed using software development techniques such as version control and continuous integration. In IaC, data flows through machine-readable definition files, rather than physical hardware, data center servers or networking infrastructures.

With IaC, entire systems are managed into a few lines of code. This is much easier to deploy quickly.

How to Start Scaling DevSecOps Practices

Within each DevOps approach to scale organizations, there are unique security areas which need to be considered. Below, we break down the seven main practices of DevOps and how they impact security in the SDLC.

Infrastructure as Code (IaC)

As mentioned above, IaC simplifies infrastructures, making it easier to reproduce systems and reducing inconsistency in code deployments. IaC works with cloud environments, posing a unique set of challenges for security. Below are some of the top security risks in IaC to consider when figuring how to scale DevSecOps:

• Misconfigured templates or modules. This could easily and “scalably” expose the attack surface. For example, a misconfigured template can give public access unnecessarily to ssh access, databases or other network resources that shouldn’t be exposed publicly.
• 3rd Party Vulnerabilities Exposure. IaC could be used to provision compute and containerized instances by including base images that are pulled from different sources. These images would come with their own share of vulnerabilities which need to be detected and addressed.
• Ghost Resources. Automating and scaling IaC has a side effect of leaving ghost resources which has security, operation and reliability ramifications.
• Privilege escalation. Developers often use privileged accounts to configure environments. When this is introduced in IaC contexts, it results in issues like hardcoded credentials inside IaC templates. Then, this might be checked in to GitHub repos.

Configuration Management

As mentioned in IaC, configuration management can greatly impact security if not properly handled. When scaling DevSecOps, administrative activities tend to be the biggest threat to configuration management security. Some examples of these activities include:

• Device inventory. It is easier to lose track as components or infrastructure nodes are instantly created, repurposed or deleted. This creates the same security risk identified in the Ghost Resources above.
• Creation & approval of baseline configurations. This is a process where baseline configuration and images are approved from a security standpoint. Such a process might exist in source code creation, but it could be missed for IaC. In this case, perhaps it is under a different team.
• Establishing change approval processes. With change approval processes, standardization is key. A system is as strong as the weakest link. If development has really good security processes but DevOps doesn’t (or vice versa) a security gap is present.
• Creating compliance reports. Most companies nowadays are under at least one or more compliance or regulation. Compliance is not a security mandate anymore and it is becoming a business requirement. Take SaaS-based startups, for example. It is unlikely nowadays to get a sales conversion without being hit with the dreadful security questionnaire. Responding incorrectly on the questionnaire could derail the deal.
• Remediation of violations. Incorrectly handling one of the above configuration management activities could leave a gap in security. Having well-documented procedures and policies for configuration will help ensure that all settings are correctly put in place.

Continuous Integration

Continuous integration (CI) is an automated practice that brings together code from multiple contributors into one project.

• Insecure use of containers. A major (and often overlooked) security risk in CI is when containers are used and have security vulnerabilities. While this article won’t dive into detail about how to secure containers, it's important to know the effect they can have on CI if they’re insecure.

While containers have a short lifespan and are often walled off from each other, they’re also usually deployed on the same IP space. Meaning, if a hacker can access one container in that cluster, they can then access any other container within that same group.

It’s also important to consider that continuous delivery and continuous deployment also have security implications. Since they’re all usually practiced together, all of these security concerns must be considered at the same time.

Continuous Delivery

Continuous delivery pipelines are automated to help move application code along the SDLC, from developers to the end user or application service. Top risks in this automation include:

• Wrong tooling which creates a low-quality user experience. This can risk misconfiguration and misuse of the tool, which would give a false sense of security.

Using unversioned code. A great security risk occurs when a bug is identified, and it is not clear who is affected, which version needs to be fixed or which customer needs their customer updated. Also, a lack of a mechanism to update software could also pose security risks. For example, the development team could identify exactly what the bug is, fix it and have a release ready. However, if the customer is not ready or willing to update the software, then the issue ultimately goes unresolved. In these cases, there may be a lack of backwards compatibility or lack of training for the people using the software.

Continuous Deployment

Continuous deployment (CD) is commonly practiced with continuous integration. CD, in itself, is an automated practice to ensure that software can be released effectively when requested. A CD process could or could not also include an approval process, depending on the organization’s policies.

While CD’s main function is to improve efficiency in the deployment of code, the practice does have some possible security risks including:

• Code with well-known vulnerabilities that is imported into the pipeline from a third party source
• Unauthorized access to code repositories
• Leaving keys or credentials in scripts

Continuous Monitoring

Continuous monitoring, or continuous security monitoring (CSM), is a practice which benefits the security of scaled DevSecOps. It is considered a threat intelligence approach that automates the monitoring of controls and vulnerabilities to then support organizational risk management decisions. A good CSM practice should provide visibility on all digital assets within your organization, including cloud storage, web applications, mobile applications, IoT and connected devices, email servers and public code repositories, among others you may have.

Integrating continuous monitoring into your DevOps practices has several key benefits, including:

• Visibility on all IT assets and controls
• Metrics on cybersecurity status
• Verification of compliance with security policies

Automated Testing

In 2018, 61% of organizations said a lack of automated or integrated security testing tools was creating challenges for their business. Like continuous monitoring, automated security testing is another protective practice. Many types of automated security testing exist, including vulnerability scanning, penetration testing as a service (PTaaS), DAST, SAST, and more.

Vulnerability scanning can automatically run analysis on code and systems to identify security weaknesses. These tools can be affordable and quick, though they don’t always provide deep insights.

DAST and SAST are quite similar in this respect. DAST tools are used for web applications through performing black box testing. DAST is more beneficial in the design and build stages. SAST tools, on the other hand, reviews source code to identify possible vulnerabilities. Though, like DAST, it is also best used before release to production.

Penetration testing as a service (PTaaS) is the most comprehensive of the tests, though it does take more time. Like the other tools, it can integrate into an existing SDLC that frequently deploys new code updates. A unique difference between PTaaS and the automated scanners, is that PTaaS uses a real human penetration tester to search for and identify security risks. This produces more in-depth findings than other testing methods.

Conclusion

DevOps is a fantastic methodology which can certainly increase the agility and efficiency of code deployment. However, all DevOps practices need to be considerate of security implications when scaling an application’s deployment. Including the security considerations above, managers also need to have their teams properly trained on application security and develop an overarching, supportive organizational culture.

References:

1. https://levelup.gitconnected.com/digestible-devops-the-7-devops-practices-8bd8b34e1418
2. http://cdn.swcdn.net/creative/v3.4/pdf/techtips/Secure_Configuration_Management.pdf
3. https://www.upguard.com/blog/continuous-security-monitoring
4. https://www.cypressdatadefense.com/blog/sast-advantage/
5. https://dzone.com/articles/common-security-challenges-in-cicd-workflows
6. https://thenewstack.io/top-5-security-risks-for-infrastructure-as-code/