Don’t neutralize the profitability of a CI/CD business by failing to consider risk. Here are best practices to ensure your software development pipeline is secure.
As adoption of the Continuous Integration/Continuous Delivery (CI/CD) approach to software development continues to snowball, many organizations are reaping the benefits. CI/CD helps organizations deploy software faster and more often, accelerating code changes in response to changing business conditions.
But the automation and complexity of CI/CD pipelines and processes can introduce significant security risks into the development process if organizations do not plan carefully. Not only do organizations need security checks built into the fast-paced workflow of the CI/CD process, but the tools and integrations of the CI/CD pipeline itself must be protected.
Here are eight key practices to help increase CI/CD security on all these fronts.
1. Strengthen your CI/CD infrastructure
The first step to excellent CI/CD security is to ensure that the infrastructure supporting the entire pipeline is properly configured, updated, and isolated to prevent lateral movement.
That means strengthening your CI/CD infrastructure, said Liv Caspi, CTO and co-founder of Legit Security.
“For example, if you’re running a Jenkins server, make sure to harden the server itself, its configuration, its authentication, and so on so that no one can hack it, or if they do, they can’t move forward.”
– Liav Caspi
Victor Gazdag, managing security consultant at NCC Group, explains that “updating fundamentals such as password policies, network segmentation, least-privilege principals for identity and updates will help prevent many attacks.”
“All the necessary barriers and manipulations to establish safety hygiene come into play here.”
– Viktor Gazdag
Brian Price, founder of Interplay, a security-focused managed services provider, agreed that software updates on CI/CD systems should be an especially high priority.
“Frenzy software updates and patching is critical on every device that touches the development chain. Turning on automatic updates is not enough; A system that manages, reports and can roll back patches.
– Brian Price
2. Establish developer guidelines and railings
Another fundamental starting point for establishing CI/CD security is to set effective security guidelines and railings for developers on how they code and operate in everyday environments, says René Milzarek, Managing Director of CrashTest Security.
“For a less security-conscious company, I suggest starting with code reviews and creating developer guidelines that address cybersecurity. Define how you handle updates to static secrets, encryption algorithms, and third-party components. Later, I’ll move on to automating some of this Steps with specific tools but remember that there is no clear answer because it depends Institution and application You need to take this decision keeping in mind the above risks at hand hand“
– Rene Milczarek
Sometimes setting up railings starts with simply putting controls in place to ensure that developers play by the same rules in a CI/CD environment as other privileged users play on some other sensitive platform. An example: implementing password sanitization and two-factor authentication.
“Developers may know they need strong, unique passwords across toolsets, but they may not actually practice it. A good password manager can dramatically improve password hygiene. Two-factor authentication is also required. It is widely available, and a password manager can act as an authenticator in most cases.
– Brian Price
3. Do the pemissions and secrets A priority
Speaking of passwords, gaining control over secrets management and identity management is one of the most valuable ways organizations can take their CI/CD security to the next level. This includes managing permissions to development systems and sensitive data stores, as well as securely storing and managing everything from passwords to tokens, encryption keys to APIs.
As Gazdag explains, even though the industry has been littered with horror stories over the years about hardcoded credentials and secrets leaking from development environments, weak to non-existent secret management is still rampant.
“When we see a lack of CI/CD best practices, my main concern is secret management and identity management. Identity management can be very complex and often over-permissioned. The scope and storage of secrets is also often overlooked and misconfigured, allowing everyone or a large audience to access them.”
– Viktor Gazdag
Ultimately, the goal should be to enforce the rule of least privilege—for developers and other people who touch these systems, as well as for the machines and automated processes that integrate into the stack.
Stavros Zavrakas, managing director of software development firm Orthogonality, said the idea was to ensure that not everyone had access to the network and pipeline, “so that it stays under control.”
“Admit only those who are required for the job. It is important to regularly check on the system inventory and remove any components that are not required for operation.
– Stavros Zavrakas
4. Remove the sole power of authority
As organizations use a kaleidoscope of tooling to build automated CI/CD pipelines, some of the biggest risks can occur when automated actions to trigger code commits or deployments are inappropriate.
Zavarkas said removing the single power of authority in the pipeline is critical to safety. “Establish control mechanisms to ensure that no authority has the authority to send sensitive code through the pipeline without verification or validation from another source from the outside,” he says.
On the developer side, organizations can reduce the attack surface by reducing permissions around source code, says Nir Waltman, founder of software supply chain security firm Arnica.
“If the company culture is to provide access to push code for all developers, implement branch protection policies to review pull request reviews and CI/CD permissions and triggers by appropriate owners.”
– Nir Waltman
5. Carefully plan when and how security tests are performed
Some of the most important practices in CI/CD are related to security testing. As organizations plan their CI/CD workflow, they should carefully plan when and how security tests are performed.
All too often, organizations rely heavily on static methods like code audits and code reviews, said Ketron Evans, principal security researcher at the Infosec Institute. “This completely ignores that many critical vulnerabilities only exist at runtime when external libraries and other code infrastructure are pulled,” says Evans.
Evans said he believes one of the most important CI/CD best practices is to weave regular dynamic testing into the development of software. “It has the most immediate impact and can identify the most critical vulnerabilities.”
Larry Maccherone, a long-time DevSecOps practitioner and DevSecOps Transformation Architect for Contrast Security, goes a step further, stating that organizations should consider the big picture with security testing as part of high-coverage automated functional testing – aka end-to-end, integration. . , or smoke testing — instead of focusing on unit tests.
“Having a high-coverage automated functional suite is my secret best practice for security conscious organizations. This is hidden because most security people think that focusing on a test suite that targets ‘quality’ rather than directly targeting security is out of their swim lane but it’s not.
– Larry McCarron
Macherone said he believes testing should be done before a pull-request merge decision, which requires a container-based CI system that will deploy the necessary resources to run the pipeline.
“It is not a trivial engineering task but it is worth it. Once you do this, you run all your tests in as parallel a manner as possible so that the pull-request completes fast enough to inform the merge decision.”
– Larry McCarron
Milzarek says getting testing right is a matter of pulling together a combination of specific tools, including a balance of software structure analysis, static code analysis and dynamic code analysis. “You need to understand that different tools have different strengths to address threats, and you need to build a security culture.”
“If your developers don’t share awareness, they will find ways to automate CI/CD checks or become demotivated by breaking builds.”
– Rene Milczarek
6. Verify the security of the build software
Caspi says that when development teams use open source components and platforms to write CI/CD automation, they don’t blindly trust the stuff they’re using to build it. This is a microcosm of the broader software supply chain security issue, but narrowly focused on the software that underlies the CI/CD pipeline.
“A factory line that produces software is software itself. And it can be targeted. For example, when you look at SolarWinds, the build server at SolarWinds was hacked to build malicious software. So, verify the build components you use, plugins, GitHub actions you depend on. Make sure you have control over the other ingredients you’re using and make sure you’re using a verified ingredient and not something so obscure that you don’t know what’s going on.”
– Liav Caspi
7. Make the CI/CD pipeline more visible
Ideally organizations should monitor and track activity across the CI/CD pipeline to detect potential malicious behavior, unauthorized access to sensitive systems and code repositories, and potential threats latent in the CI/CD pipeline.
One of the most important priorities organizations can engage in is making sure they see the entire pipeline and that the pipeline itself flows. Todd Weber, an operating partner at Ten Eleven Ventures and longtime security-focused CTO before that, said many existing cybersecurity technologies look at one or two pieces, such as repositories or code, but not all the components together.
“My biggest concern is that the ecosystem we’ve set up for this process has become so complex, with so many third-party dependencies, that no single party can monitor or even compute everything, especially now that we have CI/CD. Above sees such expansion. Pipelines with categories like Data-as-a-Service and AI-as-a-Service.
– Todd Weber
As a VC, Weber says this is the biggest opportunity for future innovators, but right now organizations must do some hard work to gain complete visibility into how the pipeline works and which third-parties affect the security of that software factory.
“Keep digging deep. There are a lot more third parties involved than you might think. Identifying all the components of your pipeline looks deceptively easy — it actually takes a lot of work.”
– Todd Weber
8. Use threat modeling
Threat modeling can also be a valuable next step for organizations that already have many of the basics of secrets management and security hygiene covered in a CI/CD environment. Gazdag explains that organizations should not only consider developing risk models for the software components of the software product they are building, but also for critical systems in the build environment.
“Continuous risk modeling will show connection boundaries, connecting elements and data flows. This will allow an organization to see the big picture of gaps and blind spots, so that they can put in place appropriate preventive and security controls. For example, developers can see where credentials are stored, used, where they came from, how they are stored, and who can access them.”
– Viktor Gazdag
A call to action
with attackers They are turning their attention to the growing development environmentOrganizations must take measures to prevent attackers from infiltrating the CI/CD infrastructure so that they can not only breach the organization but also introduce dangerous errors into their applications.
*** This is a Security Bloggers Network syndicated blog from the ReversingLab blog written by Erica Chikowski. Read the original post here: https://blog.reversinglabs.com/blog/8-cicd-security-best-practices-software-pipeline