CI/CD is an important part of the application development cycle. However, criminals are exploiting vulnerabilities in CI (Continuous Integration)/CD (Continuous Delivery) pipelines to steal sensitive information, mine cryptocurrencies, and deliver malicious code.
Recent cyber attacks have exploited vulnerabilities in CI/CD pipelines and developer tools, which highlights the urgent need to improve the security of developer infrastructure. The most typical case is the Codecov supply chain attack, which also reminds users that confidential information should not be stored in CI/CD environment variables regardless of how secure the environment is.
The Codecov attackers invaded the Bash uploader used by thousands of developers, successfully stealing credentials, keys, and API tokens from customer environments and hiding for two months without being discovered, and it is said that they have compromised hundreds of restricted customer networks. Similarly, attacks on automation tools (such as Jenkins, GitHub Actions, and cloud-native container environments) have also further prompted enterprises to explore and deploy effective defense measures for these tools.
The following are some best practices to ensure the security of the CI/CD pipeline.
Firstly, please do not store sensitive information in the CI/CD environment.
The reason why the Codecov supply chain attack was successful lies in the fact that the attacker leaked environment variables containing hardcoded sensitive information, including passwords, tokens, and keys. Some of the credentials allowed the attacker to access the company's private GitHub repositories, from which further data leakage could occur from these private repositories that should have contained confidential data.
Although multiple Codecov customers including HashiCorp, Twilio, Rapid7, and Monday.com have disclosed the impact of supply chain attacks, the most far-reaching data breach so far has been at the Japanese e-commerce giant Mercari. After the Codecov attack, more than 27,000 records related to financial information, merchants, business partners, company employees, contractors, and various entities associated with Mercari's customers were leaked to unauthorized external attackers.
Although these attacks may have started from the Codecov vulnerability, some people also question why personal identifiable information (PII) such as customer financial records is stored in private GitHub repositories.
Similar concerns have been raised about the GPG private keys stored in CI/CD environments by HashiCorp. These are keys released by HashiCorp for signing and verifying software versions. Before the key is revoked, attackers can abuse the key to forge HashiCorp's signature on the release of malicious software. A developer said on Twitter: 'Why isn't anyone talking about the issue of HashiCorp storing their signature key as ENV in Vault?'
Enterprises need to reconsider what information can be stored in CI/CD tools, environment variables, and private GitHub repositories. If an application needs to store credentials or tokens in these places, it is best to store the credentials in an account or resource with the lowest permissions, which is only necessary to complete the task and is often referred to as the principle of least privilege. This way, even if private information is exposed in an unprecedented attack, the loss can be controlled.
Reviewing automatic pull requests and scheduled tasks
CI/CD automation tools like GitHub Actions allow developers to set up scheduled tasks for their code repositories, such as automatically reviewing and processing incoming pull requests. But what if a contributor with malicious intent submits a pull request to an open-source project? What happens then?
In April 2021, GitHub Actions were abused by attackers who submitted automated pull requests to hundreds of repositories, with the goal of mining cryptocurrencies using GitHub's infrastructure. This large-scale attack occurred after a vulnerability in GitHub Actions was exposed at the beginning of February.
At the lowest permission level, these pull requests can abuse GitHub's servers to mine cryptocurrencies or execute malicious code from attackers. If the project leader is negligent and merges these pull requests, they will introduce malicious code into their repository and the broader software supply chain. In May, GitLab reported that attackers on its platform misused the 'free minutes' (quota) allocated to new accounts to carry out similar cryptocurrency attacks.
Because the essence of CI/CD automation tools like GitHub Actions and GitLab is to provide convenience for the automation of key tasks, managing them becomes a challenge. It is possible that a feature intended to be used responsibly is quickly turned into a security vulnerability after being misused by threat actors.
GitHub recently announced new features to combat the abuse of its Actions platform by encrypted attackers. Pull requests from first-time contributors will need to be manually approved by repository collaborators with write access before any action workflow runs. GitHub Product Manager Chris Patterson said in a blog post: 'When first-time contributors open a pull request, they will see a message that maintainers must approve their action workflow before it can run.'
Leading CI/CD solutions and DevOps platforms can follow GitHub's approach and add some security checks to prevent malicious actors from massively misusing their infrastructure.
Three, strengthen and regularly audit cloud-native containers
Practice makes perfect, and standard best practices have great reference significance. For example, ensuring that production container configurations are correct and reinforcing common attack vectors, including protecting pipeline configurations.
However, simple misconfigurations are sometimes hard to find. So, is there a vulnerability in the Docker environment? Therefore, it is necessary to conduct regular security audits of containers to identify weaknesses, scan container images and manifest files to find common security issues, and these measures are still very helpful.
Investing in reliable cloud-native container security solutions is also a wise move, as they can automate much of the work. A large number of security vulnerabilities are reported every year, and they are difficult to detect.
In addition, as companies adopt the Kubernetes framework and Docker containers to deploy their applications, container security solutions with built-in web application firewalls can detect and block suspicious network traffic early on. This can prevent greater damage, even if the attacker can penetrate the container and gain initial access.
Four, integrate deep code scanning to automatically perform code quality checks
Before code is officially submitted, automated tools are needed to identify code quality issues, security vulnerabilities, and errors such as memory leaks or race conditions, which can ensure an effective strategy for the security of the CI/CD pipeline from the very beginning. Although the focus is mainly on preventing network attacks, even minor errors can also have a large-scale impact. For example, Fastly's global outage caused major websites worldwide to go offline.
Solutions like GitHub code scanner or Sonatype's Lift can be seamlessly integrated into existing coding workflows and provide basic protection for developers. Ultimately, an organization's goal should be to support its developers in doing their work well, while also preventing errors or security vulnerabilities from being introduced into applications as much as possible. This requires a synergy between the development and security teams. Reminding developers of potential oversights while they are coding can save everyone's time and ensure the entire CI/CD workflow from the very beginning.
Fifth, patch the latest CI/CD tool vulnerabilities as soon as possible
In March 2021, attackers exploited a cryptocurrency mining botnet named z0Miner to mine Monero (XMR) cryptocurrency on vulnerable Jenkins and ElasticSearch servers. By exploiting remote code execution (RCE) vulnerabilities in internet servers, attackers attempted to infect and take control of the automated infrastructure for their criminal activities.
Notably, last year it was reported that attackers were using Jenkins servers to carry out distributed denial-of-service (DDoS) attacks. The event was traced back to a UDP amplification reflection DoS attack vulnerability, tracked as CVE-2020-2100, which affected versions below Jenkins v2.219 and Jenkins LTS 2.204.1.
Once these serious vulnerabilities are discovered, patch the automated tools and pipelines immediately, which is crucial for ensuring the security of the CI/CD infrastructure.
Sixth, verify the integrity of updates before applying them
Applying the latest updates and patches sounds like a reasonable measure, but it seems difficult to determine whether the updates have been tampered with? For decades, the suggestion to 'update to the latest version' has been a mantra of security experts, but after the SolarWinds supply chain attack event, this suggestion has been challenged.
In the SolarWinds incident, malicious updates to the Orion IT product allowed attackers to distribute malicious code to more than 18,000 downstream customers. However, the 'local upgrade feature' of the Passwordstate password manager was compromised again, distributing malicious updates to Passwordstate users. Therefore, blindly applying product updates may not be a good thing.
In the case of Codecov, a simple integrity check revealed a vulnerability that had been present for two months. A customer noticed that the checksum (hash value) of the Bash Uploader hosted on the server was different from the legitimate checksum listed in Codecov's GitHub repository, and immediately contacted Codecov. They then fixed this issue.
Therefore, the depth defense method requires verification of the integrity of any updates, patches, and downloads to exclude the risk of attacks from complex supply chain attacks.
Reference links
评论已关闭