The Current State of CI/CD Security, and How to Prevent Common Mistakes

An ever-growing need for faster and structured development has meant that CI/CD tools have become integrated into the core of an organization's development processes.

CI/CD tools help automate most of the basic, repetitive tasks of the application development process from end to end—from running basic indent checks on the code submitted to deploying compiled code into production servers.

And with the growing number of functions and features that CI/CD tools possess, they too require a certain level of access to perform these various functions. Consider, for example, a CD tool that deploys compiled code into production servers: this tool would require the same level of access and/or amount of permissions that would be required of a human to automate the deployment process.

Now that software builds are being completely automated by CI/CD processes, however, it’s entirely possible for attackers to silently inject malicious code into the build process itself. This has been seen quite often recently, with attackers injecting cryptocurrency miners' code into various software builds while going unnoticed.

This is why using misconfigured or vulnerable CI/CD tools can lead to a significant increase in your organization's attack surface.

• Recent CI/CD security incidents
• The most common CI/CD mistakes that lead to data breaches and network intrusions
  • Hardcoded credentials
  • Misconfigured containers
  • Misconfigured or misused environment variables
  • Misconfigured CI/CD tools
  • Outdated CI/CD tools
• How does Attack Surface Intelligence help detect critical CI/CD cybersecurity issues?
• Wrapping up

¶Recent CI/CD security incidents

Breaches within CI/CD tools often originate from running misconfigurations or out-of-date software. With modern breaches frequently automated, it doesn’t take long for an “automated bot” (similar to those seen attempting SSH logins all over the internet) to begin scanning for hosts running these out-of-date versions, whether through banner grabbing or by simply brute forcing a vulnerability PoC on a publicly accessible CI/CD tool.

A recent GitLab vulnerability allowed servers running out-of-date GitLab installations to be exploited into running DoS attacks, which, multiplied by the number of servers running the obsolete GitLab version, resulted in internet DDoS attacks exceeding 1 TB/s.

Beyond sending out internet attacks (an illegal act in most countries), GitLab has suffered authorization-related vulnerabilities which have allowed unauthorized users to enter your instance as well. Similarly, there have been ACL-related vulnerabilities too, allowing for compromised yet authorized accounts to push and pull data from unauthorized access areas within your GitLab instance.

While these vulnerabilities have been patched in future builds of GitLab, it remains critical to stay on top of these updated releases and, in turn, to update your instances (most often on your own) to reduce your attack surface.

Similarly, the popular open-source CI/CD tool Jenkins has been burdened with quite a few CVEs recently, ranging from XSS/CSRF vulnerabilities to permission bypass vulnerabilities that threaten to expose sensitive information to attackers.

Looking at the above examples, one might think that using hosted or SaaS CI/CD tools, and avoiding self-hosted tools, seems like an obvious solution—but hosted CI/CD tools can pose their own security challenges. A recent vulnerability saw Travis CI injecting environment variables via pull requests in the form of .travis.yml files, and considering how Git works, simply deleting the file from within Git wasn’t enough to secure the project's various secrets.

Similarly, even if your Git repository was private and secrets weren’t leaked out via the repository itself if your deployment process uses Git and clones the repository onto a production instance, any user accessing your in-production web app would be able to download or view the .travis.yml file containing your application/build secrets.

¶The most common CI/CD mistakes that lead to data breaches and network intrusions

¶Hardcoded credentials

Just as a human would require certain credentials to access a machine (such as a server) to deploy built code onto it, CI and CD tools require similar, if not the same, levels of access to build and deploy code into production servers.

These credentials stored within CI/CD systems can be anything from SMTP servers used to send email alerts regarding build completion or build errors, to something potentially more dangerous like SSH/root-level access to server systems that would be used to deploy built code.

Let's consider an official CircleCI example:

version: 2.1
executors:
  my-executor:
    docker:
      - image: cimg/ruby:3.0.3-browsers
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference

jobs:
  my-job:
    executor: my-executor
    steps:
      - run: echo outside the executor

Under the auth section for the docker container, we see a variable defined $DOCKERHUB_PASSWORD which is injected/replaced by CircleCI during the build process so that the password is hidden in the configuration file itself. However, if one doesn’t use these variables and defines the password, credentials can be stolen if the config file is made public by error.

Hard-coded credentials are a common source of compromises originating from CI/CD systems.

¶Misconfigured containers

Containers allow organizations to ensure complete software consistency between different environments. Linux containers have become commonplace in production, development, and build environments.

Similarly, CI/CD tools make use of containers to ensure that the software being built is using the same exact setup as the production environment. Using misconfigured containers here can lead to attackers entering containers created by CI/CD—and injecting malicious code into the software, as seen in our previous blog post Popular Misconfigurations that Make Containerized Apps Vulnerable to Attacks.

¶Misconfigured or misused environment variables

Environment variables have become an alternative to using hard-coded variables, but as has often been seen with environment variables, they are global to the build process and can be exploited or leaked since they are not limited to their access scope.

Another source of misused environment variables originates from the usage of Git, where public repositories often get mistaken commits that contain .env files, often used for storing environment variables. Removing these files from Git is considered a solution, but given that Git contains a history of every file and change, it doesn’t take a lot for attackers to go through the repository history and fetch these credentials from past deleted files.

With team members working all over the world, teams frequently rely on messaging tools like Slack to publish information about build statuses.

Jenkins uses pipelines to enhance your CI/CD processes with features that include announcing the build status through Slack, during or after the build process; for example:

import groovy.json.JsonOutput
// Add whichever params you think you'd most want to have
// replace the slackURL below with the hook url provided by
// slack when you configure the webhook
def notifySlack(text, channel) {
    def slackURL = 'https://hooks.slack.com/services/xxxxxxx/yyyyyyyy/zzzzzzzzzz'
    def payload = JsonOutput.toJson([text      : text,
                                     channel   : channel,
                                     username  : "jenkins",
                                     icon_emoji: ":jenkins:"])
    sh "curl -X POST --data-urlencode \'payload=${payload}\' ${slackURL}"
}

As shown above, if the Slack hook is exposed to the public rather than being used as an environment secret/variable, Slack teams can wind up being spammed or targeted even further, by attackers giving them clues (such as that of your team’s very use of Slack). To prevent situations like these, the hook URL must be configured as a secret and not exposed in plain text in the configuration files.

¶Misconfigured CI/CD tools

CI/CD tools come with a range of features, allowing for the automation of various time-consuming manually run tasks or steps within a software build process. CI/CD tools often include testing steps as well, which can further be used to automate checking whether an automated build is “good” or not.

Misconfiguring these features can lead to attackers gaining access to the CI/CD tool itself, thereby allowing them to modify or inject code during the build process itself.

GitLab is a popular open-source, self-hosting tool used for in-house Git and CI/CD.

Also available are tools like CircleCI, popular amongst various GitHub users due to its strong integration with GitHub. Using CircleCI requires building a YAML-formatted configuration file called config.yml, which contains various “instructions” that CircleCI follows to run the build process.

The configuration file is not limited to basic build instructions, however—it can also contain various parameters like passwords.

For now, let’s take a look at an official CI configuration:

version: 2.1

# Define the jobs we want to run for this project
jobs:
  build:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    steps:
      - checkout
      - run: echo "this is the build job"
  test:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    steps:
      - checkout
      - run: echo "this is the test job"

# Orchestrate our job run sequence
workflows:
  build_and_test:
    jobs:
      - build
      - test:
          requires:
            - build

As shown above, one can reference private docker hub container templates as well, via the username/password parameter. But if this config file leaks due to its public visibility in your GitHub repository, attackers can gain access to your private docker hub container templates.

Similarly, popular self-hosted CI/CD software GitLab recently shipped with the default root user login credentials “5iveL!” As this password was often left as-is, GitLab instances were inevitably compromised.

Non-self-hosted CI/CD tools are susceptible to misconfigurations, too. CircleCI too can be very easily misconfigured by sharing a project's config.yml exposed to the public. This config.yml contains every single step that has to be performed by CircleCI which can contain hostnames and other sensitive information such as SSH keys of production or testing deployments where the built code would be sent.

¶Outdated CI/CD tools

Just like any other tool or software, CI/CD tools too can fall prey to hacks, owing to the vulnerabilities they bring with them.

Using outdated CI/CD tools with vulnerabilities can lead to attackers compromising the CI/CD software itself and in turn being able to inject malicious code during the build process itself.

A recent CVE example of this would be a popular, open-source CI/CD tool like Jenkins, allowing attackers to bypass certain protection mechanisms and thereby access confidential areas of Jenkins itself.

¶How does Attack Surface Intelligence help detect critical CI/CD cybersecurity issues?

Attack Surface Intelligence helps automate your organization's security outlook, with automated scanning for the latest CVEs and various possible security threats (such as software misconfigurations). This ensures that your organization’s build processes are completely safe and secure. And when it comes to CI/CD tools, we got you covered—as shown in the following screenshot:

Below is a brief list of other CI/CD risks we detect. Bear in mind, that this is not a comprehensive list but an example:

¶Wrapping up

With the ever-growing dependence on CI/CD tools in an organization's build process, it’s important to secure the build process end-to-end, right from the code to the deployment of the built code, and this includes the CI/CD tool itself.

As end-to-end automation plays a key factor in the dependence on CI CD tools, automating scanning and the alerting of security vulnerabilities present within the tools themselves is necessary due to security misconfigurations or inherent vulnerabilities. Using hosted/SaaS solutions is often seen as the solution for reducing an organization's security requirements, but as seen recently, hosted/SaaS tools can also inject security vulnerabilities into your projects. Incorporating continuous scanning, detection, and alerting of possible vulnerabilities within your organization’s build processes plays a key role in ensuring their security, no matter what they are.

Using the Attack Surface Intelligence platform makes it possible to automate all your vulnerability scanning, detection, and alerting tasks and keep them up to date, to ensure that your organization is able to focus its time and resources on code while shielding your build process from attackers.

Book Your Demo Now

ESTEBAN BORGES

Esteban is a seasoned cybersecurity specialist, and marketing manager with nearly 20 years of experience. Since joining SecurityTrails in 2017 he’s been our go-to for technical server security and source intelligence info.

Sign up for our newsletter today!

Get the best cybersec research, news, tools,
and interviews with industry leaders

Subscribe