Securing Your CI/CD Pipeline: Exploring the Dangers of Self

Home » Security Bloggers Network » Securing Your CI/CD Pipeline: Exploring the Dangers of Self-Hosted Agents

Continuous Integration/Continuous Deployment (CI/CD) pipelines have become crucial to modern software development practices. CI/CD pipelines can significantly improve development efficiency and software quality by automating the process of building, testing, and deploying code. Most modern CI/CD platforms (like GitHub actions, Circle CI, etc.) offer an option to run the pipeline process over a self-hosted runner – an agent hosted by the user instead of the CI/CD platform, to execute jobs on their own infrastructure.

With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs. Additionally, you can install software available on your local network and choose an operating system not offered by the platform. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud environment. The alternative is to use a build-as-a-service runner offered by SaaS CI/CD platform where the user has no control over the environment.

One might use self-hosted runners for several reasons:

Compliance and privacy: many organizations are bound to strict regulations that prevent them from building and sending sensitive data to SaaS build services. For example, organizations from the financial or gov sectors often prefer running the build on their internal servers.

Special specifications: when building software for uncommon operating systems or different architectures, the build process needs a machine with specs that match their requirements. This is also the case in many embedded, IoT, or client-side applications.

Cost saving: organizations with existing cloud infra / private cloud can save costs if they choose to use self-hosted runners and manage resource pools themselves.

Given the reasons above, self-hosted runners have gained popularity in many organizations. However, self-hosted runners can pose significant security risks, especially if untrusted workflows run on them. Malicious programs may compromise the build machine, and escape from the runner's sandbox, exposing access to the machine's network, secrets, and more.

Using a self-hosted runner can offer advantages, such as improved performance and increased environmental control. However, there are also security risks associated with it:

Self-hosted runners require dedicated servers or virtual machines (VMs) to operate. The security of these servers is crucial. If the server hosting the runner is not properly secured, it can become a potential entry point for attackers.

Self-hosted runners typically require privileged access to the repository and the system on which they are installed. It's important to carefully manage access control and restrict the permissions granted to the runner. Unauthorized access to the runner could lead to potential data breaches or malicious code execution.

Failure to isolate self-hosted runners from critical systems and sensitive data increases the risk of a compromised runner leading to unauthorized access and potential data breaches. Without proper isolation, attackers who gain control of the runner may be able to move laterally within the network, potentially compromising other systems and data.

Neglecting to regularly update and monitor the self-hosted runner software exposes the system to known vulnerabilities and leaves it susceptible to attacks. Without timely security patches and updates, attackers can exploit known weaknesses in the runner software. Insufficient monitoring increases the chances of undetected malicious activities and delays in responding to security incidents, potentially resulting in extended periods of unauthorized access or damage.

Using a self-hosted runner without restricting access to authorized users allows anyone to exploit compute resources for activities like crypto-mining and other activities that can inflict financial damage.

GitHub Actions have gained significant adoption in recent years as a powerful CI/CD automation tool for software development. With GitHub Actions, developers can automate tasks like building, testing, and deploying software, allowing them to focus on more important tasks. GitHub Actions offers a wide range of customization options, including pre-built actions, community-built actions, and the ability to create your own actions. As more and more organizations adopt GitHub Actions, it is becoming an increasingly important tool for modern software development.

GitHub also provides the option to use self-hosted runners. You can add self-hosted runners at various levels in the management hierarchy, including repository-level runners dedicated to a single repository, organization-level runners that can process jobs for multiple repositories in an organization, and enterprise-level runners that can be assigned to multiple organizations in an enterprise account.

During our research, we’ve scanned over three million public repositories. For each repository, we’ve inspected the different GitHub Actions files and extracted the runner used by each job. We have learned that out of 3,106,445 public repositories checked, 43,803 are using self-hosted runners. Using a self-hosted runner in a public repository increases the aformantioned risks dramatically, as any GitHub user in the world could initiate an attack.

GitHub provides organizations with several options to configure their self-hosted runners. Users need to verify that their organization is following configuration best practices to prevent potential risks.

Runner groups:When a self-hosted runner is defined at the organization or enterprise level, GitHub can schedule workflows from multiple repositories onto the same runner. Consequently, a security compromise in these environments can result in a wide impact. To help reduce the scope of a compromise, you can create boundaries by organizing your self-hosted runners into separate groups.

Disallowing public repositories:If you use self-hosted runners with public repositories and do not properly configure access controls, an attacker can potentially gain access to your self-hosted runner machine by creating a pull request that runs a malicious workflow. This can allow the attacker to execute arbitrary code on your machine, access sensitive data, or perform other malicious actions, depending on the level of access your self-hosted runner machine has.

For example, an attacker could modify the code in a pull request to execute commands that install malware on your self-hosted runner machine or steal sensitive data from your organization. This can lead to serious consequences, including data breaches, loss of intellectual property, and damage to your organization's reputation.

To avoid the risk, it's advised to use self-hosted runners only with private repositories. You can limit the attack surface and reduce the risk of unauthorized access to your self-hosted runner machine.

Access control:You can restrict what organizations and repositories can access runner groups. Additionally, you can restrict the runners to specific workflows. For more information, see "Managing access to self-hosted runners using groups."

In a blog post posted by @ryotkak, we can see an example of how self-hosted runners impose risk and, if used incorrectly, can lead to sensitive information theft. ryotkak shows an example of a workflow created by GitHub. The vulnerable workflow is part of the actions framework itself. Under specific conditions, the self-hosted runner was reachable to any attacker. The token stolen in that example was a token owned by a developer from GitHub itself which attackers could have leveraged to carry a wide-impact software supply chain attack.

In this demo, we show how we created a public GitHub repository with a self-hosted runner. This example shows an external malicious collaborator performing several attacks.

In the first image, we can see the runner pwn_me is added to the runners group pool.

Our next step was to configure a workflow that runs on our self-hosted runner and is triggered by the pull_request trigger:

Now, all an attacker needs to do is create a fork with a modified workflow file that retrieves the runner's secrets and sensitive information. An example of a job created by a malicious contributed:

This video shows how we created a pull request from a fork and gained access to the /etc/passwd file as an example.

The following best practices are relevant to all CI/CD self-hosted platforms. Whether you use GitHub Actions Jenkins, GitLab, or any other platform, these guidelines will help keep your self-hosted runners safe and your software supply chain secure:

Make sure no sensitive information is stored over the runner. For example, private SSH keys, and API access tokens, among others.

Make sure the runner has minimum network access to other internal assets. For example, Azure or AWS management consoles or metadata services. Management consoles and metadata services are panels the cloud vendor provides to retrieve cloud environment management or perform information. An attacker with access to these resources can change the victim's cloud configuration or extend their attack surface. The amount of sensitive information in this environment should be minimal. You should always be mindful that any user capable of invoking workflows has access to this environment.

Ensure all services inside the runner are up-to-date and free from known vulnerabilities.

Avoid persistency – make sure each job runs on a clean, fresh agent container/VM

If using containers, run at minimum permissions.

Avoid running containers in privileged mode.

Make sure the container is mounted to dedicated devices and does not share resources with the host.

Run only a single container per host.

Use HTTPS for communication: Use HTTPS to encrypt communication between your self-hosted runner and GitHub, and avoid using unencrypted HTTP.

Overall, self-hosting CI/CD pipelines can be risky if not properly managed. To minimize the potential dangers, it's essential to follow security best practices, such as regularly updating self-hosted agents, implementing access controls, and monitoring for suspicious activity. Additionally, organizations should consider using cloud-based CI/CD services that can provide better security and reliability than self-hosted solutions.

The Legit Security platform connects to your build environment to detect attack attempts in your pipeline and much more. We identify different misconfigurations, such as a public repository that uses a self-hosted runner. If you are concerned about these security risks and others across your software supply chain, please contact us  to request a demo on our website.

*** This is a Security Bloggers Network syndicated blog from Legit Security Blog authored by Nadav Noy. Read the original post at: https://www.legitsecurity.com/blog/securing-your-ci/cd-pipeline-exploring-the-dangers-of-self-hosted-agents