Blogs

Setting up GitHub Runners on DigitalOcean

Category
Software development
Setting up GitHub Runners on DigitalOcean

Introduction

Using GitHub Actions is free for private repositories and it comes with a certain amount of free minutes and storage. Organization using GitHub Free has 2 000 minutes (per month) at its disposal, which is also related to architecture on which the job is executed. 

Jobs that run on Windows and macOS runners that GitHub hosts consume minutes at 2 and 10 times the rate that jobs on Linux runners consume. For example, using 100 macOS minutes, would consume 1 000 minutes included in the account, which can be described using simple formula:

100 used minutes = 10 building minutes x 10

An option that emerges as the best solution is using a self-hosted GitHub runner. Self-hosted runners can be used with a repository, at organization or enterprise level. In this article we will focus on self-hosted runners at organization level. 

Using self-hosted runners on organization level enables configuring processes for multiple repositories. It offers more control over hardware, operating system, software etc. but also bears some other responsibilities (previously handled by GitHub) like updating software and OS along with security issues.

With self-hosted runners there is no need to have a clean instance for every job execution (like with GitHub-hosted runners).

Self-hosted runners open connections to GitHub to check if any jobs are queued for processing, so there is no need to allow GitHub to make inbound connections to self-hosted runners.

Note: Self-hosted runners are recommended to use with private repositories.

Setting up DigitalOcean Droplet

Digital Ocean droplet is a Linux-based virtual machine (VM) that runs on top of virtualized hardware.

Adding public key to DigitalOcean account

Note: It’s safe to share SSH public key because it cannot be used to recreate the 
          private key.

To add an SSH public key to the Digital Ocean account, log in to the control panel. In the “Account” section click “Settings”, then click the “Security” tab at the top of the page. In the “SSH keys” section, click “Add SSH Key”.

Copy the public key and paste it into the “SSH key content” field.

Tip: Key files are saved in a hidden SSH folder in the home directory and the public
        key has the .pub extension. On Linux distribution, the key is typically 
        /home/user/.ssh/id_rsa.pub

There is a field for the key’s name which will be used to identify the added key in the DigitalOcean control panel. Upon key creation it will be displayed on the “Security” page and it will be automatically used in the droplet created process, if the “SSH keys” option is selected for the “Authentication” method.Now, instead of using a set root password, a private key will be used to sign in to Droplet.

Creating a new Droplet using DigitalOcean web interface

Process of creating a DigitalOcean droplet is fairly simple using a control plane. Once logged in DigitalOcean, in the top right corner there is the “Create” drop-down which allows creation of different resources like droplets, databases, volumes, firewalls, load balancers and many more.

Once Droplet is selected, new page with all available options for droplet’s creation will be triggered containing following steps:

1. Select one of multiple Linux images like Ubuntu, CentOS, Fedora etc.:

2. Select cost plan according to your needs (for this article, we used simple machine for demo purposes):

3. Select datacenter region (the recommended is to use region closest to you) and VPC Network will be automatically created according to selected region:

4. Select the authentication method (we will use the SSH keys setup in previous step):

5. For final step chose a hostname for your machine along with some additional settings:

Upon droplet creation, connection using SSH, as root user can be established (droplet_ip_address will be displayed in control panel, after droplet is created):

1 | $ ssh root@droplet_ip_address

Create new user and grant administrative privileges

Using a system as root user is not a good practice, so creation of a regular user for daily use is recommended:

1 | $ adduser ag04

Command execution will trigger a set of questions, like account password, full name, work phone etc., most questions can be left at default values except account password.

Next step is to set up sudo privileges for newly created user, which will allow the user to execute administrative tasks as the root user. 

Add user ag04 to sudo group:

1 | $ usermod -aG sudo ag04

The above command will modify the default user settings, including the sudo group in the list of groups a user already belongs to. With append  (-a) argument, the user is added to the supplementary group. Append argument is to use only with groups (-G) option.

Setting up a basic firewall

UFW (Uncomplicated Firewall) is a firewall configuration tool that is specially designed to simplify the process of configuring firewall and it comes with Ubuntu servers. It is used to make sure that only connections to certain services are allowed on the server. It allows or blocks incoming and outgoing connections to and from the server.

OpenSSH, the service which allows the connection to the server, has a profile registered within UFW.

To get started, check all current available profiles:

1 | $ ufw app list

Output should be similar to the following:

1 | Available applications:
2 |   OpenSSH

Allow SSH connections:

1 | $ ufw allow openSSH

Enable the firewall:

1 | $ ufw enable

If all steps were successful, the output should look like:

1 | Firewall is active and enabled on system startup

Check if SSH connections are allowed:

1 | $ ufw status

Output will be similar to the following:

1 | Status: active
2 |
3 | To               Action     From
4 | --               ------     -----
5 | OpenSSH          ALLOW      Anywhere
6 | OpenSSH (v6)     ALLOW      Anywhere (v6)

Upon setting UFW, the firewall is blocking all connections except for SSH, if any additional services are installed, firewall settings should be adjusted to allow acceptable traffic in.

Enabling external access for regular user

After a regular user is created for daily use it must be configured so it can be used for SSH connection into the account directly.

Tip: Until login verification with a regular user is confirmed, it is recommended to
        stay logged in as root user.

To log in successfully, the local public key must be copied to the new user’s ~/.ssh/authorized_keys file, but since SSH keys were used to create the droplet, the public key is already added in the root account’s ~/.ssh/authorized_keys file. 

The next step is to copy the directory structure to the newly created user account in the existing session.

There are multiple commands that can be used to perform that action, but we will use rsync since that is the simplest way to copy all the files and the correct ownership and permissions.Copy the root user’s .ssh directory, preserve the permissions and modify the file owners:

1 | $ rsync --archive --chown=ag04:ag04 ~/.ssh /home/ag04

Check if everything is setup correctly run (preferably in other terminal session):

1 | $ ssh ag04@droplet_ip_address

Above command should establish a connection to a server without being prompted for the remote user’s SSH password for authentication.

Adding a GitHub self-hosted runner to an organization

Warning: In order to add a self-hosted runner on an organization level, owner
                 permission is required.

Self-hosted runners, when added at organization level, can be used to process jobs for multiple repositories. Self-hosted runner can be created using following steps:

1. Under organization, go to “Settings”:

2. In the left sidebar, under “Actions”, select “Runners”

3. Click “New Runner”:

Above selection will trigger a new window with detailed instructions on how to install GitHub Actions application for multiple platforms, in this article installation steps for Linux distribution is used.

Installation steps for GitHub Actions application

1. Create and move inside a new directory:

1 | $ mkdir actions-runner && cd actions-runner

2. Download the installation package:

1 | $ curl -o actions-runner-linux-x64-2.286.0.tar.gz -L
2 | https://github.com/actions/runner/releases/download/v2.286.0/
3 | actions-runner-linux-x64-2.286.0.tar.gz

3. Extract the downloaded package:

1 | $ tar xzf ./actions-runner-linux-x64-2.286.0.tar.gz

Extracted file contains:

1 | actions-runner-linux-x64-2.286.0.tar.gz bin config.sh env.sh
2 | externals run.sh

4. Create the runner using the configuration script:

1 | $ ./config.sh –url https://github.com/ag04 --token token_value

Running the configuration script will trigger “Self-hosted runner registration” section in which runner group, runner name, tags and working directory are configured, and it will look similar to the following (for testing purpose all default values for runner creation are selected):

Warning: If there is only a Default runner group, then all created runners must be
                 added to it, until a new group is created.

5. Start the runner using script:

1 | $ ./run.sh

If everything is setup correctly, runner will start listening for jobs:

1 | √ Connected to GitHub
2 |
3 | Current runner version: '2.286.0'
4 | 2022-01-17 13:41:05Z: Listening for Jobs

All above settings and runner’s status can be inspected in GitHub’s “Actions” section:

Configuring application to run as a service

In order to simplify application running there is an option to configure GitHub Actions application to run as a service so it’s automatically started when the machine starts.

In the compressed file that was extracted in the previous step, there is a script called svc.sh which can be used to install a managed application as a service.

Note: Before configuring a self-hosted runner application as a service, it’s 
          recommended to stop the application if it’s currently running.

Script has multiple options which can be check by starting script without any argument (must be started with sudo):

1 | $ sudo ./svc.sh

Generated output will display all available arguments:

1 | Usage:
2 | ./svc.sh [install, start, stop, status, uninstall]
3 | Commands:
4 |    install [user]: Install runner service as Root or specified user.
5 |    start: Manually start the runner service.
6 |    stop: Manually stop the runner service.
7 |    status: Display status of runner service.
8 |    uninstall: Uninstall runner service.

Install self-hosted application as a service using install argument:

1 | $ sudo ./svc.sh install

Upon installation service will be inactive and it can be started using start argument:

1 | $ sudo ./svc.sh start

Service should be up and running now, which can be checked using status argument:

1 | $ sudo ./svc.sh status

Command output will contain information about service’s state and logs:

1  | /etc/systemd/system/actions.runner.ag04.github-runner.service
2  | ● actions.runner.ag04.github-runner.service - GitHub Actions Runner 
3  | (ag04.github-runner)
4  |     Loaded: loaded (/etc/systemd/system/actions.runner.ag04.github-runner.service; 
5  |             enabled; vendor preset: enabled)
6  |     Active: active (running) since Mon 2022-01-17 13:49:57 UTC; 6s ago
7  |    Main PID: 2922 (runsvc.sh)
8  |       Tasks: 23 (limit: 1136)
9  |      Memory: 34.8M
10 |     CGroup: /system.slice/actions.runner.ag04.github-runner.service
11 |             ├─2922 /bin/bash /home/ag04/actions-runner/runsvc.sh
12 |             ├─2929 ./externals/node12/bin/node ./bin/RunnerService.js
13 |             └─2942 /home/ag04/actions-runner/bin/Runner.Listener run --startuptype    
14 |                    service
15 |
16 | Jan 17 13:49:57 github-runner systemd[1]: Started GitHub Actions Runner 
17 |                 (ag04.github-runner).
18 | Jan 17 13:49:57 github-runner runsvc.sh[2929]: Starting Runner listener with startup 19 |                 type: service
20 | Jan 17 13:49:57 github-runner runsvc.sh[2929]: Started running service
21 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: √ Connected to GitHub
22 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: Current runner version: '2.286.0'
23 | Jan 17 13:50:00 github-runner runsvc.sh[2929]: 2022-01-17 13:50:00Z: Listening for Job

To test if runner is connected to GitHub and that everything is working correctly, minor code changes are required and that is where labels come in handy.

Labels are used to organize self-hosted runners based on their characteristics. There are two types of labels; default and custom labels. Default labels are created upon Actions application installation and they are:

  • self-hosted: Applied to all self-hosted runners
  • linux, windows or macOS: Applied depending on operating system
  • x64, ARM or ARM64: Applied depending on hardware architecture

On a project example it would mean that GitHub Actions will trigger build, based on build.yml file in .github/workflows directory.

Label update is done in the YAML workflow file (e.g. build.yml). Default value for runs-on parameter is ubuntu-latest, which needs to be replaced with self-hosted, so that job section look like:

1 | jobs:
2 |   validation:
3 |     runs-on: self-hosted

The GitHub repository contains at least one workflow, defined as a separate YAML file in the .github/workflows directory. Each workflow is triggered by one or more events, which can be some internal GitHub event like a push request, scheduled event like a cron job, etc.

A workflow consists of one or more jobs, which are basically a set of commands that will be executed once the workflow is triggered. When workflow is triggered, all of its jobs run in parallel, by default, but they can also be configured to run sequentially if needed. Each job runs on a specific runner marked with a different label.

Using custom labels with self-hosted runners

Using a custom label to a runner allows us to configure a job to only execute on runners with that label. Unlike default labels which are fixed and cannot be removed or changed, custom labels can be added, assigned etc., but cannot be manually deleted (any unused labels will be automatically deleted within 24 hours).

Custom labels can be created with configuration script (part of GitHub Actions installation package) on self-hosted runner:

1 | $ sudo ./config.sh --labels test-runner

The label is created if it does not already exist.

Custom labels allow sending jobs to particular types of self-hosted runners, based on how they’re labeled. A self-hosted runner that matches all the assigned labels will then be eligible to run the job.

For example, multiple labels (both default and custom) can be combined for one job:

1 | jobs:
2 |   validation:
3 |     runs-on: [self-hosted, test-runner]

  • self-hosted: Run this job on a self-hosted runner.
  • test-runner: Custom label assigned to self-hosted runner with specific hardware configuration.

Install Docker Engine on runner machine

Docker is a platform that allows building, testing and deploying applications. Docker packages software into standardized units called containers. Containers allow applications to run in resource-isolated processes. They’re similar to virtual machines, but containers are more portable, more resource-friendly, and more dependent on the host operating system.

Example of GitHub Actions usage is to build and publish Docker images, so in order to successfully execute that task, the machine on which runner is installed must have Docker Engine installed.

The Docker installation package is available on the official Ubuntu repository, but in this demo we will install it from the official Docker repository.

A good practice when installing a new package on a system is to check for available updates and if necessary update the whole system.

Check if there are any new updates available:

1 | $ sudo apt update

Update all available packages:

1 | $ sudo apt upgrade

Shortcut to check and update all packages using combination of above mentioned commands:

1 | $ sudo apt update && sudo apt upgrade -y

Install a few prerequisites packages which lets apt use packages over HTTPS:

1 | $ sudo apt install apt-transport-https ca-certificates curl
2 |   software-properties-common

Add the GPG key for the official Docker repository to the system:

1 | $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | 
2 |   sudo apt-key add -

Add the Docker repository to APT sources:

1 | $ sudo add-apt-repository "deb [arch=amd64] 
2 |   https://download.docker.com/linux/ubuntu focal stable"

Install Docker Engine:

1 | $ sudo apt install docker-ce

Check if Docker is successfully installed:

1 | $ systemctl status docker

If installation process was successful, output will be similar to:

1  | ● docker.service - Docker Application Container Engine
2  |      Loaded: loaded (/lib/systemd/system/docker.service; enabled; 
3  |              vendor preset: enabled)
4  |      Active: active (running) since Thu 2022-01-27 10:51:55 UTC; 31s ago
5  | TriggeredBy: ● docker.socket
6  |        Docs: https://docs.docker.com
7  |    Main PID: 88078 (dockerd)
8  |       Tasks: 8
9  |      Memory: 29.9M
10 |      CGroup: /system.slice/docker.service
11 |              └─88078 /usr/bin/dockerd -H fd:// 
12 |                       --containerd=/run/containerd/containerd.sock
13 |
14 | Jan 27 10:51:54 github-runner dockerd[88078]: 
15 |                 time="2022-01-27T10:51:54.787623149Z" level=warning msg="Y>
16 | Jan 27 10:51:55 github-runner systemd[1]: Started Docker Application 
17 |                 Container Engine.
18 | Jan 27 10:51:55 github-runner dockerd[88078]: 19 |   
                     time="2022-01-27T10:51:55.654941680Z" level=info msg="API >

Above command output can be interpreted as that Docker service is successfully installed and enabled to start on boot (as machine starts).

Note: In order to start/stop/restart Docker daemon, sudo must be used. 
          e.g. to stop Docker daemon: $ sudo systemctl stop docker

Execute the Docker commands without sudo

By default, the docker command can only be run by the root user or by a user in the docker group, which is automatically created during Docker’s installation process. Any attempt to run the docker command without prefixing it with sudo or without being in the docker group, will result with following error:

1 | Got permission denied while trying to connect to the Docker 
2 | daemon socket at unix:///var/run/docker.sock: Get 
3 | "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": 
4 | dial unix /var/run/docker.sock: connect: permission denied

To avoid using sudo with every docker-related command, add username to the docker group (e.g. username set up at the beginning of the article):

1 | $ sudo usermod -aG docker ag04

To apply the new group membership, logout of the server and back in, or use the su command:

1 | $ su - ag04

Check if username is added to the docker group:

1 | $ groups

The output of above command should look like:

1 | ag04 sudo docker

Another thing that needs to be checked in order to successfully execute a job on a newly created self-hosted runner are docker.sock permissions.

By default docker.sock file has the following permissions:

1 | srw-r---- 1 root docker 0 Jan 27 10:51 docker.sock

Default permissions will not allow successful job execution, but result with similar error as for username:

1 | Got permission denied while trying to connect to the Docker  
2 | daemon socket at unix:///var/run/docker.sock: Post 
3 | "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=
4 | %7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota
5 | =0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&
6 | labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&
7 | shmsize=0&t=4746e9%3Ac00f274114c741e6ba192ef78629f7d1&target=
8 | &ulimits=null&version=1": dial unix /var/run/docker.sock: 
9 | connect: permission denied

Quick solution to this problem is to change permissions of docker.sock file (add read and write permission for other users):

1 | $ sudo chmod 666 /var/run/docker.sock

The permissions, after command is executed:

1 | srw-rw-rw- 1 root docker 0 Jan 27 10:51 docker.sock

Conclusion

Using self-hosted runners on organization level offers more control over hardware, operating system, software etc. but also bears some other responsibilities like updating software and OS along with security issues. Self-hosted runners are flexible in terms of supported architectures (x64, ARM64, ARM32)  and operating systems, multiple Linux (RHEL, CentOS, Ubuntu, Fedora, Mint, etc.) and Windows distributions (Windows 7, 8, Windows Server, etc.) along with macOS distributions.

With self-hosted runners there is no need to have a clean instance for every job execution.

When configuring self-hosted runners it’s important to keep in mind what type of project will be running and based on that select type of machine for GitHub Actions application; if it’s simple Java project basic machine (e.g. 2 CPU/2GB RAM) will perform just fine, but if we are talking about complex NodeJS-based or Angular/React project than machine with better configuration will be required in order for job to perform smoothly.

Another important thing to keep in mind is that one self-hosted runner can only run one job at a time, when no available runners are idle, the subsequent jobs will be in queue until available runners are idle.

In this article the basic installation and usage scenario is demonstrated and by that it is safe to say that there are many fields that can be updated in terms of security, optimization, auto scaling, etc.

References

Next

Blog

Going native with JHipster and Spring Boot

Going native with jHipster - Agency04

Company

Our people really love it here

Evolution of expertise

The agency was founded in 2014 by seasoned industry veterans with experience from large enterprises. A group of Java engineers then evolved into a full-service company that builds complex web and mobile applications. 

Flexibility is our strong suit — both large enterprises and startups use our services. We are now widely recognized as the leading regional experts in the Java platform and Agile approach.

We make big things happen and we are proud of that.

Personal development

If you join our ranks you’ll be able hone your coding skills on relevant projects for international clients.

Our diverse client portfolio enables our engineers to work on complex long term projects like core technologies for Delivery Hero, Blockchain and NFTs for Fantasy Football or cockpit interface for giants like Strabag. 

Constant education and knowledge sharing are part of our company culture. Highly experienced mentors will help you out and provide real-time feedback and support.

Contact

We’d love to hear from you