Deploying Gitlab with Podman

For my private projects I run a self-hosted GitLab instance deployed with the official Community Edition Docker image. In addition to Git repository management GitLab comes packed with a lot of features such as Continuous Integration/Deployment, Wikis, Kubernetes cluster integration and much more. Those looking for a minimal Git solution should probably look elsewhere. I have heard a lot of positive about Gitea.

Feature heavy as it is, and realistically speaking, GitLab is a bit too excessive for my own use cases. I could easily replace it with managed solutions such as GitLab.com, GitHub or Azure DevOps. But where’s the fun in that?

Table of contents

• GitLab CE with Podman
  • Update: Bind to privileged port
• Prerequisites
• Prepare the host
• GitLab Configuration
• podman run
• Manage container lifecyle with systemd

GitLab CE with Podman

When it comes to deploying and running the containerized version of GitLab I use Podman as the container engine. Podman is really evolving into a great alternative to Docker. One of the top benefits is that it runs daemon-less which allows for true rootless mode out of the box.

Please note that in the following example I do in fact run the container as root. The only reason for this is that I have not (yet) setup a proxy in front of GitLab that would eliminate the need to expose privileged ports from the container, which requires it to run as root.

Update: Bind to privileged port

More recent Linux kernel versions does in fact come with the capability to lower the unpriviliged port range. For example to bind a rootless container to port 80 you can either add net.ipv4.ip_unprivileged_port_start=80 to /etc/sysctl.conf or run command:

sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80

Prerequisites

• VM or bare-metal running Linux/amd64 (there’s also unofficial images for arm64, which I have not tried, but they should in theory work fine even on Raspberry Pi’s. Especially Pi4 4GB or 8GB versions.).

• 2GB — 4GB available RAM. The official recommendation is 4GB but I have managed to run it with as low as 2GB, although on some occasions Out-Of-Memory related restarts would be triggered. The sweet-spot that has been working well for me is 3GB total RAM to the host VM with 2.5GB dedicated to the container.

• Podman (latest version 2.2.1 at the time of writing).

Prepare the host

GitLab requires a location on the host system to store persistent data in-between container restarts and version upgrades. In this example I use /opt/gitlab as the base directory.

1. First export the base directory path to environment variable as it will be used in further configuration.

export GITLAB_HOME=/opt/gitlab

2. Create the base directory

mkdir -p $GITLAB_HOME

3. Now create sub-directories for storing application data, configuration and log files.

mkdir $GITLAB_HOME/data $GITLAB_HOME/config $GITLAB_HOME/logs

The base directory and sub-directories should now have the following structure:

gitlab
├── config
└── data
└── log

GitLab Configuration

There are a lot of parameters available to configure in GitLab. For a minimal working configuration only external URL and SSH port (if a custom port other than 22 is used) needs to be defined in order to have GitLab generate correct Git URLs. External URL is the IP or FQDN that GitLab should be reachable with. For example if GitLab should only be reachable on the internal network the host IP can be used.

Configuration can be read from environment variables or file $GITLAB_HOME/config/gitlab.rb. I use the latter approach and template the configuration in gitlab.rb file and keep it version controlled.

1. Define external URL and custom SSH port and create $GITLAB_HOME/config/gitlab.rb.

export GITLAB_EXTERNAL_URL=https://192.168.1.10 GITLAB_CUSTOM_SSH=2222

cat << EOF > $GITLAB_HOME/config/gitlab.rb
   external_url '$GITLAB_EXTERNAL_URL'
   gitlab_rails['gitlab_shell_ssh_port'] = $GITLAB_CUSTOM_SSH
   ## Uncomment to disable Let's encrypt
   # letsencrypt['enable'] = false
EOF

See below for an example of a more extensive configuration where outbound SMTP, custom SSL certificates, disabling not used features and some performance tweaks are included:

external_url '$GITLAB_EXTERNAL_URL'
gitlab_rails['gitlab_shell_ssh_port'] = $GITLAB_CUSTOM_SSH
gitlab_rails['time_zone'] = 'Europe/Stockholm'
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['initial_root_password'] = '$GITLAB_ROOT_PASSWORD'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = '$EXTERNAL_SMTP_SERVER'
gitlab_rails['smtp_port'] = $EXTERNAL_SMTP_PORT
gitlab_rails['smtp_user_name'] = '$EXTERNAL_SMTP_USERNAME'
gitlab_rails['smtp_password'] = '$EXTERNAL_SMTP_PASSWORD'
gitlab_rails['smtp_domain'] = '$EXTERNAL_SMTP_SERVER'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_tls'] = true
gitlab_rails['gitlab_email_from'] = '$EXTERNAL_SMTP_ADDRESS'
registry['enable'] = false
letsencrypt['enable'] = false
nginx['ssl_certificate'] = '/etc/gitlab/ssl/server.crt'
nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/server.key'

## Optimizations
puma['worker_processes'] = 2
puma['per_worker_max_memory_mb'] = 250
sidekiq['concurrency'] = 9
postgresql['shared_buffers'] = "256MB"

## Disable Grafana & Prometheus for now
prometheus_monitoring['enable'] = false
grafana['enable'] = false

Note: By default GitLab will try to issue a certificate via Let’s Encrypt Staging CA. GitLab will throw an error during startup if an IP in the private RFC1918-block is used as external URL:

Error executing action `create` on resource 'acme_certificate[staging]'

To resolve either use a FQDN in external_url or add letsencrypt['enable'] = false in gitlab.rb to disable Let’s Encrypt

podman run

1. Initial run of GitLab container:

sudo podman run -d --name gitlab \
--publish 443:443 --publish 80:80 --publish $GITLAB_CUSTOM_SSH:22 \
--memory=2560m \
--hostname $GITLAB_EXTERNAL_URL \
--volume $GITLAB_HOME/config:/etc/gitlab \
--volume $GITLAB_HOME/logs:/var/log/gitlab \
--volume $GITLAB_HOME/data:/var/opt/gitlab \
gitlab/gitlab-ce:13.7.4-ce.0

At the time of writing the latest version of GitLab CE was 13.7.4-ce.0 — feel free to change this to a different version

2. Tail the log with:

podman logs -f gitlab

First time starting up will take around 5-10 minutes. Consecutive container runs will start up much more quickly.

When a similar message is displayed startup is completed:

Running handlers:
Running handlers complete
Chef Infra Client finished, 345/1429 resources updated in 02 minutes 14 seconds
gitlab Reconfigured!

If container startup fails with:

cat: /var/opt/gitlab/gitlab-rails/VERSION: Permission denied
Installing gitlab.rb config...
cp: failed to access '/etc/gitlab/gitlab.rb': Permission denied

SELinux might be enabled on the host OS. To grant the container permissions to mounted volumes append :Z to the container directory in the run command:

sudo podman run -d --name gitlab \
--publish 443:443 --publish 80:80 \
--publish $GITLAB_CUSTOM_SSH:22 \
--memory=2560m \
--hostname $GITLAB_EXTERNAL_URL \
--volume $GITLAB_HOME/config:/etc/gitlab:Z \
--volume $GITLAB_HOME/logs:/var/log/gitlab:Z \
--volume $GITLAB_HOME/data:/var/opt/gitlab:Z \
gitlab/gitlab-ce:13.7.4-ce.0

3. Access https://192.168.1.10 (IP/FQDN that was used with $GITLAB_EXTERNAL_URL). Ignore the TLS Insecure error and verify that the login page is visible.

While there take the opportunity to set a default root password for the instance.

Manage container lifecyle with systemd

So far the GitLab instance is running within the container but it will not start up automatically if the host is rebooted or perform clean-up of previous container references if stopped manually.

That responsibility can be delegated to systemd by defining a service for the GitLab container.

Podman comes with a neat feature that generates the required systemd unit file.

1. Generate the unit file:

podman generate systemd gitlab \
   --files --restart-policy=always \
   --new --name --time 60

For definitions of the arguments reference Podman offical documentation

2. Copy the generated unit file to /etc/systemd/system:

sudo cp container-gitlab.service /etc/systemd/system

3. By default KillMode=none is specified in the unit file. This is being deprecated by systemd. Let’s fix that by replacing it with TimeoutStopSec instead:

sudo vim /etc/systemd/system/container-gitlab.service

Set the value to 70 seconds to ensure it does not override ExecStop timeout value that is set to 60 seconds.

   # container-gitlab.service
   # autogenerated by Podman 2.2.1
   # Tue Jan 19 21:21:47 UTC 2021

   [Unit]
   Description=Podman container-gitlab.service
   Documentation=man:podman-generate-systemd(1)
   Wants=network.target
   After=network-online.target

   [Service]
   Environment=PODMAN_SYSTEMD_UNIT=%n
   Restart=always
   ExecStartPre=/bin/rm -f %t/container-gitlab.pid %t/container-gitlab.ctr-id
   ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-gitlab.pid --cidfile %t/container-gitlab.ctr-id --cgroups=no-conmon --replace -d --name gitlab --publish 443:443 --publish 80:80 --publish 2222:22 --memory=2560m --hostname https://192.168.1.10 --volume /opt/gitlab/config:/etc/gitlab:Z --volume /opt/gitlab/logs:/var/log/gitlab:Z --volume /opt/gitlab/data:/var/opt/gitlab:Z gitlab/gitlab-ce:13.7.4-ce.0
   ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-gitlab.ctr-id -t 60
   ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-gitlab.ctr-id
   PIDFile=%t/container-gitlab.pid
- KillMode=none
+ TimeoutStopSec=70
   Type=forking

   [Install]
   WantedBy=multi-user.target default.target

4. Now enable and start the service:

systemctl enable --now container-gitlab.service

5. Verify that the service started:

systemctl status container-gitlab.service

If everything went well it should be active (running):

● container-gitlab.service - Podman container-gitlab.service
   Loaded: loaded (/etc/systemd/system/container-gitlab.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-01-19 22:04:29 UTC; 5min ago

GitLab has now successfully been deployed with Podman!

Questions? Errors spotted? Feel free to comment below contact me!