deadlime

If you can read this post, the experiment was successful. I managed to migrate my development tools from a cloud virtual machine to an actual physical machine humming behind my back. But let's not rush forward that much and go back to the beginning.

I have a virtual machine running a couple of development-related tools. It has a Gogs Git server, a Jenkins instance, a Docker Registry, and a Composer repository (a static HTML site generated by Satis). It doesn't get a lot of traffic, and the applications besides Jenkins are pretty lightweight. But the Jenkins is a bit problematic. For example, sometimes, it cannot build Docker images because it runs out of memory.

That's one of the reasons why I started to build a small home server (4 CPU cores, 16 GB of memory in a 20x20x6 centimeters little box) to migrate all the applications. It's running a Docker Swarm as a single manager node with Traefik for the routing and Portainer for managing the Docker stacks. However, initial tests showed that moving the Git server won't be a smooth sail, so I didn't start the migration up until now.

Without the resource constraints, I decided to go with a GitLab installation. It can replace all four applications mentioned before: it operates as a Docker Registry, as multiple Package Registries (and Composer is among them), and Jenkins can be replaced with GitLab CI. Maybe my life would be easier this way (nope).

The problem

SSH is an old piece of furniture. It doesn't have such fancy accessories as the SNI support for TLS. There's no easy way to have a proxy or load balancer-like application that can route incoming connections based on specific criteria (like the target host of the connection) to different backend applications. It's a huge problem for us because the server already has an SSH server on port 22, and the Git server running in Docker would also want to start an SSH server on port 22. If that's not available, it could bind to port 2022, for example, but that would transform this:

$ git clone git@git.example.com:group-name/repository-name.git

Into this:

$ git clone ssh://git@git.example.com:2022/group-name/repository-name.git

Alternatively, we could create a .ssh/config similar to this on every machine we want to clone repositories:

.ssh/config

Host git.example.com
    Port 2022

Not a big issue, but I didn't want to make this compromise. Certainly, there are other solutions, like the article the Gogs Docker image documentation mentions, but that also feels too hacky to me.

The solution?

Then I got an idea. I don't know why exactly now. This migration project was on pause for quite a while. We could assign multiple IP addresses to the machine, the host SSH could listen on one of the IP addresses, and the Git SSH could listen on the other IP address, both on port 22.

$ host example.com
example.com has address 192.168.0.23
$ host git.example.com
git.example.com has address 192.168.0.24

Such elegance, such beauty. I immediately started working on the implementation. First, I needed a second IP address. On a local network, it's not a big issue. Even if the machine doesn't have an extra network adapter, it could be solved. My host machine runs Debian, so I needed to do the following things:

/etc/network/interfaces

auto enp3s0:0
iface enp3s0:0 inet dhcp

auto enp3s0:1
iface enp3s0:1 inet static
  address 192.168.0.24
  netmask 255.255.255.0

As you can see, a network adapter can have multiple IP addresses. It gets the first one from the good old DHCP server from the router (based on the MAC address, it's also a fixed IP address), and the second one is a static one the machine sets to itself. One little change in the host SSH config and we are good to go:

/etc/ssh/sshd_config

ListenAddress 192.168.0.23

Now we can switch to the Docker side of things. Although it wouldn't be necessary, and it doesn't make much sense, I passed the connection through Traefik. This way, it is the one and only entry point in Docker.

traefik.yaml

version: "3.2"
services:
  traefik:
    image: traefik:v2.7.0
    command:
      - "--entrypoints.gitssh.address=:22"
    ports:
      - target: 22
        host_ip: 192.168.0.24
        published: 22
        protocol: tcp
        mode: host
    deploy:
      mode: global

These are just the relevant parts. The complete config looks a lot like what we created in a previous post. Unfortunately, this first try ended up with an error that it could not recognize the host_ip key, so I changed it to the - 192.168.0.24:22:22 short format, and that was good enough for it.

Everything works. I started a Gogs server to test it out. I could clone a repository and push back some data. Everything was awesome. Until I closed my host SSH connection to the server. I simply couldn't reconnect. It looked like the Git SSH was running on the other IP address as well. But that's impossible, right? We just told them to listen on a single IP address.

I needed to take a technical break until I found a VGA cable and an unused keyboard to restore my proper SSH connection with the server. I also managed to find out with a manual run of docker stack deploy that binding to an IP address was ignored in my config, but Portainer didn't share this irrelevant piece of information with me.

After some research, I even found a Github issue that says that Swarm services can only bind to 0.0.0.0.

It looked like I had to throw my beautiful solution into the trash. Of course, I could run the Git server outside of Swarm, but that's like running it on port 2022, and I couldn't allow that.

The solution!

After a couple days, I got another idea. We will listen on port 2022. I know it sucks, but let me finish. If we get a connection on address 192.168.0.24 on port 22, we could redirect it to port 2022. We only need the following two iptables rules for that:

# iptables -t nat -I PREROUTING 1 -d 192.168.0.24/32 -p tcp -m tcp --dport 22 -j REDIRECT --to-port 2022
# iptables -A INPUT -i enp3s0 -p tcp -m tcp --dport 2022 -j ACCEPT

A slight drawback is that the Git SSH server is accessible on port 2022 too, but we may allow this bit of compromise here. There is nothing left to do but install and set up the GitLab server and migrate all the old data and processes into it. That wasn't a walk in the park either, but I may tell that story another time.