deadlime

I feel like I painted myself into a corner here. It's getting harder and harder to find time-related titles for these posts. Yet here we are again with a little bit of progress on the project.

In the last episodes, we couldn't convince CLion to do as we wished and PyCharm wasn't that cooperative either (The time has (not yet) come), but at least we printed out a nice little case around the clock (Time in a Box), that we don't use.

Today we'll continue the journey by taking a step back and trying to tame CLion once again, then we'll have a chance to experience the deep dark horrors known as time zones.

Development environment on a Pi

The purpose of hobby projects (at least for me) is to have as much fun as possible (depending on your definition of 'fun', I guess, but we'll see that later). I didn't want to continue the project in MicroPython since I do enough Python in my day-to-day life, I needed some change. And I'm stubborn.

The original idea was to take a regular Raspberry Pi, connect the Pico to the Raspberry Pi's GPIO instead of the Picoprobe (Debugprobe), plug in a monitor/keyboard/mouse, and start developing on that. The idea slightly changed when I couldn't find a suitable HDMI converter for the Rasberry Pi. The new plan is to try to convince CLion running on a Windows machine to use the Raspberry Pi as a build environment via SSH.

Theoretically, the Pi is the most suitable environment for Pico development. After a few simple commands, everything should just work:

$ wget https://raw.githubusercontent.com/raspberrypi/pico-setup/master/pico_setup.sh
$ chmod +x pico_setup.sh
$ ./pico_setup.sh

Just a bit of waiting and we're done... unless you happen to be the (increasingly less) happy owner of a Raspberry Pi 5 because at the time of writing this script wasn't able to handle the new model. Luckily I found this pull request, which got me halfway there, but I also needed a new OpenOCD interface config, which I can't remember where I got it from:

/usr/local/share/openocd/scripts/interface/raspberrypi5.cfg

adapter driver linuxgpiod

adapter gpio swclk 24 -chip 4
adapter gpio swdio 23 -chip 4

With that, I managed to run one of the examples from the Pimoroni repository:

$ openocd -f interface/raspberrypi5.cfg -f target/rp2040.cfg -c "program gfx_pack_demo.elf verify reset exit"

Despite all this, I went with the Picoprobe at the end. I read somewhere that this driver is slower than the Picoprobe, so I wired it up again and connected it to the Pi with a USB cable. This way I could use this good old OpenOCD command:

$ openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "program gfx_pack_demo.elf verify reset exit"

The lion's share of the work

Now we just need to get CLion to do its magic. In theory, it has the support, let's see what we can do with it. First, it needs an SSH config to connect to the Pi:

Then a remote toolchain that uses this SSH connection:

It's probably also worth setting up a deployment to make file synchronization work (although it won't, but we'll come back to that in a moment):

And finally, a debug configuration to run our code.

Here we have a few interesting bits worth mentioning. For GDB, I chose to run on the remote host, so the 'target remote' args can connect to localhost (gdb and openocd both run on the Pi). In the 'GDB Server args' area, the first two parameters (interface and target setting) used to be enough, the adapter speed is probably optional. And the program... well, that's an interesting story.

In an ideal world, the 'Upload Executable' part would be, say, 'If updated' instead of 'Never', but if that's turned on, it would try to use the file from the Windows machine, but that isn't there, because ELF files generated on the Pi aren't synced by CLion.

If I turn off the setting mentioned in the ticket and hit Tools > Resync with Remote Host, it does download the ELF file, but somehow it still doesn't end up on the Pico. Based on the output, it looks like it's starting to do it, but ultimately nothing happens. I wonder why it even needs to download it (and then upload it back to the Pi) if everything is already there? A mystery.

So I'd rather call the program manually, but that way the reset breaks the connection to the debugger, which is not so nice if you want to... you know... debug. I ended up making two configurations, one that can upload (program part included) and the other that can only debug (no program part). It's not an ideal solution, but it works.

The console output of our Pico can be accessed by running minicom -b 115200 -o -D /dev/ttyACM0 on the Pi.

The right time

In the Python version, we connected to Wi-Fi and obtained the time from an NTP server. Now we need to do the same with the C SDK. In reality, this was done before the Python version, but let's ignore that.

First of all, the SNTP module needs some configuration in the CMakeLists.txt file:

add_definitions(
  -DSNTP_SERVER_DNS=1
  -DSNTP_SERVER_ADDRESS="hu.pool.ntp.org"
  -DSNTP_SET_SYSTEM_TIME=sntp_set_system_time
  -DSNTP_STARTUP_DELAY=0
)

The sntp_set_system_time part is a C function that we have to define. We'll also need some libraries from the SDK:

target_link_libraries(
  pico-clock
  hardware_rtc
  pico_cyw43_arch_lwip_threadsafe_background
  pico_lwip_sntp
  pico_stdlib
  pico_time
)

The function mentioned before to handle the time we get from the NTP server:

#include "hardware/rtc.h"
#include "lwip/apps/sntp.h"
#include "pico/cyw43_arch.h"
#include "pico/stdlib.h"
#include "pico/time.h"
#include "pico/util/datetime.h"

bool sntp_finished = false;

void sntp_set_system_time(uint32_t sec) {
  datetime_t datetime;
  time_to_datetime(sec, &datetime);

  rtc_set_datetime(&datetime);

  sntp_stop();
  cyw43_arch_disable_sta_mode();

  sntp_finished = true;
}

We turn off the SNTP module and disconnect from the Wi-Fi after the successful time synchronization. Now we just need to start the whole thing:

int main() {
  stdio_init_all();
  cyw43_arch_init_with_country(CYW43_COUNTRY_HUNGARY);
  rtc_init();

  cyw43_arch_enable_sta_mode();
  while (cyw43_arch_wifi_connect_timeout_ms("SSID", "secret", CYW43_AUTH_WPA2_AES_PSK, 30000)) {
    printf("failed to connect.\n");
  }

  sntp_setoperatingmode(SNTP_OPMODE_POLL);
  sntp_init();

  while (!sntp_finished) {
    printf("waiting for time.\n");
    sleep_ms(100);
  }

  datetime_t now_utc;
  char datetime_buf[256];

  while (true) {
    rtc_get_datetime(&now_utc);
    datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_utc);
    printf("%s\n", datetime_buf);
    sleep_ms(1000);
  }
}

If all's well, the Pico connects to the Internet, talks to the NTP server and we start to see the time on the console every second:

Tuesday 27 August 20:47:13 2024
Tuesday 27 August 20:47:14 2024
Tuesday 27 August 20:47:15 2024
...

The only problem is that it's in UTC, so we need a solution to convert it to our local time.

My time with the time zones

A desktop clock wouldn't be worth much if it could only display the time in UTC. We receive a Unix timestamp from the NTP server, but the RTC (Real-time clock) module already expects a datetime_t struct and it returns one as well, so we need to use that when displaying the time.

In theory, mktime can make a Unix timestamp from a date and time struct, and localtime can make a date and time struct in the appropriate time zone from a Unix timestamp. The pico/util/datetime.h part of the SDK also provides some helper functions we will use, but they are just wrappers around mktime and localtime. datetime_to_time turns a datetime_t into a Unix timestamp, and time_to_datetime turns a Unix timestamp into local time (in a datetime_t).

But there's a little twist. Both directions work with local time and we need to be able to tell one to work from UTC, and the other to use the time zone we are currently in. But how could we achieve this?

The only starting point I had was the tzdata package, which contains time zone data files. Maybe I can find something there, like uploading the contents of a time zone file to the Pico somehow. This did not lead to the solution, but I found out that the time zone names we use (like Europe/Budapest) are actual files in the /usr/share/zoneinfo directory.

I also ran into the environment variable called TZ, whose value could be, say, Europe/Budapest, which we now know is actually the file /usr/share/zoneinfo/Europe/Budapest, so it's a dead end. Or is it?

After a bit more searching I found a manual with the information I needed: the TZ variable isn't for just file paths, it can contain a time zone definition as well. Based on the manual and the information on the related Wikipedia page the definition of Europe/Budapest would look something like this:

TZ=CET-1CEST,M3.5.0/2,M10.5.0/3

The base time zone is CET which we need to subtract one hour from to get the UTC time. The period of summer daylight saving is called CEST. Summer daylight saving starts on the last Sunday of March (M3.5.0) at 1 AM UTC which is 2 AM in CET (because this format requires the local time). The end of summer daylight saving is the last Sunday of October (M10.5.0), at 1 AM UTC as well, which is 3 AM in CEST.

If I understand this correctly then the time zone file would give you more rules if they changed in the past or will change in the future. For example, if you run a zdump -v Europe/Budapest | less command, then you can see that in 1916 the start of summer daylight saving was the last Sunday of April at 11 PM.

With this power we can convert the time from the RTC module to the correct local time, we just need to adjust the value of the TZ variable before the datetime_to_time and time_to_datetime calls.

datetime_t now_utc;
datetime_t now_local_time;
char datetime_buf[256];

while (true) {
  rtc_get_datetime(&now_utc);

  setenv("TZ", "UTC", 1);
  time_t timestamp;
  datetime_to_time(&now_utc, &timestamp);

  setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  time_to_datetime(timestamp, &now_local_time);

  datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_utc);
  printf("utc time:   %s\n", datetime_buf);
  datetime_to_str(&datetime_buf[0], sizeof(datetime_buf) / sizeof(char), &now_local_time);
  printf("local time: %s\n", datetime_buf);

  sleep_ms(1000);
}

Everything looks good... on the first iteration.

utc time:   Tuesday 27 August 21:02:03 2024
local time: Tuesday 27 August 23:02:03 2024
utc time:   Tuesday 27 August 21:02:04 2024
local time: Tuesday 27 August 21:02:04 2024
utc time:   Tuesday 27 August 21:02:05 2024
local time: Tuesday 27 August 21:02:05 2024

But only on the first iteration, then it went downhill. It looks like the two functions are somehow stepping on each other's toes. I checked out multiple implementations of the two functions but I couldn't find out the problem. In the end, I went with a 'hotfix' instead of a proper solution: I found an mktime implementation in the Linux kernel source that doesn't care about time zones and things started to work.

The real enlightenment came when I was writing this post and re-read the TZ documentation. Maybe it just needs to know how much is the difference compared to UTC even if we want UTC? This, fortunately, solved the problem, I had to replace the setenv("TZ", "UTC", 1) part with setenv("TZ", "UTC+0", 1). The plain UTC would probably refer to the file /usr/share/zoneinfo/UTC, which we don't have here.

utc time:   Tuesday 27 August 21:08:56 2024
local time: Tuesday 27 August 23:08:56 2024
utc time:   Tuesday 27 August 21:08:57 2024
local time: Tuesday 27 August 23:08:57 2024
utc time:   Tuesday 27 August 21:08:58 2024
local time: Tuesday 27 August 23:08:58 2024
...

Everything was nice but after like 10 minutes the clock started to display UTC time again. You can question my definition of a fun hobby project at this point. I know I did.

I fired up the debugger, but at one point I only saw assembly code in CLion, so I had to get the source. After a bit of research, I found out that the Newlib C standard library provides the mktime and localtime functions (among many others).

I ran a sudo apt install newlib-source command on the Pi and got the source in the /usr/src/newlib/newlib-3.3.0.tar.xz file so I could point the CLion in the right direction. In the deepest parts of this library I found out that when the localtime tries to load the value of the TZ variable, there is nothing there. This made the setenv function to the new suspect. We call it, obviously but it didn't do its thing after the 6000th run or so.

After another long search, I arrived at this part of the code:

  if (!((*p_environ)[offset] =  /* name + `=' + value */
  _malloc_r (reent_ptr, (size_t) ((int) (C - name) + l_value + 2))))
    {
      ENV_UNLOCK;
      return -1;
    }

I verified that the setenv really returns -1 at the point when everything goes wrong. I'm not a C expert, but based on that malloc I jumped to the conclusion that we ran out of memory. I made a little program to try this out:

int counter = 0;
int res;
while (true) {
  res = setenv("TZ", "UTC+0", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }

  res = setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }
  ++counter;
}

As expected, I got the message:

oops: 5217

I was wasting my time with the mktime and localtime source code, the problem was the setenv function all along. If we take a look at the code we can see if the new value of the environment variable is longer than the old value we allocate a new chunk of memory that will not be freed. And our code does almost nothing else than setting a longer value for an environment variable. What if we keep the two values on the same length? Would it work? I filled the shorter one with extra spaces and tried it out.

int counter = 0;
int res;
while (true) {
  res = setenv("TZ", "UTC+0                       ", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }

  res = setenv("TZ", "CET-1CEST,M3.5.0/2,M10.5.0/3", 1);
  if (res == -1) {
    printf("oops: %d\n", counter);
    break;
  }
  ++counter;
}

No more oops from this one, the problem is solved, so I can put the time conversion part back and check out if everything still works.

utc time:   Tuesday 27 August 21:38:21 2024
local time: Tuesday 27 August 22:38:21 2024
utc time:   Tuesday 27 August 21:38:22 2024
local time: Tuesday 27 August 22:38:22 2024
utc time:   Tuesday 27 August 21:38:23 2024
local time: Tuesday 27 August 22:38:23 2024
...

This is something new. We lost an hour somewhere. At this point, I had already acquired enough routine to suspect that mktime could not interpret the TZ value correctly because of all the extra spaces. I put an extra comma after UTC+0, to see if that would soften the parser's heart of stone...

setenv("TZ", "UTC+0,                      ", 1);

... and to my surprise, it worked!

utc time:   Tuesday 27 August 21:39:22 2024
local time: Tuesday 27 August 23:39:22 2024
utc time:   Tuesday 27 August 21:39:23 2024
local time: Tuesday 27 August 23:39:23 2024
utc time:   Tuesday 27 August 21:39:24 2024
local time: Tuesday 27 August 23:39:24 2024
...

Huge success. After several days of suffering fun and laughter, we finally got the Pico clock to show the correct local time. Hopefully, it will handle the end of the summer daylight saving properly. We'll see on the last Sunday in October. Maybe we will have some nice output on the LCD display by then.