deadlime

Now with extra fingers for the more efficient hacking experience!

Have you ever wondered what happens behind the scenes when you send an HTTP request? How do the bits know where to go when you ping someone on the local network? Today, we will try to find answers to such questions.

At first, I didn't know how to approach the problem. Do I take an HTTP request and dig down to the bottom or start from the bottom until I'm able to construct an HTTP request? In the end, I decided on the latter.

It may be a bit more alien at the beginning, but I hope it'll make sense in the end. We'll go along the layers defined in the OSI model, so let's start with the hardware.

Physical layer

Chances are that everyone has an Ethernet network at home, something like this:

The modem, the router, and the switch may happen to live in the same box.

It's called the physical layer: a bunch of network devices, tied together with copper strings (or wireless devices tied together with magic). Each network device has a unique MAC address, assigned in the factory. There's a lot more interesting stuff to look into here (like how devices communicate with each other about which speeds they support), but personally, there's not much more I can tell you about it, so let's move on to the next layer, the data link layer.

Data link layer

At this layer, the zeros and ones we know so well finally appear. We have what's called the Ethernet frame, which is a unit of data that we can send at a time. It looks something like this:

Interesting that the sender needs to tell who the sender is. What happens if someone else's MAC address is entered there? Is it possible to do something naughty with this?

Two interesting things to note here. One is MAC spoofing, where we are not happy with the MAC address assigned by the hardware vendor and we want to change it. Say because our ISP's device only works with a fixed MAC address. Or there are Android phones, which by default connect to a Wi-Fi network with a random MAC address so that the phone cannot be tracked between networks.

The other interesting thing is MAC flooding, where the attacker sets up a bunch of random MAC addresses as senders. This fills up the entire MAC address table of the switch, leaving no room for the real MAC addresses. This usually has the consequence that the recipient MAC address of an incoming real packet will not be found in the MAC address table, causing it to be sent out to everyone. This allows attackers to peek into the contents of packets that are not intended for them.

The observant reader may also notice that there is no mention of the so-called IP addresses that we normally use. For that, we need to move on to the next level, which is the network layer.

Network layer

There are several interesting protocols to be mentioned here, all of which will be placed in the data part of the Ethernet frame. First, we need to know what MAC address is associated with a given IP address.

Address Resolution Protocol

ARP helps us with this, we can send a request to which the owner of the IP address can respond. The message structure:

Let's add IP addresses to the above diagram and see a concrete example of a request and response.

Say Alice wants to ping Bob, so she asks which MAC address the IP address 192.168.1.103 belongs to. The request (the green part belongs to the Ethernet frame, the blue part to the ARP request):

The IP addresses are in hexadecimal format, CyberChef is a nice tool for converting. In a request packet, the MAC address of the recipient can be anything, its value will be ignored.

Bob receives this request and since his IP address is 192.168.1.103, he sends a reply:

Here again, the question arises, what happens if Bob is not the only one who replies to the request, but the evil Mallory as well? Could Alice be sending her messages for Bob to the wrong place?

The devices have something called an ARP cache, which holds the IP address - MAC address mappings so that you don't have to ask every time. For Alice, it might look something like this:

$ ip -br neigh
192.168.1.103                           eth0             0a:00:00:00:00:03
192.168.1.104                           eth0             0a:00:00:00:00:04
192.168.1.1                             eth0             0a:00:00:00:00:01

This cache is also updated when an ARP response is received without being requested, which means that if Mallory starts flooding the network with fake ARP responses (telling Alice that he is the router, telling the router that he is Alice), and forwards packets passing through it to the original recipients, he can intercept (or even alter) Alice's Internet traffic without Alice noticing anything.

Internet Protocol

The following protocol is IP, which finally gives us IP addresses and the ability to exchange data between two IP addresses. The IP packet structure:

Internet Control Message Protocol

Since we mentioned ping earlier, we should mention ICMP. It's a strange beast, it belongs to the network layer, but it feels like it should be in the transport layer. It's wrapped into an IP packet in the same way as UDP or TCP, only it's not for data transport. In the case of ping, the message is structured somewhat like this:

Alice already knows Bob's MAC address, so she can finally send the ping she originally wanted, which Bob can then reply to. The request looks something like this (the green part is for the Ethernet frame, the blue part is the IP packet, the red part is ICMP):

To which Bob sends the following reply:

It's getting complicated, and we are far from the end. Did you notice that there was no mention of ports? It wasn't by mistake, at this point the concept of a port does not exist, it is time for us to move up yet another level.

Transport layer

If you've ever opened sockets in any programming language, you'll be familiar with the protocols found here. Let's start with the easy one.

User Datagram Protocol

As mentioned above, UDP is the easier one. There's no guarantee that the packet will arrive, no retransmission for lost packets, you just yell into one end of the pipe and hope that the other end will hear it. A packet looks like this:

The sender port is also optional, if it is not set to zero, then the response packets are expected on that port.

Transmission Control Protocol

And this brings us to the famous and popular TCP. The cornerstone of the pack-everything-in-your-browser-and-serve-it-over-HTTP-based Internet. Until the wide adoption of HTTP/3, which uses UDP. A packet looks like this:

It's not just the structure of the packet that is important, but the little dance that the client and server do to exchange data. The connection must be established and closed, and both parties have to acknowledge that they have received the data sent by the other.

Establishing a connection

1. the client sends a SYN packet
   (the sequence number is 0, as this is the first packet from the client)
2. the server replies with a SYN, ACK packet
   (the sequence number is 0, as this is the first packet to the server, the acknowledgment number is 1, as the sequence number received before was 0 and no data was included, so the next packet will be expected to have the sequence number of 1)
3. the client responds with an ACK packet
   (the sequence number is 1, the acknowledgment number is 1)

Data exchange

1. the client sends 10 bytes of data
   (the serial number is 1)
2. the server replies with an ACK packet
   (the sequence number is 1, and the acknowledgment number is 11 since the previous sequence number was 1 and 10 bytes of data were received)
3. the server sends 100 bytes of data
   (the sequence number is 1)
4. the client replies with an ACK packet
   (the sequence number is 11, the acknowledgment number is 101)

Closing the connection

1. the client sends a FIN packet
   (the sequence number 11)
2. the server replies with a FIN, ACK packet
   (sequence number 101, acknowledgment number is 12)
3. the client replies with an ACK packet
   (the sequence number is 12, and the acknowledgment number is 102)

Application layer

We elegantly skip two layers, the session layer and the presentation layer. The session layer contains the SOCKS protocol for example, and the scope of the presentation layer often merges with the application layer.

The application layer can be, for example, HTTP. Using the knowledge gained above, let's see what happens when Alice runs a simple curl www.example.org command.

To be able to tell this, we need to know Alice's network settings. Suppose something like this is configured:

auto eth0
iface eth0 inet static
      address 192.168.1.102
      netmask 255.255.255.0
      gateway 192.168.1.1
      dns-nameservers 1.1.1.1

• we can do nothing with the www.example.org domain, we need an IP address
• the configured DNS IP address is not on the local network, so the request has to be sent to the router (gateway)
• send an ARP request to find out the MAC address of the router
• send a UDP packet to the MAC address of the router with the DNS IP address in the IP packet
• the router sees that it is not the recipient, so it forwards the packet to the Internet (NAT and connection tracking are also involved here so that the router's public IP address is visible in the packet on the way out and it knows who to forward the reply to)
• a reply UDP packet arrives from the Internet, the router sees that it is not the recipient
• because of connection tracking, the router knows where to forward the packet
• forward UDP packet to Alice
• IP address of www.example.org is X.X.X.X
• establish a TCP connection, make an HTTP request
• the address X.X.X.X is not on the local network, so TCP packets are sent to the MAC address of the router with the corresponding destination IP address
• reply packets are forwarded by the router to Alice
• closing the TCP connection
• ARP requests are probably not needed at this point because they are cached

Another question that may arise is how to know that an IP address is not on the local network. From the address/netmask/gateway settings above, a routing table is generated that looks something like this:

$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

I don't say I know exactly how this works, but I would guess that it will pick the narrowest match or start from the end and work backward and pick the first match, but the point is, if something is in the range 192.168.1.0-192.168.1.255 it will just simply send it out to the appropriate MAC address, for all other IP addresses it will send the packet to the MAC address associated with the IP address 192.168.1.1. The router has a similar routing table to decide what to do with the incoming packets.

Summary

I hope I have managed to get a glimpse of what happens "underneath" us when we use the Internet. There are a lot of things going on through many layers and we have only managed to scratch the surface a little bit.

We didn't even get very far, just ventured as far as the router. What's beyond that... is a world of its own, with things like DSL, SDH, PPP, MPLS, BGP, OSPF, and a bunch of other acronyms I don't even know about. And yet, in most cases, our packets get to the right recipients. What is this if not magic?