When I thought about what I wanted to learn about during my batch at the Recurse Center, I reflected on the parts of my previous job I never understood very well, and networking was at the top of the list. I had previously watched MIT OpenCourseWare’s lectures on Computer Systems Engineering, but didn’t feel that I had a good grasp of the subject, and read the Wikipedia page on the seven-layer model of networking many, many times without actually absorbing the information. I searched for beginner-friendly networking tutorials without success. When I asked more experienced engineers at my job how I could get better at solving networking problems, the answers basically boiled down to “be wrong a lot and eventually you’ll be right”. (This seems to be true for pretty much everything in engineering.) Now that I had time to write code solely for the purpose of personal improvement, I decided to start the process of being wrong a lot by making networking a focus of my time at RC.
My original goal was to implement a very bare-bones version of TCP in Rust.
I used the libpnet library to abstract away networking layers below the transport layer
(where TCP lives) without abstracting away the TCP protocol itself, as Rust’s TcpListener does.
TCP is a complex protocol, and implementing every detail of the TCP spec would not be a good use of my time
(I knew I would lose interest long before then).
So instead, I aimed for the minimum possible code that would do something, anything that I could demo or that would produce some visual confirmation of working.
What’s the smallest piece of functionality I can start with? Let’s send a SYN packet to localhost, the very first step in opening a new TCP connection!
How would I know if my code worked? tcpdump to the rescue!
You can use tcpdump to print out all the packets sent to a given network interface,
or dump them into a file and open it with Wireshark to inspect more easily.
My code sent a lot of badly formed packets, but luckily, tcpdump will provide helpful feedback on why your packet is no good.
Here, I’m printing out all the packets on the lo interface (a.k.a. localhost),
and tcpdump warns me that the data offset in the header is set incorrectly:
lee@lee-pc:~$ sudo tcpdump -i lo -X -v
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
16:30:56.898509 IP (tox 0x0, ttl 64, id 7212, offset 0, flags [DF], proto TCP (6), length 84)
localhost.8989 > localhost.8000: tcp 60 [bad hdr length 4 - too short, < 20]
0x0000: 4500 0054 1c2c 4000 4006 2076 7f00 0001 E..T.,@.@..v....
0x0010: 7f00 0001 231d 1f40 0000 0000 0000 0000 ....#..@........
0x0020: 1002 0000 af57 0000 0000 0000 0000 0000 .....W..........
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0000 ....
Here, the packet’s checksum (used to detect data corruption) is wrong:
lee@lee-pc:~$ sudo tcpdump -i lo -X -v
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
15:33:48.540537 IP localhost.8989 > localhost.8000: Flags [S],
cksum 0x000f (incorrect -> 0x6f57), seq 0:44, win 0, length 44
0x0000: 4500 0054 efc3 4000 4006 4cde 7f00 0001 E..T..@.@.L.....
0x0010: 7f00 0001 231d 1f40 0000 0000 0000 0000 ....#..@........
0x0020: 5002 0000 000f 0000 0000 0000 0000 0000 P...............
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0000 ....
Finally, I managed to produce a packet that tcpdump was happy with!
lee@lee-pc:~$ sudo tcpdump -i lo -X
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
16:38:13.702638 IP localhost.8989 > localhost.8000: Flags [S],seq 0:44, win 0, length 44
0x0000: 4500 0054 269b 4000 4006 1607 7f00 0001 E..T&.@.@.......
0x0010: 7f00 0001 231d 1f40 0000 0000 0000 0000 ....#..@........
0x0020: 5002 0000 6f57 0000 0000 0000 0000 0000 P...oW..........
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0000 ....
16:38:13.702650 IP localhost.8000 > localhost.8989: Flags [R.], seq 0, ack 45, win 0, length 0
0x0000: 4500 0028 0000 4000 4006 3cce 7f00 0001 E..(..@.@.<.....
0x0010: 7f00 0001 1f40 231d 0000 0000 0000 002d .....@#........-
0x0020: 5014 0000 6f44 0000 P...oD..
… but wait, what’s up with that second packet?
It’s coming from localhost:8000 (my destination), directed to localhost:8989 (my source). Do I have anything running on localhost:8000?
lee@lee-pc:~$ sudo lsof -i :8000
lee@lee-pc:~$
Nope. Just to be sure, I sent my packet to localhost:8002 instead, and again, the response appeared.
After some Googling, I realized this was just my kernel doing what it was built to do!
It doesn’t recognize the incoming packet, so it tells the other party to terminate the connection by sending an RST (reset) packet.
This realization was awesome because of how clearly it exposed a gap in my understanding of networking.
It’s hard to describe my mental model of TCP before starting this project, but it probably went something like this:
HTTP is built on top of TCP, and when I navigate to https://google.com, my browser is issuing an HTTP request.
It does DNS resolution of Google’s hostname, somehow (for me, dig google.com resolves to the IP address 172.217.12.174).
Therefore, my browser constructs a TCP packet, and sends it off towards this IP address.
How does it get there? Magic, probably.
In short, because I had spent most of my time as a developer in the application layer,
my networking knowledge existed almost entirely in that context.
I hadn’t thought about why I could use TCP to communicate with a Postgres Docker container at localhost:5432,
or how my internet browser could talk to my router.
My mental model wasn’t completely wrong, but it didn’t explain why this RST packet appeared in my terminal;
after all, I had no applications listening on localhost:8000.
In hindsight, it feels painfully obvious that my kernel knows how to respond to TCP packets.
This experiment also generated some interesting follow-up questions, such as: if my kernel doesn’t expect to receive a random SYN packet, what kind of conversations does it expect to happen on localhost? There are lots of applications using my localhost interface right now (e.g. Spotify, Chromecast); what are they doing? And will I still be able to send packets to localhost (and will my kernel respond) if I put the code into a Docker container? (Spoiler alert: sort of. I tried this because I was looking for a way around running my code with elevated privileges.)
I’ve definitely accomplished my loosely-defined goal of “learning more about TCP”, but there’s still a lot to explore. Maybe designing the TCP connection interface will give me a chance to learn more about asynchronous programming in Rust, or I could dive deeper into container networking. It’s clear that reading textbooks and watching lecture videos won’t substitute for writing networking-related code and fixing bugs, but I’d like to have a more clearly defined goal for my next project. I hope this post will encourage readers to experiment with technology they’d like to understand better, even if it seems daunting. Happy debugging!