Touch Keyboard, kid.
...returning to my roots post-layoff.
Kids today! The Gen Zers or whatever they would be called, have a terrific phrase: Touch Grass. Which is basically a way of saying “Get off your computer and go outside. Touch Grass, dude.” It’s kind of an insult? I dunno, there’s a lot of stuff said on discord during Fortnite that sounds insulting, but I always kinda loved this one.
The big fat Google layoff 1 hit in early 2023, which was a banner year for snowfall in the Wasatch mountain range. I chose to ski out my feelings that season and I booked about 28 days on various mountains.
It was a deeply cleansing experience. I indeed touched ‘grass’ (tundra?) and recentered myself, dealt with some feelings2, etc. But even during the layoff-prompted ski vacation, I started thinking of not just “Touching Grass” but “Touching Keyboard.”
The reality of being in management at a large tech company is you actually get little in the way of just fun programming done. I was a good manager, to be sure, and stayed in touch with the technology of the firm so as to best do the work, but programming time was rare. With the layoff coming at around the same time as some really kickin’ ai coding partners hitting maturity, I decided to start having fun again.
In 2015, a dear friend and I started a side project game company and we were approaching alpha launch by early 2024. In early 2023, the back end systems were simply not up to the task. This was all happening as Intel was releasing some deeply flawed processors.
Using fire-breathing desktop cpus for serving games is not new. Single core performance is a huge thing for game engine capacity planning. Yes, it uses other cores, even efficiency cores, for the helper threads and the rest, but your main game event loop should sit on the fastest core you can keep boosting.
In 2022/2023, that meant an i9 whose base clock was 3.3 and could boost to 5.8ghz. If you can manage your thermals, you can keep in boost the whole time. Epyc server chips would often top out at 2.4ghz and a less maintainable 3.7ghz (or 3.2 in the case of Xeon) boost. And, before you can ask, cloud server providers offerings of high speed CPUs were and are wildly expensive for a game company that isn’t Ubisoft or Blizzard3. We’re talking potentially twice as much as the parts cost for a i9 system every single month.
Looking at the power logs for these machines, they average out somewhere around 135 watts of power consumed, which at 14 cents/kwh is about 13$/month in power for twice the clocks. And no hidden surprise billing and egress switch traversal monitoring or whatever upsell the cloud providers decide is somehow desired or okay.
There are alternative providers, which we used for user locality reasons, which were cheaper, but boy, renting other people’s computers is pretty spendy :-)
Sadly, these 13th (and 14th) generation i9s would hit a state where they would start requesting more power for a core from the motherboard than the core could handle. The result would be a chip that gets too hot and outstrips the coolers ability to remove heat as well as very real problem with the lines and connections on the die and processor.
Basically, they were cooking themselves.
A lot of people using i9s also overclock (read, overvoltage) their processors and in the process of doing so, made it even harder to dump all that heat. We didn’t overclock as a policy, but lots of folks buy these i9s specifically so they can.
The result under Linux is a really weird crash where the computer would halt but would show on the console whatever was sitting there when it crashed. We were not running motherboards with IPMI, as it would add sometimes 500$ to the cost of the motherboard and instead we had PI-KVMs in front of a KVM switch. When Linux would crash like this, no keyboard commands, no wake-on-lan commands would suffice, and we’d have to actually press the button on the server to restart it.
I ended up putting a bunch of $20ish super hackable shelly plugs in front of these machines to allow me to reboot them from anywhere in the world, essentially replacing $500/machine with $20 and some fraction of the $450 I spent on the switch and kvm which could handle 16 machines.
This economy of scale, though, prompted me to write a monitoring system that would keep an eye on these machines and prompt me when they entered into this state so I could reboot them4.
I wrote this system with my pal copilot to get back into what was a ‘modern’ pythonic programming much as I had done to write the ansible playbooks and such to manage the machines in concert with k8s.
Rapidly I got to the point where the system would run as a service on a few machines (redundancy FTW!) and would alert me via discord and email when a machine hit this vexing state. Then I would run the reboot command. I didn’t feel comfortable having the program do it, but I’ve recently started removing the restraints and letting it sort out the machines that hit this bug.
The thing about programming with copilot/codespaces and chatgpt-4o windows alongside is that it is *so* much fun to program this way.
I’ve been programming since I was 12 years old, and I absolutely love it. A good part of why I fell in love with free software and open source was because no matter what my problem was I could just *look* at how it was done. If I have a bug I can drill all the way down to the bare metal practically and see what I’m doing wrong and can fix it. I could take that code and by simply acknowledging the wishes of the person as expressed in the oss license, I could even re-use that code in my programs.
In using the LLMs to co-program with me, I have that same rush. I can ask what's the right way to do something in a language that’s new to me, like gdscript5, I can ask the smartest way to implement a json gateway for my systems monitoring tool, how to read the power consumption from a shelly plug for logging and turn it off and on again, etc…
And what does it produce? Well, it produces …90%… the right code, and I spend some time getting the llm to get the rest of the way or failing that, doing it myself. I enjoy the process so much and I have to encourage anyone reading this to cozy up to your favorite LLM and start Touching Keyboard again6.
it was done so oddly so I think of it as “babies first layoff”
Speaking of layoffs and feelings, please read Quintin’s post, it’s terrific
Small players rarely enjoy the discounts of a Ubisoft or Spotify.
And before you say it, yes, there’s a million services and programs that do this, I know, but I wanted something fast, no cost, and suited to my use case.
Touch grass too, it’s really good for you to get outside.