Fazal Majid's low-intensity blog

For a long time I avoided e-ink devices. E Ink Corporation effectively holds a monopoly on eletrophoretic e-ink technology, and that has kept panel prices high. When combined with very slow refresh rates, e-ink devices are unitaskers like eBook readers, yet priced the same as versatile devices like tablets with much better display quality. Unlike some, I also find I prefer reading on LCD or OLED screens than e-ink. The sole saving grace of the technology is the display does not need to be powered on, which can lead to devices with outstanding battery life measured in weeks.

I have recently softened my stance and now own three e-ink devices:

• PocketBook Verse e-reader. Unlike, say, a Kindle, it is not locked in to a content provide, and has a microSD card reader for expansion (I have 180GB of eBooks, more than most e-readers can handle), and it is fairly inexpensive (I paid £105 for mine).

• A M5Stack Paper S3. There is currently a fad for the Xteink X4, but this device is superior: it’s open to custom firmware without having to pay a premium for a Developer Edition (I installed a community build of CrossPoint Reader, but TRMNL is available) and has a touchscreen.
• A Seeed Studio reTerminal E1002. This has a 7.3" Spectra 6 display that can display in six colors (black, white, red, yellow, blue and green), has a batter life measured in weeks because it only wakes up every half hour or so to fetch updates and sleeps in a very low-power state in-between

The e-readers are fairly nondescript. I wanted to see why people rave about the benefits of e-ink for reading, and am not convinced, I still find reading on a phone or iPad or Google Pixel Tab (with GrapheneOS, of course) a much more pleasant reading experience.

The E1002 is a more interesting device. I’ve wanted for a while to have a dashboard at the entrance with weather and transit data so I can avoid specific Tube lines if they are out of service. There isn’t any conveniently located plug nearby so using an old tablet is not an option. I learned a few months ago about TRMNL, who make devices that would fit the bill, but the E1002 has the advantage of a color, not monochrome display, so I sprung the £100 or so for it on AliExpress.

Seeed Studio offer a low-code environment called SenseCraft HMI, but it is too simplistic for my needs, and more importantly, it is cloud-dependent, which is simply not acceptable. The E1002 ships with reTerminal firmware tied to SenseCraft HMI, but they also offer a TRMNL-compatible firmware. TRMNL is also cloud-based, and they want you to pay a one-time $50 fee for the privilege of using a non-TRMNL device, but they do support running a custom server instead, so I vibe-coded one gTRMNL, that basically fetches a web page, renders it using an embedded Chromium browser and dithers it onto the 6 colors of the display using the Floyd-Steinberg algorithm. I also vibe-coded a simple dashboard in PHP to display the basic data I am interested in:

and after dithering it would look like this:

Unfortunately, when I reflashed with the Seeed TRMNL firmware 1.6.7, I was not getting all the colors. Green was turned to blue, blue was turned to black, yellow was turned to red. I tried installing TRMNL upstream firmware 1.7.7, and go similar results, except this time blue was turned into white, and green into yellow. I found a GitHub issue describing the problem, but no solution.

Long story short, I spent the afternoon identifying the root cause of the problem, which was that Seeed Studio had started implementing support for the E1002 in the TRMNL firmware, but hadn’t gotten around to implementing 6 colors support and were using only 4 colors as TRMNL does have a device with a 4-color screen. With assistance from Claude, I spent the afternoon implementing a fix and now it works perfectly:

The dithering makes for fuzzy text, and at some point in my copious spare time I will write a dedicated dashboard that generates the PNG directly without using a browser so I can control the colors and dithering, but it’s perfectly usable for now.

Until the 1970s, there were only four TV networks in the US. Then cable led to an explosion, from 28 in 1980, 79 in 1990 to well over a thousand today. Part of this was cheaper distribution via cable, but also because technology like computer video editing reduced the cost of production.

The flip side of this explosion was a fragmentation of audiences. The big four networks went from having 20-30% of the population and 80-90% of prime-time TV viewership to 20-30% prime time audience and low single digits of the population. This had all sorts of consequences, including politically as there is no longer a widely shared frame of reference, or someone like Walter Cronkite to tell Nixon the Vietnam War was over and lost.

The same fragmentation is coming for software via LLMs, for the same structural reasons. The SaaSpocalypse is overblown, because writing code is only a small part of the cost of producing software, developing requirements, integration, testing and maintenance are far larger components. Thus the dream of DIY enterprise software will remain that, a dream. Software will remain the preserve of companies that can manage the development process, but the dropping cost of coding will increase the number of software houses, and their addressable market will shrink, as it did for the big four TV networks. This will have an impact on the economics, as the cost of development and marketing, even if lowered by LLMs, will also be spread over a smaller customer base.

One of the downsides of self-hosting critical applications like email on your homelab is that if you lose connectivity, especially when you are travelling, you are out of luck. It’s happened to me twice. The first time I had to ask a colleague (Hi Jason!) to go get my spare keys from the building super and reboot my home server. In the other instance, I walked my wife over the phone through the steps of rebooting our OpenBSD home router that runs on a somewhat dubious computer sourced from AliExpress with an Intel N100. I actually ordered an industrial-grade Asus NUC 13 Rugged N50 to replace it, but in a variant of the Heisenberg effect, the original machine started working flawlessly, go figure.

On some of my HP machines (Z workstations and EliteDesk 8xx Mini), the firmware includes Intel AMT/IME spyware management firmware. You can install the MeshCommander software to get a poor man’s version of the IPMI remote management facility included in most servers. IPMI usually includes remote KVM, i.e. being able to control the computer over the network as if you were directly in front of its keyboard, mouse and monitor. KVM stands for Keyboard, Video and Mouse, although most KVM systems also give you the ability to insert a virtual USB drive to boot into diagnostics or a rescue drive. This allows access to the BIOS and other things you can’t do from the OS itself, or recover if the OS itself has crashed.

To resolve this vulnerability, I have been equipping the majority of my key machines that don’t have Intel AMT with physical IP KVM devices. These used to be very expensive and required having some cursed version of the Java plugin installed in your browser, but recently the Pi-KVM project has opened up the market and there are now a host of relatively inexpensive (in the $100 range) devices available like the JetKVM and GL.iNet’s Comet line of IP KVMs.

GL.iNet is known for its well-regarded line of travel routers like the tiny but mighty Mango, but has been expanding into IoT and now IP KVMs. That line is now quite extensive, with:

• The basic Comet GL-RM1.
• A variant with PoE, the GL-RM1PE which also supports USB-C PD for power.
• A pro variant, the GL-RM10 (I haven’t tried it).
• More interestingly, a 5G cellular equipped model, the GL-RM10C.

Like their travel routers, the KVMs have an open operating system based on Linux with SSH and root access, and excellent support for VPN protocols actually invented in this millennium, i.e. WireGuard rather than hoary L2TP, PPTP, IKE/IPsec or OpenVPN.

I have a basic Comet, two PoE powered ones and the 5G cellular one attached to the router.

Due to the exhaustion of available IPv4 address pools, almost all cellular carriers today use some form of Carrier-grade NAT (CGNAT), which means you do not have a permanent IP address for your mobile device. Some cellular carriers will offer plans with static IP addresses, but they are extremely expensive including the per-kilobyte charges because this is a niche market, primarily enterprises wanting remote monitoring and access to satellite offices.

GL.iNet offers a cloud service for remote access and also supports Tailscale and ZeroTier. Either of these would obviate the need for an exotic data plan SIM. I don’t trust the cloud, however, and find Tailscale too invasive, so I opted instead to set up WireGuard between the GL-RM10C and a cloud server, with routes forcing it to use the 5G wwan0 interface instead of Ethernet for the tunnel.

The /etc/wireguard/wg0.conf config on the server is:

[Interface]
Address = 192.168.2.1/24, fddd::ffff/64
ListenPort = 45340
PrivateKey = <redacted>
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;iptables -A FORWARD -o %i -j ACCEPT; ip6tables -A FORWARD -i %i -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;ip6tables -A FORWARD -o %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE;iptables -D FORWARD -o %i -j ACCEPT; ip6tables -D FORWARD -i %i -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE;ip6tables -D FORWARD -o %i -j ACCEPT
 
[Peer]
PublicKey = <redacted>
AllowedIPs = 192.168.2.2/32, fddd::1/128

on the GL-KVM, it is:

[Interface]
Address = 192.168.2.2/24, fddd::1/64
#ListenPort = 45340
PrivateKey = <redacted>
 
[Peer]
PublicKey = <redacted>
AllowedIPs = 192.168.2.0/24
Endpoint = <redacted>:45340
PersistentKeepalive = 30

Add opening UDP port 45340 on the firewall, and on the KVM a /etc/init.d startup script to call route add <ip of server> wwan0 and wg-quick up wg0 at boot time, that establishes the tunnel. Since the IPs on either end are not routable, I also have HAProxy running in TCP mode on the server to allow access from the Internet:

global
    log /dev/log local0
    log /dev/log local1 notice
    daemon
    user nobody
    group nobody

defaults
    mode tcp
    log global
    option tcplog
    timeout connect 5s
    timeout client  1m
    timeout server  1m

frontend https_in
    bind <redacted>:443
    default_backend wg_https_out

backend wg_https_out
    mode tcp
    server wg0_peer 192.168.2.2:443 check

I have a £5/month SIM card and plan installed, with a 5GB quota. I only start HAProxy when I actually need it so I don’t waste any of it on script kiddies trying to break in.

The user interface is largely the same across the entire GL.iNet KVM product line, is excellent, uses native Web technologies and WebRTC to provide the remote video, so no janky VNC plugins or Java required. The video is crisp, as can be expected from a purely digital signal path, and I haven’t noticed compression artifacts, even when running over cellular.

It does have the same problem as almost all IoT devices with a Web UI, of not being able to self-provision a TLS certificate. I modified my monthly Let’s Encrypt certificate rotation script to copy the keys and certificates to /etc/kvmd/user/ssl/server.{key,crt} where the firmware expects them to be. There is an IETF effort to fix this once and for all, but it is still very much work-in-progress and probably still too complex for the average consumer to deal with.

It’s also worth noting the 5G in the Comet 5G is RedCap (reduced capability), which is a cheaper and more power-efficient version of 5G that is capped at around 100Mbps instead of the gigabit speeds full-fat 5G offers. This is unlikely to be an issue for this class of devices, as people are not buying them to play Doom remotely.

What you don’t get with the Comets is the ability to remotely power-cycle the machine you get with IPMI or AMT. They have an accessory for computers with ATX motherboards, but I haven’t had one in ages, and a Rube Goldberg-like contraption poetically named Fingerbot that physically pushes the power button. JetKVM does have an accessory that interposes between computers with a barrel DC connector and their power brick, to allow turning them on and off. Let’s hope GL.iNet is inspired to make their own, and also a USB-C one while they are at it. In the meantime, I plan on using a smart-home type Wi-Fi-controlled power switch running Tasmota to do forced power cycles.

I also have a JetKVM. It’s a cute little device, very compact (but surprisingly heavy), and I am planning to add it to my portable computer-maintenance toolkit rather than keeping it stationary like my Comets.

I have been on a bender this weekend experimenting with various LLM-capable machines in my homelab, specially the very capable yet fast Qwen3.6-35B-A3B. I haven’t found good benchmarks, though, so I ran the small Gemma4 E4B Q4_K model (4.62 GiB, 7.52B params) using llm-bench. This has two measures: prompt processing 512 (pp512) is how quickly in tokens/second the LLM can read a 512-token prompt, i.e. how good the LLM is at “reading”, and token generation 128 (tg128) is how quickly it can write 128 tokens’ worth of text, i.e. how fast it is at answering the question.

(click on the hostnames to get the raw report)

The Mac Studio performs very well at token generation, despite being a 4 year old machine, but perhaps that reflects how Llama.cpp is particularly optimized for Apple Silicon. I was also surprised at how strong the performance of the A2000 is, despite it being a fairly weak low-power graphics card meant for 2D CAD in small form-factor workstations like my HP Z2 Mini G9 where it lives. Conversely, the Strix Point performance is underwhelming, even if subjectively it performs reasonably well with Qwen 3.

It’s been a while since I last did a chocolate review. There are not many bars that can claim to be made using a patented process invented by a Nobel laureate. The Dardenne Lait Praliné (dark milk chocolate with ground hazelnuts) is one.

The special cooking process was invented in 1910 by Paul Sabatier (Chemistry Nobel, 1912) to make chocolate easier to digest for people who don’t tolerate it well. In any case, this is an organic and Fairtrade bar with a moderately high 43% cacao content (for milk chocolate).

The bar is not too sweet, and as expected has a strong chocolate taste. The texture is mostly smooth like a milk gianduia, but you can feel an ever so slightly grainy feel, as if the hazelnuts were not ground finely enough or the chocolate not sufficiently conched. Despite this minor flaw, this is an excellent bar that is relatively inexpensive for organic chocolate. The only drawback is, it may be difficult to source outside France.