Fazal Majid's low-intensity blog · Fazal Majid's low-intensity blog

The fetish for uptime

2022-07-28 IT

At one of my previous jobs, the engineers on my team had an informal competition as to who could rack up the longest uptime on their workstation (they all had Sun Solaris or Linux, of course). When the company moved to a new office, one crafty engineer managed to beat all the others by putting his Sun into the seldom-used hibernation mode to preserve his uptime when everyone else was forced to reboot.

I posit that uptime is actually a bad thing. All software has bugs, and a regular maintenance schedule to apply patches, at the very least once a month, should be part of the plan and designed into the architecture. By that token, an uptime greater than 31 days is a “code smell” for infrastructure.

PSA: iCloud Private Relay can make Safari on your iPad unusable

2022-05-24 Network IT Mac Web

After upgrading my iPad to iPadOS 15.5, Safari became unusable. It would take forever to load the Reddit login page, and many others like Dilbert.com. Opening the same in Firefox Focus had no issues.

Going into Settings / Safari / Privacy & Security / Hide IP Address and disabling it fixed this for me. Alternatively you can disable it only for specific networks (Settings / Wi-Fi / ⓘ / Limit IP Address Tracing / Off).

It seems Apple turned on iCloud Private Relay on by default for Safari in iPadOS 15.5 and presumably iOS 15.5 as well. Macs are probably next.

I can only speculate why turning it off fixes the breakage, but:

The feature routes your calls through Akamai then CloudFlare, and for whatever reason CloudFlare doesn’t seem to like my ISP, I often encounter their “prove you are human” challenges.
It may also be because Apple overrides your DNS settings for this feature to work, and if your network is locked down with something like Pi-Hole to prevent trackers, those DNS requests may not be getting through. I don’t want IoT devices or the like to bypass my DNS server, which uses Wireguard to my Cloud VPN server to ensure my ISP cannot snoop on my DNS requests (a setup I believe more secure and private than Apple’s), nor CloudFlare, nor the UK Police State. I haven’t blocked DNS-over-HTTPS servers yet as this guy does but it’s on my list. This might be interfering with iCloud Private Relay.
It may also be sabotage, as Rui Carmo points out, or as John Oliver memorably calls it, “Cable Company F∗∗∗ery”.

Batch-converting HEIC images to JPEGs on the Mac

2021-10-27 Mac Photo

TL:DR working around Apple proprietary brain damage

I use Lightroom 6 to manage my photo collection, although it is falling victim to bit rot (e.g. the face recognition module no longer works, apparently due to a licensing logic time bomb in the code). Exploitative pay-forever software subscriptions are simply unacceptable so I will not yield to Adobe’s Creative Clout bondage, and since Lightroom will not work in newer versions of MacOS, that means I am working on migrating to Darktable, albeit very slowly.

My wife does all her photography on her iPhone, and while the image quality is poor, she does take a great many photos and videos of our daughter. I decided to integrate them in my workflow.

To do so, I installed the free and excellent Photobackup app on her iPhone. It allows backing up her photos and videos using rsync to my ZFS backup server, from which I rsync them to my Mac, and then use my linkonce tool to create a parallel file hierarchy that mirrors it, but so that when I delete a photo in Lightroom, it stays deleted. That way I can remove duds without having them pop back up in Lightroom every time I do a sync.

I just realized I was missing a large number of images because they are in Apple’s obnoxious HEIF format, that they switched to around the time they introduced the mostly useless Live Photos misfeature. Lightroom 6 does not recognize the format. While you can batch convert and export HEIC files to JPEG in Preview.app, it is still a manual process.

I investigated what command-line tools are available that could be run from a cron job and there are surprisingly few. GraphicsMagick sensibly refuses to support the format because of patent concerns. Most of the others require compiling an intimidating stack of dependencies first, and because HEIF is based on the H.265 HEVC video codec, an ostensibly open (in name only) ISO format that is heavily encumbered with patents, so is HEIF and it is probably illegal to use those tools like heic2jpeg.

I opted instead to write my own heic2jpeg (no relation to the previous tool). It is a very basic conversion utility using Apple’s CoreImage framework, to piggyback on Apple’s patent licenses, and as a side benefit, it will preserve the image metadata including geoloc. The flip side is that means the tool can only run on a Mac and not on Linux or Illumos, but I can live with that.

It is also my first ever Swift project. A nice expressive language in the vein of Python or Go (except with Apple’s grotesquely long API names), but I do not expect to use it much, as I have grown disillusioned with Apple’s policies and software quality, and have no intention of indenturing myself as a sharecropper in Tim Cook’s plantation any more than to Adobe’s.

The code is in heic2jpeg.swift

To build it, assuming Swift or Xcode is installed on your Mac, just run:

swiftc -O -o heic2jpeg heic2jpeg.swift

My sync script (part of my backup script) then runs something like:

find $HOME/Pictures -name \*.HEIC -print0 | xargs -0 -P 12 -t -n 10 heic2jpeg --delete

This will run 12 processes in parallel, consuming 10 files each until all HEIC are converted (or if already converted, left alone). I find the optimal setting to be 150% to 200% of the actual cores on your system (not including Intel’s fake hyperthreading cores, which do not count).

import Foundation
import CoreImage

var jpegQuality = 0.90
let context = CIContext(options: nil)
let options = NSDictionary(
    dictionary: [kCGImageDestinationLossyCompressionQuality:jpegQuality]
)
var delete:Bool
var filename:String?
delete = false
for i in 1..<Int(CommandLine.argc) {
    filename = CommandLine.arguments[i]
    if filename == "-delete" || filename == "--delete" {
        delete = true
        continue
    }
    let srcURL = URL(fileURLWithPath:filename!)
    let destURL = srcURL.deletingPathExtension().appendingPathExtension("jpg")
    var exists:Bool
    do {
        exists = try destURL.checkResourceIsReachable()
    } catch {
        exists = false
    }
    if exists {
        print("skipping \(filename ?? "???")")
    } else {
        print("converting \(filename ?? "???")")
        let image = CIImage(contentsOf: srcURL)
        try! context.writeJPEGRepresentation(
            of:image!,
            to:destURL,
            colorSpace: image!.colorSpace!,
            options:options as! [CIImageRepresentationOption : Any]
        )
        if delete {
            print("deleting \(filename ?? "???")")
            try! FileManager.default.removeItem(at:srcURL)
        }
    }
}

How to ensure a cron job runs exclusively

2021-09-01 IT

TL:DR a simple but effective mutex for cron jobs

Often you need to run a job periodically, e.g. backing up files, but the job could take more time than the interval allotted between runs, and you do not want multiple instances of the process to be running at the same time. For instance, bad things happen when multiple rsync processes are trying to synchronize the same folders to the same destination. Thus you want a mutex, something that ensures only one copy of the process can run at any given time.

There are approaches using lock files, but if the computer reboots or the job crashes, the lockfile will not be deleted and all subsequent runs of the job will fail. Some advocate using flock() or fcntl(), but those calls are finicky with strange semantics, e.g. fcntl will release a lock if any related process closes the file.

My solution to deal with this is to bind an IPv6 localhost ::1 socket to a given port. Only one process can do this, and thus it’s a very effective mutex. No lock files to cause havoc, no dealing with the dark and buggy corners of advisory file locking.

For shell scripts, simply replace the #!/bin/sh with #!/somewhere/bin/lock 2048 where 2048 is the port number you will use to enforce the lock (greater than 1024 if you do not want to deal with the hassles of privileged ports). If you want the jobs to wait and not exit immediately if they fail to acquire the lock, just change the line to #!/somewhere/bin/lock w2048

The code is in lock.c. Just compile using:

gcc -O2 -o lock lock.c

clang -O2 -o lock lock.c.

#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <netinet/in.h>
#include <inttypes.h>
#include <sys/time.h>
#include <string.h>

extern char **environ;

int main(int argc, char **argv) {
  int sock, port, status, exit_on_fail;
  char *port_start, *port_end = NULL;
  struct sockaddr_in6 sin6;
  struct timeval timeout;

  if (argc < 3) {
    fprintf(
      stderr,
      "Usage:\n"
      "\t#!%s [w]<port:1-65535> (first line of script instead of #!/bin/sh)\n"
      "\t\tor\n"
      "\t%s [w]<port:1-65535> -c \"cmd [args...]\"\n\n"
      "\tw: wait if we could not get the port\n",
      argv[0], argv[0]);
    return -1;
  }
  
  exit_on_fail = 1;
  port_start = argv[1];
  if (port_start[0] == 'w') {
    exit_on_fail = 0;
    port_start++;
  }
  port = strtol(port_start, &port_end, 10);
  if (port_end != port_start + strlen(port_start)) {
    printf("port %s invalid format, must be integer between 1 and 65535\n",
           port_start);
    return -2;
  }
  if (port < 1 || port > 65535) {
    printf("port %d invalid, must be between 1 and 65535\n", port);
    return -3;
  }

  sock = socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP);
  if (sock == -1) {
    perror("could not create socket");
    return -4;
  }

  sin6.sin6_family = AF_INET6;
  sin6.sin6_port = htons(port);
  sin6.sin6_addr = in6addr_loopback;

  status = -1;
  while (status < 0) {
    status = bind(sock, (const struct sockaddr *) &sin6, sizeof(sin6));
    if (status < 0) {
      if (exit_on_fail) {
        /* perror("could not bind socket"); */
        return -5;
      }
      timeout.tv_sec = 1;
      timeout.tv_usec = 0;
      /* fputs("sleeping...\n", stderr); */
      select(0, NULL, NULL, NULL, &timeout);
      
    }
  }
  /* default to /bin/sh if no args are supplied, so we can do something like:
     #!lock 2048
     instead of
     #!/bin/sh
  */
  argv[1] = "/bin/sh";
  execvp("/bin/sh", &argv[1]);
}

Setting HTTP headers for a static site on AWS CloudFront

2021-04-28 Network

TL:DR This is way, way more complicated than it needs to be

For a very long time, I ran this site off my cloud server in the US. When I moved to London, I started experiencing the painful impact of the ~100ms latency on the loading time for images and videos, and decided to move to a Content Delivery Network (CDN) with global reach. Unfortunately, most CDNs have steep minimum spend requirements that are excessive for a low-traffic site like this one. Amazon’s CloudFront is an exception, and my hosting costs are in the vicinity of $20 per month, which is why I settled for it despite my dislike for Amazon.

Serving a static site is not just about putting content somewhere to be served over HTTPS. You also need to set up HTTP headers:

Cache-Control headers to ensure static content isn’t constantly checked for changes.
Security Headers to enable HSTS and protect your users from abuse like Google FLoC, sites that iframe your content or XSS injection.

In my original nginx configuration, this is trivial if a bit verbose, just add:

expires: max;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
add_header Content-Security-Policy "default-src 'self' https://*.majid.info/ https://*.majid.org/; object-src 'none'; frame-ancestors 'none'; form-action 'self' https://*.majid.info/; base-uri 'self'";
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy no-referrer-when-downgrade;
add_header Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; execution-while-not-rendered 'none'; execution-while-out-of-viewport 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; layout-animations 'none'; legacy-image-formats 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; navigation-override 'none'; oversized-images 'none'; payment 'none'; picture-in-picture 'none'; publickey-credentials-get 'none'; sync-xhr 'none'; usb 'none'; vr 'none'; wake-lock 'none'; screen-wake-lock 'none'; web-share 'none'; xr-spatial-tracking 'none'; notifications 'none'; push 'none'; speaker 'none'; vibrate 'none'; payment 'none'";
add_header Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), serial=(), sync-script=(), trust-token-redemption=(), vertical-scroll=(), notifications=(), push=(), speaker=(), vibrate=(), interest-cohort=()";

Doing this with CloudFront is much more complicated, however. You have to use a stripped-down and specialized version of their AWS Lambda “serverless” Function-as-a-Service framework, Lambda@Edge. This is very poorly documented, so this is my effort at rectifying that. When I first set this up, only Node.js and Python were available, but it seems Go, Java and Ruby were added since. I will use Python for this discussion. The APIs are quite different for each language so don’t assume switching languages is painless.

In the interests of conciseness, I am going to skip the parts about creating a S3 bucket and enabling it for CloudFront. There are many tutorials available online. I use rclone to deploy actual changes to S3, and make an AWS API call using awscli to trigger a cache invalidation, but software like Hugo has built-in support for AWS. Here is my deployment target in my Makefile:

deploy:
	git push
	git push github master
	-rm -rf awspublic
	env HUGO_PUBLISHDIR=awspublic hugo --noTimes
	-rm -f awsindex.db
	env HUGO_BASE_URL=https://blog.majid.info/ ./fts5index/fts5index -db awsindex.db -hugo
	rclone sync -P awspublic s3-blog:fazal-majid
	rsync -azvH awspublic/. bespin:hugo/public
	scp awsindex.db bespin:hugo/search.db
	ssh bespin svcadm restart fts5index
	aws cloudfront create-invalidation --distribution-id E************B --paths '/*'

First of all, even though Lambda@Edge runs everywhere CloudFront does, you cannot create functions everywhere, so you will need to go to the Lambda functions console then switch your region to US-West-1 in your AWS Console drop-down menu (even though my CloudFront and S3 are in eu-west-2 (London).

Click on the Create Function button.

Then choose Author from scratch, give a name (in my case, SecurityHeaders) and choose the Python 3.8 runtime.

In the development environment, click on lambda_function.py to edit the code of your function.

Click on Deploy (which is really more of a Save button), then press the orange Test button. Choose the Event Template cloudfront-modify-response-header. Save it, e.g. TestHeaders and click again on the Test button to verify the function executes without exceptions.

Here is the code I use:

def lambda_handler(event, context):
    cf = event["Records"][0]["cf"]
    response = cf["response"]
    headers = response["headers"]
    headers['strict-transport-security'] = [{
      "key": "Strict-Transport-Security",
      "value": "max-age=31536000; includeSubDomains; preload"
    }]
    headers['content-security-policy'] = [{
      "key": "Content-Security-Policy",
      "value": "default-src 'self' https://*.majid.info/ https://*.majid.org/; object-src 'none'; frame-ancestors 'none'; form-action 'self' https://*.majid.info/; base-uri 'self'"
    }]
    headers['x-frame-options'] = [{
      "key": "X-Frame-Options",
      "value": "SAMEORIGIN"
    }]
    headers['x-xss-protection'] = [{
      "key": "X-Xss-Protection",
      "value": "1; mode=block"
    }]
    headers['x-content-type-options'] = [{
      "key": "X-Content-Type-Options",
      "value": "nosniff"
    }]
    headers['referrer-policy'] = [{
      "key": "Referrer-Policy",
      "value": "no-referrer-when-downgrade"
    }]
    headers['feature-policy'] = [{
      "key": "Feature-Policy",
      "value": "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; execution-while-not-rendered 'none'; execution-while-out-of-viewport 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; layout-animations 'none'; legacy-image-formats 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; navigation-override 'none'; oversized-images 'none'; payment 'none'; picture-in-picture 'none'; publickey-credentials-get 'none'; sync-xhr 'none'; usb 'none'; vr 'none'; wake-lock 'none'; screen-wake-lock 'none'; web-share 'none'; xr-spatial-tracking 'none'; notifications 'none'; push 'none'; speaker 'none'; vibrate 'none'; payment 'none'"
    }]
    headers['permissions-policy'] = [{
      "key": "Permissions-Policy",
      "value": "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), serial=(), sync-script=(), trust-token-redemption=(), vertical-scroll=(), notifications=(), push=(), speaker=(), vibrate=(), interest-cohort=()"
    }]
    headers['x-fm-version'] = [{
      "key": "x-fm-version",
      "value": str(context.function_version)
    }]
    # caching
    if "request" in cf and "uri" in cf["request"]:
      url = cf["request"]["uri"]
      ext = url.split('.')[-1].lower()
      if url.endswith('/') or ext in ('html', 'gif', 'png', 'jpg', 'jpeg', 'ico', 'css', 'js', 'eot', 'woff', 'mp4', 'svg'):
        headers['expires'] = [{
          "key": "Expires",
          "value": "Thu, 31 Dec 2037 23:55:55 GMT"
        }]
        headers['cache-control'] = [{
          "key": "Cache-Control",
          "value": "max-age=315360000, immutable"
        }]
        
    return response

You will need to modify the hardcoded value for Content-Security-Policy, most likely you don't want your images and assets to be only served from https://*.majid.info/... Also, I cache all HTML forever in the browser, which may be more aggressive than you want if you update content more frequently than I do.

Before you can set up the hook, you will need to deploy your code to Lambda@Edge.

Now, this is very important. There are 4 different places a Lambda@Edge function can hook into.

If you deploy your function in the wrong place, most likely you will cause HTTP 500 errors until you can delete the bad trigger and redeploy, a process that takes an interminable 5–10 minutes to percolate through the CloudFront network (ask me how I know...). The hook (event trigger in Lambda@Edge parlance) is Viewer Response, unfortunately the deployment dialog defaults to Origin Request.

Click the disclaimer checkbox and press the Deploy button. It will take a few minutes to deploy to CloudFront, and then you can use curl or your browser’s developer console to verify the headers are sent. I include a header X-FM-Version to verify which version of the function was deployed.

fafnir ~>curl -sSL -D - -o /dev/null 'https://blog.majid.info/hsts-preload/'
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 26260
x-amz-id-2: 3ndAsEvUgHDhUYxok9kDnaNCUeQ8QMCbVURoiyjQHc699mrHQvJpN7xwgUeAp7Ir/9Pd1sLwtOU=
x-amz-request-id: 0NDCZD7JEG55903A
date: Wed, 28 Apr 2021 17:55:17 GMT
x-amz-meta-mtime: 1618529322.342819304
last-modified: Thu, 15 Apr 2021 23:28:59 GMT
etag: "33eb01a86db2b3f800c7bee0b5c10c11"
server: AmazonS3
vary: Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
content-security-policy: default-src 'self' https://*.majid.info/ https://*.majid.org/; object-src 'none'; frame-ancestors 'none'; form-action 'self' https://*.majid.info/; base-uri 'self'
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: no-referrer-when-downgrade
feature-policy: accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; execution-while-not-rendered 'none'; execution-while-out-of-viewport 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; layout-animations 'none'; legacy-image-formats 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; navigation-override 'none'; oversized-images 'none'; payment 'none'; picture-in-picture 'none'; publickey-credentials-get 'none'; sync-xhr 'none'; usb 'none'; vr 'none'; wake-lock 'none'; screen-wake-lock 'none'; web-share 'none'; xr-spatial-tracking 'none'; notifications 'none'; push 'none'; speaker 'none'; vibrate 'none'; payment 'none'
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), serial=(), sync-script=(), trust-token-redemption=(), vertical-scroll=(), notifications=(), push=(), speaker=(), vibrate=(), interest-cohort=()
x-fm-version: 22
expires: Thu, 31 Dec 2037 23:55:55 GMT
cache-control: max-age=315360000, immutable
x-cache: Hit from cloudfront
via: 1.1 f655cacd0d6f7c5dc935ea687af6f3c0.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS54-C1
x-amz-cf-id: QkB3rN2hWiI8ah_EJ3x3bvjgbm_BrqhFG1GJ_f4po-Mc2rs_TjTF-g==

Needless to say, because of the convoluted nature of this process, and the high likelihood of making mistakes, you should test this on a non-production site before you try this on a live site.

If by error you associated the lambda function with the wrong event trigger, you can delete it by going through the different deployed versions of your function, finding the trigger and deleting it.