Reverse Engineering HMRC Basic PAYE Tools to Understand Platform Support

Summary

HMRC provides a Basic PAYE Tools (BPT) application for small companies to do pay as you earn (PAYE) returns. The tool required Rosetta when opening on my Macbook (only x86_64 build), leading to an ensuing decompilation and exploration of the source code and history of the project from publicly available resources. We found that BPT is built with Qt6, Python 2.7, and Django v1.6. Python 2.7 has no official arm64 target. We took the time to understand what it would take for BPT to be migrated to a version of Python and Django that would support arm64

Introduction

Hello Readers,

I recently started a business and needed to set up PAYE (Pay As You Earn) for some employees. For small businesses HMRC provides a Basic PAYE Tools (BPT) application. I downloaded and went to install it but it required Rosetta 2, the x86_64 translation layer for macOS. At this point, not having installed Rosetta yet and knowing about its oncoming sunset, I looked at the other versions available on the HMRC webpage. A Linux version of the application was available as well. This led me to ask whether it was possible to run the application on my Linux server, and if there was a better way than to stream a window to my laptop. I don’t particularly like the idea of X forwarding and VNC if I can help it.

Why the Mac build needs Rosetta

Before digging into the linux version, it would be interesting to understand why the macOS version needed Rosetta. What part of the binary needed x86_64 architecture. In the accompanying README in the macOS .pkg, HMRC says “BPT is built exclusively for x64 (64-bit Intel/AMD compatible) operating systems. macOS is supported on Apple Silicon-based Mac computers via the Rosetta 2 emulation layer.” Okay, this tells us that their targets are x86_64 across their different images, but doesn’t tell us why the binaries can’t be built for arm64, what in their toolchain is impeding their targets?

The Mac build is published as payetools-rti-<version>-osx.zip, 180 MB with a .pkg inside. We can use xar to extract the archive. This releases a pbzx payload, holding a cpio. Unfortunately Apple doesn’t publish any specifications for pbzx (typical). However the community has been able to build out an informal specification through trial and error. pudquick’s write-up and matteyeux’s C implementation are the usable references. Following this reference we understand that there is a magic pbzx, then 16-byte chunk headers, and then we xz per resulting chunk. This leaves us with four layers to open, with only two of them specified anywhere. Fortunately file reports the architectures directly, because a universal binary carries a fat header listing each slice. Moreover, every Qt framework in it is a universal binary, x86_64 and arm64, whereas every binary HMRC built themselves is x86_64 only:

We also find that the updater is named osx-x86_64, and the installer declares it. Distribution in the .pkg carries <options ... hostArchitectures="x86_64"/>. HMRC states the target themselves. This means that the Qt arm64 slices are already sitting in the package but are never executed (inflating the binary size!). This tells us HMRC’s own binaries are the cause of the restriction, but it would be nice to understand deeper why. The name of one of the slices gives the game away, libpython2.7.dylib has no first party arm64 compilation target. Python 2.7 reached end of life (EOL) in January 2020, 10 months before Apple shipped the first Apple Silicon device in November 2020. There is no official arm64 macOS build of CPython 2.7. The runtime predates the architecture. cx_Freeze links the frozen app against libpython. No arm64 libpython, no arm64 app. This however doesn’t tell us why they are still using CPython 2.7, a language that has been sunset for almost 6 years at this point.

# Four layers: zip -> pkg -> xar -> pbzx -> cpio. None of it documented.
BASE=https://www.gov.uk/government/uploads/uploaded/hmrc
curl -fL -o osx.zip "$BASE/payetools-rti-26.1.26134.173934-osx.zip"
unzip -q osx.zip -d osx

# a .pkg is a xar archive
cd osx && xar -xf payetools-rti-26.1.26134.173934-osx.pkg
cat uk.gov.hmrc.bptrti.pkg/PackageInfo        # 13,500 files, installs to /Applications

# the Payload is Apple's pbzx: magic, then 16-byte chunk headers, xz per chunk
od -An -tx1 -N4 uk.gov.hmrc.bptrti.pkg/Payload      # 70 62 7a 78  = "pbzx"

cat > pbzx.py <<'EOF'
import sys, struct, lzma
f=open(sys.argv[1],'rb'); out=open(sys.argv[2],'wb')
assert f.read(4)==b'pbzx'
f.read(8)                                   # flags
while True:
    hdr=f.read(16)
    if len(hdr)<16: break
    unc,comp=struct.unpack('>QQ',hdr)
    data=f.read(comp)
    if len(data)<comp: break
    out.write(data if comp==unc else lzma.decompress(data))
EOF
python3 pbzx.py uk.gov.hmrc.bptrti.pkg/Payload payload.cpio

# pull out only the executables and read their Mach-O headers
mkdir ext && cd ext
cpio -idm --quiet '*/MacOS/*' '*libpython*' < ../payload.cpio
find . -type f -exec file {} \;
# -> Qt frameworks: universal x86_64 + arm64
# -> everything HMRC built: x86_64 only, including libpython2.7.dylib

What is the window?

So the question becomes, What is actually providing the interface for BPT? Looking at the Linux Image, it is 448 MB, with 322 MB of that being from Qt. We find that the browser bundled is Chromium (shocker) and there are only two interesting strings in the whole 1 MB binary: 46729 and http://127.0.0.1:%1%2. If that doesn’t look like the backend and port, I don’t know what does. We could probably expose the http on 46729 across the network and be able to bypass the chromium middleman. There isn’t much else within the Qt - Browser section, so let us shift our focus to the actual BPT binaries from HMRC.

The same app in two wrappers

Looking at the BPT binaries from HMRC, the Mac image opened in why the Mac build needs Rosetta, and Linux follow an identical design from what is the window?. The only thing that meaningfully differs is the packaging. Both Images use a Qt WebEngine shell in front of a frozen Python 2.7 server.

du -sh squashfs-root/usr/*
strings -a squashfs-root/usr/bin/bptshell | grep -aiE '^(Qt[A-Za-z]+|libQt)' | sort -u
strings -a squashfs-root/usr/bin/bptshell | grep -aiE '46729|127\.0\.0\.1|http://' | sort -u
# the only two interesting strings in 1 MB:
#   46729
#   http://127.0.0.1:%1%2

So run just the server

The server lives in usr/bptserver/ — 121 MB, libpython2.7.so, and rti.linux. rti.linux is an 18 MB stripped ELF. There isn’t any loose .pyc and no available source code. There is also no documentation on running it standalone (this isn’t intended is it?). If we run strings rti.linux | grep -i cx_freeze we obtain:

cx_Freeze__init__
cx_Freeze__init__.pycPK
libpython2.7.so.1.0

This tells us something important! PK after a .pyc filename is a zip local-file-header signature. 0x04034b50, section 4.3.7 of PKWARE’s APPNOTE.TXT, the format’s defining document. There is a zip appended to the executable! Let us run unzip -l rti.linux. This produces 6,867 files 👀. Huzzah we have progress!

BASE=https://www.gov.uk/government/uploads/uploaded/hmrc
curl -fL -o release.zip "$BASE/payetools-rti-26.1.26134.173934-linux.zip"
unzip -q release.zip

head -c 64 Basic_PAYE_Tools-*.AppImage | od -An -c | head -2
# "AI\002" at offset 8 = 0x414902, the AppImage type 2 magic
chmod +x Basic_PAYE_Tools-*.AppImage
./Basic_PAYE_Tools-*.AppImage --appimage-extract                # no FUSE required

file squashfs-root/usr/bptserver/rti.linux
strings -a squashfs-root/usr/bptserver/rti.linux \
  | grep -aiE 'cx_freeze|libpython' | sort -u
#   cx_Freeze__init__.pycPK   <- "PK" = a zip appended to the ELF

unzip -l squashfs-root/usr/bptserver/rti.linux | tail -3        # 6867 files

rti.linux is manage.py

Now let us turn our attention to __main__.pyc which is only four lines:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
execute_from_command_line(sys.argv)

It contains django-admin. runserver, dumpdata, shell, huh this is everything we need! This isn’t an opaque binary shockingly, this is just a seriously old Django project in a trenchcoat. We can invoke runserver and it comes online… awesome. Now two headless instances run out of the box, let’s see what happens when we connect from the laptop. 400. After some digging, it was just a whitelist that needed to be altered, where ALLOWED_HOSTS is ['127.0.0.1', 'testserver']. So let us just bind anywhere non-local, and HMRC’s own settings flip it to ['*']. Now from 127.0.0.2: 400 → 302, awesome. ['*'] plus no authentication. We could now bind to the Tailscale interface if I wanted to access BPT from my laptop without Rosetta, awesome. Fuzzing a little, --server-only and mode = server exist but are unfortunately undocumented. Don’t exactly know what they do but aren’t necessary at the moment.

mkdir -p bytecode
unzip -q -o squashfs-root/usr/bptserver/rti.linux -d bytecode
chmod -R u+rwX bytecode                     # archive entries are read-only
pycdc bytecode/__main__.pyc                 # -> execute_from_command_line(sys.argv)

# it is django-admin, so just run it
./rti.linux runserver 127.0.0.1:46730 --noreload

# from another host you get 400 - ALLOWED_HOSTS is loopback-only
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Host: elsewhere:46730' http://127.0.0.1:46730/bptrti/datareview/

# binding non-locally flips ALLOWED_HOSTS to ['*']; 127.0.0.2 proves it safely
printf '[General]\nbind_address = 127.0.0.2\n' > rti.cfg

Updates are now my problem

But we’ve made a right mess of the binary at this point, and one of the key things about BPT is that it updates when tax policy changes. We’ll need to reimplement this ourselves. This is because originally the Qt shell drove auto-update (we kinda gutted the Qt shell). To start with, we can look at the manifest the app itself polls. Looking at the Qt shell closely we find that realtimepayetools-update-v26.xml is the endpoint, under gov.uk’s HMRC uploads path. gov.uk 301s these endpoints to assets.publishing.service.gov.uk, so we follow the redirects to the actual endpoints. Each major’s manifest advertises its successor: v14 advertises v15.0, and v15 advertises v16.0. The update channel thus acts like a linked list through the major versions. There is no authentication on any of the endpoints, the binaries can be received with simple GET. I was able to walk the entire range, not just the recent ones, and was able to produce a current list of accessible releases. The root of the chain is the unversioned realtimepayetools-update.xml, which advertises 14.0.14063.106. v1 to v13 all 404, with the same gov.uk body as v27:

{"_response_info":{"status":"not found"}}

All fourteen versions are still downloadable on all three platforms, Linux, macOS, and Windows. The versions span 12 years 2 months: 14.0 built 4 March 2014, 26.1 built 14 May 2026. The versionId is the build number and it is a date: 26134 is 2026 day 134, 14 May. 15048 is 2015 day 48. Every release is stamped with when it was built.

14.0.14063.106      major 14   built 2014 day 063
15.0.15048.300      major 15   built 2015 day 048
...
25.1.25092.226      major 25   built 2025 day 092
26.1.26134.173934   major 26   built 2026 day 134

This could imply that the major version is the calendar year, leading to the conclusion that there was never a version 1 through 12. Version 13 would be 2013, the first year of RTI (Real Time Information, the scheme that requires employers to report each pay run to HMRC as it happens), and those clients had no versioned manifest to poll — they used the unversioned one, which is why it still exists and still points at 14.0. It is the v13 manifest without the suffix. v1 to v13 404 because the numbering scheme never produced them. Confirmed from inside the oldest build rather than inferred. 14.0 is a 32-bit i386 static ELF with a zip appended — the same trick as rti.linux, twelve years earlier — except the central directory is wrapped, so unzip refuses it and the local headers have to be carved by hand. Carving buildconstants.pyc out of it:

$ pycdc buildconstants.pyc | grep -E 'VERSION_FULL|UPDATE_XML|LISTENPORT'
BUILD_VERSION_FULL    = '14.0.14063.106'
BUILD_UPDATE_XML_PATH = 'realtimepayetools-update-v14.xml'
PRODUCT_LISTENPORT    = 46729

Interestingly port 46729 has not moved in twelve years either. Also the magic number in that 2014 .pyc is 62211. The same Python 2.7. So BPT has shipped Python 2.7 for at least 12 years 5 months — that is only as far back as the downloads go, so the real figure is longer — and for 6 years 7 months since the runtime reached end of life. The chain skips releases. v23 advertises 24.1 and v24 advertises 25.1, so 24.0 and 25.0 existed and are not reachable this way. The revision suffix is unguessable, so the chain gives you the history it chooses to, not all of it. The jump at v24 is not packaging. 70 MB to 175 MB is a change of browser engine.

BASE=https://www.gov.uk/government/uploads/uploaded/hmrc

# sweep the whole range, not just the recent majors
for v in $(seq 1 32); do
  code=$(curl -sL -o /tmp/m.xml -w '%{http_code}' --max-time 15 \
         "$BASE/realtimepayetools-update-v$v.xml")
  [ "$code" = 200 ] || continue
  ver=$(tr -d '\n' < /tmp/m.xml | sed -n 's|.*<version>\([^<]*\)</version>.*|\1|p')
  echo "v$v -> $ver"
done

# and the unversioned one, which is the root of the chain
curl -sL "$BASE/realtimepayetools-update.xml" | grep -E '<version>|<versionId>'

# are the historical releases still there? (parallel, HEAD only)
printf '%s\n' 14.0.14063.106 15.0.15048.300 ... 26.1.26134.173934 \
| xargs -P 14 -I{} sh -c \
  'curl -sIL --max-time 20 "'"$BASE"'/payetools-rti-{}-linux.zip" \
   | awk "BEGIN{IGNORECASE=1}/^HTTP/{c=\$2}/^content-length:/{l=\$2}END{print \"{}\", c, l}"'

# patch files, per platform - the answer differs by platform
for p in win linux osx; do
  curl -sIL -o /dev/null -w "patch-$p %{http_code}\n" \
    "$BASE/payetools-rti-patch-26.1.26134.173934-$p.zip"
done

From PyQt4 to Qt6

The size jump is the visible edge of the only structural change in BPT’s history. Carving the payload out of each installer shows exactly what the window was made of.

The evidence

If we investigate the v14.0, we can see it carries a broad PyQt4:

PyQt4/QtWebKit.pyc   PyQt4/QtOpenGL.pyc   PyQt4/QtScript.pyc   PyQt4/QtSql.pyc
PyQt4/QtTest.pyc     PyQt4/QtGui.pyc      PyQt4/QtCore.pyc     PyQt4/QtSvg.pyc

Likewise if we look at v24.1, it carries a trimmed down PyQt4 instance. The time between v14 to v24.1 is about ten years later, but v24.1 still uses Qt4 and WebKit:

PyQt4/QtWebKit.pyc   PyQt4/QtCore.pyc     PyQt4/QtNetwork.pyc  PyQt4/sip.pyc

This however changed in the v25.1 and v26.1 releases; neither carry Python Qt bindings at all:

$ unzip -l v25/squashfs-root/usr/bptserver/rti.linux | grep -ciE 'PyQt|/sip\.pyc'
0
$ ls bytecode/ | grep -iE '^PyQt|^sip'
$

v25.1 is the first AppImage, with the shell split out into its own binary:

$ ls v25/squashfs-root/usr/bin/
bptshell  qt.conf
$ strings -a v25/squashfs-root/usr/bin/bptshell | grep -aoE 'libQt6WebEngine[A-Za-z]*' | sort -u
libQt6WebEngineCore
libQt6WebEngineWidgets

The v25.1 Linux “patch” was the 4,371-byte notice telling users to go and download it by hand. Indicating that the version change required a complete reinstall due to upgrading from PyQt4 to Qt6.

So What?

PyQt4 and QtWebKit were used from at least v14 and the Qt 4.8’s open-source support ended at the end of 2015 calendar year. QtWebKit was removed from Qt in 5.6, March 2016. So v24.1 shipped a browser engine that had been out of Qt for 8 years, and a toolkit unsupported for 8 years 2 months. v25.1 finally updated the browser from WebKit to Chromium. The bundling of libQt6WebEngineCore.so.6 alone is 159.3 MB, and WebEngine on top with its resources and translations is 213 MB after decompression. This is the main cause of the binary size increase from v24.1 to v25.1. The codebase grew across the same period, but nowhere near enough to explain it: 2,529 .pyc in v14.0, 6,111 in v24.1, 6,352 in v25.1, 6,867 in v26.1.

Going forward what does this tell us about HMRC?

Well for starters it tells us the separation of the browser and HMRC’s binary was only possible in the past couple years. We would have been stuck doing this surgery ourselves to get the http advertised publicly.

Surprisingly, this uplift from PyQt4 to Qt6 shows that HMRC are capable of moving from one dependency to another when the source code needs it. They moved the entire GUI out of Python and into a separate C++ process, changed packaging format, and pushed users through a manual reinstall. However they did this all while leaving Python 2.7 and Django 1.6.

# the pre-25 installers are BitRock: an ELF with a zip payload, but the central
# directory is wrapped, so unzip refuses it. Carve the local headers instead.
python3 - <<'EOF'
import re, struct, zlib
d = open('payetools-rti-24.1.24086.542-linux','rb').read()
for m in re.finditer(b'PK\x03\x04', d):
    o = m.start()
    try: (v,fl,meth,t,dt,crc,csz,usz,nlen,elen) = struct.unpack('<HHHHHIIIHH', d[o+4:o+30])
    except struct.error: continue
    if not 0 < nlen <= 512: continue
    name = d[o+30:o+30+nlen]
    try: name = name.decode('utf-8')
    except Exception: continue
    if re.match(r'^[\w./\-+ ]+$', name):
        print(name)                     # -> PyQt4/QtWebKit.pyc, and 6110 others
        # body: d[o+30+nlen+elen : ...+csz], zlib.decompress(body, -15) if meth==8
EOF

# 25.1 onward is an AppImage, so it opens the normal way
./Basic_PAYE_Tools-25.1.25092.226-x86_64.AppImage --appimage-extract
ls squashfs-root/usr/bin/                                    # bptshell appears here
strings -a squashfs-root/usr/bin/bptshell | grep -aoE 'libQt6WebEngine[A-Za-z]*' | sort -u
unzip -l squashfs-root/usr/bptserver/rti.linux | grep -ciE 'PyQt|/sip\.pyc'   # 0

# where the size went
ls -la squashfs-root/usr/lib/ | sort -k5 -nr | head -3
du -ch usr/lib/libQt6WebEngine*.so* usr/resources/ usr/translations/ | tail -1

What is decompiled?

Okay but what goodies does the binary piñata have? Well the binary has a magic number. The magic number is the first two bytes of any .pyc:

$ od -An -tx1 -N4 bytecode/bptrti/urls.pyc
 03 f3 0d 0a
$ od -An -tu2 -N2 bytecode/bptrti/urls.pyc
 62211

62211 is 0xf303. CPython’s own Python/import.c on the 2.7 branch defines it, and accounts for all four bytes:

#define MAGIC (62211 | ((long)'\r'<<16) | ((long)'\n'<<24))

03 f3 is 62211 little-endian; 0d 0a is the \r\n the macro appends. So: Python 2.7. That picks pycdc and dates the codebase. The runtime is shipped alongside, and corroborates this finding:

$ ls squashfs-root/usr/bptserver/ | grep -i python
libpython2.7.so.1.0

The same file documents the rest of the header. write_compiled_module writes the magic, then the source mtime as a 32-bit field, then the marshalled code object. So bytes 4–7 are the mtime of the source it was compiled from. Knowing that the bytes 4-7 are mtime at compile time turns out to be extremely useful.

od -An -tu2 -N2 bytecode/bptrti/urls.pyc     # 62211 = Python 2.7
od -An -tx1 -N8 bytecode/bptrti/urls.pyc     # bytes 4-7 = source mtime

What does HMRC change in a release?

Okay, now that I’ve taken over the process of updating the BPT app, I need to look at what changes between versions. To do this I first hash every .pyc in v26.0 and v26.1, compare: 6,195 of 6,867 changed. 90% of the application in a point release. This doesn’t seem to be correct at all. I then picked files that really shouldn’t have changed between releases: django/db/models/base.pyc, calculators/__init__.pyc. I then ran cmp -l on all three: bytes 5, 6, 7, 8. I then implemented the skip of the initial 8 bytes, and then reran the hash comparison. Only 11 files changed, huzzah! The full bytecode changeset of a release is 11 files.

OLD=../26.0.26069.145147/bytecode
NEW=./bytecode

# naive: 6195 of 6867 "changed"
cd "$NEW"
while IFS= read -r f; do
  [ -f "$OLD/$f" ] || continue
  [ "$(sha256sum "$f" | cut -d' ' -f1)" != "$(sha256sum "$OLD/$f" | cut -d' ' -f1)" ] \
    && echo "CHANGED $f"
done < <(find . -name '*.pyc' | sed 's|^\./||') | wc -l

# what actually differs in a file that cannot have changed
cmp -l "$OLD/django/db/models/base.pyc" "$NEW/django/db/models/base.pyc" | awk '{print $1}'
# -> 5 6 7 8   (the mtime, every time)

# header-aware: skip 8 bytes -> 11 changed
while IFS= read -r f; do
  a=$(tail -c +9 "$f"      | sha256sum | cut -d' ' -f1)
  b=$(tail -c +9 "$OLD/$f" | sha256sum | cut -d' ' -f1)
  [ "$a" != "$b" ] && echo "CHANGED $f"
done < <(find . -name '*.pyc' | sed 's|^\./||' | sort)

Can I trust the decompiler?

Now the real question. Can I trust the decompiler? The real v26.1 changes between v25.1: _table_4_pension_check(), Pensions Act 2014 Table 4. State Pension age 66 → 67 for DOBs 6 Apr 1960 – 5 Mar 1961. Docstring intact. Statute legible. Same hunk:

if None >= datetime.date(1960, 4, 6):     # should be `dob`
pensionable_date = None.date(2019, 3, 6)  # should be `datetime`
if dob <= dob:                            # nonsense

What can I get out of decompiling the pyc? I can get a general grasp of what was being implemented, but I can’t entirely reverse engineer mechanically at the moment. To be fair, I just want to poke around at this point to see what makes the application tick.

pycdc "$OLD/r26_lib/dates.pyc" > /tmp/a.py 2>/dev/null
pycdc "$NEW/r26_lib/dates.pyc" > /tmp/b.py 2>/dev/null
diff -u /tmp/a.py /tmp/b.py

Is there an API under this?

If I wanted to update the frontend, or for that matter if HMRC wanted to update the frontend, it would be great if there was a REST API or the like to hit when the user interacts with the platform. I was able to find urls_ajax.pyc, which looked initially promising: data/employer, data/payment, automate/employee/create. However all of these were CSRF- and session-bound plumbing for the app’s own JavaScript :(. Only the helpers like datehelpers/subtract return clean JSON! There is absolutely no API with the application. A new frontend would need to drive the same Django forms (yuck) or add a JSON layer on top (also yuck).

How does the browser talk to it?

Okay, so how does communication work? The browser talks to the frontend over http (makes sense). Server-rendered Django HTML (yuck why do you need dynamic pages on a bloody installed application?) and then uses CSRF and session middleware. Interestingly, libQt6WebChannel is not linked. And there is no bridge object into the binary. One IPC path exists: QT_PIPE in django_offline/connector.py. There are two consumers in code, however the interesting one is dead!:

def openfile(request):
    '''DEPRECATED'''
    return {'path': ''}

grep -rla 'DjangoOfflineConnector' bytecode/          # only 3 files
pycdc bytecode/django_offline/connector.pyc           # the QT_PIPE
pycdc bytecode/bptrti/views/execute/filehelpers.pyc | head -25   # openfile: DEPRECATED
strings -a squashfs-root/usr/bin/bptshell | grep -aiE 'webchannel' | sort -u

Why is the codebase shaped so strangely?

Honestly the codebase is really strange, with how it is set up and maintained. But if we step back and understand that this application needs to keep all tax year computations around, it makes a bit more sense. There are in total 232 per-year template dirs! BPT is shipped with a sqlite3.db. It contains 674 tables, 479 of them cloned per tax year, roughly 48 tables a year! HMRC obviously doesn’t believe in versioning rules against a stable schema, and instead clone the schema annually. This is, awkwardly enough, the correct approach: amended prior-year returns must compute under that year’s rules, and cloning removes any regression risk for closed years. It does however cause some interesting idiosyncrasies, for example there is no year-agnostic Employee! This makes tax-year top of mind for any feature. Every single layer inside of the application touches the tax-year being calculated for.

DB=squashfs-root/usr/bptserver/sqlite3.db
sqlite3 "$DB" "select count(*) from sqlite_master where type='table';"        # 674
sqlite3 "$DB" "select count(*) from sqlite_master where type='table'
               and name GLOB '*_r[0-9][0-9]_*';"                              # 479
sqlite3 "$DB" "select name from sqlite_master where type='table'
               and name GLOB '*_r[0-9][0-9]_*';" \
  | grep -oE '_r[0-9]{2}_' | sort | uniq -c
ls squashfs-root/usr/bptserver/templates | grep -cE '_r[0-9]{2}$'             # 232

What actually leaves the building?

Okay, so one of the interesting things we can look at while the guts are pulled out is what actually is included in messages to HMRC from BPT. What things are sent and what are held back? Nicely this doesn’t require any decompilation. The message formats ship as plain XSD and Schematron. There are 88 files under static/schemas/, generated by CoreFiling for HMRC. This combines with the local data model, which is readable from the shipped sqlite3.db.

Every submission carries: Government Gateway SenderID, password as Method: clear inside TLS. Plus both tax office references, and an IRmark (the integrity hash HMRC’s gateway checks a submission against) — base64(SHA1(c14n(body))). Additionally ChannelRouting/URI = 9205, HMRC’s vendor ID for BPT, with product name and version. The FPS (Full Payment Submission) is the substantive message, the one sent on every pay run. 189 fields, covering identity (including NINO, the employee’s National Insurance number), employment context, year-to-date totals, and the payment itself:

identity    NINO?  name  address?  BirthDate?  Gender  PassportNumber?
            PartnerDetails?
context     OffPayrollWorker?  OccPenInd?  DirectorsNIC?(AN|AL)
            Starter?(StartDate, StartDec, StudentLoan?, PostgradLoan?,
                     Seconded?, OccPension?, StatePension?)
            EmployeeWorkplacePostcode?  PayId?  IrrEmp?  LeavingDate?
YTD         TaxablePay  TotalTax  StudentLoansTD?  PostgradLoansTD?
            BenefitsTaxedViaPayrollYTD?  EmpeePenContribns{Paid,NotPaid}YTD?
payment     PayFreq  PmtDate  WeekNo  MonthNo  PeriodsCovered
            HoursWorked (4 bands, top one "30 or more")
            TaxCode @BasisNonCumulative? @TaxRegime?(S|C)
            TaxablePay  ->  TaxDeductedOrRefunded
            StudentLoanRecovered? @PlanType(01|02|04|05)
            FlexibleDrawdown?  TrivialCommutationPayment?*
benefits    Car*  Make FirstRegd CO2 ZeroEmissionsMileage? Fuel Price
                  AvailFrom CashEquiv AvailTo? FreeFuel?
NI          NIletter  GrossEarningsForNICs{InPd,YTD}
            AtLELYTD  LELtoPTYTD  PTtoUELYTD
            TotalEmpNIC{InPd,YTD}  EmpeeContribns{InPd,YTD}

The EPS (Employer Payment Summary) is monthly adjustments: statutory recoveries, NIC compensation, CIS (Construction Industry Scheme) deductions suffered, apprenticeship levy, repayment bank details. EmpAllceInd is employment allowance as a bare indicator. No amount. The EPS never states what the employer thinks it owes. No liability field. HMRC derives the debt from the FPS stream and applies recoveries against it. The EYU (Earlier Year Update) is retired. Schemas run 2016-17 to 2019-20 then stop. From 2020-21 corrections go through another FPS. BPT ships both, because of the per-year cloning.

ls squashfs-root/usr/bptserver/static/schemas/taxyear_2026/
# .xsd = structure, .sch = business rules, both plain XML, no decompiling needed

for f in squashfs-root/usr/bptserver/static/schemas/taxyear_2026/*.sch; do
  echo "$f: $(grep -c '<sch:assert' "$f") asserts"
done

# readable rule list straight out of the schematron
grep -o 'id="a_[^"]*"[^>]*>[^<]*' \
  squashfs-root/usr/bptserver/static/schemas/taxyear_2026/FullPaymentSubmission-2027-v1-0.sch \
  | sed 's/ diagnostics="[^"]*"//'

# the field tree comes from the XSD; walk element refs and named complexTypes
python3 xsdtree.py FullPaymentSubmission-2027-v1-0.xsd IRenvelope 12

Can HMRC check the maths?

Yes

What can HMRC not do?

The main thing is that previous-employment pay and tax are never transmitted. employees_r26_taxdetails keeps previouspay/previoustax. The FPS has no element for them. This means a mid-year job change on a cumulative code cannot be verified from one FPS. HMRC must join it to the prior employer’s stream (yuck). Employer-side NI banding is withheld. BPT computes six secondary thresholds. ST, UST, AUST, VUST, FUST, IZUST. Three bands go out, all employee-side. Claim relief for an apprentice, veteran or freeport workplace and HMRC gets a smaller number and a postcode. payeresult keeps the whole intermediate calculation and sends none of it. taxfreepay, payadjustment, regulatorylimit, taxnotdeducted, taxpaymethod.

Contractual gross is stored as paymentactualgrossannualsalary and never sent. Salary sacrifice leaves no trace. It reduces taxable pay and NIC-able earnings, then vanishes. An employer sacrificing too much and one paying less are identical on the wire. Non-payrolled benefits go via P11D. Hours are four buckets. RTI is a reconciliation format, not an audit format. It proves internal consistency and lets HMRC cross-check employers against each other. It does not carry the employment facts and is not trying to. Consequence: 9205 is BPT’s vendor ID. Own submissions need their own, and HMRC recognition.

DB=squashfs-root/usr/bptserver/sqlite3.db
# what BPT computes and keeps locally
sqlite3 "$DB" '.schema employee_payments_r26_payeresult'
sqlite3 "$DB" '.schema employee_payments_r26_nicresult'
sqlite3 "$DB" '.schema employees_r26_taxdetails'      # previouspay / previoustax

# what it is allowed to send
SCH=squashfs-root/usr/bptserver/static/schemas/taxyear_2026
grep -icE 'previouspay|previoustax|PrevEmp' "$SCH/FullPaymentSubmission-2027-v1-0.xsd"   # 0
grep -oE 'name="[^"]*(ST|UEL|LEL|PT)[^"]*"' "$SCH/FullPaymentSubmission-2027-v1-0.xsd" | sort -u
# -> AtLELYTD, LELtoPTYTD, PTtoUELYTD only. No secondary thresholds, no above-UEL.

What would it take to get off Python 2.7

Now the creeping question once Python 2.7 was spotted, what would it take to get off Python 2.7?

To get off Python 2.7, it wouldn’t just be a language migration. It’s mainly a Django migration. The framework version is not documented anywhere, but funnily enough it is leaked in the bytecode:

$ pycdc bytecode/django/__init__.pyc | head -4
# Source Generated with Decompyle++
# File: __init__.pyc (Python 2.7)

VERSION = (1, 6, 0, 'final', 0)

Django 1.6.0 released in November 2013 and reached EOL in 2015.

An example target for the migration might be Python 3.11 which would need Django 4.1 or newer. A staggering jump of fifteen major releases. Something to make note of is that the project still uses South. South however has been dead since Django grew its own:

$ pycdc bytecode/south/__init__.pyc | grep __version__
__version__ = '0.8.2'
$ grep -o "'south'" decompiled/mysite/settings.py
'south'

South 0.8.2, still in INSTALLED_APPS. Django replaced it in 1.7, in 2014.

There are 273 South migration files across 169 migration directories to convert:

$ find bytecode -path '*/migrations/*.pyc' ! -name '__init__.pyc' | wc -l
273
$ find bytecode -type d -name migrations | wc -l
169

There are also a few Django APIs in active use that no longer exist:

pycdc bytecode/django/__init__.pyc | grep VERSION         # (1, 6, 0, 'final', 0)
pycdc bytecode/south/__init__.pyc  | grep version         # 0.8.2
find bytecode -path '*/migrations/*.pyc' ! -name '__init__.pyc' | wc -l    # 273
find bytecode -type d -name migrations | wc -l                             # 169

APP="bptrti bptrti_submission bptrti_pct bpt_shared bpt_settings calculators
     calculator_sncp calculator_spbp calculator_spp mysite django_offline
     r26 r26_lib r16 r17 penutils"
cd bytecode
find $APP -name '*.pyc' | wc -l                           # 2336
for id in iteritems has_key basestring unicode xrange cPickle urllib2 StringIO; do
  printf '%-12s %s\n' "$id" "$(grep -rla "$id" $APP | wc -l)"
done

# removed Django APIs still in the settings
grep -nE 'formtools|TransactionMiddleware|MIDDLEWARE_CLASSES|TEMPLATE_DIRS' \
  ../decompiled/mysite/settings.py
grep -rl 'from django.conf.urls import patterns' ../decompiled | wc -l      # 32

The actual blocker of a migration

There are 35 generated Schematron modules, three or four per tax year, 2016-17 through 2026-27. Each one embeds a dead XML stack:

from Ft.Xml.Xslt import PatternList, parser, Stylesheet, Processor
from Ft.Xml.XPath import Compile as CompileXPath
from Ft.Xml.Xslt import XsltFunctions, Exslt
from Ft.Xml.Domlette import NonvalidatingReader
from amara import domtools
import cStringIO

Ft is 4Suite: a full XSLT 1.0 and XPath engine, with C extensions. Amara sits on top of it. Both are bundled:

$ pycdc bytecode/amara/__config__.pyc | grep VERSION
VERSION = '1.2.0.2'

Their PyPI records say the rest. These dates are from the JSON API (I hope you get as much a fright as I did!):

$ curl -s https://pypi.org/pypi/4Suite-XML/json | jq -r '.info.version'
1.0.2
$ curl -s https://pypi.org/pypi/4Suite-XML/json \
    | jq -r '.releases["1.0.2"][0].upload_time[:10]'
2006-12-26
$ curl -s https://pypi.org/pypi/Amara/json | jq -r '.info.requires_python'
>=3.12

4Suite’s last release was 26 December 2006. Nineteen years ago. It only has Python 2 support evidently. The Amara on PyPI today is 4.1.0 and needs Python >= 3.12, so it is not the same library. The 1.x line BPT uses is dead.

These modules are run in the submission path, the main point of the application. They validate every FPS and EPS before it leaves. They must keep working for every year still open to amendment. On paper this is the death knell for the migration…

cd bytecode/bptrti_submission/submissions
find . -name '*stron.pyc' | wc -l                 # 35
for d in taxyear_*; do echo "$d $(ls $d/*stron.pyc 2>/dev/null | wc -l)"; done

# what they import
grep -rla 'Ft\.Xml|from Ft' . | head
grep -rla 'amara' . | head
pycdc taxyear_2026/nvr12stron.pyc | head -25
# -> Ft.Xml.Xslt, Ft.Xml.XPath, Exslt, Ft.Xml.Domlette, amara.domtools, cStringIO

Except… it’s not!

The .sch Schematron sources ship alongside the generated modules (Huzzah!). There are a total of 88 files under static/schemas/, one set per tax year. If the generated Python is a faithful derivative of the .sch, the .sch is the source of truth and 4Suite is replaceable. We can check this!

Rule by rule, generated module vs shipped schema

All the related 35 generated *stron.pyc modules decompile cleanly in the binary. Each rule appears as:

expr = CompileXPath(u'count(fps:Keys/fps:Key) > 0')
if not Conversions.BooleanValue(expr.evaluate(xpath_ctx)):
    WRITER.text(u'At least one key must exist in the IRheader')
    DIAGNOSTICS[u'errorCode.r1005'](xpath_ctx)

So each rule yields an id, an XPath string and a message. The .sch yields the same three. This led me to extract both sides, normalise whitespace, and compare by ids. The result across all 35 module/schema pairs:

sch-rules        1220
py-rules         1220
identical-xpath  1220
xpath-diff          0
only-in-sch         0
only-in-py          0

All matches! 22 further assertions sit in the 2016-17 and 2017-18 NVR schemas, which have no generated module of their own. This isn’t a bug, validation.py sets NVR_MODULE = nvr12stron, one year-independent module used for every year.

The generated Python contains no logic the .sch does not. It is a compilation target.

And the replacement runs

So let us try to compile the .sch to something newer. All 37 shipped .sch files compile under lxml’s isoschematron. All 37 modules execute end to end against a test document without error. None reports an assertion id that is not in its own schema. Only the 2026-27 FPS fires against an FPS document. The rest correctly match nothing. Namespace targeting works. Two stub FPS documents differing only in an IRheader key and a gender field:

bad.xml   valid=False  failures=7
   a_r1005          5004  At least one key must exist in the IRheader
   a_NILETTER.0     7849  If [GENDER] is 'M', [NILETTER] cannot equal 'B','E','I','T'
   ...5 more from the stub being incomplete

good.xml  valid=False  failures=5
   ...the same 5. Both targeted rules cleared.

The two rules I wanted to fire did, and the other five held, success! It resolves diagnostics too: r1005 carries HMRC error code 5004, NILETTER.0 7849. Those are the gateway’s own error numbers, recovered from files that ship in the box.

Why hasn’t HMRC done the migration?

Government? I don’t know?

What would force HMRC’s hand? - Rosetta

Apple announced the wind-down at WWDC 2025, in their own words:

Rosetta was designed to make the transition to Apple silicon easier, and we plan
to make it available for the next two major macOS releases - through macOS 27 -
as a general-purpose tool for Intel apps to help developers complete the
migration of their apps. Beyond this timeframe, we will keep a subset of Rosetta
functionality aimed at supporting older unmaintained gaming titles, that rely on
Intel-based frameworks.

Confirmed at WWDC 2026 with the announcement of macOS 27 “Golden Gate”.

BPT’s Mac build is x86_64 only. So Basic PAYE Tools stops running on Apple Silicon around autumn 2027. That is roughly fourteen months from now (gulp). The exception does not help. HMRC’s payroll application is not an unmaintained gaming title. Fixing it means an arm64 build. An arm64 build means an arm64 libpython. There is no arm64 CPython 2.7, so it means Python 3, so it means Django 1.6 to 4.2 or 5.2. The deadline is on the Mac build. It requires the full migration.

Is HMRC ready?

No

What is their current state?

BPT 26.1 shipped in May 2026, eleven months after Apple’s announcement. It contains:

Let’s put some ages on those to give some context. As of 22 August 2026:

The XML stack in the submission path predates the iPhone (lol). Django 1.6 has been superseded for 11 years 11 months — 1.7 landed September 2014. Every one of those twelve annual releases was a decision to ship it again.

There isn’t any indication of a start to a migration. Not a partial port, not a compatibility shim, not a branch. Nothing has started. The v26.0 to v26.1 diff says the same thing. Eleven changed files: a State Pension age rule, an NI letter check, a visibility flag, build stamps. No infrastructure. Their own README still lists supported macOS as 15 ("Sequoia") and 16 ("Tahoe"). Tahoe is macOS 26. That numbering was replaced at the same WWDC that announced the Rosetta wind-down. The document naming the dependency has not tracked the announcement. It names Rosetta 2 as the supported path for Apple Silicon, with no end-date caveat.

cd bytecode
APP="bptrti bptrti_submission bptrti_pct bpt_shared bpt_settings calculators
     calculator_sncp calculator_spbp calculator_spp mysite django_offline
     r26 r26_lib r16 r17 penutils"

# any sign of a Python 3 migration in HMRC's own 2,336 modules?
grep -rla 'six\.moves|import six' $APP | wc -l          # 0
for m in python_2_unicode_compatible sys.version_info PY2 PY3 __future__; do
  printf '%-30s %s\n' "$m" "$(grep -rla "$m" $APP | wc -l)"
done
# careful: a bare grep for "six" matches "six months" and "sixteen_years_back".
# Filter to real imports or the number is a false positive.

What they can actually do

• Full uplift (yippee)

Python 3, Django 5.2, drop South and 4Suite. Correct, and years of work against eleven cloned tax years. Fourteen months looks unlikely from a standing start.

• Build Python 2.7 for arm64 themselves (ball and chain)

Unsupported, but people have done it. The C extensions are the problem, and 4Suite has C extensions. Buys years, fixes nothing.

• Drop macOS (???)

Supported platforms are already only Windows, macOS and Ubuntu. Cheapest option, and it strands every Mac-based small employer.

• Move it server-side. (Just create another product at that point?)

Solves the architecture problem by deleting the desktop app. Which is a different product, and a much larger decision than a port.

Does Windows have the same problem?

No

Where does that leave us?

This whole project started because I needed BPT working for my own business without installing Rosetta. I’ve been able to do that by moving the core logic and frontend onto a Linux server and connecting to it over Tailscale; a resounding success in my book. The only upkeep on my behalf now is to update the program manually when necessary, a small price to pay.

The follow-up question, however — whether BPT will still be available on macOS 28 after general-purpose Rosetta is removed — is uncomfortable. The actual bytecode seems to be platform agnostic and doesn’t use anything specific to x86_64. However there have been no signs of a migration away from Python 2.7 and Django 1.6.

But looking at the broader picture, which I find more interesting, this isn’t a specific problem for HMRC or BPT. Rosetta’s wind-down is a single, well-telegraphed, four-year-notice end of life. Rosetta’s sunset will take out an enormous amount of otherwise working software, very much in the same vein as the PowerPC transition Apple did onto the x86 architecture in yesteryear. Almost no software in a typical consumer application is inherently x86_64-dependent. In the case of BPT, and applications more generally, it is due to a toolchain/bootstrap dependency that locks the application to an architecture, doubly so if development doesn’t migrate along with the version updates and EOLs of those toolchains.

Moving forward, the question to ask is: what would it take to build something in 2026 that is agnostic to instruction set from the ground up? And not agnostic in the aspirational sense, but in the sense that a target you never once thought about during initial development can be added later without a rewrite. Say RISC-V. There is a plausible world in five years where someone wants BPT, or your thing, running on a RISC-V box, and you would have had no business spending initial development effort on that target, no hardware to test it on, and no users asking. Would you be able to add the target as a build change rather than a rewrite?

This framing helps us understand what other contemporary choices people have made. Bun’s port from Zig to Rust ran in July 2026: 535,496 lines of Zig rewritten to over a million lines of Rust in eleven days, some 6,500 commits, up to 64 parallel Claude agents across four worktrees, and 5.9 billion uncached input tokens, 690 million output tokens and 72 billion cached input reads — about $165,000 at API pricing. The LLM-assisted port worked, in that the metrics the team cared about improved. However the benefit of this approach has to be weighed against the costs, and it doesn’t generalise to a reverse engineering situation like this one. The $165,000 wasn’t paid directly. Oven was acquired by Anthropic in December 2025, so the tokens were internal. It also required the two things that stranded software doesn’t have: an engineer who knew the entire codebase intimately, and an extremely robust test suite. HMRC may have both, but the public definitely has neither. There is no test suite bundled (nor should there be, really). There is no source (more debatable).

if None >= datetime.date(1960, 4, 6):     # should be `dob`
pensionable_date = None.date(2019, 3, 6)  # should be `datetime`
if dob <= dob:                            # nonsense

This leads us to the main idea I want to argue: an LLM-assisted solution is a last resort. It has been shown to be a legitimate option for one-off migrations (the Bun case), however it doesn’t seem to be a wise move to rely upon automated porting of binaries for the Rosetta sunset, with no test suite or source code available.

What might actually work is a little more boring. Software development should keep architecture surface front of mind in timelines when the package has a user base. Building an understanding of dependencies and of critical code infrastructure that is outsourced to other libraries is important, especially when their EOL is flagged. I think taking on the burden of absorbing maintenance for an EOL dependency is a road to nowhere, but it might be the only choice as a stop gap. If you want your community to be able to port your software in the future, source code and test suites are mandatory, however I understand that is inadvisable for most products that are commercial (BPT is free but still sensitive). And for sanity, attempt to build for many targets initially once scope allows, as it’s much easier to increase targets from two to three than from one to two.

None of this is new ground or was exotic in 2013 when BPT was seemingly built. It is just one of the many examples of how software and technology is as much a living product and service as it is something you ship once and call done. Finally, my own answer to “will BPT still work in fourteen months” is yes, but only because I moved it off the Mac myself. Every other Mac-based small employer should probably be looking for the exit already.

Versions, end-of-life dates, and where they would have to go

Links are inline throughout the notes above; this is the same set as one table. All checked and resolving on 2026-08-21. Release dates for the dead packages come from the PyPI JSON API, not memory.

Useful beyond the table:

• Django’s full deprecation timeline — the actual migration checklist
• Django 2.0 release notes — where Python 2 support was dropped
• Python version support status and the Python EOL table
• South’s documentation, still up, still describing a 2014 world
• Schematron itself

Formats and magic numbers

What the bytes in this document mean, and who says so.

The Rosetta deadline

• Installing and using Rosetta 2
• Apple Developer news, where deprecation notices are posted

Arm market share and Windows emulation

Vendor and analyst estimates, measuring different things over different periods. Directional, not precise.

HMRC’s own material

• Basic PAYE Tools
• RTI internet submissions support for software developers — where the schemas and error codes are published