Server migration post

1 files changed, 205 insertions, 0 deletions

diff --git a/_posts/2026-06-07-migration-time-again.md b/_posts/2026-06-07-migration-time-again.md
new file mode 100644
index 0000000..3d93622
--- /dev/null
+++ b/_posts/2026-06-07-migration-time-again.md
@@ -0,0 +1,205 @@
+---
+layout: post
+title: Server migration time (again)
+lang: en
+categories: tech
+date: 2026-06-07 13:44 +0200
+---
+
+It's a long weekend in Germany (for me) again. Thursday was the "Feast of Corpus
+Christi" (sarcastically translated as "Happy Cadaver Day" sometimes), and I took
+Friday as a vacation day. Plenty of time for "projects". This is an attempt to
+write up a digest of the thoughts I… let's say spit out on
+[Mastodon](https://woof.tech/@uvok) over the previous days.
+
+## HomeAssistant
+
+> It actually began a bit earlier. I bought a used car back in April. My old car
+> was threatening to get some expensive repairs. And with the gas prices going
+> up, I said "fuck it" and took a look the the various portals for used cars. I
+> might've panicked/overreacted a bit, and decided for a Seat Mii Electric
+> shortly before my Brussels vacation (on which I also wanted to write a short
+> blog article as well, some time), as in, getting fear of all the used cars
+> being sold very quickly and me being left with nothing.  In retrospect, there
+> is indeed a very high demand for used electric cars, so it was probably a good
+> decision.  Anyway, I couldn't be happier. I never had *so much fun* driving
+> before!
+>
+> Anyway, this is one of these cars which already have an eSIM built-in. It's
+> disabled by default, but if you activate it (via customer support), you can
+> use the SEAT Connect app and see various status reports, and get telemetry
+> data (like how much you drove, how much energy you used). In a furry EV chat,
+> some furries even had their cars integrated into HomeAssistant.
+
+This reminded me… I do have an HomeAssistant, which was still running HA
+2024.something. Since it only runs in my own network, I don't see a problem
+with that, actually. I never really updated because I keep on reading about
+breakages by upgrades. My setup basically is running this in Proxmox:
+
+- 1 VM with Zigbee2MQTT (Z2M)
+- 1 VM with HA OS (including the MQTT server)
+
+With so much time on my hands, I decided "just clone the existing VM, import the
+new HA OS disk into the new VM, backup and restore, and see if it works. If so,
+I'll use it, otherwise I'll revert.". Yeah, only no. That didn't work out that
+easily. I ended up with unavailable values in the dashboard of the new
+installation.
+
+Since the MQTT server runs inside HA, Z2M will only connect to the broker of the
+old installation. So I had to push an item onto the mental stack, "Create an LXC
+container running Mosquitto". So I did that.
+
+Only, wait. The default Proxmox LXC Alpine template is stupid. It doesn't have
+Dropbear installed. So inserting the SSH key in the container config wizard
+*doesn't do anything*. So I looked up Distrobuilder to create an LXC template
+which does have Dropbear installed -- and enabled -- by default.
+
+Only, then I had to remember how I set up my internal certificates, since I
+wanted to secure the broker with TLS.  Thus, I had to (stack.push) set up
+uacme/ualpn first on that container first.  Which put some "mental load" on me
+trying to keep in mind the end goal.
+
+Anyway, I got this sorted out in the end, and now I have an up-to-date HA
+running, only… uhh… I think my Z2M is still not up to date, because I would need
+to setup a newer node version first.
+
+Why did I upgrade HA OS again… oh right, I was poked to integrate my SEAT car
+into it. Only, Volkswagen[^1] decided to change/shutdown their API, or
+something. This seems to be a long-going battle, which started with VW simply
+changing an OAuth endpoint, but in the meantime they probably changed more. At
+least I got basic info on my desktop PC using
+[CarConnectivity](https://github.com/tillsteinbach/CarConnectivity-connector-seatcupra)
+
+[^1]: A Seat Mii is basically a Volkswagen e-up
+
+At this point, I realized I still have NetBox running, and should probably
+document the VMs there as well. *sighs*
+
+## Uptime Kuma
+
+Apropos of Node.
+For "monitoring", I use [Uptime Kuma](https://uptime.kuma.pet/) (UK). I haven't
+upgraded that in some time as well, and they actually had a major upgrade. After
+figuring out which repo I need for getting an appropriate node version[^2]<sup>,
+</sup>[^3], I was able to update UK successfully.
+
+[^2]: Apparently, NodeSource is the way to go?
+[^3]: Maybe I should just run it in Podman… But eh, Proxmox isn't naturally
+      suited for that.
+
+During that, I actually decided to approach another problem I had again and
+again, my local network certificates (step-ca) expiring, because I made mistakes
+in my automations[^4]. So I created an HTTPS monitor, only to figure out that UK
+only notifies of expiries via notifications, but doesn't show that in the
+dashboard.  Oh well, I can live with that.
+
+[^4]: Specifically, the certificate always fails to update after every OpenWRT
+      upgrade. Either I forgot to create the directories in the image builder,
+      or I forgot to set `chmod +x` in some script, or I forgot to include the
+      step-ca root into the trusted certificates (which is always ugly, because
+      every distribution does it differently, and it depends whether you have
+      OpenSSL installed, which brings `update-ca-certificates`. Only, on OWRT, I
+      don't want to install that, so apparently you have to append your cert to
+      the certificate store/bundle manually.
+
+Another problem I encountered was that my UK runs in a "VM VLAN", and I
+configured my router (running OpenWRT) to not allow incoming (to the router)
+connections from that VLAN. So UK can't directly query the certificate expiry
+that way.  I ended up with a manual/push monitor, where the notifier script runs
+on Proxmox itself. Ergh. I find it ugly, but it works.
+
+## Blog migration!
+
+For quite some time now, I wanted all my uvok.de / uvokchee.de services running
+on the same server. But… dependencies! Take this blog for example. It's a Jekyll
+site. Build via Buildbot. From a git repo hosted with gitolite. And it also
+runs Hatsu, for making my blog ActivityPub-connectable. You see where this is
+going. This is how it went:
+
+- So, I started with the gitolite repos. Luckily, they provide [helpful
+  documentation](https://gitolite.com/gitolite/install.html#moving-servers) for
+  exactly that. Nice! That went relatively smoothly.
+- Aside: It's really annoying you can't just do a `su - user` anymore to login
+  as a different user, if you expect to have systemctl working. No, you have to
+  do a `machinectl shell user@ /bin/bash`.
+- Then I set up buildbot again. Only I decided I might as well build the blog
+  inside Podman.
+  * No more worries about system ruby version. No more installing the same gems
+    every time. This is gonna go great.
+  * "This should be simple." Just write a
+    [Containerfile](https://git.uvok.de/blog/tree/_ci/Containerfile) with the
+    needed gems installed and run Podman for building the Jekyll site.
+  * I actually started with creating the image on my local machine, pushing it
+    to the Hetzner server, only to find out it doesn't work. Well, yes, amd64
+    binaries don't run on an arm64 machine. *facepalm*
+  * Wait, why does Podman emit warnings?
+    I was getting messages about Podman not being able to connect to the user
+    session, and falling back to cgroupfs.  Do I need `enable-linger` for
+    buildbot? Do I need to enable the Podman socket? No, that didn't change
+    anything, either.
+  * Using `podman --remote` in the build step didn't work at all. I got a
+    permission denied.
+  * What's a `DockerLatentWorker`, buildbot? Oh, you need buildbot-worker
+    installed into the container for that. Not what I want.
+  * Oh, I had `PrivateTmp` and `PrivateDevices` specified in the service file.
+    Apparently this lead to the permission problems.
+  * Oh, and apparently, it's easier (and more secure?) to just use a user unit,
+    instead using a system unit with `User=` and `Group=`. So I moved that
+    around, and removed all the sandboxing directives from the unit file[^7].
+  * And, for completeness: No, I won't blame systemd here, I doubt it would have
+    gone better under SysVinit.
+- Uh, so… where was I again? Oh yeah, building the blog.
+  So, a forced buildbot build (so, manual action) works now. I'll worry about
+  the push hooks later. [^5]
+- Next up, the webserver.
+  I thought, while I'm at it, I might as well get rid of Certbot, and
+  let Caddy handle the certificates automatically. While still running Nginx in
+  the backend, because apparently, you're more likely to find DokuWiki and
+  Nextcloud snippets for Nginx than for Caddy.
+  - At this point it occurred to me, "Oh, I still have Authelia set up." I
+    ended up migrating that to Caddy completely. Luckily, the Authelia docs have
+    snippets for that.
+- So, of course, to go live with Caddy, as in, getting the TLS certificates, I
+  need to point the domains to the new server. So, a DNS update is in order as
+  well.
+- I ended up migrating Hatsu[^6] as well, and update it while I'm at it. Hatsu
+  is written in Rust.
+  * You know what, I'll use Podman for that as well! So I don't need to install
+    the rust/cargo toolchain on the system.
+  * *types `podman run ... cargo build`*
+  * … Hey, why doesn't my VPS react anymore? Shit.
+  * *logs into the Hetzner console and reboots/shutdowns the VPS*
+  * *types `podman run -m 2g --cpus 1 `*
+  * Ah, now it compiles without crashing my server!
+
+[^5]: That was always a pain point in the past. I used the scripts in the
+      examples of the buildbot repo, which depend on twisted, so I had to make
+      sure the required modules were installed inside whatever happens in
+      gitolite's update hook. Very ugly. Maybe I'll look up a simple curl call.
+
+[^6]: The service offering ActivityPub integration.
+
+[^7]: I really wish there was a built-in unit generator in systemd, I always end
+      up having to do a web search for a template.
+
+So, after several hours of work, wanting to bang my head against the wall, and
+head scratching later: YAY! My blog is now on the other server! And Hatsu works as well!
+
+I still have "get my Funkwhale running again" on my maybe-todo-list. To be
+honest, I'm not sure if it's worth the effort. I uploaded some guitar pieces I
+played there, but… my heart is not really in it.
+
+I might look into what else I can run in Podman in my homelab, but I am not
+convinced running it directly on the Proxmox host is a good idea. Especially
+with all the networking stuff. I'm glad I got the bridging with VLAN figured out
+in the first place.
+At the same time, the machine might be too weak to take another VM on which I
+can run it. It's a "just-for-fun" project anyway. It's not like I run anything
+mission-critical on that.
+
+Oh, and I still need to migrate the Gemini version of my blog! Not sure if I can
+and should do it this weekend as well. Weekend is for relaxation, after all. :)
+I ended up philosophizing about how dissatisfying "administration stuff" is for
+me, because at the end of the day, you don't *see* the result of your work, as
+in, you don't have a physical artifact.
+