My Immich server threw a wall of 5xx errors out of nowhere, so I did what any self-hosted homelabber would do: SSH into the box and start digging. First stop was PostgreSQL, since Immich leans on it hard for pretty much everything. Running `docker compose ps` inside `/opt/immich` showed the database container stuck in a crash loop — restarting and dying within seconds, which explained why the rest of the stack looked "healthy" while search and uploads were completely broken.
To figure out *why* Postgres kept dying, I pulled the last 100 lines with `docker compose logs database --tail 100`. The logs pointed straight at disk space, so I confirmed it with a quick `df -h`. Sure enough, the root filesystem was sitting at 94G out of 98G used — zero available. A full disk means Postgres can't write to its data directory, and that's exactly what was tanking the container on every restart.
Since this is just my lab server, I didn't need to reach for anything fancy like resizing a volume. I remembered a couple of old VMs sitting around unused — a Linux Mint box and a Windows 11 install — that were prime candidates for cleanup. After checking with `virsh domblklist` to find the actual qcow2 file paths (since `virsh list` doesn't show sizes or paths), I confirmed both VMs were shut off and safe to remove.
Deleting them wasn't totally painless — Linux Mint went down clean with `virsh undefine`, but Windows 11 threw an error about not being able to undefine a domain with attached NVRAM, since it was a UEFI VM. The fix was adding the `--nvram` flag to force the undefine, then removing the qcow2 disk image and the leftover Windows ISO sitting in Downloads for good measure.
After all that cleanup, root jumped from 0 available to 44G free, and Postgres flipped from "restarting" to "healthy" almost immediately. A quick search for "cat" in the Immich UI confirmed everything was back online.
Lesson learned: keep an eye on disk usage on your servers. If I run out of space again, it's probably time to actually upgrade the drive instead of playing VM Jenga.
I've also put together a full walkthrough video covering every command and log output shown here, so if you want to watch the whole troubleshooting process step by step, check that out alongside this post.
