Media on pl.m0e.space and other stuff is gone.

How I discovered this (funny story)

Some quite time ago someone sent me a message about minecraft server not working. Pelican (previously Pterodactyl) panel with its “docker-runner” software was down for some reason. I saw an issue with db, but I was too distracted by the work, so I forgot about it for some time. Secondly, one of our users sent me message with media uploading issue, I tried to fix it, but self-hosted Minio S3 was returning Access Denied all the time. I was too distracted by the work, so I planned to do it after my work is done (unfortunately I forgot a quite about it and it was night, so yeah).

Today I tried to fix Minio, but it kept sending me errors like “Permission denied”. I tried to chmod the directory and that’s the moment, where I spotted the issue - filesystem was in “read-mode”. I tried to check btrfs logs and it said something like 20k+ write errors and 600+ read errors and “forced readonly”. Tried to check mdadm - everything is good, but when I tried to check SMART on every disk, that’s what happened - /dev/nvme3n1 keeps spamming Input/Output error. After reboot the disk is gone, raid was RAID0, so it’s unrecoverable.

What happened and what happens next?

I still don’t know if it’s a disk or the nvme x4 controller (small chance, but whatever), but if it is disk failure - congratulations, we’ve lost at least 1 year of media in pl.m0e.space (maybe I could find old backups when I migrated from GMhost or from docker, when it was in local-mode, but it was a long time ago), pelican panel with minecraft (i still have backups, but for some reason I don’t have the newest backups for current minecraft server), wiki server and it’s data about our minecraft server and maybe some other stuff that I don’t remember.

If it recovers somehow - we can restore all of this, but that’s a very small chance, unfortunately.

Anyway I’ll restore Minio S3 on HDD and make replica or smth like that (remotely or locally - we’ll see, I’ll probably won’t sleep today)

Conclusions

DO THE FUCKING BACKUPS, don’t wait tomorrow or weekends - DO IT NOW. Now I’ll spend some cash to make at least 2 backup storages for all of the servers + increase storage size for hogyoku (Git server, Status page and XMPP)

Also i fucking hate myself