Status updates and other server news for https://mecha.garden
Last night the Sharkey server inexplicably stopped receiving inbox requests, rejecting them with a 401 status code. I've investigated and for the life of me cannot find a reason for this, nor what the initial incident might have been.
Basically go outside or something, we'll be back soon I hope.
You may notice that mecha.garden is kind of slow today (read: occasionally unreachable). This is apparently due to network congestion at our VPS host, and is an issue they're aware of but there's no one available to work on right now. Hopefully whoever is hogging the network (reportedly 50MB/s in and out) finishes up whatever they're doing soon, but until then (or until the ops people at cyberia get off work) we're stuck with basically no bandwidth.
Oops ᕕ( ᐛ )ᕗ – E
It looks like our hosting provider is having issues, not sure at the moment when this will be resolved. They are working on it but depending on the root cause it could be a lengthy recovery. Rest assured the database is safe so if we need to jump to a different provider or something nothing will be lost except the existing job processing queues. I'll keep this post updated with info as I get it
UPDATE:
Disk issues have forced an early migration to a new server which is currently underway, still no ETA but the fix is in the works at least
UPDATE 2/10/24:
It looks like the migration is complete, everything should be up and running again. Sorry for the unexpected downtime, the new server should be more stable. (☞ ͡° ͜ʖ ͡°)☞
Well we're pretty far over the ~2 hour downtime I wanted, so here's a little status update.
The first two rounds of database migrations went off without issue, thought they took a bit longer than I had anticipated. I had to re-patch a change I had made to allow the instance to talk to the connection pool for the database which I had forgotten about. Fortunately Sharkey is much quicker to build than Firefish, and the image is considerably smaller as well, which is a nice bonus. Currently the last round of migrations is running, but there's a lot of changes that need to be made so it will probably be another hour minimum before we're back online.
TLDR: everything's going fine it's just slower than I'd guessed (￣┰￣*)
This weekend I am planning on migrating mecha.garden to a different Misskey fork, Sharkey. This is due in large part to the recent drought of activity on the Firefish git repo, and my personal dissatisfaction with the direction and pace of Firefish's development in general.
Because Firefish and Sharkey share a common ancestor the migration should go smoothly, hopefully only needed an hour or two of downtime. Currently I'm planning on that happening around noon EST on Saturday the 13th. The majority of the time will be spent running database migrations, and one that's done there shouldn't bee much else that needs to be done.
Sharkey is a soft fork of Misskey, compared to Firefish's hard fork of Misskey V12(I believe). It should have most of the features you like in Firefish, plus some of their own (user data export, listenbrainz integration 👀). There are some features that will be missing (get your quote-renotes in now), but there are also some superfluous features that have been cut down (did you know there's an NSFW-detecting machine learning model in Misskey?). All in all the user experience should be similar, with the benefit of all the development and optimization Misskey has received since Firefish forked off, hopefully making things run smoother (and perhaps a bit faster).
As always if things go south, you can check here for updates. Otherwise, see you on the flipside (☞ﾟヮﾟ)☞
- E
You might currently be noticing that mecha.garden is down. Rest assured it is indeed down and it is indeed my fault sort of.
A confluence of circumstances involving SSL certificate renewal and hosting provider server migrations has rendered the postgres server kaput due to an issue currently out of my control. I don't really have an ETA right now on when it will be resolved, so check back here regularly as when I find anything out I will edit this post :(
EDIT: Dec 3, 2023
We're back in action. The root cause was that I restarted the postgres box after renewing the SSL cert, but the VPS wasn't able to restart because of an issue the provider (Capsul) was/is experiencing with IP address provisioning relating to a hardware migration. The issue had to be resolved manually by an admin so it took a while before anyone could get to it. Really this was just a perfect storm that resulted in a relatively lengthy downtime, and I don't think there was really anything I could have done differently other than simply not needing to renew a cert at the same time that Capsul was experiencing networking issues.
If you're reading this then that means I've successfully configured the Garden's brand new Writefreely powered blog :)
This will be used for status updates, general server news, moderation decisions, and whatever else. Writefreely is federated so you should be able to follow this blog at @info@blog.mecha.garden.