Spyke

Syndicated from the fediverse. Read and engage on the original instance.

View original on kbin.earth
kbinEarth·kbin.earth metabyjwr1

Addressing kbin.earth stability issues

As I'm sure y'all have noticed, the server has been having stability issues the past week. Every once in a while, the server seems to "crash"; it will time out on all network requests for about 5 minutes, happening several times within an hour. Additionally, there have been several times over the past week when the server queues got backed up, resulting in incoming and outgoing federation traffic being delayed.

At first, I assumed the server hardware had simply become insufficient to handle all of the network and federation traffic we have been receiving, potentially due to a DDOS attack. I was honestly starting to think I would need to upgrade the server hardware, which is ridiculous, as the VPS is already quite powerful and is certainly expensive enough.

Luckily, @green_copper helped with debugging and noticed the server timeouts only occur when the microblog section is viewed (and sometimes when the combined section is viewed). So rather than a lack of hardware being the issue, I believe it actually has to do with the recent Mbin upgrade. My guess is that an SQL query was changed in one of the commits between Mbin v1.9.1 and v1.10.0, which resulted in a widely inefficient query, causing the server to freeze up and time out on requests.

I have and will be investigating the issue this week. I will start by analyzing which database queries are causing the timeouts, but if needed, roll back to Mbin v1.9.1 and try to figure out which specific Mbin commit introduced the bug. Additionally, I may have access restricted only to signed-in users while I'm working on this.

In the meantime, if you guys can refrain from viewing microblogs (or combined view), that would keep the server accessible for everyone and help me work on it faster, as I can't do anything while the server is frozen. If you want any realtime updates on the server issue (or anything else related to kbin.earth), feel free to join the Matrix room.

Thanks for sticking around and for your patience!

View original on kbin.earth
34
Comments17

17 replies

green_copper
kbin.earth

Thanks for sticking around and for your patience!

And thank you for dedicating your time and money providing us with a nice corner of the internet ^^

15
Skua
kbin.earth

While attempting to find where to change my default home page view off of the combined view, I mis-clicked and pressed the home button and then the server crashed. So I apologise for being a moron. Thank you for all your work with the server

6
celeste
kbin.earth

Went in and switched my default view off combined. Good luck! This sounds like a complicated issue, but I hope it actually turns out to be simple

5
BentiGorlichreply
gehirneimer.de

that was actually mostly a pg-bouncer. I have no idea why it is happening though...

I just have a script that checks for this line and restarts php-fpm if necessary:

#!/bin/bash

lineCount=`tail -n 1 /var/log/postgresql/pgbouncer.log | grep query_wait_timeout | wc -l`

if [ $lineCount -gt 0 ]; then
  systemctl restart php8.3-fpm.service
  logger "Restarting php because pgbouncer has query_wait_timeout messages"
else
  echo "No"
fi
2
Maeve
kbin.earth

The prime24/nordpublications jackasses have switched tactics and are creating accounts still without posting their vacuous rubbish. Watch new accounts, in not saying it's them but they're highly suspect, imo.

3
jwr1reply
kbin.earth

Yup, go for it! The issue should be resolved for now.

1
Maeve
kbin.earth

Just wondering if you've considered Anubis rather than cloudflare?

1
jwr1reply
kbin.earth

I have considered it in the past, but there are a number of reasons I am still using Cloudflare.

2

You reached the end

Addressing kbin.earth stability issues | Spyke