I self host! Very nice having an instance all to myself.

I wonder if these are real users or if someone wrote a script to register users via the lemmy API… 🤔

Have you tried enabling the slow query logs @[email protected]? I went through that exercise yesterday to try to find the root cause but my instance doesn’t have enough load to reproduce the conditions, and my day job prevents me from devoting much time to writing a load test to simulate the load.

I did see several queries taking longer than 500ms (up to 2000ms) but they did not appear related to saving posts or comments.

Oh, Big-O notation? I never thought I’d see someone else mention big O notation out in the wild!

:high-five:

Welcome here, and thanks for creating the CR community.

Is it Thursday yet?

OMG! You are right. It’s my time to shine!

Note: this seems like it has something to do with the database, and something getting royally messed up post upgrade.

After trying all sorts of network hacks and updates, I eventually just decided to backup my Postgres container, and nuke it.

With a fresh Postgres DB running along with 0.18.0, my self hosted site is back online. Of course, my local post history and all my subs are gone... but at least my site is operational again.

I'd advise anyone self-hosting to not upgrade to 0.18.0 yet.

Yes. Absolutely does happen on other instances that have thousands of users.

That actually sounds like something I would have enjoyed. I joined Reddit around the time it started taking over, I think.

I really hope someone is doing some level of performance testing on those changes to make sure the changes fix the performance issues.

That’s pretty neat! I’ve honestly never seen it mentioned on Reddit before, so got a bit excited to see someone mention it here, admittedly maybe too excited.

The footer still shows the old versions, by the way. It does feel snappier than earlier for sure.

I’m setting up my own instance to mess around with as well. I’ve got it running via Docker. The SSL setup was a pain, not really documented well. Also pictrs is giving me issues complaining about DB access.

The redirect is tricky and works in some browsers, but not others.

Edit: save took about 10 seconds, which is better than 20-30 seconds!

Edit 2: the footer shows the updated versions, now.

Actually the approval email was in my spam folder. My bad!

Thanks for looking into the save issue. I’m looking into setting up my own instance now on a digital ocean droplet maybe, so I can maybe debug a few things on my own.

Would be funny if it was missing an index and doing a full table scan for some odd reason...

I focus on application performance in my day-to-day work, and missing indexes, greedy upoptimized queries, etc, are the root of a lot of issues. Hopefully you can get to the bottom of it.

Quick note: I'm not seeing a big delay (10+ seconds) when posting or saving on lemmy.ml, or my own instance.

I suppose I could, but, I've honestly spent the majority of today on lemmy answering "support" questions for people lol... Maybe I can try to take a look tomorrow. 🤷

Actually, saving edits on lemmy.ml is also slow, about 4-5 seconds. It’s probably a combination of user load and non-optimized queries.

I did see it, thanks. I’m hoping to find some time to contribute this week.

Too slow! 😀

That sounds correct to me.