Spyke
announcements·AnnouncementsbyDemigodrick

Hexbear has been temporarily defederated - UPDATED now federated

Original text below this.

Following release of lemmy 0.19.18 we've refederated as this release should fix the bug that caused the issue.

TL;DR

We’ve temporarily defederated from Hexbear due to a Lemmy bug with very deeply nested comment threads.

A thread there triggered repeated crashes on our server, causing errors like 502 pages and “Lemmy is starting” messages. Defederating stops the issue for now.


Announcement

Due to technical issues, we’ve temporarily defederated from Hexbear until a Lemmy update is available that fixes issues with deeply nested comment chains.

There is a known bug in Lemmy (see: https://github.com/LemmyNet/lemmy/issues/6435 ) where very deeply nested comments can trigger excessive recursion during federation. When Lemmy processes these comments, it recursively fetches and verifies parent comments, which can eventually lead to stack overflows.

Under normal circumstances this happens rarely (we’ve been seeing it maybe once per day), but it becomes much more problematic when multiple new comments are added to an already deeply nested thread. Each new activity can trigger processing of the same deep chain again.

In this case, a thread on Hexbear received a large number of additional replies in a very deep comment chain.

This caused Lemmy to repeatedly process that chain, leading to stack overflows, federation worker exhaustion and timeouts. Simply put, parts of the server were crashing, too many tasks piled up at once, and requests started timing out and failing to load

You may have see this on the website with 502 errors or the lemmy error screen, and on apps it may have presented you with API timeout errors or "Lemmy is starting" errors.

For a visual representation, this graph shows the memory drop each time the server restarts:

The flat bit to the left is good, everything is fine. The choppy bit to the right, not so good, everything is not fine.

Usually its a one-off comment causing this crash, however in this case the user spent a good portion of time bumping the thread, and we had to process each one of those, each causing a crash, restarting the server, and then crashing on the next in the queue, and so on.

I did try removing the offending community from Lemmy.zip to prevent this from happening (It's quite common behavior in that community to bump threads I think), however we still process all the activities from that community - the only certain fix for now is to defederate until a version of lemmy is released that fixes this.

The graph is back to improving now:

Hope that all makes sense!

Demigodrick

View original on lemmy.zip
lemmy.world

So... Hexbear circlejerked so hard it theatrened the fabric of the fediverse?

Called it

44

Thank you for the transparency!

I hope this will be fixed quickly. While people from hexbear often annoy me - that is, if I even understand what they’re trying to say - I love that .zip only defederates very rarely. It’s why I came here after lemm.ee went down.

Lemmy.zip: come for the federation policy, stay for the transparent communication.

40
db0
lemmy.dbzer0.com

Thanks so much for doing the legwork on this. I was going nuts trying to figure out where seemingly random downtimes were coming from. It felt like a DOS and this cause explains why.

Out of curiosity, how did you trace this root cause?

32
lemmy.zip

I noticed in the logs before every timeout there were lots of "verify" words appearing, and in each iteration of that statement there were more and more verify words. Honestly had no idea what it meant at the point, only that I didn't recognise it from looking at lemmy logs previously, it always appeared before a crash, and it felt suspicious.

Here's an example from some logs before a crash:

2026-03-15T21:47:22.670586Z  INFO HTTP request{http.method=POST http.scheme="https" http.host=lemmy.zip http.target=/inbox otel.kind="server" request_id=2cc6dc65-571d-4a69-9733-5e80e455c00b}:receive:community:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify: activitypub_federation::fetch: 
Fetching remote object https://hexbear.net/comment/7004776

thread 'actix-server worker 18' has overflowed its stack

fatal runtime error: stack overflow

I pinged some logs over to Nutomic on matrix, who thought it might have been related to nested comments, and then I noticed Dessalines had made the linked thread, which matched pretty much with what I was seeing behaviour and logs-wise.

Usefully the logs link the object it's fetching, and 9 times out of 10 its a deeply nested hexbear thread! Or someone from another instance commenting on a nested hexbear thread. Nutomic confirmed the behaviour based on the logs in the issue, and I'm pulling the logs when I get chance to see what other threads are causing it to crash, although hopefully the fix will make it's way into 0.19.18 beta 3 so I can stop worrying about it!

28
db0reply
lemmy.dbzer0.com

Did you also see the db cpu spiking during this period?

5

No, no meaningful cpu spikes I could make out anywhere, although admittedly I was focusing on the lemmy server container mostly

3

Thanks boss. Very happy we have you to look after the behind-the-scenes adventures.

29

tfw ur meaningless internet debate goes so deep it crashes a instance.

20

Its amusing and tracks that hexbear is triggering a bug from chains being too long.

17
lemmy.ml

Not the first time the posting power of the hexbears has brought down servers

13

Ahh that explains the behavior I was seeing earlier. Much appreciated!

7

Hexbear embodying the banner of its Slop community:

3

Is this why .ml has been slow as fuck lately? I thought it was getting DDoSed or something lol

1

you are free to block any instance you want in your personal settings. Usually lemmy.zip and piefed.zip are not known to block a lot instances, there were discussions about that in the past.

22
Arelinreply
lemmy.zip

It's one of the few cool instances so I'd rather not

4

While I disagree with your opinion of Hexbear, I do hope federation comes back for you to enjoy the community you like.

1
KairuBytereply
lemmy.dbzer0.com

Also one to jump to conclusions and from what I’ve seen, their mods will do what they want if they think they know better than their admins.

Granted it’s anecdotal, since I blocked them after my ban, but the fact that it happened and never got addressed says all I personally need to know.

1
KairuBytereply
lemmy.dbzer0.com

I got an instance ban: “Stepping in for admins”. The “pedo apologia” was for explaining what Yandre meant.

1

Oh cool, yeah that’s the post where one of their admins said I was in the proverbial clear after I explained the situation, right before the mod that made that post decided they knew better than the instance admin and instance banned me.

Fun times, thanks for taking the time to search for it.

1

We should really give this a while. Take a few months and make sure the fix is set before we bring them back in. Maybe even a full year.

-11
lemmy.today

Why do you even care? You're on Lemmy.world which already defederated.

37
lemmy.world

So, just because I'm unaffected I should be OK with it continuing?

Edit: I'm so glad that this instanced community is brought together by the fact that no one is part of the same community. Maybe you should block Lemmy.world if you really don't like us taking part in conversations regarding the state of Lemmy.

-41
ramble81reply
lemmy.zip

You’re on .world, your opinion doesn’t matter. I’m native here on .zip and one of the reasons I am here is because the admins don’t defederate unless it’s a technical reason like this. They leave me to act like an adult and choose which instances I want to block or not.

Enjoy your walled garden of babysitting by the admins.

34

I’m native here on .zip and one of the reasons I am here is because the admins don’t defederate unless it’s a technical reason like this. They leave me to act like an adult and choose which instances I want to block or not.

Exactly the same here. Thank you @[email protected]!

4

Except this conversation is NOT about "the state of Lemmy", it's a comm specifically about this instance.

And no, butting your head in where it clearly doesn't belong is not appreciated.

24

I think they meant to imply that your opinion in the matter doesn't matter since you're not affected AND trying to influence other people's experience...

21

It's the myth of consensual sex federation meme, but it's hexbear, lemmy.zip saying "I consent" and this goober saying "I don't"

13

You reached the end