[RESOLVED] lemmy.ca sends every activity out 3 times?
Hi. Just looking at what lemmy.ca sends my server:
From nginx log:
your.ip.address - - [20/Feb/2024:03:48:47 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://lemmy.ca/"
your.ip.address - - [20/Feb/2024:03:48:49 +0000] "POST /inbox HTTP/1.1" 400 0 "-" "Lemmy/0.19.3; +https://lemmy.ca/"
your.ip.address - - [20/Feb/2024:03:48:57 +0000] "POST /inbox HTTP/1.1" 400 0 "-" "Lemmy/0.19.3; +https://lemmy.ca/"
Dumping the JSONs reveals 3 duplicate files, all containing:
"id": "https://lemmy.ca/activities/create/0968bf93-ac05-4178-bf3f-28cdc65fc338"
"actor": "https://lemmy.ca/u/HairyOldCoot"
"object": {
"id": "https://lemmy.ca/comment/7454502"
"content": "That was an episode of Sliders."
"published": "2024-02-20T03:48:45.810068Z"
These are 3 Creates, not 1 Create and 2 Updates or anything. I've chosen a comment 'cos that's public info, but it's the same for all other activity too (Votes, Deletes, etc).
lemmy.ca isn't alone in this. endlesstalk.org sends everything twice (info posted here), as does lemmy.sdf.org and mander.xyz, but lemmy.ca is the only one I've seen send stuff 3 times. Other instances (e.g. aussie.zone, feddit.nl, lemm.ee, lemmings.world, lemmy.world, programming.dev, reddthat.com, sh.itjust.works, slrpnk.net) don't, but there's nothing obvious like lemmy version numbers to indicate the difference.
Whilst I'd prefer you didn't send me stuff 3 times, I'm mostly mentioning it because I doubt it's good for your own resources to send every activity by every lemmy.ca user out multiple times. Also, the OPs of my community are from Mastodon, so I imagine you're sending them the same message 3 times too ...
9 replies
Hi, /u/Shadow recently modified our configuration by adding additional containers (3 lemmy, 3 lemmy-fed, 3 lemmy-ui) so that we can restart each of them without causing an outage.
https://lemmyverse.link/lemmy.ca/post/15548660
An issue was also discovered yesterday (9 hours ago approx) where all three federation instances were using index=1 in the docker-compose. Maybe the issue is already fixed, according to the timestamps you have vs when we applied some changes?
Thanks. This seems a likely answer - running 1,2 or 3 containers would explain the differences across instances. I hope it can be resolved though.
This should have been fixed yesterday before your post, although it was taking some time to catch up. Are you still seeing this behavior from us?
Thanks for posting this!
I've turned logging back on, so I'll let you know once someone from lemmy.ca interacts with ![email protected]
Thanks, I just sent a subscribe
I need to dig into the lemmy code, but I think the federation index / count system may be broken.Oh, well I only got one copy each of your Subscribe / Unsubscribe / Subscribe actions, so that's good! Many thanks.
Awesome, glad to hear it. Thanks again for letting us know about the problem.
Thank you for the heads up... In my case, I am running a single lemmy docker container for mander.xyz, so the reason for the duplicates is not yet clear to me. I will investigate.
Did you find out anything unusual that might be causing the duplicate activities? We have an issue for this now so we can hopefully resolve it.