We are going to do some maintenance on Reddthat.

We are going to upgrade:

• Lemmy to v0.19.20
• Postgres to v18
• Frontends if they need it
• And performing some optimisation on the Database while we are at it to try and help with the random long loads/failures we are having.

All of this was tested and it will should take around 15 minutes for the updates. Once the updates are done, we will be back and then while we are online we will be performing the database optimisations.

We will be online after the updates.

World Clock link to Translate 09:00 to your local timezone:

• https://www.worldtimebuddy.com/?qm=1&lid=100%2C2147714%2C2867714%2C2643743%2C5128581%2C5368361&h=100&date=2026-10-23&sln=9-10&hf=0

Our standard Status Page for updates incase everything goes pear shaped:

https://status.reddthat.com/

Due to the PHP Exploit (see https://www.php.net/archive/2026.php#2026-10-20-1 or any tech news site) we are unable to send emails for the time being.

The mail software we were using seems to have been using PHP for part of its stack. And out of an abundance of caution I've taken it offline.

Probably back in 24 hours. (Assuming its had a release by then)

Stay safe out there.

Edit: Our Mail server is back online

Bad day to be us...

https://jackalsyn.com/posts/php-up-2026.html

Apparently created a PoC just from the video of the original PoC. But PHP and others have said there are zero mitigations. So doesn't matter what you do... You're fucked.

Today I made a new Template for our Go-Away anti-bot protection and am up-streaming the solution so that all of lemmy admins can have a drop in replacement.

When I was creating it I made it for us and by default it looks like this: (The Reddthat one has our icon instead ofcourse)

I named the template file reddthat-challenge.gohtml and when I was making it more 'generic' I renamed it from reddthat and typed lemmy...

Turns out I typed lemmmy instead! It's technically valid, ansible-lint and ansible-playbook --syntax-check both pass without issue! Our test PR deployment also was successful because like all of the other items, everything was correct and the one thing it doesnt do is issue a 'reload' or a dry-run. Even then when issuing a dry-run it says the container will be recreated, which is an expected outcome as I made a change to the go-away proxy, so I expect it to restart...

--- a/templates/docker-compose.yml
+++ b/templates/docker-compose.yml
@@ -170,7 +170,7 @@ services:
     volumes:
       - "goaway_cache_main:/cache"
       - "./policy.yml:/policy.yml:ro"
-      - "./lemmmy-challenge.gohtml:/lemmy-challenge.gohtml:ro"
+      - "./lemmy-challenge.gohtml:/lemmy-challenge.gohtml:ro"
     environment:
       GOAWAY_CLIENT_IP_HEADER: "X-Real-Ip"

Our monitoring was/is were all returning success?!?

This is a screenshot from https://status.reddthat.com/:

(This service is Betterstack)

It was setup to only alert when "the url becomes unavailable" (which is any code outside than 2XX). Previously Cloudflare used to pass through the error code? As this has always alerted me whenever the backend failed?)

Here is a memory graph you can see the red 'swap' be basically static until I tested the template and then the swap came back up again and then at 10:53 (00:53 UTC) it was restarted and never came back!

(This service is hetrixtools)

And here is my own on-site monitoring service showing all green!

(This is uptime-kuma. (Note: This goes directly to the box and not via cloudflare))

I can probably dig deeper into the logs but as I went out to lunch and my commit at 10:52 coincides with the exact 'increase'/lack of increase seen above we can be certain it's related!

A 3 hour outage and I didn't know about it because all 3 uptime monitoring solutions all agreed we were online!

Resolution:

I'm so sorry!

Turns out uptimekuma which is my fallback for the online serivces was setup to monitor the PORT only directly on the box. So as our webserver was still running as far as it was concerned, it was technically "online".

• I've added a few more checks to uptime-kuma so instead of just being a port check, there is a HTTPS check with keyword. So the page has to load, and the keyword needs to be on the page otherwise I'll get notified.
• I've updated Betterstack to check only a 200 HTTP code. Which is what Hetrixtools was already set too.
• I've updated Hetrixtools to also check for "Reddthat" in the page.
• Hetrixtools also can monitor the docker service, so if the docker service goes down I get a message now.
• I'm thinking about adding a way to punch into the box from uptimekuma too and monitor the docker container itself, as Hetrixtools doesnt seem to do that, but that might be overkill? Or not? This will probably be the only way in which I can tell?

Even after 15 years (oof now I feel old) doing "IT" I am still amazing at how no matter how many systems you put in place to try and catch issues, assumptions and unfortunate circumstances can rear its ugly head.

Our automated testing all passed.
Our external monitoring solutions passed.
Our internal monitoring solutions passed.
Our syntax checker all passed. Our service still went down.

Cheers,

Happy May!

Tiff

Also happy June? What was I thinking...

The above is probably enough of an update. Things are mostly normal. Lemmy devs are nearing 1.0 release, which will probably mean a big push across the internet and more people joining. (Hopefully picking Reddthat :P ).

I've been struggling to accept all the new people in a timely manner. As I like to sleep for at least 6 hours and don't always wake up and immediately open Reddthat (Shocking I know!), sometimes people end up getting to around the 12 hour mark for their applications to be accepted. Which I can see being a problem, especially when they don't supply an email address. I'm working on a better solution to alert me when there are applications that need processing.

Also did you know I renewed the domain again? That means it's been 3 Years now!

Honestly insane. This Lemmy thing is probably the biggest thing I've done for a community on the internet for a long time! Here's to 3 years more!

<3

Tiff

Donation Thingy:

Note: On Liberapay, donations are paid in advance, but you are more than welcome to make it recurring monthly instead of paying yearly. Don’t worry too much about the “fees”. It’s just the cost of doing business via the credit card duopoly.

💸 “Expenses”:

• April Costs: ~A$150
• May Costs: ~A$150

Still tracking around the ~150 mark per month. I managed to cull some assets from our S3 bucket (ie; our old backups and all the dev buckets) to bring our costs down a little bit, so when the June bill come in the next 3 days I'll update it.

⭐ Donation “Statistics”:

• New Donators in May: 0
• Lost Donators (Who did not renew): 0
• Total Weekly: ~A$26.84 (Trending Down)
• (“Monthly”: 26.84×52÷12 = ~A$116.30)
• Our Public Donators: <3
• AppleStrudel
• asqapro
• bitwize
• ~1903711
• Matthew Fennell

🥅 Goal: 26.84 / 60.00

Want a month dedicated to you? -> https://liberapay.com/reddthat

PS: don’t like fees? Use Crypto (Litecoin/Monero) for even better transaction fees than credit cards for your donation. (See the main sidebar for addresses). And validate them again on liberapay too if you want to ensure I get those dollary doos!

::: spoiler spoiler

How much is a coffee in your part of the world? If you can shout me a coffee per month, we'll easily crush our donation goals. :::

We took some time today to reboot the server with the new kernel to protect us against dirtyfrag/copyfail etc/etc.

Reddthat then failed to come back online because of a slight miss-configuration / DNS shenanigans.

This was the first time we'd rebooted since our move to our dedicated server and in classic fashion something happened!

::: spoiler Actual issue explained localhost resolved to ::: (IPv6) and our lemmy containers only exposed a local port on IPv4 127.0.0.1.

Why on earth it didn't forward the traffic who knows but I told nginx who's boss and then we were back up and running.

I ended up specifying 127.0.0.1 to force it all via IPv4 so you can all get back to the memes asap. :::

Yes, it's always DNS.

Cheers,

Tiff

Donation Plug Goes Here: https://liberapay.com/reddthat/

An interesting take on the Amerocano

Hello Friends!

The silent admin has come out of hibernation to give you an update. As we've been moderating, approving, and all around enjoying the content reddthat & the lemmyverse provides I figured we might as well have an update.

Reddthat has been upgraded to the latest lemmy verison 0.19.18. 🥳

See the link in the post for the main updates, as well as the 0.19.17 and 0.19.16 versions. But honestly... there hasn't been many changes that are relatable for end users. The Lemmy devs have been pushing out only serious fixes. The only one that is even slightly relevant is 'Display UserBadges for Bot, Banned and Deleted users in all PersonListings'

It looks like this:

We've been kicking along but have seen a huge increase in our image/video storage! Which has unfortunately meant that I'm getting charged close to $44USD/m for our storage, and we are now further away from breaking even. So all the savings we manged to find with our server migration has been offset with our increase in storage usages. This happened a couple months ago and I don't have a pretty graph showing the increase, but happened in Feb based on the financials.

We regular serve about 1 to 2 TB per month from the server and have apparently "saved" 1TB over the past 30 days.

Something else that is interesting (to me at least), and might be part of the current state of AI nonsense is that even though we have always blocked 'old.reddthat.com' from the search engines it is still significantly queried more than other hostnames. Including one of our old frontends 'next' which has been removed for a "long time".

The DNS query map shows the closest datacentre for those queries. Interestingly there are some tiny tiny tiny datacenters there.

The total number of connections also is an indication of how popular the lemmyverse is. As this includes the server to server connections, not just our regular users it is a bit polluted but shows how many connections we need to respond to on a regular basis.

And if you've made it to the end, I want to make a special shoutout to everyone for being around, having fun, and being respectful to our fellow netizens. Unfortunately there have been a few bad actors so I would like to reiterate our No. 1 Rule. Remember the human! (no harassment, threats, etc.) (Yeah... While it's certainly not number 1 in the list, it should be!) On a whole Reddthat users are always special in my heart and we have been very "stable" in the number of users. For those of you who regularly, comment, vote, post, and use reddthat thankyou!

An an extra special shoutout to those who "put their money where their mouth is" and support Reddthat financially. I honestly wouldn't be doing this after all these years without your continued support.

Cheers,

Tiff

Note: On Liberapay, donations are paid in advance, but you are more than welcome to make it recurring monthly instead of paying yearly. Don't worry too much about the "fees". It's just the cost of doing business via the credit card duopoly.

💸 "Expenses":

• Jan Costs: ~A$122
• Feb Costs: ~A$156
• March Costs: ~A$148
• April Costs: ~A$150

I expect future costs to stay around the 150 mark. We were so close to being self sufficient but as always, if we get close to our goal of $60/w we'll finally be in a solid position to stay forever online.

⭐ Donation "Statistics":

• New Donators in Jan,Feb,March,April: 0
• Lost Donators (Who did not renew): 1
• Total Weekly: ~A$27.36
• ("Monthly": 27.36×52÷12 = ~A$118.56)
• Our Public Donators: <3
  • AppleStrudel
  • asqapro
  • bitwize
  • ~1903711
  • Matthew Fennell

🥅 Goal: 27.36 / 60.00

Want a month dedicated to you? -> https://liberapay.com/reddthat

PS: don't like fees? Use Crypto (Litecoin/Monero) for even better transaction fees than credit cards for your donation. (See the main sidebar for addresses). And validate them again on liberapay too if you want to ensure I get those dollary doos!

A word on limitations

Little Snitch for Linux is built for privacy, not security, and that distinction matters. The macOS version can make stronger guarantees because it can have more complexity. On Linux, the foundation is eBPF, which is powerful but bounded: it has strict limits on storage size and program complexity. Under heavy traffic, cache tables can overflow, which makes it impossible to reliably tie every network packet to a process or a DNS name. And reconstructing which hostname was originally looked up for a given IP address requires heuristics rather than certainty. The macOS version uses deep packet inspection to do this more reliably. That's not an option here.

For keeping tabs on what your software is up to and blocking legitimate software from phoning home, Little Snitch for Linux works well. For hardening a system against a determined adversary, it's not the right tool.

I like this.

It could be worse... People could use Azure sign in for securing their infrastructure! #joke

Massive French Torrent Tracker that earned 5-8 Million Euros per year has been leaked.

Hey #AntiMeme

Feel like becoming a moderator for a next to no content community? Every felt like your calling was to resurrect a community to only eventually leave it, fulfilling the communities inherit nature.

Let Me Know

Tiff

Hey #WebComics ,

We'd like to have a moderator who can keep the webcomics community alive and give it a nice refresh.

Please comment here (&/or send me a PM directly) if you wish to become a mod :) .

Thanks,

Tiff

I found some time. and in 15 minutes from this post we will go down for ~1 hour to ensure we have complete data consistency.

• 09:00 UTC to 10:00 UTC

See you soon!

HI!!!!!!!!

I've made my position pretty clear how this was the wrong move by our government...

Services that eSafety considers will be age-restricted social media platforms

Facebook
Instagram
Kick
Reddit
Snapchat
Threads
TikTok
Twitch
X (formerly Twitter)
YouTube

Services that eSafety considers will not be age-restricted social media platforms

Discord
GitHub
Google Classroom
LEGO Play
Messenger
Pinterest
Roblox
Steam and Steam Chat
WhatsApp
YouTube Kids

The best time for an update, while hopped up on candy , coffee, and playing trick or treat on the neighbours, enjoying family, friends or just a relaxing day. I hope wherever you are, you are enjoying the holiday season(s).

Reddthat has been super stable and nothing major to report except for pictrs crashing recently (as my lack of updates may have proven).

This sat in my drafts for way too long! But now's the time to finally update you all as there are actually items to update you on!

Moderation

The other admins haven't been online as much lately and I'm doing most of the moderation. So I may end up asking people if they would like to become administrators of reddthat. Mainly I need some people in US time zones and EU time zones. (Which should nearly be all of you 😅!). The majority of the work results in actioning reports and ensuring we don't let bots/people not even trying to read the application process into Reddthat.

If you would like a position, please PM me with:

• Your timezone/Active Hours
• What makes you a good candidate for an Admin. (Prev experience?)

Storage storage Storage. New Server!?

Edit: we completed it in https://reddthat.com/post/57007804

As I went to purchase the new server in June/July right after posting our last update the company IMMEDIATELY jacked up their prices. It must have been a typo because it's now for $135/m. Which would give us 2x500GB and 32GB of Ram. The processor is a Xeon E-2136 , which only comes with 12 threads! With a single thread rating of 2700, which is about double of what OVH is giving us.
So hopefully our database will be even faster once we make the change.

On OVH, we managed to finally get the storage increase and it's worked without any problems. Which was perfect timing really. We now sit at around 75% usage of our 200GB drive. So I'm hoping to get 2-3 more months out of it and pray to the tech gods there are some black friday dedicated server deals or even Christmas ones which will allow us to get a better deal.

And that was what happened. We managed to get a deal on a dedicated server in OVH which was around the $90/m . So only a little bit more than what we are paying now. Which is great. From a brief benchmark we get at least 1.5x faster CPU which equates to shorter scheduled tasks, faster queries, and more resources available to showing everyone their own subscriptions and memes. That's not inclusive of the 500GB ssds, which will allow for our database to grow to 2-3x our current total. Which means this should last us around 4-6 years (at the current growth of Lemmy as a whole).

I've been getting it working over the past 2 weeks Oz. We're taking this time to also upgrade to Debian Trixie, which is currently not supported by the Lemmy ansible repo. So I will be upstreaming that change once I fixed it. This includes keeping it up to date so that when we do the swap over in a week's time it will be as short as 5 minutes and as long as 15 minutes for the swap over to happen. I'm pretty sure it will be closer to 15mins as even at 1Gb/s our 100GB database will take around 800s to completely migrate at the theoretical maximum. (And a brief 15 minutes, is better than mucking around with postgres' HA).

This server change is probably going to happen on the 27th/28th. I'll add another post outlining the timeline on the day.

Cheers,
Tiff

Note: On Liberapay, donations are paid in advance, but you are more than welcome to make it recurring monthly instead of paying yearly. Don't worry too much about the "fees". It's just the cost of doing business via the credit card monopoly.

💸 "Expenses":

• August Costs: ~A$116
• September Costs: ~A$117
• October Costs: ~A$121 (Increased storage costs and falling AUD/USD)
• Nov Costs: ~A$125
• Dec Costs: ~A$125

⭐ Donation "Statistics":

• New Donators in Aug,Sept,Oct: 2
  • Thank you again! September and October was dedicated to both of you!
• New Donators in Nov, Dec: 0
• Total Weekly: ~$24.54
• ("Monthly": 24.54×52÷12 = ~$106.34)
• Our Public Donators:
  • AppleStrudel
  • ~1890351
  • Matthew Fennell

🥅 Goal: 24.54 / 60.00

Want a month dedicated to you? -> https://liberapay.com/reddthat

PS: don't like fees? Use Crypto (Litecoin/Monero) for even better transaction fees than credit cards for your donation. (See the main sidebar for addresses). And validate them again on liberapay too if you want to ensure I get those dollary doos.