Storage full
I tried to upload an image, but got the error IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }
I'm assuming it's a different storage area than the main database, otherwise this will fail too...
I tried to upload an image, but got the error IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }
I'm assuming it's a different storage area than the main database, otherwise this will fail too...
What! We use a s3-like service for that :( Investigating
Turns out there is a bug with pictrs which doesn't clean up temp files. So we had ~88GB of temp data!
And since we moved to a dedicated pictrs server (so the main Lemmy procesees & DB could benefit) I didnt add that server to our monitoring 🫢
It's fixed now and I'll be adding disk monitoring once I get a moment away from $job
We are on 0.4.7 and 0.5.0 fixes this bug (apparently).
I was waiting for the "dust to settle" as it were on the 0.5 release but will schedule the upgrade this weekend sometime.
That's how it goes, if this was a universe where you'd monitored that, the bug wouldn't exist. Thanks for fixing 👍
Good news!