Been working on a project where we need to keep an eye on competitor pricing pages and feature comparison tables. The data changes frequently enough that manual checking doesn't scale, but it's not structured data so scraping is painful.

What we ended up doing is taking scheduled screenshots of specific pages and diffing them visually. Sounds dumb, but it actually works better than trying to parse HTML that changes layout every other week.

The setup

We tried a few approaches before landing on something that stuck:

Attempt 1: DOM diffing. Wrote a script that grabbed the page HTML, stripped non-content tags, and compared. Broke constantly because class names changed, ads rotated, and layout shifts threw false positives everywhere. Lasted about a month.

Attempt 2: Text extraction + diff. Better, but lost all context. Knowing that some text changed doesn't help when you can't see where on the page it changed. Pricing buried in a table looks very different from pricing in a hero banner.

What worked: Visual monitoring. Take a full-page screenshot on a schedule, overlay it with the previous one, highlight the pixel differences. You immediately see what moved, what changed, what disappeared. We've been using SnapshotArchive for this — it handles the scheduling and keeps a visual history so you can scrub back through changes over time.

Where it's been most useful

Pricing pages, obviously. But also:

• Feature comparison tables (competitor added a feature we don't have? good to know early)

• Landing page A/B tests (you can literally watch competitors test different copy)

• Job postings (hiring for "AI engineer" tells you something about their roadmap)

• Changelog/release notes pages

The screenshot approach catches things that text-based monitoring misses entirely. A redesigned pricing page with the same prices but different emphasis? DOM diff says "no change." Visual diff says "they made the enterprise plan way more prominent."

Curious what others do

Is anyone doing this kind of competitive monitoring? What's your approach — manual, scripted, or some tool? We've been happy with the visual route but I'm sure there are workflows I haven't considered.

Been dealing with this more often lately. Tests pass on my machine, I push, and CI blows up. Usually it's one of these:

• Different Node/Python/whatever version
• Missing env vars that exist in my .env but not in CI secrets
• File system case sensitivity (macOS vs Linux)
• Some flaky test that depends on timing

My current debugging flow is pretty basic: check the logs, compare versions, run the exact same Docker image locally if I can. But it still eats 20-30 minutes each time before I figure out the actual problem.

Anyone have a more systematic approach? Like a quick checklist you run through before you even look at the logs?

Also curious — do you replicate your CI environment locally with something like act (for GitHub Actions) or just trust the remote runner?

Had a backup script running via cron for months. Worked fine until it didn't — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.

The obvious answer is "check your logs" but let's be honest, nobody's reading cron logs daily for 15 different scheduled tasks across 4 servers.

What's your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.

Honest question. I've got automated daily pg_dump backups going to S3 and I check that the files are there, but I've never actually tried restoring one on a fresh instance to see if it works.

Feel like this is one of those things where you assume it's fine until you desperately need it and find out the dumps were corrupted for 3 months.

Anyone have a setup where restores get tested automatically? Or is that overkill for side projects?

Spent an hour today renaming env vars across three services to make them "consistent." Broke staging in the process because one service cached the old values. Should've just left the mess alone — it worked fine before I touched it.

Working on a project where I need to grab screenshots of pages on a schedule — mostly for QA and keeping a visual history of what changed and when.

Right now I'm running Puppeteer in a cron job on a cheap VPS. It works, but honestly it's a pain to maintain. Chromium eats RAM like crazy, sometimes it hangs and the whole thing needs a restart, and the screenshots come out wrong on pages that load content dynamically.

A few things I've been struggling with:

• Pages with lazy-loaded images — half the time the screenshot fires before everything renders
• Cookie consent banners blocking the actual content
• Memory usage goes through the roof when I try to do more than ~50 pages in a batch

I've looked into Playwright as a replacement but from what I can tell the resource usage is about the same. Also tried running headless Chrome in Docker which at least makes cleanup easier, but didn't solve the core problems.

Curious what others are using. Are screenshot APIs worth it for this kind of thing, or is self-hosting still the way to go? Anyone running something similar at scale?

I run a few side projects and I've gone through different stages of monitoring them. First it was just checking manually if the site loads. Then I added a simple curl ping in cron. Then I started tracking response times, certificate expiry, even visual changes on pages.

At some point I realized I was spending more time building monitoring than the actual product. Classic trap.

Curious what other devs use for keeping an eye on their stuff. Do you go with a hosted service, self-host something like Uptime Kuma, or just wing it with scripts?

Hey everyone! Just signed up on Lemmy. I've been running self-hosted services for a while now and looking forward to learning from this community. Glad to be here.