Enjoying the company of others while also finding it emotionally draining is such a strange paradox.

The Iran war excursion is now in it's 11th 6th week. Everyone is having such a blast great time.

I recently made a huge mistake. My self-hosted setup is more of a production environment than something I do for fun. The old Dell PowerEdge in my basement stores and serves tons of important data; or at least data that is important to me and my family. Documents, photos, movies, etc. It's all there in that big black box.

A few weeks ago, I decided to migrate from Hyper-V to Proxmox VE (PVE). Hyper-V Server 2019 is out of mainstream support and I'm trying to aggressively reduce my dependence on Microsoft. The migration was a little time consuming but overall went over without a hitch.

I had been using Veeam for backups but Veeam's Proxmox support is kind of "meh" and it made sense to move to Proxmox Backup Server (PBS) since I was already using their virtualization system. My server uses hardware raid and has two virtual disk arrays. One for VM virtual disk storage and one for backup storage. Previously, Veeam was dumping backups to the backup storage array and copying them to S3 storage offsite. I should note that storing backups on the same host being backed up is not advisable. However, sometimes you have to make compromises, especially if you want to keep costs down, and I figured that as long as I stayed on top of the offsite replications, I would be fine in the event of a major hardware failure.

With the migration to Proxmox, the plan was to offload the backups to a PBS physical server on-site which would then replicate those to another PBS host in the cloud. There were some problems with the new on-site PBS server which left me looking for a stop-gap solution.

Here's where the problems started. Proxmox VE can backup to storage without the need for PBS. I started doing that just so I had some sort of backups. I quickly learned that PBS can replicate storage from other PBS servers. It cannot, however, replicate storage from Proxmox VE. I thought, "Ok. I'll just spin up a PBS VM and dump backups to the backup disk array like I was doing with Veeam."

Hyper-V has a very straight forward process for giving VM's direct access to physical disks. It's doable in Proxmox VE (which is built on Debian) but less straight forward. I spun up my PBS VM, unmounted the backup disk array from the PVE host, and assigned it as mapped storage to the new PBS VM. ...or at least I thought that's what I did.

I got everything configured and started running local backups which ran like complete and utter shit. I thought, "Huh. That's strange. Oh well, it's temporary anyways." and went on with my day. About two days later, I go to access Paperless-ngx and it won't come up. I check the VM console. VM is frozen. I hard reset it aaaannnnddd now it won't boot. I start digging into it and find that the virtual HDD is corrupt. fsck is unable to repair it and I'm scratching my head trying to figure out what is going on.

I continued investigating until I noticed something. The physical disk id that's mapped to the PBS VM is the same as the id of the host VM storage disk. At that point, I realize just how fucked I actually am. The host server and the PBS VM have been trying to write to the same disk array for the better part of two days. There's a solid chance that the entire disk is corrupt and unrecoverable. VM data, backups, all of it. I'm sweating bullets because there are tons of important documents, pictures of my kids, and other stuff in there that I can't afford to lose.

Half a day working the physical disk over with various data recovery tools confirmed my worst fears: Everything on it is gone. Completely corrupted and unreadable.

Then I caught a break. After I initially unmounted the [correct] backup array from PVE it's just been sitting there untouched. Every once in a great while, my incompetence works out to my advantage I guess. All the backups that were created directly from PVE, without PBS, were still in tact. A few days old at this point but still way better than nothing. As I write this, I'm waiting on the last restore to finish. I managed to successfully restore all the other VM's.

What's really bad about this is I'm a veteran. I've been in IT in some form for almost 20 years. I know better. Making mistakes is OK and is just part of learning. You have to plan for the fact that you WILL make mistakes and systems WILL fail. If you don't, you might find yourself up shit creek without a paddle.

So what did I do wrong in this situation?

• First, I failed to adequately plan ahead. I knew there were risks involved but I failed to appreciate the seriousness of those risks, much less mitigate them. What I should have done was go and buy a high capacity external drive, using it to make absolutely sure I had a known good backup of everything stored separately from my server. My inner cheapskate talked me out of it. That was a mistake.
• Second, I failed to verify, verify, verify, and verify again that I was using the correct disk id. I already said this once but I'll repeat it: storing backups on the host being backed up is ill advised. In an enterprise environment, it would be completely unacceptable. With self-hosting, it's understandable, especially given that redundancy is expensive. If you are storing backups on the server being backed up, even if it's on removable storage, you need to make sure you have a redundant offsite backup and that it is fully functional.

Luck is not a disaster recovery plan. That was a close call for me. Way too close for my comfort.

LITTLE ROCK, Ark. – Gov. Sarah Huckabee Sanders said she was asked to leave a Little Rock restaurant after her office claims that the restaurant’s owner said employees felt “threatened and uncomfortable.”

According to a Wednesday night statement from the governor’s office, Sanders, while accompanied by an Arkansas State Police protection detail, was having lunch on Friday at the Little Rock restaurant called The Croissanterie.

“After the group had been seated at two separate tables for about an hour and fifteen minutes and had paid and tipped the restaurant staff, The Croissanterie’s owner approached a member of the Governor’s Executive Protection Detail and asked the Governor to leave because her presence was ‘making employees threatened and uncomfortable’ because of her political views,” the governor’s spokesperson said.

The governor’s spokesperson stated that Sander’s group began to leave the restaurant when a man standing with members of the staff, “yelled ‘it’s time to go’ and made a crude hand gesture in the Governor’s direction.”

The governor also shared a statement:

“Last week I was having lunch with two other moms at a restaurant when the owner approached a member of the State Police Executive Protection Detail and said my presence made their employees feel threatened and told us to leave. Arkansans are known for their warm hospitality, and while that restaurant certainly doesn’t meet that standard, my administration will continue to focus on lifting Arkansans up, not tearing others down with discrimination and hate.”

KARK 4 News has reached out to The Croissanterie for comment, but has not yet heard back.

...unthinkable. Kind of like abandoning your wife in the middle of a serious, life-threatening illness. And then doing the same thing to the wife you abandoned your first wife for.

I don't even know that much about Hallie Schoffner but Tom Cotton must be really nervous to be grasping at straws this desperately.

These kind of messages are rich considering Tom Cotton has spent the last 11 years in the Senate doing nothing for the average Arkansan aside from trying to make their lives more difficult.

WASHINGTON—Circulating a new advisory that highlighted what it called the “worst of the worst,” the Department of Homeland Security released a watchlist Friday warning Americans about the presence of mothers driving cars. “Please be on the lookout for the following mothers, who are confirmed to be in possession of dangerous vehicles and should be met with extreme caution,” said DHS Secretary Kristi Noem, who stressed that the women on the list were known caregivers operating inside the United States and cited intelligence gathered by the Department of Justice, the Federal Bureau of Investigation, and local Parent Teacher Associations. “If you see a minivan with a ‘Baby on board’ sticker, do not engage—many of these women are armed with juice boxes and will not hesitate to use a wet wipe on your face. Maintain a safe distance from any compact SUVs playing Bluey on their entertainment systems. The DOJ has opened an anonymous tip line for reporting any sightings of a car seat in your area.” Noem also warned the public to avoid locations where the exceedingly nurturing individuals were known to congregate, like the school pickup line around 3 p.m.