Posts tagged: backup

The Faulted Disk: harbor Replacement Writeup

2026-10-06T21:30:00Z

The sequel to panicking-led-to-losing-my-desktop — this time the monitoring actually caught the disk dying, and nothing was lost.

What happened #

harbor is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from tank. One side of the mirror, a Seagate Exos ST12000NM0127 (serial ZJV4QFLB, /dev/sdb), went FAULTED with 14 read + 22 checksum errors.

mirror-0                              DEGRADED     0     0     0
  ata-ST12000VN0008-2PH103_ZTM0NFDW  ONLINE       0     0     0
  ata-ST12000NM0127_ZJV4QFLB         FAULTED     14     0    22  too many errors

The IronWolf mirror side carried the pool — No known data errors. ZFS redundancy did exactly its job.

The difference from last time #

Last failure: no monitoring, found out by accident months later, desktop died.

This failure: SigNoz + node-exporter’s ZFS collector → node_zfs_zpool_state{state="degraded"} → alert rule → Gotify → my phone. The gotify notification fired before I knew anything was wrong.

Diagnosis #

Before zpool clear or replace — check SMART. smartctl-exporter already scrapes all disks into SigNoz, so I didn’t even need sudo:

Reallocated_Sector_Ct raw = 3024 and counting
Offline_Uncorrectable value/worst still 100 but raw errors climbing
SMART overall: still PASS (SMART’s overall bit is conservative until threshold)

3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don’t clear.

The swap (the annoying part) #

Hot-plug is never as smooth as it should be:

zpool offline harbor <old> to stop writes to the dead drive — actually skipped; drive fell off the bus on its own
New IronWolf ZTN1CQ07 in hand → plugged into ghost’s SATA → didn’t enumerate
Forced rescan (echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan) → nothing
Moved it to the old drive’s port → nothing
Sabrent USB dock on aurora → dock enumerated as 0B device, no disk behind it → reseated + replugged → drive spun up and appeared: /dev/sdd, 10.9T, old ZFS partition table on it (used drive — part1/part9 layout)
Back to ghost, direct SATA → enumerated as sdb

Lesson: “spins up but doesn’t enumerate” on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.

The replace #

sudo zpool replace harbor \
  ata-ST12000NM0127_ZJV4QFLB \
  ata-ST12000VN0008-2PH103_ZTN1CQ07

Resilver: 8.80T in 18h38m, 0 errors (~154M/s). Pool stayed online and usable the whole time.

One gotcha: after resilver, zpool status showed errors: 1 data errors — but zpool status -v showed an empty error list. The corrupted data was already repaired; the counter was just stale. sudo zpool clear harbor → clean.

The monitoring that made this possible #

Wired during this session (all now in SigNoz):

node_zfs_zpool_state — pool health (node-exporter zfs collector)
smartctl-exporter — SMART attributes incl. reallocated sectors (the early-death signal)
zfs-metrics.sh textfile bridge — sanoid --monitor-* exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success
Alert rules → Gotify → phone

The replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.

Remaining homework #

Buy the replacement spare (the shelf’s empty now)
10Fold datasets are garbage + have zero snapshots — destroy
Every scheduled job emits cron_last_success_epoch now — if a job dies silently again, I’ll know

Same failure as April, opposite outcome. Redundancy did the protection; monitoring did the detection. You need both.

Co-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.

Panicking Led to Losing My Desktop

2026-05-13T08:24:00Z

## False Sense of Security I thought I had backups handled... can you imagine how the rest of this post is going to go with that intro? To be fair, I do have backups figured out on my NAS - simple ZFS + sanoid/syncoid + replica pool + off-site backup with simple restore pathways. However, my desktop has been another story entirely. My desktop OS didn't support ZFS when I started checking it out, and I spent weeks thinking through how I would backup my HOME directory and projects mostly. I landed on a solution that I did validate once, but it fell off my radar and lo' and behold that was problematic... So that backup was based on restic for my home directory, but it was lazy. I verified it one time but I had built it with ai, thought I understood the restic repo part, and then promptly moved on with my life never buttoning it all up. That home directory backup got too big for where I was going to end up restoring it. My desktop system was installed on a 4 TB NVMe drive and due to the circumstances spawning this blog post I was gonna have to drop to a 500 GB boot drive with some extra disks as the storage layer. Overall it looked like: - A 4 TB SSD that was going bad - old OS - A 500 GB SSD, that was going to be my new operating system boot disk - A 2 TB SSD that was originally going to be this external storage volume anyways but I never set it up because the version of Aurora I was running didn't have ZFS, I was married to the idea of using ZFS, so I never ended up taking advantage of the space. However it was moot to me because my boot drive was 4 TB, high quality drive, so I was "just sure" I didn't need it. - AND a 4 TB rust disk as well, which was already a ZFS pool, left over from a previous desktop configuration, and admittedly I had forgotten it was even in the system. ## The Storm If it wasn't clear the problem is that my super-nice high-speed 4TB NVMe drive was going bad, like really bad. Eventually my OS stopped booting, it was even difficult to live-boot from any other ISO due to, I think ultimately, that disk causing such extreme latency in the start-up processes that they just failed. So I quickly found myself with little-to-no access to my primary desktop's data... ## Where It Went Wrong What I did is I live booted into an Ubuntu server environment (which took blood sweat and tears to successfully get into), mounted my home directory from the 4 TB SSD, and tried to continue my restic backup to my NAS, like an idiot. But at the same time I also tried to prune it by only backing up a few projects because I was getting worried about time. This was the first primary mistake - trying to muck with my backup script under duress. Then over the course of the whole thing it ended up taking over a week to solve this when it could've been 2-3 days. So say it with me kids - "Don't make decisions under duress" ## Climbing Out I downloaded opencode and had it help me write the right excludes syntax in my restic backup script and got it back up going. That went ok but opencode agents had no historical context for why anything was the way it was, and frankly an agent would've been misled thinking the backup solution was much more solid than it was due to how I documented it. Agents also miss things... in my chat sessions it knew about the other 2 available disks on the desktop system, I could have done a fresh backup to the 4 TB spinning rust disk no problem: install zfs, mount the pool, change target of restic, run full... that would've been beautifully simple. But instead I trimmed it down and backed not-everything up to the NAS over the network, and to a different backup target nonetheless... SMH. As I started to consider which OS I was going to go with next I failed to install Pop_OS! or Ubuntu onto the new disc... Then I tried Omarchy and the install script just looped. So, I reinstalled Aurora onto the new 500 GB disk and then quickly realized I don't have Firefox tabs, my SSH keys are in that restic backup, my ssh config, api keys in hidden files.... Everything is in that restic backup... The backup that's too big to restore to my new boot drive. But you know what I have? That 2 terabyte disk mounted just fine as a ZFS dataset. And I could mount the 4 TB rust disk with zfs as well because this version of Aurora has zfs working flawlessly! ## Hindsight What I should've done is so simple... While in that ubuntu live environment I should've just either updated restic to be a local backup to the 4 TB rust disk, or rsync'd my home directory to it plain and simple... I got all in my head about not backing up python venvs, node_modules, etc. that I didn't think to just basically carbon copy it all to a healthy disk and then prune it later. Then I could've synced everything back over that I needed to the new Desktop's $HOME and then scheduled the rsync or restic again to that locally mounted disk. ## The Detail I Left Out The keen reader might stop to think... why not just mount the old 4TB disk and copy what you need to your new desktop? And that's a prudent question... However, in order to get anything installed I had to physically remove the 4TB SSD from the motherboard, which was basically a full PC tear-down. From there I was able to at least boot in and out of iso's like you'd otherwise expect, and I have a USB/NVMe adapter so I planned to mount the old drive and copy things over from there... But sadly... it won't mount. it's dead-dead and it appears that anything I didn't save in my days-long-panicked-state is just. gone. I feel pretty stupid to have not taken advantage of the 2 available disks local to the machine, to have naively copied stuff over and dealt with the organization later once my OS was back up. I tried to be smart and efficient and ended up wasting so much time and losing quite a lot of "stuff"... ideas, blog posts that I never committed, etc. ## Current Status So a few lessons... 1. untested backups are not backups 2. false backups might be worse than none, although I did at least save a few things so maybe the jury is out here 3. making decisions while stressed out will lead to missing obviously better pathways... slow down, talk it out As for my current status - I'm working on [[desktop-setup-2026]] and recovering what I can from my haphazard'd rsyncs in the live ubuntu env I got into. I'm also setting up a new Linux laptop at work at the same time so maybe I'll hve some workflow changes to write about in the future. For now, it's nice to be forced to accept that not every idea was that important, the good stuff will come back around, and ultimately computers and shit are just things, they're not life.