Posts tagged: zfs Archive

The Faulted Disk: harbor Replacement Writeup

2026-10-06T21:30:00Z

The sequel to panicking-led-to-losing-my-desktop — this time the monitoring actually caught the disk dying, and nothing was lost.

What happened #

harbor is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from tank. One side of the mirror, a Seagate Exos ST12000NM0127 (serial ZJV4QFLB, /dev/sdb), went FAULTED with 14 read + 22 checksum errors.

mirror-0                              DEGRADED     0     0     0
  ata-ST12000VN0008-2PH103_ZTM0NFDW  ONLINE       0     0     0
  ata-ST12000NM0127_ZJV4QFLB         FAULTED     14     0    22  too many errors

The IronWolf mirror side carried the pool — No known data errors. ZFS redundancy did exactly its job.

The difference from last time #

Last failure: no monitoring, found out by accident months later, desktop died.

This failure: SigNoz + node-exporter’s ZFS collector → node_zfs_zpool_state{state="degraded"} → alert rule → Gotify → my phone. The gotify notification fired before I knew anything was wrong.

Diagnosis #

Before zpool clear or replace — check SMART. smartctl-exporter already scrapes all disks into SigNoz, so I didn’t even need sudo:

Reallocated_Sector_Ct raw = 3024 and counting
Offline_Uncorrectable value/worst still 100 but raw errors climbing
SMART overall: still PASS (SMART’s overall bit is conservative until threshold)

3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don’t clear.

The swap (the annoying part) #

Hot-plug is never as smooth as it should be:

zpool offline harbor <old> to stop writes to the dead drive — actually skipped; drive fell off the bus on its own
New IronWolf ZTN1CQ07 in hand → plugged into ghost’s SATA → didn’t enumerate
Forced rescan (echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan) → nothing
Moved it to the old drive’s port → nothing
Sabrent USB dock on aurora → dock enumerated as 0B device, no disk behind it → reseated + replugged → drive spun up and appeared: /dev/sdd, 10.9T, old ZFS partition table on it (used drive — part1/part9 layout)
Back to ghost, direct SATA → enumerated as sdb

Lesson: “spins up but doesn’t enumerate” on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.

The replace #

sudo zpool replace harbor \
  ata-ST12000NM0127_ZJV4QFLB \
  ata-ST12000VN0008-2PH103_ZTN1CQ07

Resilver: 8.80T in 18h38m, 0 errors (~154M/s). Pool stayed online and usable the whole time.

One gotcha: after resilver, zpool status showed errors: 1 data errors — but zpool status -v showed an empty error list. The corrupted data was already repaired; the counter was just stale. sudo zpool clear harbor → clean.

The monitoring that made this possible #

Wired during this session (all now in SigNoz):

node_zfs_zpool_state — pool health (node-exporter zfs collector)
smartctl-exporter — SMART attributes incl. reallocated sectors (the early-death signal)
zfs-metrics.sh textfile bridge — sanoid --monitor-* exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success
Alert rules → Gotify → phone

The replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.

Remaining homework #

Buy the replacement spare (the shelf’s empty now)
10Fold datasets are garbage + have zero snapshots — destroy
Every scheduled job emits cron_last_success_epoch now — if a job dies silently again, I’ll know

Same failure as April, opposite outcome. Redundancy did the protection; monitoring did the detection. You need both.

Co-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.

Backups are dope

2025-05-27T19:42:27Z

I accidently chown'd -R an app directory and it totally screwed up the database folder. Luckily I zfs replicate my docker volumes to another drive even on the same host, so a quick [rsync] from /harbor/encrypted/docker/manyfold to /tank/encrypted/docker/manyfold got me back up and running since I hadn't replicated the messed up permission set yet Q: How to link to my rsync like a pro post? A: Wikilinks `[[rsync-like-a-pro]]` [[rsync-like-a-pro]]

SMB with ZFS on Ubuntu

2025-02-20T09:18:35Z

sudo apt-get install -y samba \ then set sharesmb=on \ chown the user \ smbpasswd -a \ then mount with user/password from other machine

Recovering from zpool corruption when you’re short a drive

2025-02-03T21:08:56Z

can only mount tank RO

so can’t rename
also can’t detach which is what I’d want

Could import pool witn -N to not mount any datasets
detach and attach
resilver…

D and uninterruptable sleep

2024-12-11T10:48:10Z

Htop #

I recently have been having significant home server issues, and that’s not the point of this - today I learned what D state is when looking at htop.

Apparently this means “uninterruptable sleep” and it’s a dev’s nightmare…

Context #

The issue I was having was that some zfs rollback commands were hung - for hours… I wasn’t sure what was going on, rollbacks should be instant but I figured it was just an artifact of these issues.

Turns out I still don’t know what locked the disks up but I learned why <C>-c did nothing… the more you know

lsof to find what’s using your filesystem

2023-04-09T13:32:38Z

lsof | grep /tank/nas shows me what is using my nas at any time!

Changing ZFS key for child datasets of encrypted dataset after migration

2023-04-08T20:12:22Z

➜ pihole sudo zfs load-key -L file:///path/to/.zfs.tank.key tank/encrypted/vms/arch-sandbox ➜ pihole sudo zfs change-key -o keylocation=file:///path/to/.zfs.tank.key -o keyformat=raw tank/encrypted/vms/arch-sandbox Need to load-key for each individual dataset, then change key location to be a file instead of the prompt

Systemd timer for syncoid

2022-12-21T11:38:27Z

I have a bash script called syncoid-job which boils down to a barebones -

#!/bin/bash

syncoid --no-sync-snap --sendoptions=w --no-privilege-elevation $SYNOIC_USER@$SERVER:tank/encrypted/nas tank/encrypted/nas

I want to run this script hourly but as my user (notice the no-privilege-elevation flag)

First - create a systemd unit file at /etc/systemd/system/syncoid-replication.service

[Unit]
Description=ZFS Replication With Syncoid

[Service]
Type=oneshot
ExecStart=/$HOME/dotfiles/syncoid-job
User=$USER
Group=$GROUP

[Install]
WantedBy=multi-user.target

Then we save the unit file, enable the service, and then start it

systemctl enable syncoid-replication.service
systemctl start syncoid-replication.service

Note this will run that script… so be ready for syncoid to do its thing

Now for the timer… We create /etc/systemd/system/syncoid-replication.timer

[Unit]
Description=Run syncoid-replication every hour

[Timer]
OnCalendar=hourly

[Install]
WantedBy=timers.target

Hit it with a systemctl enable syncoid-replication.timer and you’re in business!

Limit zfs list to avoid docker vomit

2022-10-20T06:39:18Z

zfs list has a flag -r, but if you use zfs driver for docker then you’ll get flooded with every docker volume in the world. zfs list -r -d N will limit the dept of the print out, so zfs list -r -d 2 gives me tank, tank/encrypted, tank/encrypted/docker -> but then I don’t see all the continer volumes

Quick setup of ZFS encrytped datasets with sane permissions

2022-10-06T19:58:01Z

Assuming you have a pool called tank…

And assuming you have an encrypted dataset (See Jim Saltar’s short intro)

Create a group for permissions - in my case I have one called home
Then if there’s anything in /tank/encrypted his it with chgrp -R home /tank/encrypted to give the home group ownership
Next we need to make sure that the members of home can do the writing… so chmod 775 -R /tank/encrypted will do the trick
Finally we want to make sure that all data created inside our dataset has the same set of permissions with chmod g+s /tank/encrypted and chmod g+w /tank/encrypted

Benchmark your disks with fio

2022-08-27T13:43:22Z

Intro

I use ZFS at home in my homelab for basically all of my storage… Docker uses ZFS backend, all my VMs have their .qcow2 images in their own zfs datasets, and all my shares are ZFS datasets. I love ZFS but my home hardware presently is the opposite of expensive or new… Thankfully I’ve had a lot of my orginal homelab simply given to me but the cost of this is that I didn’t put my machines together, I didn’t choose the disks, and I definitely didn’t do the research I would’ve otherwise done had I bankrolled my server personally…

The Problem #

I run glances on basically all my machines and for the longest time I have been seeing big time iowait issues. Now, since everything was free I’ve largely been able to ignore that however I’m now after some better performance which I think means new hardware!

Here is a random screenshot of my glances homepage at time of writing - The only major load on my server is some ffmpeg transcoding (about 60% CPU utilization)…

As you can see… there’s a lot of issues and I don’t even know what they mean.

fio

I heard about fio through a friend and decided to try it out quick. It installs with apt on ubuntu quick and easy…

Jim Saltar has a good blog post on it here

Basically it’s a handy tool for benchmarking your disks and the blog dives into what types of metrics matter - it’s not just throughput, but also latency, iops, etc.

Tests #

I ran a few basic commands inside a new zfs dataset on my server tank/fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1 > single-4KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1 > 16-parallel-64KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1 > single-1MiB-random-write.txt

Results #

The single 4 KiB random write:

WRITE: bw=7836KiB/s (8024kB/s), 7836KiB/s-7836KiB/s (8024kB/s-8024kB/s), io=523MiB (548MB), run=68317-68317msec

The 16 parallel 64KiB random writes:

WRITE: bw=93.9MiB/s (98.4MB/s), 5599KiB/s-6303KiB/s (5734kB/s-6454kB/s), io=7642MiB (8013MB), run=81310-81418msec

The single 1MiB random write:

WRITE: bw=81.2MiB/s (85.1MB/s), 81.2MiB/s-81.2MiB/s (85.1MB/s-85.1MB/s), io=8177MiB (8574MB), run=100699-100699msec

Summary

So I don’t fully understand these numbers yet… 80-100 MiB/s isn’t super fast and that’s across a parallelized workload… The single threaded workloads have awful performance so this tells me something is wrong… I have a few ideas…

ZFS dataset config options such as ashift or the blocksize might be way misconfigured
The disks/pool which came from a TrueNAS/FreeBSD machine may have some artifacts that I need to clean up
The SAS controller I am using, which I flashed with IT firmware to get it into JBOD mode might be messed up
The data cables themselves could be a problem…

Points 3 and 4 are less likely given that the write speed does increase in the parallelized job but I’m a newbie so it’s time to dive in!

ZFS Permissions for Sanoid/Syncoid

2022-07-08T15:54:58Z

zfs allow -u $USER clone,load-key,create,destroy,mount,mountpoint,receive,send,rollback,compression,snapshot,hold,keylocation,bookmark tank

load-key only needed if using encrypted datasets

Self-hosted Docker registry with proxy pull through

2022-07-06T06:44:35Z

I decided that I want to self-host all my docker images for the purposes of regularly rebuilding and security scanning. The first step is to set up a registry, which coincidently enough you can do with a Docker container 😛!

Instructions for setting up the proxy are on the offidical docker docs here

Deployment

I chose to deploy with Portainer because I already use it for monitoring all my docker containers. I deploy most container with Ansible but use Portainer to setup a few more adhoc or non-git driven applications.

How I use Nextcloud for safe central storage

2022-05-19T21:17:42Z

Setup admin
External Storage extension
Add my nas zfs dataset
chown -R www-data:www-data on anything nextcloud uploads to.

Nextcloud permissions with ZFS and Ansible-NAS

2022-05-19T13:55:20Z

TL;DR

As the nextcloud docs say… if you want to write to an external volume that location has to be writeable by the user/group www-data on the host system… so if that makes sense to you then this TIL probably isn’t a ton of value.. if not however, read on :)

Case Study

You want to self-host your own cloud and use a smart file system for convenience… Nextcloud and ZFS are pretty common goto answers for each of those problems.

My home NAS is built on ZFS and among other things I have a zpool named tank and nested in there is a tank/nas dataset with several child zfs datasets under that.

I want to use nextcloud mainly for auto-uploading photos from my wife’s and my phones for automatic backups. The issue is that the nextcloud application (I run in Docker) is fixed as the www-data user and so any volume/folder that you want nextcloud to write to needs to be permissioned such that www-data owns it… but I don’t want www-data to own everything in my NAS… so what’s a girl to do?

Solution

Well, one way to go is to just utilize docker volumes, write the data in the container to /var/www/html and let that be the place your data backsup to.

I still wanted nextcloud to automatically write right to my NAS so I created a nextcloud-upload directory inside of tank/nas/media/photos (photos cause that’s all that gets automatically uploaded)

Then I chown -R www-data:www-data /tank/nas/media/photos/nextcloud-upload so that just that sub-folder is owned by www-data. Now everyone’s happy!

See ZFS snapshot disk usage

2022-05-19T06:05:23Z

As I was cleaning up my NAS recently I noticed that I ran out of storage even though my disk usage looked pretty low… turns out I was keeping a mega-ton of ZFS snapshots and due to my own ignorance at the time didn’t realize the storage cost of this!

zfs list-o space tank/home will show the disk usage of the dataset and snapshots in tank/home

Remove ZFS Dataset Specific Snapshots

2022-05-19T05:49:16Z

I started my homelab journey being super naive about ZFS and how to manage the filesystem… that bit me in the butt when transfering a ton of files out of folders and into datasets because ZFS is copy on write so I was essentially duplicating my storage until I got a hair smarter about removing files after they’re moved (rsync –remove-source-file ftw). But I had a ton of snapshots of child datasets with a ton of data that I just never will need, so I learned zfs list -H -o name -t snapshot tank/dataset1/dataset2 will list just the snapshots for dataset2 and if you pipe that into xargs -n1 zfs destroy then you have a way to clear out some snapshots you don’t need!