Posts tagged: linux

💭 Tyblog | systemd has been a complete, utter, unmitigated success

2025-07-11T11:00:54Z

Here’s my thought on Tyblog | systemd has been a complete, utter, unmitigated success

I found this post on systemd this morning, and although I’ve been a Linux user for around 6 years now, there’s some levels of it that are pretty foreign to me. One of the things I never got was the drama around systemd - but that’s because I wasn’t around in the beforefore times, I don’t use cron THAT MUCH, and I’ve only had to setup a few systemd unit files - not enough to generate a real opinion on it. But lately I’m looking into more automation at home - and some stuff is fun to put into an overkill of a platform for a homelab, like temporal, but some stuff should probably be run in more hardened ways (backups for example) - so getting a little download this morning on systemd from a clearly seasoned Linux user is a nice addition to my hot cup of coffee.

Note

This is one of [[ my-thoughts ]]. I picked this up from Waylon Walker(https://thoughts.waylonwalker.com). It’s a short note that I make about someone else’s content online. Learn more about the process Thoughts

💭 Self-Hosting A Cluster On Old Phones | Hackaday

2025-07-06T11:30:17Z

Here’s my thought on Self-Hosting A Cluster On Old Phones | Hackaday

Came across a blog and article on using a mobile OS to self-host a cluster on old phones… I happen to keep several old android phones lying around and although I’m not presently super interested in figuring out an android/arm based homelab, I am excited for the next coming years when I’m sure it’ll only get easier, phones will only be more powerful, and perhaps we’ll be running full blown CV pipeline in k8s clusters running on Pixel devices!

Note

💭 Linux Explained: What is The Linux Kernel? - YouTube

2025-06-07T11:28:18Z

Here’s my thought on Linux Explained: What is The Linux Kernel? - YouTube

I’m getting sucked into the Linux is a kernel not an operating system crowd because tonight I realized I couldn’t describe what a kernel is so I YouTubeda few things and before you know it you joined a decades old war before finishing loading the dishwasher

Note

I was wrecked by a weird combo of » and -e

2025-04-23T08:44:59Z

TL;DR #

If state matters then check it in the beginning or handle it on a failure… Let me explain

I ran into some trouble recently almost losing some encrypted data… Now this would be the second time I’ve had that happen, so I’m going to write a little bit about it now that I figured it out

The Stage #

Here’s the scenario - I use ansible-vault to keep some sensitive data in a few public repos, and I keep the keys I use in Bitwarden Secrets Manager.

I use just as a command runner and have common just encrypt and just decrypt recipes throughout many justfiles. It might look something like this


get-vault-key:
    bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'

encrypt:
    #!/bin/bash
    set +e
    just get-vault-key > key
    ansible-vault encrypt ./secret-file --vault-password-file key
    ansible-vault encrypt ./secret-file2 --vault-password-file key
    rm key

decrypt:
    #!/bin/bash
    set +e
    just get-vault-key > key
    ansible-vault decrypt ./secret-file --vault-password-file key
    ansible-vault decrypt ./secret-file2 --vault-password-file key
    rm key

Spoiler #

Above is a good example of the 2 recipes, but prior to this morning they looked like this


get-vault-key:
    bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'

# BAD EXAMPLES! DO NOT DO THIS

encrypt:
    set -e
    just get-vault-key >> key
    ansible-vault encrypt ./secret-file --vault-password-file key
    ansible-vault encrypt ./secret-file2 --vault-password-file key
    rm key

decrypt:
    set -e
    just get-vault-key >> key
    ansible-vault decrypt ./secret-file --vault-password-file key
    ansible-vault decrypt ./secret-file2 --vault-password-file key
    rm key

Problem #

So here’s the story - in real life I started using distrobox when I ran into this and I thought it was an issue with different versions of ansible installed in distrobox and on my desktop. After a few more odd deployment workflows I think I’ve determined that the real problem is that >> appends to a file (see the issue yet?), but if my recipes fail (say I try to encrypt an already encrypted file) then the command exits but what’s left on my disk? key is still there… and the next time I go to run a command I’ll echo the real key to the key file again and if I called encrypt when the first or more files in the list were decrypted then they’ll be encrypted not with my key, but with my key repeated as many times as I ran a command with just get-vault-key >> key before ever removing the key file!


nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
Danger, Will Robinson

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just encrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Encryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
$ANSIBLE_VAULT;1.1;AES256
39616332393662346639633030633766666336323939346138633238626239363733316431333737
6463663763366434376437303335643431663431326135300a636261663636336139323033316232
37666439383131393631333332353731633661396431633834326432363936613331623135666565
6364363039663933620a326563323931643632323031396664646363613636613562366166386439
63653661653037393538356461323239396630643338393231306163343964623866

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Decryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
Danger, Will Robinson

So as you can see, I have secret-file here, and I can encrypt and decrypt just fine. But if I have that >> in the recipe, add a second file, and a failure in a command - well then I’ll start getting confused…

Follow along with the workflow of adding a file that I’ll need encrypted

Only secret-file is in the just recipe here - I add secret-file2 after

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
ERROR! input is not vault encrypted data. /var/home/nic/projects/personal/pype.dev/secret-file is not a vault encrypted file for /var/home/nic/projects/personal/pype.dev/secret-file
error: Recipe `decrypt` failed with exit code 1

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
✗ echo "Good - secret-file is already decrypted! So lets add another file"
Good - secret-file is already decrypted! So lets add another file

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ echo "Foo Bar Bam Baz" > secret-file2                                    

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ echo "Added secret-file2 to my Just recipes"
Added secret-file2 to my Just recipes

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just encrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Encryption successful
Encryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
ERROR! Decryption failed (no vault secrets were found that could decrypt) on /var/home/nic/projects/personal/pype.dev/secret-file for /var/home/nic/projects/personal/pype.dev/secret-file
error: Recipe `decrypt` failed with exit code 1

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
✗ echo "Oh no!"

So what happened?

Recall that my recipes originally had set -e at the top and used >> to cat my secret to a file key. Well, when the fir| decryption recipe failed the key file was left behind. Then when I went to encrypt both of my files the recipe got the key again and made a keyfile like this:

secretKey
secretKey

The encryption recipe encrypted the files with this key, removed the key file, and then the next decryption recipe failed because when it got the key and made the file it looked like

secretKey

So the solution in the end is to remake the key file every time (> instead of >>), or check if it exists, or handle the error when the recipe fails, or just keep going (set +e)

Fin #

And that’s how I lost some secrets in my homelab once, and how I almost lost a second time if I hadn’t noticed the key file in my file tree

SMB with ZFS on Ubuntu

2025-02-20T09:18:35Z

sudo apt-get install -y samba \ then set sharesmb=on \ chown the user \ smbpasswd -a \ then mount with user/password from other machine

Jellyfin container updates for HWE + AMD

2024-12-14T11:27:15Z

I use LSIO Jelyfin container for the easy addon they provide for AMD GPUs but I couldn’t get trickplay to work with HWE…

There was almost NOTHING on the internet about the error, and all the threads were about BSD systems…

Thankfully someone posted on the formum here but the only answer was to literally upgrade stuff in the container…

Someday maybe I’ll build off of LSIO to add this, but until then I shell’d in and homelab’d the hell out of it

THIS IS INSIDE THE CONTAINER - I use Portianer to make it easy


apt update && apt install -y curl gpg

mkdir -p /etc/apt/keyrings

curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor -o /etc/apt/keyrings/rocm.gpg

cat <<EOF | tee /etc/apt/sources.list.d/rocm.sources

Types: deb

URIs: https://repo.radeon.com/rocm/apt/latest

Suites: ubuntu

Components: main

Architectures: amd64

Signed-By: /etc/apt/keyrings/rocm.gpg

EOF

apt update && apt install -y rocm-opencl-runtime

rsync like a pro

2024-12-11T10:52:23Z

I am moving a hefty amount of data to a new ZFS pool due to some corruption and I want to avoid using zfs send/recv for this just to make sure I don’t propagate any corrupted data to my new pool.

I’ve used rsync for simple things before but I needed this to be a little smarter and I wanted to see simple progress without flooding my terminal with a billion filenames.

TLDR #

TLDR: rsync -aHAX --chmod=Da+s --info=progress2 --inplace --exclude='encrypted/docker/frigate-media' /tank/ /harbor/

Explanation #

-aHAX: Preserves attributes (archive mode, hard links, ACLs, extended attributes). –chmod=Da+s: Ensures the setgid bit is applied to directories. –info=progress2: Provides detailed progress information, including overall data transfer stats. –inplace: Writes directly to the destination file, avoiding temporary files (useful for large files). –exclude=‘encrypted/docker/frigate-media’: Excludes the specified path (relative to the /tank root). /tank/ /harbor/: Ensures the contents of /tank are copied directly into /harbor.

D and uninterruptable sleep

2024-12-11T10:48:10Z

Htop #

I recently have been having significant home server issues, and that’s not the point of this - today I learned what D state is when looking at htop.

Apparently this means “uninterruptable sleep” and it’s a dev’s nightmare…

Context #

The issue I was having was that some zfs rollback commands were hung - for hours… I wasn’t sure what was going on, rollbacks should be instant but I figured it was just an artifact of these issues.

Turns out I still don’t know what locked the disks up but I learned why <C>-c did nothing… the more you know

hostnamectl to easily change hostname

2024-12-06T07:25:59Z

hostnamectl is apparently a linux utility for easily changing your hostname in a variety of ways


❯ hostnamectl --help
hostnamectl [OPTIONS...] COMMAND ...

Query or change system hostname.

Commands:
  status                 Show current hostname settings
  hostname [NAME]        Get/set system hostname
  icon-name [NAME]       Get/set icon name for host
  chassis [NAME]         Get/set chassis type for host
  deployment [NAME]      Get/set deployment environment for host
  location [NAME]        Get/set location for host

Options:
  -h --help              Show this help
     --version           Show package version
     --no-ask-password   Do not prompt for password
  -H --host=[USER@]HOST  Operate on remote host
  -M --machine=CONTAINER Operate on local container
     --transient         Only set transient hostname
     --static            Only set static hostname
     --pretty            Only set pretty hostname
     --json=pretty|short|off
                         Generate JSON output

See the hostnamectl(1) man page for details.

I learned there’s transient and static hostnames, so that’s cool…

The thing I needed was hostnamectl --static hostname babyblue-aurora

pretty sweet tool

DNS Broke After Reboot - Ubuntu 22.04

2024-11-22T08:08:40Z

I rebooted by server and DNS broke randomly. I have no idea if it was from a kernel update or what but that’s the issue with Ubuntu I guess…

After much toil and none of the other options working for me (sorry to not have those documented here) this is what got me the vic from this SO Post

sudo mkdir /etc/systemd/resolved.conf.d/ sudo $EDITOR /etc/systemd/resolved.conf.d/dns_servers.conf

Most folks probably are good with google (8.8.8.8) and cloudflare (1.1.1.1)

[Resolve]
DNS=8.8.8.8 1.1.1.1

But I decided to use tailscale

[Resolve]
DNS=100.100.100.100

Then restart systemd-resolved

sudo systemctl restart systemd-resolved

Restart KDE Plasma

2024-11-08T15:53:52Z

Plasma shits the bed a little too often on Fedora for me right now but I finally have a quick fix…


sudo killall plasmashell

kstart plasmashell

docker-remote-add

2024-09-17T06:19:00Z

Add from url??

ADD http://example.com/cars.csv /tmp/cars.csv

Unpack automatically!? (.tar, .tar.gz, .tgz, .bz2, .tbz2, .txz, .zip)

ADD myapp.tar.gz /opt/myapp/

Docker copy and chown

2024-09-17T06:18:09Z

COPY –chown=myuser:mygroup source-file target-file

Interesting IPs between Jellyfin clients and server depending on tailscale and server address

2024-07-12T06:24:57Z

When connecting from my phone to jellyfin I’m seeing some interesting patterns.

Scenarios #

Tailscale IP of phone is listed as local network to jellyfin #

Wifi: off Tailscale: on Use exit node: on LAN access: on Jellyfin: LAN IP

Jellyfin sees 192.168.1.1, my router address

Wifi: off tailscale: on Use exit node: on LAN access: on Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Q: This might be because of traefik somehow

Wifi: off tailsacale: on Use exit node: on LAN access: off Jellyfin: LAN IP

Jellyfin sees the 192.168.1.1

Q: Why did this work even work?

Wifi: off tailsacale: on Use exit node: on LAN access: off Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Wifi: off tailsacale: on Use exit node: off LAN access: off Jellyfin: LAN IP

Jellyfin sees the 192.168.1.1

Q: Why did this work?

Wifi: off tailsacale: on Use exit node: off LAN access: off Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Wifi: on tailsacale: of Use exit node: off LAN access: off Jellyfin: LAN IP (via pihole DNS)

Jellyfin sees the IP of my phone

Upgrading your kernel can F you up… whoops

2024-06-26T10:21:59Z

https://forum.proxmox.com/threads/update-error-with-coral-tpu-drivers.136888/#post-608975 I lost TPU for frigate, this saved me ass

DHCP Restart to Save Ubuntu 22.04 Server Networking

2023-12-31T20:26:50Z

I moved a computer to a remote location for an off-site backup but when it was powered on it wouldn’t show up on any networks. A solution that got me back in was a friend restarting the dhcp client for me:

sudo dhclient -r -v <interface> && sudo dhclient -v <interface>

Refresh Nextcloud Groupfolders after messing around on the filesystem

2023-09-23T12:45:06Z

Exec in as www-data and run ./occ groupfolders:scan folder_id -v (the -v to see what it’s doing)

Convert Word Doc to PDF with Headless Libreoffice

2023-03-09T06:48:38Z

I’ve been using paperless-ngx to manage all my documents, but every once in a while I’ll get a .docx file to deal with…

Turns out Libreoffice has a headless mode a pdf converter built-in!

libreoffice --headless --convert-to pdf /path/to/file.docx --outdir /path/to/output/directory

Note that --outdir is in fact a directory, not the path to a file

i3-Like keyboard mapping in Pop_OS

2023-01-12T05:51:25Z

I was introduced to tiling window managers through i3, which I use heavily on one of my machines. I have switched to Pop_OS! at home though, which has a tiling window mode but the keybindings are not what I’m used to for i3. I wanted to at least navigate workspaces how I’m used to doing (cause I set workspace 3 for communication apps, 1 for my terminal, etc…)

Here’s how I set keybindings for:

<Super> + <number> sends me to that numbered workspace
<Shift> + <Super> + <number> moves the window I’m focused on to workspace number

#!/bin/bash
gsettings set org.gnome.mutter dynamic-workspaces false 
gsettings set org.gnome.desktop.wm.preferences num-workspaces 8 
gsettings set org.gnome.shell.keybindings switch-to-application-1 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-2 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-3 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-4 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-5 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-6 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-7 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-8 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-9 [] 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-1 "['<Super>1']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-2 "['<Super>2']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-3 "['<Super>3']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-4 "['<Super>4']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-5 "['<Super>5']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-6 "['<Super>6']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-7 "['<Super>7']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-8 "['<Super>8']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-9 "['<Super>9']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-10 "['<Super>0']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-1 "['<Super><Shift>1']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-2 "['<Super><Shift>2']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-3 "['<Super><Shift>3']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-4 "['<Super><Shift>4']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-5 "['<Super><Shift>5']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-6 "['<Super><Shift>6']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-7 "['<Super><Shift>7']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-8 "['<Super><Shift>8']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-9 "['<Super><Shift>9']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-10 "['<Super><Shift>0']"

Use non-standard named ssh keys with github

2023-01-03T08:34:50Z

I was getting (publickey denied) when trying to push to GH using ssh. When I tested the connection I saw that a bunch of keys in ``~/.ssh/ were being attempted

✗ ssh git@github.com -vv

...

debug1: Will attempt key: /home/nic/.ssh/id_rsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/nic/.ssh/id_xmss 
debug1: Will attempt key: /home/nic/.ssh/id_dsa 

...

debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

None of those were the key I setup with GH. So I added an entry into ~/.ssh/config:

Host
github.com
User git
Port 22
Hostname github.com
IdentityFile ~/.ssh/my_custom_github_key
TCPKeepAlive yes
IdentitiesOnly yes