Quick tips and learnings Archive

Agents can miss obvious things too

2026-04-03T14:39:51Z

I am working with MiniMax M.2 to vibe up some webapp ideas for Nexus and I just spent many minutes not having the app render correctly because the agent mounted the wrong directory into the docker compose stack. I noticed it about 5 seconds after opening the file, but wasted nearly a half hour thinking it was a docker issue in distrobox…

Danger

Clankers are still just clankers man

Prettier Docker ‘ps’ Command

2025-12-29T05:33:05Z

The docker ps command is very useful, but I hate reading the output. Turns out, you can make it prettier:

docker ps

–format “table” is implied with the command.


❯ docker ps --format "table" | grep can
9b716a7d1ab0   postgres:17                                               "docker-entrypoint.s…"   13 hours ago    Up 13 hours (healthy)            0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp                                                cannalyzer-db-1
f7708ea0c112   adminer                                                   "entrypoint.sh docke…"   13 hours ago    Up 13 hours                      0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp                                                cannalyzer-adminer-1
93cd67719ed4   frontend:latest                                           "/docker-entrypoint.…"   13 hours ago    Up 13 hours                      0.0.0.0:5173->80/tcp, [::]:5173->80/tcp                                                    cannalyzer-frontend-1
f517625fca98   traefik:3.0                                               "/entrypoint.sh --pr…"   13 hours ago    Up 13 hours                      0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp           cannalyzer-proxy-1
f4daa036216e   schickling/mailcatcher                                    "sh -c 'mailcatcher …"   13 hours ago    Up 13 hours                      0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp   cannalyzer-mailcatcher-1

But you can pass a template string to the --format option, like so:

docker ps --format "table {{.Names}}"


✗ docker ps --format "table {{.Names}}" | grep can
cannalyzer-db-1
cannalyzer-adminer-1
cannalyzer-frontend-1
cannalyzer-proxy-1
cannalyzer-mailcatcher-1

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"


❯ docker ps --format "table {{.Names}}\t{{.Ports}}" | grep can
cannalyzer-db-1             0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp
cannalyzer-adminer-1        0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp
cannalyzer-frontend-1       0.0.0.0:5173->80/tcp, [::]:5173->80/tcp
cannalyzer-proxy-1          0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp
cannalyzer-mailcatcher-1    0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp

Picture #

I noticed my template isn’t folding the codeblocks in a way that actually makes this post look like I’m lying!

COLUMNS Env Var For Nicer Screenshots

2025-12-08T05:16:55Z

I was working on UV Run + PEP 723 Is A Match Made In Heaven and I wanted to grab a screenshot of a TUI, but my terminal is full screen and the TUI was then super stretched… Nice for working and using it, but not nice for a screenshot going into a blog post… see the difference export COLUMNS=120 does here!

COLUMNS=120 mytui.py

mytui.py

Check SWAP in htop

2025-11-15T06:52:14Z

I struggle with maxxing swap on all my machines, I am sure it’s a neovim config that I’ve propagated everywhere and I’m too lazy to figure it out. Instead I just swapoff and swapon kind of often… but this morning I had to know how to see what’s using all that swap….

The way to see SWAP usage by process in htop is:

open htop
Slam F2 (f if you’re just in old school top)
see the screenshot, find M_SWAP
move M_SWAP to Active Columns

Credit to this superuser post

Use Jellyfin Tags For Content Moderation

2025-10-08T10:21:01Z

Jellyfin brings in ratings and allows you to set custom ratings for shows, which is nice for things like “block anything worse than TV-14 for my kid’s account”. But what if my firend’s kids can watch shows I don’t want my kids watching? Turns out tags exist and are great!

Tag the content #

First is to actually tag the content - in the UI it’s simple.

Login with admin
Manage Metadata
Click the content you want to tag and find the Tags section
Add your tag… for example blocked:girls

Filter on the account #

Go to Dashboard -> Users -> Library Access -> Parental Control -> Block items with tags

Sending SMS with SMTP

2025-10-04T09:38:58Z

I’m cooking something up over here that will include SMS message notifications as a feature and I’ve been doing some reading on different providers…

Many specialize in bulk SMS messaging, like for campaigns and promotions. So this pricing model doesn’t work for me.

The providers that do quick single SMS messages are often $0.005 to $.04 cents per message which isn’t insanity but I’m not exactly backed by anyone…

Well today I learned that you can actually send SMS messages via SMTP! I wouldn’t bank on total reliability here but for testing and initial roll out of what I want I think this will be perfect.

And because I’m just that kind of guy, here’s some code you can uv run smtp2sms.py and get yourself a handy little text. I’m going to build a simpler module but you can get a gmail and gmail app password within about 10 minutes. see here

# /// script
# requires-python = ">=3.10"
# dependencies = [
# ]
# ///
import os, smtplib
from email.mime.text import MIMEText

SMTP_APP_USER=os.environ.get("SMTP_APP_USER") # me@gmail.com
SMTP_APP_PASSWORD=os.environ.get("SMTP_APP_PASSWORD")
SMTP_SERVER=os.environ.get("SMTP_SERVER")  # smtp.gmail.ch

# NOTE: I haven't verified all the carrier domains - this was provided by ChatGPT

carriers = {
    "att": {
        "sms": "txt.att.net",
        "mms": "mms.att.net"
    },
    "tmobile": {
        "sms": "tmomail.net",
        "mms": "tmomail.net"
    },
    "verizon": {
        # "sms": "vtext.com", # not reliable
        "sms": "vzwpix.com",
        "mms": "vzwpix.com"
    },
    "sprint": {
        "sms": "messaging.sprintpcs.com",
        "mms": "pm.sprint.com"
    },
    "googlefi": {
        "sms": "msg.fi.google.com",
        "mms": "msg.fi.google.com"
    },
    "uscellular": {
        "sms": "email.uscc.net",
        "mms": "mms.uscc.net"
    },
    "boost": {
        "sms": "sms.myboostmobile.com",
        "mms": "myboostmobile.com"
    },
    "cricket": {
        "sms": "sms.cricketwireless.net",
        "mms": "mms.cricketwireless.net"
    },
    "metropcs": {
        "sms": "mymetropcs.com",
        "mms": "mymetropcs.com"
    },
    "virgin": {
        "sms": "vmobl.com",
        "mms": "vmobl.com"
    }
}
contacts = {
    "me": {"carrier": "verizon", "number": "1234567890"},
}
msg = MIMEText("Hello from a python script? What?!")
msg["From"] = SMTP_APP_USER

SEND_TO = "me"

# Complex for my future use cases - deal with it
msg["To"] = f"{contacts[SEND_TO]['number']}@{carriers[contacts[SEND_TO]['carrier']]['sms']}"

with smtplib.SMTP(SMTP_SERVER, 587) as email_handler:
    email_handler.starttls()
    email_handler.login(SMTP_APP_USER, SMTP_APP_PASSWORD)
    email_handler.sendmail(SMTP_APP_USER, [msg["To"]], msg.as_string())

Docker Compose Build From Git Repo

2025-09-12T08:45:20Z

You can reference a git repo in compose’s “build” argument for a service as of this PR

Zsh alias for uv scripts

2025-09-08T06:33:13Z

It’s apparently advantageous for uv to have a specific shebang set that uv init doesn’t add, no problem, we can have a zsh alias for it

alias add-shebang='f(){ 
  if [[ ! -s $1 ]]; then 
    echo "#!/usr/bin/env -S uv run --script" > "$1"; 
  elif head -n1 "$1" | grep -q "^#!"; then 
    echo "Shebang already present in $1"; 
  else 
    sed -i "1i #!/usr/bin/env -S uv run --script" "$1"; 
  fi
}; f'

Then if you have a python script from uv init --script you can add-shebang myscript.py to add the shebang.

Pair AceGamer Wireless PS4 Controller With PC

2025-08-14T06:51:12Z

I bought a $20 PS4 knock-off controller recently as my PS4 controllers are 12 years old (to me!) and I bought them used before that… the instructions for a $20 controller are obviously bad but a godsend redittor discovered the hidden secret to pair this device with my computer! Instructions (for ps4) say to plug it in, hit the home button, and you’re paired. Now, that didn’t work on PC obviously, but it ALSO didn’t work on my PS4… Unfortunte! It works wired just fine, but turns out the magic to send the thing into traditional Bluetooth Pairing Mode is to hold the home and share buttons down - wait for a rapid few blinks - then you’ll see the controller in your bluetooth pairing menu!

Migrate Github Issues to Kanboard

2025-08-07T14:43:27Z

I am working on shifting everything, or as much as I reasonably can, related to dev for myself to on-prem, including git and CI/CD. So for Quadtask I had a bunch of Github issues that I wanted to migrate to my kanboard instance, gpt-5 did a bang-up job on this script

It gets each issue, creates a tag for each label, creates the kanboard ticket, then closes the github issue


#!/bin/bash

# GitHub to Kanboard Issue Migration Script
# Uses GitHub CLI (gh) and curl to migrate issues

# Exit on error
set -e

# Check required environment variables
for var in KANBOARD_URL KANBOARD_TOKEN GH_TOKEN QUADTASK_KANBOARD_PROJECT_ID; do
    if [ -z "${!var}" ]; then
        echo "Error: $var is not set"
        exit 1
    fi
done

# GitHub repository (default: pypeaday/quadtask)
GITHUB_REPO=${GITHUB_REPO:-"pypeaday/quadtask"}

# Kanboard API endpoint
KANBOARD_API="$KANBOARD_URL/jsonrpc.php"

# Get all open issues from GitHub
echo "Fetching open issues from $GITHUB_REPO..."
ISSUES_JSON=$(gh issue list --repo "$GITHUB_REPO" --state open --limit 1000 --json number,title,body,labels)

# Count issues
ISSUE_COUNT=$(echo "$ISSUES_JSON" | jq '. | length')
echo "Found $ISSUE_COUNT open issues"

# Process each issue
echo "$ISSUES_JSON" | jq -c '.[]' | while read -r issue; do
    # Extract issue details
    NUMBER=$(echo "$issue" | jq -r '.number')
    TITLE=$(echo "$issue" | jq -r '.title')
    BODY=$(echo "$issue" | jq -r '.body // ""')
    # Extract label names as a JSON array
    LABELS_JSON=$(echo "$issue" | jq -c '[.labels[]?.name] // []')
    
    # Prepare Kanboard task data
    REQUEST_DATA=$(jq -n \
        --arg method "createTask" \
        --argjson id 1 \
        --arg jsonrpc "2.0" \
        --arg title "$TITLE" \
        --arg description "$BODY" \
        --arg project_id "$QUADTASK_KANBOARD_PROJECT_ID" \
        '{
            "jsonrpc": $jsonrpc,
            "method": $method,
            "id": $id,
            "params": {
                "title": $title,
                "description": $description,
                "project_id": $project_id
            }
        }')
    
    # Create task in Kanboard
    echo "Creating task: $TITLE"
    RESPONSE=$(curl -s -X POST \
        -H "Content-Type: application/json" \
        -u "jsonrpc:$KANBOARD_TOKEN" \
        -d "$REQUEST_DATA" \
        "$KANBOARD_API")
    
    # Check for errors
    if echo "$RESPONSE" | jq -e '.error' > /dev/null; then
        echo "Error creating task: $(echo "$RESPONSE" | jq -r '.error.message')"
    else
        TASK_ID=$(echo "$RESPONSE" | jq -r '.result')
        echo "Created task ID: $TASK_ID"

        # If there are labels, set them as Kanboard tags on the task
        if [ "$(echo "$LABELS_JSON" | jq 'length')" -gt 0 ]; then
            echo "Setting tags on task $TASK_ID: $(echo "$LABELS_JSON" | jq -r 'join(", ")')"
            TAGS_REQUEST=$(jq -n \
                --arg jsonrpc "2.0" \
                --arg method "setTaskTags" \
                --argjson id 2 \
                --arg project_id "$QUADTASK_KANBOARD_PROJECT_ID" \
                --arg task_id "$TASK_ID" \
                --argjson tags "$LABELS_JSON" \
                '{
                    jsonrpc: $jsonrpc,
                    method: $method,
                    id: $id,
                    params: {
                        project_id: $project_id,
                        task_id: $task_id,
                        tags: $tags
                    }
                }')

            TAGS_RESPONSE=$(curl -s -X POST \
                -H "Content-Type: application/json" \
                -u "jsonrpc:$KANBOARD_TOKEN" \
                -d "$TAGS_REQUEST" \
                "$KANBOARD_API")

            if echo "$TAGS_RESPONSE" | jq -e '.error' > /dev/null; then
                echo "Warning: Failed to set tags: $(echo "$TAGS_RESPONSE" | jq -r '.error.message')"
            else
                echo "Tags set successfully"
            fi
        fi

        # Close the corresponding GitHub issue to avoid duplicates
        TASK_URL="${KANBOARD_URL}/?controller=TaskViewController&action=show&task_id=${TASK_ID}"
        echo "Closing GitHub issue #$NUMBER with comment linking to Kanboard task..."
        if gh issue close "$NUMBER" --repo "$GITHUB_REPO" --comment "Migrated to Kanboard task $TASK_ID: $TASK_URL" >/dev/null; then
            echo "Closed GitHub issue #$NUMBER"
        else
            echo "Warning: Failed to close GitHub issue #$NUMBER"
        fi
    fi
    
    echo "----------------------------------------"
done

echo "Migration completed"

Add yourself to www-data to view your Nextcloud data on the filesystem

2025-07-17T07:38:09Z

I am working on using gotify cli for notifying me of nextcloud uploads and I’ll start by running the gotify cli as my user so I’ll need to be able to see the directory in the filesystem that Nextcloud stores the data in.

That folder on disk is owned by www-data (see Nextcloud permissions with ZFS and Ansible-NAS

Turns out it’s my computer and I can do what I want so I just add myself to the www-data group instead of running the gotify cli as root

Bonus

You can remove users from groups with gpasswd -d username groupname

Lua type hinting - undefined global

2025-07-11T06:08:39Z

I have tons of linting errors in my nvim config, partly because I use Lazyvim plus a set of poorly written customizations that have traveled with me for a while. I try to re-work things and right now I’ve been revisiting my developer setup. Well the LSP rrors as I’ve gone through my neovim config are ticking me off - and this was the biggest one “Undefined global: vim”. I saw vim.fn everywhere but the LSP hated it… turns out vim is a given global and you can help your lsp with this:

---@diagnostic disable: undefined-global
-- vim is a global in Neovim Lua scripts

For example:


---@diagnostic disable: undefined-global
-- vim is a global in Neovim Lua scripts
return {
  "pypeaday",
  dir = vim.fn.stdpath("config") .. "/lua/pypeaday",
  config = function()
    local daily = require("pypeaday.daily")
    vim.api.nvim_create_user_command("DailyNote", daily.check_and_open_daily_note, {})
    vim.api.nvim_create_user_command("Daily", daily.check_and_open_daily_note, {})
    vim.api.nvim_create_user_command("BackLinks", daily.find_backlinks, {})
    vim.keymap.set("n", "<leader>dn", daily.check_and_open_daily_note, { desc = "Open daily note" })
    vim.keymap.set("n", "<leader>df", daily.find_daily_files, { desc = "Find daily files" })
    vim.keymap.set("n", "<leader>dl", daily.find_backlinks, { desc = "Find backlinks" })
  end,
}

Nostr hex key

2025-07-02T08:43:32Z

you can hexify your Nostr key when you need to sometimes which I needed to for Posts tagged: postiz (link with a thought I’m sure I have on postiz)


# /// script
# requires-python = ">=3.8"
# dependencies = [
#     "bech32",
# ]
# ///
#
import os

import bech32

# Set your nsec... key to NOSTR_PRIVATE_KEY or bake it in...
nsec_key = os.environ.get("NOSTR_PRIVATE_KEY")


def decode_nsec(nsec_key):
    hrp, data = bech32.bech32_decode(nsec_key)
    if hrp != "nsec":
        raise ValueError("Not a valid nsec key")
    decoded = bech32.convertbits(data, 5, 8, False)
    return bytes(decoded).hex()


# Example usage
raw_hex = decode_nsec(nsec_key)
print(f"Raw hex private key: {raw_hex}")

💭 slash pages

2025-06-12T11:41:43Z

Here’s my thought on slash pages

Waylon’s mentioned slash pages before, but as I work on building my blog again and writing more regularly thanks to “thoughts” etc. I’d like to start putting together some meaningful slash pages for myself… I’m not sure what to put out there yet but this thought can serve as a todo for me

Note

This is one of [[ my-thoughts ]]. I picked this up from Waylon Walker(https://thoughts.waylonwalker.com). It’s a short note that I make about someone else’s content online. Learn more about the process Thoughts

Double check your CIDR blocks!

2025-06-07T07:27:31Z

tl;dr

Copy paste - don’t rely on typing correctly…

The Issue

I use tailscale at home and generally love it. One thing I use it for is whitelisting - so I have some apps publically available and my homelab ones have DNS with SSL certs by LE but I whitelist to only my private IPs so that I can use https at home basically. However I have had a few issues with a couple apps, and I kind of chaulked it up to “network protocol this” “need to figure out that header that” but ultimately I was being lazy and the issues weren’t catastrophic… it was things like:

couldn’t get to my NVR by DNS, so I’d have to use tailscale_ip:port
couldn’t login with my user in jellyfin cause I lock me down to LAN only since I dont’ use a password
gotify wouldn’t connect using DNS

These things aren’t huge but they are annoying and today I decided to figure it out.

Typo! #

I’m not an expert yet but I think the main problem was the CIDR block I was using in my traefik config.

This’ll shock some of you but 10.64.0.0/10 and 100.64.0.0/10 are not the same blocks!

Check your CIDRs

It’s always DNS, and even when it’s not, it’s still probably networking

Backups are dope

2025-05-27T19:42:27Z

I accidently chown’d -R an app directory and it totally screwed up the database folder. Luckily I zfs replicate my docker volumes to another drive even on the same host, so a quick [rsync] from /harbor/encrypted/docker/manyfold to /tank/encrypted/docker/manyfold got me back up and running since I hadn’t replicated the messed up permission set yet

Q: How to link to my rsync like a pro post?

A: Wikilinks <a href="/rsync-like-a-pro/" class="wikilink" data-title="rsync like a pro" data-description="I am moving a hefty amount of data to a new ZFS pool due to some corruption and I want to avoid using for this just to make sure I don't propagate any..." data-date="2024-12-11">rsync like a pro</a> rsync like a pro

Aurora U-Blue ZRAM Config

2025-05-27T13:51:27Z

I keep running out of space with my swap getting maxxed out… I don’t know why but U-Blue uses Zram already and apparently I can easily override the defaults:

in usr/lib/systemd/zram-generator.conf


# This config file enables a /dev/zram0 device with the default settings:
# — size — same as available RAM or 8GB, whichever is less
# — compression — most likely lzo-rle
#
# To disable, uninstall zram-generator-defaults or create empty
# /etc/systemd/zram-generator.conf file.
[zram0]
zram-size = min(ram, 8192)

And so I just made my own file in /etc/systemd/zram-generator.conf:


[zram0]
zram-size = min(ram, 16384)

Pandas.DataFrame info always answers and is sometimes right

2025-05-26T09:08:08Z

TIL

Today I learned that the .info() of a pandas.DataFrame will always give you an answer, but it is wildly difficult to know how accurate it is, because it depends on the underlying data types. If everything is a NumPy Dtype, then the result you get is the amount of memory NumPy is using under the hood, which is very accurate. But as soon as you have something like a Python string in the data frame, now pandas will tell you how much memory the NumPy data types are using, and it will give you some quick estimate of how much memory space the Python string types are using because numpy is just storing a pointer to that data in memory, but this is far less accurate because it doesn’t actually know what the strings are.

Memory Usage Deep or True #

To get an honest answer about a DataFrame’s memory usage, you need to pass memory_usage='deep' to the .info() method.

# This is the default
In [50]: df.info(memory_usage=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 372.0+ bytes

Notice that the memory usage has a + in it? You can force a deeper analysis by passing memory_usage='deep'

In [51]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 792.0 bytes

SearXNG

2025-05-25T08:33:10Z

I heard about SearXNG on a couple podcasts and saw it trending on GitHub several times before I finally decided to stand it up. I used it transparently when trying out khoj, a self-hosted AI LLM agent playground kind of a thing, but I’ve also been messing around with Open-WebUI to have a self-hosted ChatGPT-like experience. I don’t know if SearchXNG used to be harder to set up, but it was pretty simple with a Docker Compose up and a couple configuration options given my personal homelab setup. Using it feels very nice.

services:
  redis:
    container_name: redis
    image: docker.io/valkey/valkey:8-alpine
    command: valkey-server --save 30 1 --loglevel warning
    restart: unless-stopped
    volumes:
      - ./data/redis:/data
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

  searxng:
    container_name: searxng
    image: docker.io/searxng/searxng:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./data/searxng:/etc/searxng:rw
    env_file: .env
    environment:
      # - SEARXNG_BASE_URL=https://${SEARXNG_HOSTNAME:-localhost}/
      - UWSGI_WORKERS=${SEARXNG_UWSGI_WORKERS:-4}
      - UWSGI_THREADS=${SEARXNG_UWSGI_THREADS:-4}
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

I love the self-hosted aspect. I love not seeing any ads on my search results. I did a few searches where I know what results to expect and it did okay. It is good right now at filtering out garbage in my results.

So I look forward to tweaking it and using it as a search backend with open web UI. Next on my list is having enough resources to run Ollama and a stable diffusion generator at the same time and have image generation working through open web UI.

Grafana + Gotify for push notifications

2025-04-07T10:12:51Z

It’s easy to configure push notifications through Gotify from Grafana by setting up a generic Webhook Alerting Contact Point…

The parameters are:

HTTP Method: POST
URL: http:///message
Authentication is done with an app token generated from Gotify, and you put it in Authorization Header: Credentials in the Grafana configuration

See image

Monolith - download and save site as html

2025-03-16T16:03:31Z

TIL: Monolith #

I needed to save a website earlier and this was just what I needed…

https://github.com/Y2Z/monolith

SMB with ZFS on Ubuntu

2025-02-20T09:18:35Z

sudo apt-get install -y samba \ then set sharesmb=on \ chown the user \ smbpasswd -a \ then mount with user/password from other machine

Using my own domain on nostr

2025-01-24T06:28:19Z

After generating a keypair on a nostr app (I started with primal) I copied my public key to /.well-known/nostr.json and put that at the root of my site at https://pype.dev.

The file looks like this:


{
  “names”: {
    “nic”: “npub1q3fsec2vcv99v80ga72dlv90qwkqmuxqcr6mdyumcmpkgudlhrespyurfj”
  }
}

So now I can use nic@pype.dev as on nostr because the client will verify the pubkey associated with my profile with my domain!

big upgrade from @primal.net

Deployments are not StatefulSets

2025-01-15T18:07:57Z

I was debugging some ArgoCD stuff earlier today and I love using k9s to explore my k8s resources.

The TLDR is that I put some bad env vars in the global values for my ArgoCD pods, this caused the webUI to hang. An easy way to push things through is to open k9s, view the Deployments, and edit the env vars there so the pods can be in the right state to restart and pickup the changes (you’ve obviously fixed the values file already right?)

Well the Notification Controller was the pod hanging, and I couldn’t find it in the Deployments view! This blew my mind… until I realized that it was a Stateful Set.

So, this is not a comparison of the 2 - this is just a note to say that all I had to do was find the StatefulSets in k9s and do the same workflow for editing what I needed to edit to get the pods into workable state while working on the values file…

docker context (and an issue to question your sanity)

2024-12-19T05:59:30Z

docker contexts are great, would recommend putting them in your prompt though (via starship or something else)… here’s why

I like to manage my containers remotely - I have a nice development setup on my desktop and I try to keep my server as bare-bones as possible. For a while I’ve been using ansible which makes it easy to manage configuration etc on other machines. But I recently learned about docker contexts and I’m planning to scale down my homelab management to just docker-compose stacks rather than a bunch of super complicated ansible playbooks

So, setting up a context is easy - it’s basically an ssh connection to another machine!

docker context create koober --docker "host=ssh://nic@koober"

koober is one of my dev machines and my ~/.ssh/config is setup such that I can ssh nic@koober, this makes the context work really seamlessly.

So there’s the default context (the machine you’re on) and now I have koober

To use it you run docker context use koober

And then to check we can ls the contexts

docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT               ERROR
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   
koober *                                             ssh://nic@koober

Notice the * - that indicates that’s our current context.

Trouble #

Now here’s where things get hairy… you’ve gotta be super-aware of what context you’re using. I have an indicator in my starship prompt that shows the current context, but since I’m new to using them I kind of didn’t notice it until I ran into this issue…

I’m working on a python application in docker but was not able to execute the entrypoint even though I KNEW the file was there… let’s take a look

Example #

Here’s a minimal hello world applycation in docker to illustrate the issue

# main.py
print("hello world")

# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Run the main.py script as the container's main process
CMD ["python", "main.py"]

services:
  hello-world:
    build: .
    volumes:
      - .:/app

Notice volume mounting in my project directory . to /app as a common practice to develop inside the container

Here’s where I started to question my sanity…

✗ docker compose up                                            
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | python: can't open file '/app/main.py': [Errno 2] No such file or directory
hello-world-1 exited with code 2

python can't open file? hmm… Let’s take a look at the image

First let’s make sure we get the image name right

✗ docker container ls -a              
CONTAINER ID   IMAGE                                           COMMAND                  CREATED          STATUS                       PORTS                                                                                  NAMES
3d7285fc39e2   docker-context-example-hello-world              "python main.py"         10 minutes ago   Exited (2) 44 seconds ago                                                                                           docker-context-example-hello-world-1

Now we can docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world

❯ docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world  
root@ee46d0e22de8:/app# python main.py
Hello, World!
root@ee46d0e22de8:/app#

WHAT THE HECK??

What happened… #

What happened turns out to be pretty simple once we realize I’m using contexts…

koboer is a remote context, the docker run and docker compose up commands are interacting with the docker socket on that machine.

So if I compose up the stack notice that there’s a volume bind mount in there - well those do not work with contexts (or at least I’m not aware of hose to make it work) and so the /app directory was getting blown away essentially with an empty overlay…

But when running with just docker run with no volume mount, the code was copied in during the build and is right where we expect it…

Let’s prove it #

❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@903f591c0384:/app# python main.py
python: can't open file '/app/main.py': [Errno 2] No such file or directory
root@903f591c0384:/app#

Adding a `-v .:/app` to match the compose file, we get the same error...

If we switch to the default context we are back up and running as expected

✗ docker context use default
default
Current context is now "default"

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker compose up         
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | Hello, World!
hello-world-1 exited with code 0

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@4045b6aa8883:/app# python main.py
Hello, World!
root@4045b6aa8883:/app#

Successful runs on both accounts with the volume mount

TLDR #

Context is king

Add a healthcheck to your FastAPI app

2024-12-15T15:41:36Z

I’m building a few FastAPI apps to throw in docker and run on my homelab… I wanted to add healthchecks and here’s a simple way to do it

Make sure to install curl in the dockerfile (near the top for effeciency)

# Install curl with minimal dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Then I recommend making compose files even for single image deployments

services:
  app:
    build: .
    volumes:
      - type: bind
        source: .
        target: /app
    environment:
      - PYTHONPATH=/app
      - DOCKER_ENV=true
      - UV_VIRTUALENV=/opt/app-env
    user: "1000:1000"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

Then finally you’ll need a /health endpoint


@app.get("/health", response_class=HTMLResponse)
async def health_check():
    """
    A health check endpoint that returns a status message.
    """
    return "<html><body><h1>Service is healthy</h1></body></html>"

statically.io to help me out

2024-12-14T13:35:31Z

statically.io offers a CDN that I’m going to try to lean into for images in my blog. My hope is that the builds get and stay fast, and that page loads are fast too.

Example:


<img src="https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/skimpy-zsh.png" alt="Skimpy ZSH" title="A fancy data summary in the shell" />

Jellyfin container updates for HWE + AMD

2024-12-14T11:27:15Z

I use LSIO Jelyfin container for the easy addon they provide for AMD GPUs but I couldn’t get trickplay to work with HWE…

There was almost NOTHING on the internet about the error, and all the threads were about BSD systems…

Thankfully someone posted on the formum here but the only answer was to literally upgrade stuff in the container…

Someday maybe I’ll build off of LSIO to add this, but until then I shell’d in and homelab’d the hell out of it

THIS IS INSIDE THE CONTAINER - I use Portianer to make it easy


apt update && apt install -y curl gpg

mkdir -p /etc/apt/keyrings

curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor -o /etc/apt/keyrings/rocm.gpg

cat <<EOF | tee /etc/apt/sources.list.d/rocm.sources

Types: deb

URIs: https://repo.radeon.com/rocm/apt/latest

Suites: ubuntu

Components: main

Architectures: amd64

Signed-By: /etc/apt/keyrings/rocm.gpg

EOF

apt update && apt install -y rocm-opencl-runtime

rsync like a pro

2024-12-11T10:52:23Z

I am moving a hefty amount of data to a new ZFS pool due to some corruption and I want to avoid using zfs send/recv for this just to make sure I don’t propagate any corrupted data to my new pool.

I’ve used rsync for simple things before but I needed this to be a little smarter and I wanted to see simple progress without flooding my terminal with a billion filenames.

TLDR #

TLDR: rsync -aHAX --chmod=Da+s --info=progress2 --inplace --exclude='encrypted/docker/frigate-media' /tank/ /harbor/

Explanation #

-aHAX: Preserves attributes (archive mode, hard links, ACLs, extended attributes). –chmod=Da+s: Ensures the setgid bit is applied to directories. –info=progress2: Provides detailed progress information, including overall data transfer stats. –inplace: Writes directly to the destination file, avoiding temporary files (useful for large files). –exclude=‘encrypted/docker/frigate-media’: Excludes the specified path (relative to the /tank root). /tank/ /harbor/: Ensures the contents of /tank are copied directly into /harbor.

D and uninterruptable sleep

2024-12-11T10:48:10Z

Htop #

I recently have been having significant home server issues, and that’s not the point of this - today I learned what D state is when looking at htop.

Apparently this means “uninterruptable sleep” and it’s a dev’s nightmare…

Context #

The issue I was having was that some zfs rollback commands were hung - for hours… I wasn’t sure what was going on, rollbacks should be instant but I figured it was just an artifact of these issues.

Turns out I still don’t know what locked the disks up but I learned why <C>-c did nothing… the more you know

hostnamectl to easily change hostname

2024-12-06T07:25:59Z

hostnamectl is apparently a linux utility for easily changing your hostname in a variety of ways


❯ hostnamectl --help
hostnamectl [OPTIONS...] COMMAND ...

Query or change system hostname.

Commands:
  status                 Show current hostname settings
  hostname [NAME]        Get/set system hostname
  icon-name [NAME]       Get/set icon name for host
  chassis [NAME]         Get/set chassis type for host
  deployment [NAME]      Get/set deployment environment for host
  location [NAME]        Get/set location for host

Options:
  -h --help              Show this help
     --version           Show package version
     --no-ask-password   Do not prompt for password
  -H --host=[USER@]HOST  Operate on remote host
  -M --machine=CONTAINER Operate on local container
     --transient         Only set transient hostname
     --static            Only set static hostname
     --pretty            Only set pretty hostname
     --json=pretty|short|off
                         Generate JSON output

See the hostnamectl(1) man page for details.

I learned there’s transient and static hostnames, so that’s cool…

The thing I needed was hostnamectl --static hostname babyblue-aurora

pretty sweet tool

Stylus for custom webpage themes

2024-11-27T06:07:39Z

the Logos web app is DISGUSTINGY bright/white - enough to actually ruin your morning…

Thankfully there’s an extension called stylus and some kind folks in the Logos community created a nice dark theme here. You simply install the extension, then go to the userstyles link <– and click install. It only changes themes for the sites configured - in this case app.logos.com

TODO: image

DNS Broke After Reboot - Ubuntu 22.04

2024-11-22T08:08:40Z

I rebooted by server and DNS broke randomly. I have no idea if it was from a kernel update or what but that’s the issue with Ubuntu I guess…

After much toil and none of the other options working for me (sorry to not have those documented here) this is what got me the vic from this SO Post

sudo mkdir /etc/systemd/resolved.conf.d/ sudo $EDITOR /etc/systemd/resolved.conf.d/dns_servers.conf

Most folks probably are good with google (8.8.8.8) and cloudflare (1.1.1.1)

[Resolve]
DNS=8.8.8.8 1.1.1.1

But I decided to use tailscale

[Resolve]
DNS=100.100.100.100

Then restart systemd-resolved

sudo systemctl restart systemd-resolved

Restart KDE Plasma

2024-11-08T15:53:52Z

Plasma shits the bed a little too often on Fedora for me right now but I finally have a quick fix…


sudo killall plasmashell

kstart plasmashell

OPNSense Bootstrap Recovery

2024-11-07T08:40:19Z

enabling DHCP WAN port (dhclient <iface>)- running the bootstrap script - sh /usr/local/sbin/opnsense-bootstrap

docker-remote-add

2024-09-17T06:19:00Z

Add from url??

ADD http://example.com/cars.csv /tmp/cars.csv

Unpack automatically!? (.tar, .tar.gz, .tgz, .bz2, .tbz2, .txz, .zip)

ADD myapp.tar.gz /opt/myapp/

Docker copy and chown

2024-09-17T06:18:09Z

COPY –chown=myuser:mygroup source-file target-file

Upgrading your kernel can F you up… whoops

2024-06-26T10:21:59Z

https://forum.proxmox.com/threads/update-error-with-coral-tpu-drivers.136888/#post-608975 I lost TPU for frigate, this saved me ass

Fonts in VS C**e

2024-05-20T06:44:07Z

Jet Brains has to be specified ‘JetBrainsMono Nerd Font Mono’

Customize K9s

2024-05-06T11:57:07Z

To customize k9s use the skins from catppuccin or the ones k9s supplies

OUT="${XDG_CONFIG_HOME:-$HOME/.config}/k9s/skins"
mkdir -p "$OUT"
curl -L https://github.com/catppuccin/k9s/archive/main.tar.gz | tar xz -C "$OUT" --strip-components=2 k9s-main/dist

Then edit your k9s config

# ~/.config/k9s/config.yml
k9s:
  ui:
    skin: catppuccin-mocha
    # ...or another flavor:
    # skin: catppuccin-macchiato
    # skin: catppuccin-frappe
    # skin: catppuccin-latte

    # ...or the transparent variants:
    # skin: catppuccin-mocha-transparent
    # skin: catppuccin-macchiato-transparent
    # skin: catppuccin-frappe-transparent
    # skin: catppuccin-latte-transparent

Other k9s skins are available here

Git fetch failing - check your config

2024-04-18T08:32:33Z

I started deploying a website to Cloudflare on a branch called pages. Similar to one of the GH Pages deployment patterns. But when my CI was pushing the branch I couldn’t see it locally…

git fetch -a wasn’t pulling any new branches, and git branch -a was only showing my development and main branches at the remote… so what gives?

I checked my git config, and to this moment I have no idea how this happened but check out my fetch config:

git config --get remote.origin.fetch
+refs/tags/*:refs/tags/*

So to fix this:

git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'

Now git fetch -a works again

> git fetch -a

From github.com:DigitalHarbor7/DigitalHarbor
   357a28a..969b027  develop    -> origin/develop
   c052ac9..6d40210  main       -> origin/main
 * [new branch]      pages      -> origin/pages

Git repo specific SSH Key!

2024-01-04T15:42:15Z

git config –add –local core.sshCommand ‘ssh -i «<PATH_TO_SSH_KEY»>’

DHCP Restart to Save Ubuntu 22.04 Server Networking

2023-12-31T20:26:50Z

I moved a computer to a remote location for an off-site backup but when it was powered on it wouldn’t show up on any networks. A solution that got me back in was a friend restarting the dhcp client for me:

sudo dhclient -r -v <interface> && sudo dhclient -v <interface>

Simple Port Forwarding OPNSense

2023-10-17T10:26:34Z

https://forum.opnsense.org/index.php?topic=8783.0

Refresh Nextcloud Groupfolders after messing around on the filesystem

2023-09-23T12:45:06Z

Exec in as www-data and run ./occ groupfolders:scan folder_id -v (the -v to see what it’s doing)

lsof to find what’s using your filesystem

2023-04-09T13:32:38Z

lsof | grep /tank/nas shows me what is using my nas at any time!

Changing ZFS key for child datasets of encrypted dataset after migration

2023-04-08T20:12:22Z

➜ pihole sudo zfs load-key -L file:///path/to/.zfs.tank.key tank/encrypted/vms/arch-sandbox ➜ pihole sudo zfs change-key -o keylocation=file:///path/to/.zfs.tank.key -o keyformat=raw tank/encrypted/vms/arch-sandbox Need to load-key for each individual dataset, then change key location to be a file instead of the prompt

Convert Word Doc to PDF with Headless Libreoffice

2023-03-09T06:48:38Z

I’ve been using paperless-ngx to manage all my documents, but every once in a while I’ll get a .docx file to deal with…

Turns out Libreoffice has a headless mode a pdf converter built-in!

libreoffice --headless --convert-to pdf /path/to/file.docx --outdir /path/to/output/directory

Note that --outdir is in fact a directory, not the path to a file

FFMPEG 10-bit videos to 8-bit

2023-01-16T13:15:53Z

ffmpeg -i input.mp4 -map 0 -c:v libx264 -vf format=yuv420p -c:a copy output.mp4

i3-Like keyboard mapping in Pop_OS

2023-01-12T05:51:25Z

I was introduced to tiling window managers through i3, which I use heavily on one of my machines. I have switched to Pop_OS! at home though, which has a tiling window mode but the keybindings are not what I’m used to for i3. I wanted to at least navigate workspaces how I’m used to doing (cause I set workspace 3 for communication apps, 1 for my terminal, etc…)

Here’s how I set keybindings for:

<Super> + <number> sends me to that numbered workspace
<Shift> + <Super> + <number> moves the window I’m focused on to workspace number

#!/bin/bash
gsettings set org.gnome.mutter dynamic-workspaces false 
gsettings set org.gnome.desktop.wm.preferences num-workspaces 8 
gsettings set org.gnome.shell.keybindings switch-to-application-1 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-2 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-3 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-4 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-5 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-6 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-7 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-8 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-9 [] 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-1 "['<Super>1']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-2 "['<Super>2']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-3 "['<Super>3']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-4 "['<Super>4']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-5 "['<Super>5']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-6 "['<Super>6']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-7 "['<Super>7']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-8 "['<Super>8']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-9 "['<Super>9']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-10 "['<Super>0']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-1 "['<Super><Shift>1']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-2 "['<Super><Shift>2']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-3 "['<Super><Shift>3']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-4 "['<Super><Shift>4']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-5 "['<Super><Shift>5']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-6 "['<Super><Shift>6']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-7 "['<Super><Shift>7']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-8 "['<Super><Shift>8']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-9 "['<Super><Shift>9']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-10 "['<Super><Shift>0']"

Use non-standard named ssh keys with github

2023-01-03T08:34:50Z

I was getting (publickey denied) when trying to push to GH using ssh. When I tested the connection I saw that a bunch of keys in ``~/.ssh/ were being attempted

✗ ssh git@github.com -vv

...

debug1: Will attempt key: /home/nic/.ssh/id_rsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/nic/.ssh/id_xmss 
debug1: Will attempt key: /home/nic/.ssh/id_dsa 

...

debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

None of those were the key I setup with GH. So I added an entry into ~/.ssh/config:

Host
github.com
User git
Port 22
Hostname github.com
IdentityFile ~/.ssh/my_custom_github_key
TCPKeepAlive yes
IdentitiesOnly yes

Reminder about ssh-copy-id for SSH and Ansible

2022-12-28T13:33:07Z

ssh-copy-id -i my.key.pub <hostname probably from tailscale> this makes sure I can run ansible from my desktop against VMs on my server easily if they have tailscale for the hostname - otherwise use the IP

Adding docker daemon.json broke docker

2022-12-21T10:02:25Z

in /lib/systemd/system/docker.service there is an ExecStart command that got placed there when I setup Docker with Ansible - it threw the -H flag which told the daemon what hosts to setup. But I added the “hosts” key in my daemon.json and it broke - so removing the -H flag from the systemd unit fixed it

suda.vim for sudo access to files

2022-12-21T09:45:34Z

I regularly need to edit system config files - take /etc/sanoid/sanoid.conf as an example… I’ll want to play with something but if I don’t start Neovim as root then I get in trouble making edits I can’t save! So suda.vim gives me :SudaWrite which let’s me write that buffer with sudo privileges even though I’m Neovim is running with my login user!

Pipe to a pager to preserve console output in SSH session

2022-12-18T15:04:02Z

I’m playing with my ansible playbook in a remote tmux session, and I’m no wiz so I don’t know the ins and outs, but I can’t scroll up to get any console log output that’s not already visible on my screen. So I’m starting to end my commands with | less so I can page through the console output!

ansible-playbook plays.yml -v --tags mytag | less

Cron for Nextcloud in Docker

2022-12-13T06:43:45Z

AJAX wasn’t cutting it, traditional crontab in containers doesn’t make much sense to me, webcron is recommended but I don’t want to register with anything outside my LAN… Turns out you can just spin up an identical container with a different entrypoint to /cron.sh that does what you need!

Note that this is a task in an Ansible playbook - but the docker-compose is straight forward

So the only thing you need to make sure of is that all the configuration options - data volumes, user permissions, etc. are identical between the containers running the cron job and the one actually hosting NextCloud. This ensures that the container running cron has proper access to the database and filesystem - or at least the same access as NextCloud proper.

- name: Nextcloud Cron Docker Container
  docker_container:
    name: nextcloud-cron
    image: "{{ nextcloud_image }}"
    pull: true
    links:
      - nextcloud-mysql:mysql
    entrypoint: /cron.sh
    volumes:
      - "{{ nextcloud_data_directory }}/nextcloud:/var/www/html:rw"
    env:
      MYSQL_HOST: "mysql"
      MYSQL_DATABASE: "nextcloud"
      MYSQL_USER: "{{ nextcloud_sql_user }}"
      MYSQL_PASSWORD: "{{ nextcloud_sql_password }}"
      NEXTCLOUD_TRUSTED_DOMAINS: "{{ nextcloud_hostname }}.{{ ansible_nas_domain }}"
      PUID: "{{ nextcloud_user_id }}"
      PGID: "{{ nextcloud_group_id }}"
      TZ: "{{ ansible_nas_timezone }}"
    restart_policy: unless-stopped
    memory: "{{ nextcloud_memory }}"

Call basicConfig to get Python log messages in iPython

2022-12-10T14:04:23Z

Logging instead of printing

I am trying to adopt logger.debug instead of print but ran into a confusing thing in ipython during Advent of Code… I riddled by script with logger.debug (yes after setting logging.setLevel('DEBUG')) but in ipython none of my log messages showed up!

import logging

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Turns out what I was missing was a call to basicConfig

import logging

# forget this and your messages are in the ether! or at least not seen in ipython...
logging.basicConfig()

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Bonus

Want your new messages to show up while iterating on something without killing the ipython kernel?

from importlib import reload
reload(logging) # to make sure you get new log messages you add while developing!

New lines in Markdown tables

2022-11-25T13:35:05Z

I wanted to break down some long lines in a Markdown table cell to make it look nicer on my blog but \n didn’t do anything for me… turns out
is the magic sauce

Column 1	Column 2
Key	Doggo ipsum many pats. Borkdrive borking doggo doing me a frighten doggorino, noodle horse heckin. what a nice floof. Pupper borking doggo you are doing me a frighten, much ruin diet.
—	—

Column 1	Column 2
Key	Doggo ipsum many pats. Borkdrive borking doggo doing me a frighten doggorino, noodle horse heckin. what a nice floof. Pupper borking doggo you are doing me a frighten, much ruin diet.
—	—

Make a series of directories fast!

2022-11-10T15:27:50Z

mkdir s{1..10} will make directories s1, s2, … s10 in one command!

Case-insensitive search in Vim

2022-10-21T06:40:21Z

/mysearch\c will match mysearch, MYSEARCH, mYSeArCh…

Limit zfs list to avoid docker vomit

2022-10-20T06:39:18Z

zfs list has a flag -r, but if you use zfs driver for docker then you’ll get flooded with every docker volume in the world. zfs list -r -d N will limit the dept of the print out, so zfs list -r -d 2 gives me tank, tank/encrypted, tank/encrypted/docker -> but then I don’t see all the continer volumes

Use the right lspsage, you dope!

2022-10-14T19:41:44Z

If you think glepnir/lspsaga.nvim config will work with tali5/lspsaga.nvim installed… you’re in for a rude awakening. Double check your forks folks!

You can embed gifs?!

2022-10-09T13:39:39Z

I learned you can embed gifs and stuch in things other than GH personal pages!

So here I’ll keep ones I like and might use later

via GIPHY

Don’t forget to load XMP!

2022-10-09T07:17:43Z

Bought some DDR4-3600 speed RAM but only seeing 2666? Load up the BIOS, find DRAM config or something similar, and make sure to load the XMP profile to get that advertised RAM speed!

My PassMark Scores

2022-10-08T18:43:29Z

Babyblue v2

Ryzen 5700x + 32 GB 3200 CL16 RAM

Venus

Ryzen 3200G + 16GB 3600 CL16 RAM

Hogwarts

Dual Xeon X5650 @2.67GHz | 90 GB 1883 DDR3 ECC RAM

Server 2 ... Change the name

Ryzen 5700G + 128GB 3200 CL18 RAM

Server 2 with XMP enabled and undervolted from 1.35 to 1.2

Ryzen 5700G + 128GB 3200 CL18 RAM

Undervolted to 1.2V (from 1.35) XMP enabled

Suddenly SSH requires a password

2022-10-07T11:14:37Z

ssh -v -i ~/.ssh/id_rsa nic@hogwarts

THen we can look at print outs

cat /var/log/auth.log also showed me that I had too wide permissions on files in ~/.ssh -> probably changed from an rsync job

Quick setup of ZFS encrytped datasets with sane permissions

2022-10-06T19:58:01Z

Assuming you have a pool called tank…

And assuming you have an encrypted dataset (See Jim Saltar’s short intro)

Create a group for permissions - in my case I have one called home
Then if there’s anything in /tank/encrypted his it with chgrp -R home /tank/encrypted to give the home group ownership
Next we need to make sure that the members of home can do the writing… so chmod 775 -R /tank/encrypted will do the trick
Finally we want to make sure that all data created inside our dataset has the same set of permissions with chmod g+s /tank/encrypted and chmod g+w /tank/encrypted

Check Your BIOS Version On Ubuntu

2022-10-02T18:21:58Z

sudo dmidecode -s bios-version

Append string to list of files with xarg

2022-09-21T11:26:02Z

❯ find . -name “requirements.in” -print0 | xargs -0 sh -c ‘for arg in “$@”; do echo “awscli” »"$arg"; done’

Destroying Tmux sessions with fzf

2022-09-15T10:27:58Z

I use Tmux and Vim for most of my workflow, but I end up with a lot of dangling tmux sessions that dont’ really need to persist… but killing them one at a time is a pain so I wrote a little script-kitty nonsense to pipe multiple choices from fzf into tmux kill-session

I defined a little function in my .zshrc

destroy() { 
    tmux list-sessions -F '#{session_name}' | fzf -m | xargs -d $'\n' sh -c 'echo "killing $0"; tmux kill-session -t "$0"; for arg;do echo "killing $arg";tmux kill-session -t "$arg"; done'
}
bindkey -s '^d' 'destroy \n'

tmux list-sessions -F '#{session_name}' prints all my active tmux sessions to the console with the format of just their name

pype.dev   main   ×1 via   v3.8.11(pype.dev)  on  (us-east-1) proxy
❯ tmux list-sessions -F '#{session_name}'
session-01
session-02
session-03
...

Pipe that to fzf -m to allow multiple choices to be made using tab

Then the nasty bit in xargs… I echo killing @0 and killing $arg because the sh -c passes the first tmux session name to @0 (it’s just what bash does) and then the rest get handled in the for loop.

Basically then I get an fzf list to choose multiple tmux sessions to destroy to clean up some RAM!

Trick to login to web server on another box on lan

2022-09-12T18:48:45Z

TODO: change title - add –listen or –host or something as 0.0.0.0… that way it listens not on localhost or 127.0.0.1, but on all addresses!

Check your SMART status with smartctl

2022-08-29T06:30:27Z

https://www.simplified.guide/linux/disk-health-check

Install

For ubuntu/debian based distros (which is what I primarly use presently)

sudo apt update -y && sudo apt install smartmontools -y

List hard drives

lsblk | grep disk is one way or sudo lshw -c disk is another

smartctl

Use a device’s logical name such as dev/sda, not a partition of the disk

sudo smartctl -t short /dev/sda

dotfiles   home   ×3  ×2  ×2 via   v3.10.6(dotfiles)  took 11s
❯ sudo smartctl -t short /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.15.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Fri Sep 23 05:59:39 2022 CDT
Use smartctl -X to abort test.

check status

sudo smartctl -H /dev/sda


dotfiles   home   ×3  ×2  ×2 via   v3.10.6(dotfiles)
❯ sudo smartctl -H /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.15.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Playing with mdformat

2022-08-23T06:56:15Z

import this; print(this); print("what is taking so long black!!")

Paperless-NGX filtering on IDs instead of values

2022-08-09T08:28:51Z

To filter for saved views from the admin console you have to use the id of the tag you are using to filter on…

Ex: filtering for tags “Inbox” doesn’t work, but for tag 18 does (the id fo the Inbox tag). I can find that ID by clicking on the tag and looking at the url bar

Mounting exFAT USB in Linux

2022-07-31T07:38:43Z

Steps

sudo fdisk -l

then look for the device and partition

get the Type column

mount

Example #


dumbledore in /media  NO PYTHON VENV SET 
❯ sudo fdisk -l

...

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdk1  *        2048 60371951 60369904 28.8G  7 HPFS/NTFS/exFAT
/dev/sdk2       60371952 60437487    65536   32M ef EFI (FAT-12/16/32)


dumbledore in /media  NO PYTHON VENV SET 
❯ sudo mount -t ntfs /dev/sdk1 /media/ventoy-usb -o uid=1000                       
NTFS signature is missing.
Failed to mount '/dev/sdk1': Invalid argument
The device '/dev/sdk1' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

dumbledore in /media  NO PYTHON VENV SET 
✗ sudo mount -t exfat /dev/sdk1 /media/ventoy-usb -o uid=1000

Xrdp-Authentication-Required-To-Create-Managed-Color-Device

2022-07-18T00:00:00Z

I just need to RDP into an Ubuntu box via Remmina and everytime I login I have to authenticate to create a color managed device… which I don’t even know what that is!

To fix it?

vim /etc/polkit-1/localauthority.conf.d/02-allow-colord.conf

polkit.addRule(function(action, subject) {
 if ((action.id == "org.freedesktop.color-manager.create-device" ||
 action.id == "org.freedesktop.color-manager.create-profile" ||
 action.id == "org.freedesktop.color-manager.delete-device" ||
 action.id == "org.freedesktop.color-manager.delete-profile" ||
 action.id == "org.freedesktop.color-manager.modify-device" ||
 action.id == "org.freedesktop.color-manager.modify-profile") &&
 subject.isInGroup("{users}")) {
 return polkit.Result.YES;
 }
 });

Give’r the :x, logout, and you’re good to go

ZFS Permissions for Sanoid/Syncoid

2022-07-08T15:54:58Z

zfs allow -u $USER clone,load-key,create,destroy,mount,mountpoint,receive,send,rollback,compression,snapshot,hold,keylocation,bookmark tank

load-key only needed if using encrypted datasets

Setup KVM to boot from local PXE server

2022-07-06T12:30:50Z

Assuming you have a PXE server running you only need small edits to the virt-manager config

cheat on your man

2022-06-23T09:35:31Z

man can be a pain to read… and there’s lots of alternatives out there and one I’ve just started playing with is cheat

man man will give you this plus a billion more lines of docs, which is useful when you need it…

MAN(1)                                                                                                                       Manual pager utils                                                                                                                      MAN(1)

NAME
       man - an interface to the on-line reference manuals

SYNOPSIS
       man  [-C  file]  [-d] [-D] [--warnings[=warnings]] [-R encoding] [-L locale] [-m system[,...]] [-M path] [-S list] [-e extension] [-i|-I] [--regex|--wildcard] [--names-only] [-a] [-u] [--no-subpages] [-P pager] [-r prompt] [-7] [-E encoding] [--no-hyphenation]
       [--no-justification] [-p string] [-t] [-T[device]] [-H[browser]] [-X[dpi]] [-Z] [[section] page[.section] ...] ...
       man -k [apropos options] regexp ...
       man -K [-w|-W] [-S list] [-i|-I] [--regex] [section] term ...
       man -f [whatis options] page ...
       man -l [-C file] [-d] [-D] [--warnings[=warnings]] [-R encoding] [-L locale] [-P pager] [-r prompt] [-7] [-E encoding] [-p string] [-t] [-T[device]] [-H[browser]] [-X[dpi]] [-Z] file ...
       man -w|-W [-C file] [-d] [-D] page ...
       man -c [-C file] [-d] [-D] page ...
       man [-?V]

DESCRIPTION
       man is the system's manual pager.  Each page argument given to man is normally the name of a program, utility or function.  The manual page associated with each of these arguments is then found and displayed.  A section, if provided, will direct  man  to  look
       only  in  that  section of the manual.  The default action is to search in all of the available sections following a pre-defined order ("1 n l 8 3 2 3posix 3pm 3perl 3am 5 4 9 6 7" by default, unless overridden by the SECTION directive in /etc/manpath.config),
       and to show only the first page found, even if page exists in several sections.

       The table below shows the section numbers of the manual followed by the types of pages they contain.

       1   Executable programs or shell commands
       2   System calls (functions provided by the kernel)
       3   Library calls (functions within program libraries)
       4   Special files (usually found in /dev)
       5   File formats and conventions eg /etc/passwd
       6   Games
       7   Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
       8   System administration commands (usually only for root)
       9   Kernel routines [Non standard]

       A manual page consists of several sections.

       Conventional section names include NAME, SYNOPSIS, CONFIGURATION, DESCRIPTION, OPTIONS, EXIT STATUS, RETURN VALUE, ERRORS, ENVIRONMENT, FILES, VERSIONS, CONFORMING TO, NOTES, BUGS, EXAMPLE, AUTHORS, and SEE ALSO.

       The following conventions apply to the SYNOPSIS section and can be used as a guide in other sections.

       bold text          type exactly as shown.
       italic text        replace with appropriate argument.
       [-abc]             any or all arguments within [ ] are optional.
       -a|-b              options delimited by | cannot be used together.
       argument ...       argument is repeatable.
       [expression] ...   entire expression within [ ] is repeatable.

       Exact rendering may vary depending on the output device.  For instance, man will usually not be able to render italics when running in a terminal, and will typically use underlined or coloured text instead.

       The command or function illustration is a pattern that should match all possible invocations.  In some cases it is advisable to illustrate several exclusive invocations as is shown in the SYNOPSIS section of this manual page.

EXAMPLES
       man ls
           Display the manual page for the item (program) ls.

       man man.7
           Display the manual page for macro package man from section 7.

But what if you don’t? #

cheat man

# To convert a man page to pdf:
man -t bash | ps2pdf - bash.pdf

# To view the ascii chart:
man 7 ascii

You get tiny examples to remind you of what you probably are trying to do!

Reset SSH key passphrase

2022-06-13T06:55:32Z

I got into a pickle where I encrypted the ssh keys I use for my SSH connections on LAN, but then I couldn’t run my ansible playbook on my server! ssh-keygen -p and leave the new passphrase blank saved my day (although password protected key files are safer!)

TL;DR - just reset it to nothing

ssh-keygen -p will let you reset the passphrase on your ssh keys (good for you! yay security!)

But I needed to remove the passphrase to quickly deploy an ansible playbook 🤓

Add colored indicators to your dataframes html representation

2022-06-04T06:12:33Z

Mike Driscoll recently tweeted about making colored out with pandas DataFrames and I just had to try it for myself

Use Case

First though… why? My biggest use case is a monitoring pipeline of mine… The details aside, the output of my pipeline is a dataframe where each row has information about a failed pipeline that I need to go look into. I dump that result to a simle html file that’s hosted on an internal site and the file is updated every couple of hours. Adding some colored indicators automatically to the rows to help me assess severity of each record would be a handy way to quickly get an understanding the state of our pipelines.

How?

The docs for the applymap method state simply:

Apply a CSS-styling function elementwise.

Updates the HTML representation with the result.

So we can write a function that returns color: {color} based on the dataframe values and when we drop that dataframe to html we’ll have some simple css styling applied automagically!

By default the function will be applied to all columns of the dataframe, but that’s not useful if the columns are different types which is usually the case. Luckily there is a subset keyword to only apply to the columns you need!

Consider my example

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.read_csv("cars.csv")

sandbox   main via 3.8.11(sandbox) ipython
❯ def mpg_color(val: float):
...:     color = "red" if val < 21 else "green"
...:     return f"color: {color}"

sandbox   main via 3.8.11(sandbox) ipython
❯ df.style.applymap(mpg_color, subset="mpg").to_html("color.html")

I want to quickly see if the mpg is any good for the cars in the cars dataset and I’ll define “good” as better than 21 mpg (not great I know but just for the sake of discussion…)

The function returns an appropriate css string and after I style.applymap on just the mpg column we get this!

	Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
0	Mazda RX4	21.000000	6	160.000000	110	3.900000	2.620000	16.460000	0	1	4	4
1	Mazda RX4 Wag	21.000000	6	160.000000	110	3.900000	2.875000	17.020000	0	1	4	4
2	Datsun 710	22.800000	4	108.000000	93	3.850000	2.320000	18.610000	1	1	4	1
3	Hornet 4 Drive	21.400000	6	258.000000	110	3.080000	3.215000	19.440000	1	0	3	1
4	Hornet Sportabout	18.700000	8	360.000000	175	3.150000	3.440000	17.020000	0	0	3	2

Samba on Ubuntu 22 needs “inherit permissions” set

2022-05-31T06:30:42Z

see samba config on hogwarts

Filtering emails with core utils

2022-05-30T06:09:27Z

email1@me.com
somebody_else@gmail.com

#! /bin/bash
# pick multiple emails from list and combine into comma seperated array
emails=`cat .../emails | fzf -m | sed 's/^\|$/"/g'|paste -sd,` 

echo $emails

Reindex Nextcloud After Adding Data via CLI

2022-05-29T15:34:33Z

My Nextcloud woes

I wrote here about setting up www-data as the owner of any directories you want nextcloud to manage. However, I regularly struggle wtih permissions issues on my NAS because of the external storage app anyways so I’ve decided to just put our photos in the spot Nextcloud would otherwise put them, and use this as healthy pressure on our family to organize our photos and put the ones we care about with the rest of our family media.

Migration

Because I had a ton of photos on the NAS anyways that I wanted moved over to Nextcloud I just rsync’d the photos directory on my NAS to the user’s photos directory in nextcloud but they weren’t showing up in the web UI!

The Fix?

As www-data I needed to php /var/www/nextcloud/occ files:scan --all inside my nextcloud docker container AFTER moving a bunch of photos off my “NAS” into the folder mounted to the nextcloud container as its data folder! Before I did this they weren’t showing up in the web UI/

arr client config

2022-05-28T13:29:28Z

TIL that when setting up download clients for radarr/sonarr/lidarr/readarr/bazarr/prowlarr that you can utilize internal DNS and instead of hardcoding an IP address of your download client server, can use just the CNAME record (ie. instead of 172.10.14.13 I can use transmission.mydomain.com… notice the lact of http(s)://… adding that won’t allow the connection to work/

Furthermore, you can use internal DNS to lookup the domain, not the subdomain, and expose the port, like mydomain.com:7878 for sonarr. This was simpler to maintain because I don’t change which ports an application exposes or utilizes hardly ever, plus I don’t need to maintain CNAME records for every service!

Add space to your LVM on Ubuntu

2022-05-26T21:33:55Z

I ran out of space on the SSD in my server when doing some file transfers but only 100GB was used of a 256 GB SSD?

LVM

When installing Ubuntu live server the default option for how to partition the disk (in my experience) has been to setup an LVM group that defaults to less than the available space. Most recently I put Ubuntu server on a 256 GB SSD but the main partition was formatted as an LVM group with 100GB of storage… I didn’t think anything of this even though I’m mostly used to EXT4.

I think the reason for LVMs is performance, but in hindsight, I don’t really care much about the performance differences, I really just want all my storage that’s fast enough

Extending the LVM

A moment of googling brought me to Ubuntu’s wiki and I learned that I can expand my LVM to the space I need…

sudo lvdisplay and sudo pvdisplay show detailed views of the logical volumes and physical volumes respectively.

Take a look at those and find the volume you need to extend. For me I found this:

  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/ubuntu-lv
  LV Name                ubuntu-lv
  VG Name                ubuntu-vg
  LV Write Access        read/write
  LV Status              available
  ...

There’s more that you’ll see but this is what’s relevant - I need to extend the ubuntu-lv logical volume in the ubuntu-vg volume group.

sudo lvextend -L +50g ubuntu-vg/ubuntu-lv gives me 50 more GB of storage which should be enough for at least tonight 🤓

RTFM

Tdarr worker nodes share the cache!

2022-05-25T06:36:59Z

When working with tdarr remote nodes, they need to have access not only to the same libraries but also the same transcode cache as the server otherwise the transcodes will fail…

Network Setup

To explain I’ll give a brief overview of my home setup

I have an old Dell PowerEdge R610 as my main server running a live server distribution of Ubuntu. I use ZFS for my NAS file system, and most of my datasets are accesible over my home network. I have a Tdarr server running in a docker container on the R610.

In my office I dailyi drive a gaming desktop with an Nvidia 2060 Super running Ubuntu as well. On that desktop I am running a Tdarr node in a docker container. The container has access to the network folders with my media.

Initial Magic

When I spun up a tdarr node on my desktop, the tdarr server running on my R610 automatically registered the node, which was freaking amazing. That magic though spoiled me and I thought that I didn’t need to read the rest of the docs…

Initial Fail

I setup a transcode cache directory on the R610 locally and a separate transcode cache on my desktop that the remote tdarr node would use. Having them separated led to 2 main issues:

Transcodes were not being migrated back to my library properly
I was running out of disk space on my desktop because tdarr wasn’t deleting the completed cache files properly.

The Fix

Make a transcode cache directory accessible on my network, and give access to that directory to the docker container running my remote tdarr node.

Local DNS with Pi-hole

2022-05-23T09:40:59Z

Spoilers #

Tailscale is way easier than this… I was doing this local DNS overwrite in Pi-hole before running tailscale and I haven’t just totally “kicked the habit” yet, so for anyone NOT running tailcale, but wanting local HTTPS and using Pi-Hole this method would work fine

Intro #

I use pihole as my DNS server at home. I run unbound as well and have a pretty standard setup from their docs.

Pi-hole does 2 primary things for me:

dns sink-hold, the primary use case I believe
local SSL for all my self-hosted apps

Caveat #

I know for sure there are better ways to do this, but there’s also worse ones… so for now this has been my pattern, and it’s only bitten me in the butt when I’ve forgotten to add the final CNAME record… which’ll make sense in a sec.

Process #

It’s really quite simple - in pihole I have a DNS record for my domain pointing to my primary server

Then for each service that I want to keep everything resolved locally for I add a CNAME

With DNS resolving this way for clients using my pihole, the networking all stays local and I still get HTTPS for my wildcard cert in cloudflare

Why? #

The reasons I do this are simple:

I’ve been doing it since I started homelabbing and it started out of misunderstanding of how networking works at all
I use a home dashboard with https:// links, and it’s nice to just use that same dashbaord publically or at home. With local DNS resolution then I can whitelist some services, but conveniently access over the public url with https but routing such that my whitelist let’s me in when it wouldn’t if I tried to access the service from an external client
it’s my homelab - I can do what I want

It’s too complicated #

Honestly, as simple as this is, it is tedious and kind of complicated… There’s options to make it better…

script any service deployment to update the pihole /etc/host file
just use tailscale…

Configure bridge network on Ubuntu 22.04 with Netplan

2022-05-22T14:55:28Z

See 02-….yaml in ansible-nas

Netplan change from Focal to Jammy

2022-05-22T13:23:17Z

I am revamping my home server and bumped myself early up to Jammy Jellyfish… however to my peril I reused my netplan config and after hitting my server with the ‘ol netplan apply I lost connection… DNS still seemed to kinda work externally, but internally nothing was up…

Turns out Netplan got a little change in how to express the gateway key in the netplan config!

Old Ubuntu 20.04 way

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      gateway4: 192.168.1.1  # <-- This changes!
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

New jammin way for Jammy Jellyfish (at least that worked for me)

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      routes:
        - to: default
          via: 192.168.1.1 
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

Subset a list based on values in another list with itertools.compress

2022-05-19T14:22:58Z

I have list [True, False, False, True] and another list [1, 2, 3, 4] and a use case where I want to filter list 2 based on list 1 to remove values that line up with the element False in list 1…. so the outcome will be [1, 4]. list(compress(list2, list1)) will do it. As long as you can create a mask for the filter than itertool.compress will be your friend!

Nextcloud permissions with ZFS and Ansible-NAS

2022-05-19T13:55:20Z

TL;DR

As the nextcloud docs say… if you want to write to an external volume that location has to be writeable by the user/group www-data on the host system… so if that makes sense to you then this TIL probably isn’t a ton of value.. if not however, read on :)

Case Study

You want to self-host your own cloud and use a smart file system for convenience… Nextcloud and ZFS are pretty common goto answers for each of those problems.

My home NAS is built on ZFS and among other things I have a zpool named tank and nested in there is a tank/nas dataset with several child zfs datasets under that.

I want to use nextcloud mainly for auto-uploading photos from my wife’s and my phones for automatic backups. The issue is that the nextcloud application (I run in Docker) is fixed as the www-data user and so any volume/folder that you want nextcloud to write to needs to be permissioned such that www-data owns it… but I don’t want www-data to own everything in my NAS… so what’s a girl to do?

Solution

Well, one way to go is to just utilize docker volumes, write the data in the container to /var/www/html and let that be the place your data backsup to.

I still wanted nextcloud to automatically write right to my NAS so I created a nextcloud-upload directory inside of tank/nas/media/photos (photos cause that’s all that gets automatically uploaded)

Then I chown -R www-data:www-data /tank/nas/media/photos/nextcloud-upload so that just that sub-folder is owned by www-data. Now everyone’s happy!

See git history about one file

2022-05-19T06:43:32Z

In vim G clog % does a git clog {current file}. You get every commit that the target file is apart of (so there might be info in those commits unrelated)

See ZFS snapshot disk usage

2022-05-19T06:05:23Z

As I was cleaning up my NAS recently I noticed that I ran out of storage even though my disk usage looked pretty low… turns out I was keeping a mega-ton of ZFS snapshots and due to my own ignorance at the time didn’t realize the storage cost of this!

zfs list-o space tank/home will show the disk usage of the dataset and snapshots in tank/home

Deleting files on remote storage from Ubuntu might not do what you think

2022-05-19T05:51:50Z

From my daily driver Ubuntu machine I often open nautilus, dolphin, etc. and delete a file here or there on my NAS… turns out Ubuntu sends thse file to .Trash-100 ON THE NAS so I’m effectively just moving that file and not freeing up any space…

Remove ZFS Dataset Specific Snapshots

2022-05-19T05:49:16Z

I started my homelab journey being super naive about ZFS and how to manage the filesystem… that bit me in the butt when transfering a ton of files out of folders and into datasets because ZFS is copy on write so I was essentially duplicating my storage until I got a hair smarter about removing files after they’re moved (rsync –remove-source-file ftw). But I had a ton of snapshots of child datasets with a ton of data that I just never will need, so I learned zfs list -H -o name -t snapshot tank/dataset1/dataset2 will list just the snapshots for dataset2 and if you pipe that into xargs -n1 zfs destroy then you have a way to clear out some snapshots you don’t need!

Plug Snapshot!

2022-05-17T15:45:49Z

:PlugSnapshot, :w ~/dotfiles/nvim/snapshot.vim… keep your config safe with git!

Filepath Completion in Neovim

2022-05-17T14:03:27Z

I’ve had Plug 'hrsh7th/cmp-path' in my plugins for ever but didn’t notice until recently that I wasn’t getting any filepath completion in vim!

Fuller setup instructions below the TLDR

TL;DR

Turns out I need to not be a dope and configure nvim-cmp to actually use it…

local cmp = require'cmp'

cmp.setup({
    -- removed rest of setup - see the rest in my dotfiles
  sources = cmp.config.sources({
    { name = 'path' },  -- This needs to be here!
    })
})

My Setup

For the sake of completeness here is how I currently (May 2022) configure completion in Neovim usin nvim-cmp

Plugins #

I keep all my plugins in plugins.vim

call plug#begin(s:plug_dir)
Plug 'neovim/nvim-lspconfig'
Plug 'hrsh7th/cmp-nvim-lsp'
Plug 'hrsh7th/cmp-buffer'
Plug 'hrsh7th/cmp-path'
Plug 'hrsh7th/cmp-cmdline'
Plug 'hrsh7th/nvim-cmp'

" For ultisnips users.
<!-- " Plug 'SirVer/ultisnips' -->
<!-- " Plug 'quangnguyen30192/cmp-nvim-ultisnips' -->

call plug#end()

Vim Settings #

My vim settings are also kept in their own file, settings.vim


set completeopt=menu,menuone,noselect

nvim-cmp configuration #

I have a cmp.lua file that gets sourced in init.lua (file structure explained below) for configuring cmp.


  -- Setup nvim-cmp.
local cmp = require'cmp'

cmp.setup({
  snippet = {
    -- REQUIRED - you must specify a snippet engine
    expand = function(args)
      -- For `ultisnips` user.
      vim.fn["UltiSnips#Anon"](args.body)
    end,
  },
  window = {
      completion = cmp.config.window.bordered(),
  },
  mapping = {
    ['<Down>'] = cmp.mapping.select_next_item({ behavior = cmp.SelectBehavior.Select }),
    ['<Up>'] = cmp.mapping.select_prev_item({ behavior = cmp.SelectBehavior.Select }),
    ['<C-d>'] = cmp.mapping.scroll_docs(-4),
    ['<C-f>'] = cmp.mapping.scroll_docs(4),
    ['<C-Space>'] = cmp.mapping.complete(),
    ['<C-e>'] = cmp.mapping.close(),
    ['<Tab>'] = cmp.mapping(cmp.mapping.select_next_item(), { 'i', 's' }),
    ['<CR>'] = cmp.mapping.confirm({
      behavior = cmp.ConfirmBehavior.Replace,
      select = true,
    })
  },
  sources = cmp.config.sources({
    { name = 'nvim_lsp' },
    { name = 'ultisnips' },
    { name = 'buffer' },
    { name = 'path' },
    { name = 'tmux' },
    })
})

The sources section is what was key for this post…

Piecing it together!

My init.vim sources plugins and then settings and then finally calls init.lua. init.lua sources my cmp.lua file and BANG! auto-completion.

More sources #

hrsh7th’s wiki for nvim-cmp is here and has example configs as well as a list of sources…

Don’t forget to configure and not just install!

my dotfiles

Forms with FastAPI and Jinja

2022-05-15T00:00:00Z

I just started using FastAPI for a home project and needed to pass back a dynamic number of values from a form rendered with jinja…

Dynamic Values

The jinja templating for rendering HTML based on something like a python iterable is nice and easy

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). it?

The way to pass back all those values is pretty straight forward (after hours of messing around that is!)

# I hate it when tutorials don't show ALL relevant pieces to the blurb
import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

Plug-Snapshot-To-Save-Your-Life

2022-05-13T00:00:00Z

If you use vim-plug for managing your vim plugins, do yourself a favor and snapshot your plugins before upgrading!

:PlugSnapshot creates a vim.snapshot file that you can use to restore your plugin versions with vim -S snapshot.vim

The snapshot file looks like this:

❯ cat snapshot.vim
" Generated by vim-plug
" Tue 17 May 2022 03:44:29 PM CDT
" :source this file in vim to restore the snapshot
" or execute: vim -S snapshot.vim

silent! let g:plugs['Telegraph.nvim'].commit = '92e472f4e83acd60eb3766168e66d02718bfefe0'
silent! let g:plugs['TrueZen.nvim'].commit = '508b977d71650da5c9243698614a9a1416f116d4'
silent! let g:plugs['ariake-vim-colors'].commit = '9fb35f1255e475631c9df24ecc5485a40360cc7b'
silent! let g:plugs['auto-pairs'].commit = '39f06b873a8449af8ff6a3eee716d3da14d63a76'
silent! let g:plugs['bufutils.vim'].commit = '4634feb1312fd73fab66cfaa860e7af3abde935b'
silent! let g:plugs['cmp-buffer'].commit = '12463cfcd9b14052f9effccbf1d84caa7a2d57f0'
silent! let g:plugs['cmp-cmdline'].commit = 'c36ca4bc1dedb12b4ba6546b96c43896fd6e7252'
silent! let g:plugs['cmp-nvim-lsp'].commit = 'e6b5feb2e6560b61f31c756fb9231a0d7b10c73d'
silent! let g:plugs['cmp-nvim-ultisnips'].commit = '21f02b62deb409ce69928a23406076bd0043ddbc'
silent! let g:plugs['cmp-path'].commit = '466b6b8270f7ba89abd59f402c73f63c7331ff6e'
silent! let g:plugs['cmp-spell'].commit = '5602f1a0de7831f8dad5b0c6db45328fbd539971'
silent! let g:plugs['compe-tabnine'].commit = '33e4af509c27da9ef2c9c3002c01e3ec031797d4'
silent! let g:plugs['coverage-highlight.vim'].commit = '864e03679ea4168661501246147893cc82020917'
silent! let g:plugs['diffurcate.vim'].commit = 'b804675072220ff7c7ebcd24a028aa4aa35f09cc'
silent! let g:plugs['fzf'].commit = '6dcf5c3d7d6c321b17e6a5673f1533d6e8350462'
silent! let g:plugs['fzf.vim'].commit = 'd5f1f8641b24c0fd5b10a299824362a2a1b20ae0'
silent! let g:plugs['git-blame.vim'].commit = '9d144b7bed5d8f1c9259551768b7f3b3d1294917'
silent! let g:plugs['harpoon'].commit = 'd3d3d22b6207f46f8ca64946f4d781e975aec0fc'
silent! let g:plugs['instant.nvim'].commit = 'c02d72267b12130609b7ad39b76cf7f4a3bc9554'
silent! let g:plugs['lsp-colors.nvim'].commit = '517fe3ab6b63f9907b093bc9443ef06b56f804f3'
silent! let g:plugs['lsp_extensions.nvim'].commit = '4011f4aec61ba59c734f5dbf52e91f258b99d985'
silent! let g:plugs['lspkind-nvim'].commit = '57e5b5dfbe991151b07d272a06e365a77cc3d0e7'
silent! let g:plugs['lspsaga.nvim'].commit = '8dde091a61ab07f639baaa82b456d3508d0aa7e8'
silent! let g:plugs['neoformat'].commit = '409ebbba9f4b568ea87ab4f2de90a645cf5d000a'
silent! let g:plugs['neovim-fuzzy'].commit = '0bef4e1a81c65fc05d31380dd74454bd67733837'
silent! let g:plugs['nerdtree'].commit = 'eed488b1cd1867bd25f19f90e10440c5cc7d6424'
silent! let g:plugs['nerdtree-visual-selection'].commit = '05427635ff053a2c542fcf8d7c3744c72575e76c'
silent! let g:plugs['nvim-cmp'].commit = 'a226b6a4ff72e5e809ed17734318233fb25c87f3'
silent! let g:plugs['nvim-compe'].commit = 'd186d739c54823e0b010feb205c6f97792322c08'
silent! let g:plugs['nvim-lspconfig'].commit = '9ff2a06cebd4c8c3af5259d713959ab310125bec'
silent! let g:plugs['nvim-lspinstall'].commit = '79ec2425d6b39cdcb69d379f3e56847f49be73eb'
silent! let g:plugs['nvim-treesitter'].commit = '10d57b3ec14cac0b6b759e1eb5594617b8a7e883'
silent! let g:plugs['nvim-treesitter-textobjects'].commit = '094e8ad3cc839e825f8dcc91352837653e365a8f'
silent! let g:plugs['nvim-web-devicons'].commit = 'bdd43421437f2ef037e0dafeaaaa62b31d35ef2f'
silent! let g:plugs['playground'].commit = '71b00a3c665298e5155ad64a9020135808d4e3e8'
silent! let g:plugs['plenary.nvim'].commit = '25b3475b97f241e6f76249747bc209e70c5d2ec8'
silent! let g:plugs['popup.nvim'].commit = 'b7404d35d5d3548a82149238289fa71f7f6de4ac'
silent! let g:plugs['syntastic'].commit = 'b7f4f71539038d33f173bfa72631737da049575a'
silent! let g:plugs['tabular'].commit = '339091ac4dd1f17e225fe7d57b48aff55f99b23a'
silent! let g:plugs['targets.vim'].commit = '8d6ff2984cdfaebe5b7a6eee8f226a6dd1226f2d'
silent! let g:plugs['telescope-fzy-native.nvim'].commit = '7b3d2528102f858036627a68821ccf5fc1d78ce4'
silent! let g:plugs['telescope.nvim'].commit = '39b12d84e86f5054e2ed98829b367598ae53ab41'
silent! let g:plugs['termopen.vim'].commit = '3194a991a18a9be2fd9fcf8c4c55fe990c04b2bd'
silent! let g:plugs['tokyonight.nvim'].commit = '8223c970677e4d88c9b6b6d81bda23daf11062bb'
silent! let g:plugs['trouble.nvim'].commit = 'da61737d860ddc12f78e638152834487eabf0ee5'
silent! let g:plugs['ultisnips'].commit = 'f5ccf0977c611ffd774ca180774959301baaffad'
silent! let g:plugs['undotree'].commit = '08e259be24d4476c1ee745dc735eefd44f90efdc'
silent! let g:plugs['vim-airline'].commit = 'c4655701431a9c79704c827fd88a4783ec946879'
silent! let g:plugs['vim-airline-themes'].commit = '97cf3e6e638f936187d5f6e9b5eb1bdf0a4df256'
silent! let g:plugs['vim-be-good'].commit = 'bc499a06c14c729b22a6cc7e730a9fbc44d4e737'
silent! let g:plugs['vim-commentary'].commit = '3654775824337f466109f00eaf6759760f65be34'
silent! let g:plugs['vim-devicons'].commit = 'a2258658661e42dd4cdba4958805dbad1fe29ef4'
silent! let g:plugs['vim-dispatch'].commit = '00e77d90452e3c710014b26dc61ea919bc895e92'
silent! let g:plugs['vim-flake8'].commit = 'a99054ef98e8fdaefa1315af4649138bcadbfdf7'
silent! let g:plugs['vim-floaterm'].commit = 'ab7876f86c05c1935eb23a193f4f276132902ac1'
silent! let g:plugs['vim-fugitive'].commit = 'f529acef74b4266d94f22414c60b4a8930c1e0f3'
silent! let g:plugs['vim-gitbranch'].commit = '1a8ba866f3eaf0194783b9f8573339d6ede8f1ed'
silent! let g:plugs['vim-indent-object'].commit = '5c5b24c959478929b54a9e831a8e2e651a465965'
silent! let g:plugs['vim-nerdtree-syntax-highlight'].commit = '5178ee4d7f4e7761187df30bb709f703d91df18a'
silent! let g:plugs['vim-polyglot'].commit = '38282d58387cff48ac203f6912c05e4c8686141b'
silent! let g:plugs['vim-pydocstring'].commit = 'f6e3c52bfaf4b4f76dab4d84e75b94199d5f3b9c'
silent! let g:plugs['vim-quicklink'].commit = '021167741588555501594e1fc31f130b16acefa0'
silent! let g:plugs['vim-repeat'].commit = '24afe922e6a05891756ecf331f39a1f6743d3d5a'
silent! let g:plugs['vim-signify'].commit = '69498f6d49f3eeac06870012416dd9bf867b84f3'
silent! let g:plugs['vim-sneak'].commit = '94c2de47ab301d476a2baec9ffda07367046bec9'
silent! let g:plugs['vim-snippets'].commit = '6f270bb2d26c38765ff2243e9337c65f8a96a28b'
silent! let g:plugs['vim-startify'].commit = '81e36c352a8deea54df5ec1e2f4348685569bed2'
silent! let g:plugs['vim-surround'].commit = 'bf3480dc9ae7bea34c78fbba4c65b4548b5b1fea'
silent! let g:plugs['vim-test'].commit = '2240d7a4b868cb594b7d83544e1b6db4df806e5e'
silent! let g:plugs['vim-tmux-runner'].commit = '54767911fd5e6e2d8e493847149e315ac2e6531a'
silent! let g:plugs['vim-ultest'].commit = '6978fd32e3ca2c1c5591884eea0d57a7ee43d212'
silent! let g:plugs['vim-visualstar'].commit = 'a18cd0e7a03311ac709595c1d261ed44b45c9098'
silent! let g:plugs['vimtex'].commit = 'dfaca59bbbf0079ab1b4f159337ae7f17d1b5289'
silent! let g:plugs['which-key.nvim'].commit = 'bd4411a2ed4dd8bb69c125e339d837028a6eea71'

PlugUpdate!

Python-Eval

2022-05-12T00:00:00Z

TODO

title = "my Title"
eval('"my" in title')

>>> True

print("hello, world"); print("formatting")

Dataframe-To-Styled-Html

2022-05-07T00:00:00Z

I wrote up a little on exporting DataFrames to markdown and html here

But I’ve been playing with a web app for with lists and while I’m toying around I learned you can actually give your tables some style with some simple css classes!

To HTML

Reminder that if you have a dataframe, df, you can df.to_html() to get an HTML table of your dataframe.

Well you can pass some classes to make it look super nice!

Classes and CSS

I don’t know anything really about CSS so I won’t pretend otherwise, but as I was learning about bootstrap that’s where I stumbled upon this…

There are several classes you can pass but I found really good luck with table-bordered and table-dark for my use case

df.to_html(classes=["table table-bordered table-dark"])

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

You try it!

Crack open ipython and make a dataframe, then df.to_html(classes=["table table-bordered table-dark"]), copy the output (minus the quote marks ipython uses to denote the string type) that into my-file.html, open that up in a browser and be amazed!

For added effeciency try using pyperclip to copy the output right to your clipboard!

pip install pyperclip and then pyperclip.copy(df.to_html(classes=["table table-bordered table-dark"]))

Dataframe-To-Markdown

2022-05-07T00:00:00Z

Pandas

pandas.DataFrames are pretty sweet data structures in Python.

I do a lot of work with tabular data and one thing I have incorporated into some of that work is automatic data summary reports by throwing the first few, or several relevant, rows of a dataframe at a point in a pipeline into a markdown file.

Pandas has a method on DataFrames that makes this 100% trivial!

The Method

Say we have a dataframe, df… then it’s literally just: df.to_markdown()

❯ df.head()

          Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
0          Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
1      Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
2         Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
3     Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
4  Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2

In ipython I can call the method and get a markdown table back as a string


mental-data-lake   new-posts via 3.8.11(mental-data-lake) ipython
❯ df.head().to_markdown()
'|    | Unnamed: 0        |   mpg |   cyl |   disp |   hp |   drat |    wt |   qsec |   vs |   am |   gear |   carb |\n|---:|:------------------|------:|------:|-------:|-----:|-------:|------:|-------:|-----:|-----:|-------:|-------:|\n|  0 | Mazda RX4         |  21   |     6 |    160 |  110 |   3.9  | 2.62  |  16.46 |    0 |    1 |      4 |      4 |\n|  1 | Mazda RX4 Wag     |  21   |     6 |    160 |  110 |   3.9  | 2.875 |  17.02 |    0 |    1 |      4 |      4 |\n|  2 | Datsun 710        |  22.8 |     4 |    108 |   93 |   3.85 | 2.32  |  18.61 |    1 |    1 |      4 |      1 |\n|  3 | Hornet 4 Drive    |  21.4 |     6 |    258 |  110 |   3.08 | 3.215 |  19.44 |    1 |    0 |      3 |      1 |\n|  4 | Hornet Sportabout |  18.7 |     8 |    360 |  175 |   3.15 | 3.44  |  17.02 |    0 |    0 |      3 |      2 |'

You can drop that string into a markdown file and using any reader that supports the rendering you’ll have a nicely formated table of example data in whatever report you’re making!

Bonus method

Just like markdown, you can export a dataframe to html with df.to_html() and use that if it’s more appropriate for your use case:


'<table border="1" class="dataframe">\n  <thead>\n    <tr style="text-align: right;">\n      <th></th>\n      <th>Unnamed: 0</th>\n      <th>mpg</th>\n      <th>cyl</th>\n      <th>disp</th>\n      <th>hp</th>\n      <th>drat</th>\n      <th>wt</th>\n      <th>qsec</th>\n      <th>vs</th>\n      <th>am</th>\n      <th>gear</th>\n      <th>carb</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Mazda RX4</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.620</td>\n      <td>16.46</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Mazda RX4 Wag</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.875</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Datsun 710</td>\n      <td>22.8</td>\n      <td>4</td>\n      <td>108.0</td>\n      <td>93</td>\n      <td>3.85</td>\n      <td>2.320</td>\n      <td>18.61</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Hornet 4 Drive</td>\n      <td>21.4</td>\n      <td>6</td>\n      <td>258.0</td>\n      <td>110</td>\n      <td>3.08</td>\n      <td>3.215</td>\n      <td>19.44</td>\n      <td>1</td>\n      <td>0</td>\n      <td>3</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Hornet Sportabout</td>\n      <td>18.7</td>\n      <td>8</td>\n      <td>360.0</td>\n      <td>175</td>\n      <td>3.15</td>\n      <td>3.440</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>2</td>\n    </tr>\n  </tbody>\n</table>'

My blog will render that html into a nice table! (After removing new line characters)

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

Git-Bisect

2022-05-03T00:00:00Z

I try to commit a lot, and I also try to write useful tests appropriate for the scope of work I’m focusing on, but sometimes I drop the ball…

Whether by laziness, ignorance, or accepted tech debt I don’t always code perfectly and recently I was dozens of commits into a new feature before realizing I broke something along the way that none of my tests caught…

Before today I would’ve manually reviewed every commit to see if something obvious slipped by me (talk about a time suck 😩)

There must be a better way

Bisect?

git bisect is the magic sauce for this exact problem…

You essentially create a range of commits to consider and let git bisect guide you through them in a manner akin to Newton’s method for finding the root of a continuous function.

How to do it?

Start with git bisect start and then choose the first good commit (ie. a commit you know the bug isn’t present in)


sandbox   bisect-post   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect start

sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good 655332b
bisect-post  HEAD         main         ORIG_HEAD
5b31e1e  -- [HEAD]    add successful print (52 seconds ago)
308247b  -- [HEAD^]   init another loop (77 seconds ago)
4555c59  -- [HEAD^^]  introduce bug (2 minutes ago)
9cf6d55  -- [HEAD~3]  add successful loop (3 minutes ago)
bcb41c3  -- [HEAD~4]  change x to 10 (4 minutes ago)
3c34aac  -- [HEAD~5]  init x to 1 (4 minutes ago)
12e53bd  -- [HEAD~6]  print cwd (4 minutes ago)
655332b  -- [HEAD~7]  add example.py (10 minutes ago)  # <- I want to start at this commit
59e0048  -- [HEAD~8]  gitignore (23 hours ago)
fb9e1fb  -- [HEAD~9]  add reqs (23 hours ago)


sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad 5b31e1e
bisect-post                                                ORIG_HEAD
HEAD                                                       refs/bisect/good-655332b6c384934c2c00c3d4aba3011ccc1e5b57
main
5b31e1e  -- [HEAD]    add successful print (5 minutes ago)  # <- I start here with the "bad" commit
308247b  -- [HEAD^]   init another loop (6 minutes ago)
4555c59  -- [HEAD^^]  introduce bug (6 minutes ago)
9cf6d55  -- [HEAD~3]  add successful loop (7 minutes ago)
bcb41c3  -- [HEAD~4]  change x to 10 (8 minutes ago)
3c34aac  -- [HEAD~5]  init x to 1 (9 minutes ago)
12e53bd  -- [HEAD~6]  print cwd (9 minutes ago)
655332b  -- [HEAD~7]  add example.py (14 minutes ago)
59e0048  -- [HEAD~8]  gitignore (23 hours ago)
fb9e1fb  -- [HEAD~9]  add reqs (23 hours ago)

After starting bisect with a “good” start commit and a “bad” ending commit we can let git to it’s thing!

Git checksout a commit somewhere about halfway between the good and bad commit so you can see if your bug is there or not.


sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad 5b31e1e
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[bcb41c3854e343eade85353683f2c1c4ddde4e04] change x to 10

sandbox   HEAD (bcb41c38) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯

In my example here I have a python script with some loops and print statements - they aren’t really relevant, I just wanted an easy to follow git history.

So I check to see if the bug is present or not either by running/writing tests or replicating the bug somehow.

In this session commit bcb41c38 is actually just fine, so I do git bisect good


sandbox   HEAD (bcb41c38) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good
Bisecting: 1 revision left to test after this (roughly 1 step)
[4555c5979268dff6c475365fdc5ce1d4a12bd820] introduce bug

And we see that git moves on to checkout another commit…

In this case the next commit is the one where I introduced a bug

git bisect bad then gives me:


sandbox   HEAD (4555c597) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[9cf6d55301560c51e2f55404d0d80b1f1e22a33d] add successful loop

At 4555c597 the script works as expected so one more git bisect good yields…

sandbox   HEAD (9cf6d553) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good
4555c5979268dff6c475365fdc5ce1d4a12bd820 is the first bad commit
commit 4555c5979268dff6c475365fdc5ce1d4a12bd820
Author: ########################### 
Date:   Tue May 3 09:00:00 2022 -0500

    introduce bug

 example.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

What happened?

Git sliced up a range of commits based on me saying of the next one was good or bad and localized the commit that introduced a bug into my workflow!

I didn’t have to manually review commits, click through logs, etc… I just let git checkout relevant commits and I ran whatever was appropriate for reproducing the bug to learn when it was comitted!

Pandas-String-Contains

2022-05-02T00:00:00Z

TL;DR

pandas.Series.str.contains accepts regular expressions and this is turned on by default!

Use case

We often need to filter pandas DataFrames based on several string values in a Series.

Notice that sweet pyflyby import 😁!

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.DataFrame({"A": ["string1", "string2", "string3"]})
[PYFLYBY] import pandas as pd

sandbox   main via 3.8.11(sandbox) ipython
❯ df

         A
0  string1
1  string2
2  string3

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains('1') | df.A.str.contains('2')]

         A
0  string1
1  string2

And this isn’t the worst thing in the world, especially for such a tiny example…

But what if we had dozens or more values to filter on?

Then it looks so much nicer to create an iterable of the values we want to filter on and join them with an apropriate regex operator (in this case | for inclusive or)


sandbox   main via 3.8.11(sandbox) ipython
❯ vals = ["1", "2"]  # iterable with whatever is appropriate for your use case

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains("|".join(vals), regex=True)]

         A
0  string1
1  string2

Fin

This is a super nice and concise way to do the kind of filtering my team does on a daily basis!

Unpack-Anywhere-With-Star

2022-04-24T00:00:00Z

Unpacking iterables in python with * is a pretty handy trick for writing code that is just a tiny bit more pythonic than not.

arr: Tuple[Union[int, str]] = (1, 2, 3, 'a', 'b', 'c')


print(arr)
>>> (1, 2, 3, 'a', 'b', 'c')

# the * unpacks the tuple into the individual elements
print(*arr)
>>> 1, 2, 3, 'a', 'b', 'c'

x, y, z, *alphas = arr

# x = 1, y = 2, z = 3
# alphas = [ 'a', 'b', 'c' ]

But @Ned Batchelder showed me via Twitter than you can arbitrarily unpack arguments based on position - it doesn’t have to be done at the beginning or the end!

x, y, *mixed, alpha = arr

# x = 1, y = 2
# mixed = [3, 'a', 'b']
# alpha = 'c'

I’m not entirely sure when I’ll need this but it definitley shows me another example of how flexible python is!

Htop

2022-04-24T00:00:00Z

htop is a common command line tool for seeing interactive output of your system resource utilization, running processes, etc.

I’ve always been super confused about htop showing seemingly the same process several times though…

The Fix… #

Just hit H…. makes the view a lot nicer 😀

Pipx

2022-04-22T00:00:00Z

pipx is a tool I’ve been using to solve a few problems of mine…

pinning formatting tools like black, flake8, isort, etc. to the same version for all my projects
keeping virtual environments clean of things like cookiecutter
python utilities I want system wide but not in the global environment, like visidata

What is it? #

pip itself is just a package manager like homebrew, apt, etc. But it is tied to a python environment. If you aren’t using a virtual environment then pip will operate inside the global installation of python.

Operating within that environment has burned me several times and now I have a strict virtual environment usage policy.

But there are still things I don’t want to have to put in every virtual environment - enter pipx

What’s it do? #

When you pipx install {package} a stand alone virtual environment gets created (by default in ~/.local/pipx/venvs). THen you can install extra dependencies with pipx inject {package} {dependency}

ex. After pipx install visidata in order to open Excel files you need to pipx inject visidata xlrd

In the example with visidata, I can then use it anywhere, in any project, without re-installing with pip in every env.

Also for the formatting tools - I configure vim to run the pipx versions of them on save - this way I don’t have to put them in every project’s virtual environment!

What about pip? #

So obviously you can’t pipx everything, nor do you want to. I see it as a safe and better alternative to global package installation.

How can you then be sure that you never pip install into the global env?

Add require-virtualenv = True to your pip.conf and you’re good to go!

With that set, if you try to pip install pandas into the global env you’ll get a message like this:


~ on  (us-east-1)  NO PYTHON VENV SET
❯ pip install pandas
ERROR: Could not find an activated virtualenv (required).

End #

Disable your system pip to keep your base python safe
Use pipx for tools you want available everywhere or don’t have to need in a virtual environment!

Fx-Json

2022-04-19T00:00:00Z

fx is an interactaive JSON viewer for the terminal.

It’s a simple tool built with Charmcli’s Bubble Tea.

Installation #

The installation with go was broken for me - both via the link and direct from the repo. Now I’m not a gopher so I don’t really know how to fix that.

Luckily npm install fx also works and got me what I needed!

Usage #

Usage is simple… fx <json file>. The Github has a few other ways such as curl ... | fx etc.

Typeddict

2022-04-15T00:00:00Z

Type hinting has helped me write code almost as much, if not more, than unit testing.

One thing I love is that with complete type hinting you get a lot more out of your LSP. Typing dictionaries can be tricky and I recently learned about TypedDict to do exactly what I needed!

The Problem #

It might not be straight up obvious what the problem is, especially if you don’t utilize tools like mypy or flake8 in your development.

My handy-dandy nvim-lsp gives me a lot of feedback when I’m coding and it’s immensely helpful.

So with the LSP giving me constant feedback here’s the issue:

from typing import Dict, List, Union

my_dict: Dict[str, Union[List[str], str]] = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}

my_dict["key_2"].pop()

With the above script you’ll get an annoying warning about using pop on key_2.

The Solution #

Maybe you can stomach getting yelled at by your LSP but I like complete silence if at all possible.

TypedDict was the saving grace.

from typing import TypedDict

MyDict = TypedDict("MyDict", {"key_1": str, "key_2": List[str]})

my_typed_dict: MyDict = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}


my_typed_dict["key_2"].pop()

I was able to import TypedDict from typing, mypy_extensions, and typing_extensions

With TypedDict you define your custom type, match the first argument to TypedDict with the name of the variable (idk why), then type hint each key you expect in the dict! It’s super easy and I think puts you into a position of being extremely explicit with your dictionary variables. This isn’t always desired or appropriate but in most of my use cases it is.

RTFM #

There’s other implementation of TypedDict and while writing this I saw that most of the docs define a class for the type like this:

from typing import TypedDict
class MyDict(TypedDict):
    key_1: str
    key_2: List[str]

my_dict : MyDict = {'key_1': 'val_1', 'key_2': ["ls_1", "ls_2"]}

pep docs

mypy docs

File-Length

2022-04-04T00:00:00Z

I have a specific need for counting the number of lines in a file quickly. At work we use S3 for data storage during our Kedro pipeline development, and in the development process we may end up orphaning several datasets. In order to keep our workspace clean I have a short utility that compares the datasets in a Kedro DataCatalog with the files in the relevant S3 location.

To get that list I run an internal tool like this:

kedro our-liter | grep s3 >> orphaned_datasets.txt

This simply parses our internal linter for the lines releated to my s3 linter utility and pipes those lines to a file.

To get a quick idea of how out of wack a pipeline is I could open the text file in vim, git it with the G and see what line number I’m on but I’m way too lazy for that…

AWK #

awk 'END {print NR}' orphaned_datasets.txt gives me the number of lines and I can alias this to whatever feels appropriate in my zshrc!

built-ins for the win!

Vim-Spell-Check

2022-04-01T00:00:00Z

Did you know you can spell check in Vim?!

Vim Spell check

Without... #

Here is a missspelled word.

  <h3>With!</h3>
  <p>Here is a <u>missspelled</u> word.</p>

What is this magic??? #

set: spell spelllang=en_us

Custom words? #

Sometimes there’s things that are words to you but not the default spell checker…

Common example: package names!

plotly, streamlit, psutil, etc etc…

You can easily add these to your vim config by hitting zw ontop of the word!

Deques

2022-03-31T00:00:00Z

I am working on a project to create a small system monitoring dashboard using the python psutil library.

The repo is here (if you want actual system monitoring please use netdata).

I’m using streamlit and plotly for the webserver, design, and plotting at the moment.

My Use Case #

I needed a way to refresh my plotly charts with a fixed window of time so that I’m able to just see relevant recent data instead of cramming all data for all time into one plot that’s 500 pixels wide…

Checking the length of arrays or lists every time I get a new piece of data feels kind of dumb and I thought “python must have a way to do this”…

“This” meaning, update values in a fixed length array without reallocating memory or recreating a copy of the list

Deques #

Enter the deque. It means “double ended queue” and is in general an Iterable that you can append values to either side or pop values from either side.

The init signature is straightforward enough and I’m sure there’s more to them than I know yet but here’s how I use it…

from collections import deque

my_deque = deque([1,2,3])

This gives us my_deque, created from an iterable, with several familiar methods like index, extend, append, etc. However there’s some new ones too such as appendleft and popleft.

my_deque.appendleft('a')
print(my_dequqe)
>>> deque(['a', 1, 2, 3])

my_deque.popleft()
>>> 'a'

<!--markata-attribution-->
print(my_deque)
>>> deque([1, 2, 3])

These are handy ways to manipulate the iterable that I needed for the arrays I plot with plotly!

See my follow-up to this on using Deques with plotly and streamlit to create a quick “dashboard” with live streaming data!

follow-up

Skimpy

2022-03-23T00:00:00Z

EDA #

I work with data a lot, but the nature of my job isn’t to dive super deep into a small amount of datasets, I’m often jumping between several projects every day and need to just get a super quick glance at some tables to get a high level view.

When I’m doing more interactive exploration I’ve graduated from Jupyter cells with df_N.head() to using an amazing tool called visidata

However, Visidata is a terminal based application and I’m often in an iPython console… so is there a way to move even faster for my super quick summary views?

yes!

Skimpy #

First thing to do is pip install skimpy and then it’s as easy to get some summary stats with skimpy <data>

This is super nice for seeing missing values in particular as well as the distribution shape of the data.

iPython #

But wait… I just said I’m normally in an iPython session but that was called from zsh.. If I’m hoping back into zsh I might as well use visidata to have more powerful exploration at my fingertips. So… can I see this table quickly without breaking my iPython workflow?

Of course you can with magic!

The above assumes you’re looking at a file, like you would in the terminal. skimpy works even better in iPython with from skimpy import skim then pass any DataFrame to skim!

Pyclean

2022-03-22T00:00:00Z

I like to keep my workspace clean and one thing that I don’t personally love looking at is the __pycache__ directory that pops up after running some code. The *.pyc files that show up there are python bytecode and they are cached to make subsequent runs a tad faster. My stuff never really needs this bonus speed boost and so I came across a neat tool called pyclean!

Pyclean #

The easiest way (in my opinion) to run pyclean is to just use pipx run.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)  took 9s
❯ ls
abcmeta.py  __pycache__  python-print-align.py  system-monitor-psutils.py

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ pipx run pyclean .
⚠️  pyclean is already on your PATH and installed at /usr/bin/pyclean. Downloading and running anyway.
Cleaning directory .
Total 1 files, 1 directories removed.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ ls
abcmeta.py  python-print-align.py  system-monitor-psutils.py

Why not bash? #

You could accomplish something similar with rm **/*.pyc or find -n '*.py?' -delete but there’s a chance you’ll find something you don’t love gone. Also this won’t help our poor Windows friends out there! pyclean is fully python so it’s OS independent.

Credits! #

repo

Psutil-01

2022-03-16T00:00:00Z

Mike Driscoll has been posting some awesome posts about psutil lately. I’m interested in making my own system monitoring dashboard now using this library. I don’t expect it to compete with Netdata or Glances but it’ll just be for fun to see how Python can solve this problem!

Repo coming soon

Example code: #

Here’s a short snippit to get used/available/total RAM and disk space (on partitions that you probably care about)


import psutil
import socket

print(f"System Memory used: {psutil.virtual_memory().used // (1024 ** 3)} GB")
print(f"System Memory available: {psutil.virtual_memory().available // (1024 ** 3)} GB")
print(f"System Memory total: {psutil.virtual_memory().total // (1024 ** 3)} GB")


print(f"Hostname: {socket.gethostname()}")

partitions = psutil.disk_partitions()

for part in partitions:
    mnt = part.mountpoint
    if "snap" in mnt or "boot" in mnt:
        continue
    disk = psutil.disk_usage(mnt)
    print(f"Usage at {mnt} on {part.device}: {disk.used // (1024 ** 3)} GB")
    print(f"Free at {mnt} on {part.device}: {disk.free // (1024 ** 3)}GB")
    print(f"Total at {mnt} on {part.device}: {disk.total // (1024 ** 3)}GB")

Bonus Ipython tip! Save this to a script called my_script.py and in Ipython you can %run -m my_script to run it!

project ↪ main v3.8.11 ipython
❯ %run -m system-monitor-psutils
System Memory used: 25 GB
System Memory available: 5 GB
System Memory total: 31 GB
Hostname: ryzen-3600x
Usage at / on /dev/nvme1n1p2: 81 GB
Free at / on /dev/nvme1n1p2: 351 GB
Total at / on /dev/nvme1n1p2: 456 GB

Mu

2022-03-15T00:00:00Z

If you work with a template for several projects then you might sometimes need to do the same action across all repos. A good example of this is updating a package in requirements.txt in every project, or refactoring a common module. If you have several repos to do this across then it can be time consuming… enter mu-repo

Mu #

mu-repo is an awesome cli tool for working with multiple git repositories at the same time. There are several things you can do:

mu status will give you the git status of every registered repo (see below)
mu sh will let you execute system level commands in every repo
mu stash will stash all changes across all registered repos
There’s literally a ton more but these are some handy ones

Registration #

mu tracks its own groups, and there is a default group when no particular one is active. It’s as simple as mu register proj1 prog2 ... to get repos registered


❯ mu register proj1 proj2
Repository: proj1 registered
Repository: proj2 registered

❯ mu status

  proj1 : git status
    On branch main

    No commits yet

    Untracked files:
    (use "git add <file>..." to include in what will be committed)
    requirements.txt

    nothing added to commit but untracked files present (use "git add" to track)

  proj2 : git status
    On branch main

    No commits yet

    Changes to be committed:
    (use "git rm --cached <file>..." to unstage)
    new file:   requirements.txt

Working with mu #

As you can see above I have two projects each with a requirements.txt added but not committed yet. Using mu I can stage this change across both repos at once.


❯ mu add requirements.txt

  proj1 : git add requirements.txt

  proj2 : git add requirements.txt

Then as you might imagine, I can make the commit in each repo


❯ mu commit -m "Add requirements.txts"

  proj1 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

  proj2 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

mu groups #

The other thing I got a lot of use out of recently was mu’s groups. At work I have about 40 repos cloned that are all based on the same kedro pipeline template. Some of these projects have been deprecated. I also have several more repos that are not kedro template - custom libraries or something. group let me utilize mu across different groups of repos.

Say proj2 is a deprecated project that I don’t need to worry about making changes to anymore. I don’t just have to unregister it, instead I can make a group called “active” and register proj1 in that group


❯ mu group add active --empty

~/personal
❯ mu group add deprecated --empty

~/personal
❯ mu group
  active
* deprecated

The * tells me which group is active. The --empty flag tells mu to not add all registered repos to that group. If I don’t want to use any groups then mu group reset will go back to the default group with all registered repos.

With groups I can register only the repos that I want to be working across in their own group and not worry about affecting other repos with my batch changes!

Abstract-Base-Class

2022-03-09T00:00:00Z

ABCMeta #

I don’t do a lot of OOP currently, but I have been on a few heavy OOP projects and this ABCMeta and abstractmethod from abc would’ve been super nice to know about!

If you are creating a library with classes that you expect your users to extend, but you want to ensure that any extension has explicit methods defined then this is for you!.

from abc import ABCMeta, abstractmethod
class Family(metaclass=ABCMeta):
    @abstractmethod
    def get_dad(self):
        """Any extension of the Family class must implement a `get_dad` method"""

class MyFamily(Family):
    pass

If I try to instantiate MyFamily I will not be allowed:


❯ my_fam = MyFamily()
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-8-ecb8e21ce815>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Can't instantiate abstract class MyFamily with abstract methods get_dad

In order for me to extend Family I have to implement the method get_dad

class MyFamily(Family):
    def get_dad(self):
        return "Me"

Now everything works as expected and I can sleep well knowing no one can extend my base class without creating methods I know they need.


my_fam = MyFamily()

my_fam.get_dad()
'Me'

Python-F-String-Align

2022-03-08T00:00:00Z

I am personally trying to use logger instead of print in all of my code, however I learned from [@Python-Hub] that you can align printouts using print with f-strings!.

This little python script shows how options in the f-string can format the printout.


import random

variables = "Foo Bar Baz Bing".split()
scores = random.sample(range(1, 11), len(variables))

print("*" * 30)
print("\n")
print("With 'varable' left aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:<10} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' right aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:>15} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' center aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:^5} | {score}")

Python-Builtin-Calendar

2022-03-08T00:00:00Z

Being lazy #

I almost exclusively use Python for my job and have been eye-balls deep in it for almost 5 years but I really lack in-depth knowledge of builtins. I recently learned of an awesome builtin called calendar that has way more than I know about for sure but I’m glad I know it’s here now!

I only needed it because I was too lazy to hard code the 7 weekdays into my module but it turns out there’s a lot of useful things like calendar.isleap()!

## Future use

I’m not exactly sure what will come my way where calendar will be super relevant but like anything, I’m just glad to know it exists for when the time arises!

Dataframe-Memory-Usage

2022-03-07T00:00:00Z

I have often wanted to dive into memory usage for pandas DataFrames when it comes to cloud deployment. If I have a python process running on a server at home I can use glances or a number of other tools to diagnose a memory issue… However at work I normally deploy dockerized processes on AWS Batch and it’s much more challenging to get info on the dockerized process without more AWS integration that my team isn’t quite ready for. So TIL that I can get some of the info I want from pandas directly!

DataFrame.info()

I didn’t realize that df.info() was able to give me more info than just dtypes and some summary stats… There is a kwarg memory_usage that can configure what you need to get back, so df.memory_usage="deep" will give you how much RAM any given DataFrame is using! Amazing tool for finding issues with joins or renegade source data files.

df = pd.read_csv("cars.csv")

df.info(memory_usage="deep")

Adblock-Coverage

2022-03-07T00:00:00Z

I run pi-hole at home for ad blocking and some internal DNS/DHCP handling.

pi hole posts on the way

One thing I’ve never put too much thought in is asking “how well am I doing at blocking?” There’s lots of ways to measure that depending on what you care about but I just learned of adblock tester. It’s awesome and gave me a quick glimpse into how my pi-hole is performing on keeping my webpages clean and my DNS history private!

Credits to d3ward for the awesome tool!

Webservers-And-Indexes

2022-03-06T00:00:00Z

I host a lot of services in my homelab, but they’re mostly dockerized applications so I have never had to care much about how content gets served up. Today I had several little concepts click into place regarding webservers, and it was a similar experience to when I started homelabing and didn’t know what a “server” was in the first place.

Servers

A “server” can have a lot of different meanings but specifically in my world it was a physical server, like my PowerEdge R610 which acts as my main “home server”. But then on my server, I have other servers… Jellyfin is my main media server - but that’s obviously not a hardware thing, that’s software. This is certainly not a groundbreaking thing but it was a tiny piece to the puzzle that I was missing… that “server” is highly contextual.

Webservers

Something that confused the heck out of me when I first started down the road of having a server was what a webserver even was… I always thought the “webserver” was just “a server that hosts a website”… and yes, that’s true, but also it wasn’t true in how I understood “server”. It turns out that across my 40-odd dockerized services I have at home that I must have about 40-odd web servers running, each docker container is spinning up its own!

So something I have wanted to do for a long time is put my theology notes online for my small group to access whenever they might want… it doesn’t need to be fancy or anything. My issue was not knowing what to even Google. I tried “How to serve up static html” but that kind of search is for people who know what a “static” site is - I am not one of those people. I kept running across nginx and apache things, wordpress and other website building tools, etc. In fact I only recently learned that JavaScript assets cann still be considered static so I am a complete baby in the web-dev space.

What I really wanted was just a simple landing page with a link to each of my “posts” which are in the form of a single html file each that I can easily export from my tiddlywiki (I have a post about tiddlywiki here)

The first win python -m http.server right in the directory I kept my html files in and that got me what I wanted functionally. But then I wanted just a hair more organization… I started looking for a way to dynamically generate an index for a directory of html files but again the verbiage of that Google search just wasn’t helping me - I didn’t want anything complicated and I knew that what I wanted had to be easy…

The Index

Luckily I randomly came across a SO that mentioned a Linux utility called tree which does exactly what I wanted!

See my TIL on tree here

So now it goes like this:

Take notes on X in my tiddlywiki
Export that tiddler to a html file
Put that html file into a notes folder in my github repo for small group notes
Use tree to generate an index.html of each of those files in the notes directory
Use python -m http.server to start a web server that lands me at the index.html and now I can click through to any post!

It’s not fancy but it’s functional… This site/blog is built with markdown and markata and I wanted way more functionality in my tech notes. But for this simple use case I learned a ton about how content gets served up on a webpage and my small group benefits from the easy access as well!

Tree

2022-03-06T00:00:00Z

I wanted a quick way to generate an index.html for a directory of html files that grows by 1 or 2 files a week. I don’t know any html (the files are exports from my tiddlywiki)…

tree is just the answer.

Say I have a file structure like this:

./html-files
├── file1.html
└── file2.html

To generate a barebones simple index.html we can use tree as follows:

tree ./html-files -H "." -L 1 -P "*.html"

and get the following:


<!DOCTYPE html>
<html>
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <meta name="Author" content="Made by 'tree'">
 <meta name="GENERATOR" content="$Version: $ tree v1.8.0 (c) 1996 - 2018 by Steve Baker, Thomas Moore, Francesc Rocher, Florian Sesser, Kyosuke Tokoro $">
 <title>Directory Tree</title>
 <style type="text/css">
  <!--
  BODY { font-family : ariel, monospace, sans-serif; }
  P { font-weight: normal; font-family : ariel, monospace, sans-serif; color: black; background-color: transparent;}
  B { font-weight: normal; color: black; background-color: transparent;}
  A:visited { font-weight : normal; text-decoration : none; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:link    { font-weight : normal; text-decoration : none; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:hover   { color : #000000; font-weight : normal; text-decoration : underline; background-color : yellow; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:active  { color : #000000; font-weight: normal; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  .VERSION { font-size: small; font-family : arial, sans-serif; }
  .NORM  { color: black;  background-color: transparent;}
  .FIFO  { color: purple; background-color: transparent;}
  .CHAR  { color: yellow; background-color: transparent;}
  .DIR   { color: blue;   background-color: transparent;}
  .BLOCK { color: yellow; background-color: transparent;}
  .LINK  { color: aqua;   background-color: transparent;}
  .SOCK  { color: fuchsia;background-color: transparent;}
  .EXEC  { color: green;  background-color: transparent;}
  -->
 </style>
</head>
<body>
        <h1>Directory Tree</h1><p>
        <a href=".">.</a><br>
        ├── <a href="./file1.html">file1.html</a><br>
        └── <a href="./file2.html">file2.html</a><br>
        <br><br>
        </p>
        <p>

0 directories, 2 files
        <br><br>
        </p>
        <hr>
        <p class="VERSION">
                 tree v1.8.0 © 1996 - 2018 by Steve Baker and Thomas Moore <br>
                 HTML output hacked and copyleft © 1998 by Francesc Rocher <br>
                 JSON output hacked and copyleft © 2014 by Florian Sesser <br>
                 Charsets / OS/2 support © 2001 by Kyosuke Tokoro
        </p>
</body>
</html>

which looks like this when you serve it up with python -m http.server

Pandas-Select-Dtypes

2022-03-05T00:00:00Z

On my team we often have to change data types of columns in a pandas.DataFrame for a variety of reasons. The main one is it tends to be an artifact of EDA whereby a file is read in via pandas but the data types are somewhat wonky (ie. dates show up as strings, or a column that should be a integer comes in as float, etc.). The best solution I think is to leverage the dtypes keyword argument in which pd.read_X method is used. However there is another way which is to coerce the data types at runtime instead of loadtime.

A handy way to do this is by using pandas.DataFrame.select_dtypes…

Here is an example of finding columns read in as datetime64 and the developer would prefer to use pandas datetimes.

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.columns:
    if df[c].dtype == "datetime64":
        df[c] = pd.to_datetime(df.c)

Here is the difference in code flow between select_dtypes and manually finding the datetype64 columns:

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.select_dtypes('datetime64'):
    df[c] = pd.to_datetime(df.c)

The difference isn’t huge but it’s the little steps in leveling up that turn script-kitty scripts into clean looking functions.

Vim-Auto-Space

2022-03-04T00:00:00Z

I ran into an issue where I had some copy-pasta markdown tables in a docstring but the generator I used to make the table gave me tabs instead of spaces in odd places which caused black to throw a fit. Instead of manually changing all tabs to spaes, or trying some goofy :%s/<magic tab character>/<%20 maybe?>/g I learned that Vim has my back…

:retab

Stow

2022-03-04T00:00:00Z

Stow is a great tool for managing dotfiles. My usage looks like cloning my dotfiles to my home directory, setting some environment variables via a script, then stowing relevant packages and boom my config is good to go…

cd ~
git clone <my dotfiles repo>
cd dotfiles
# env variable stuff ignored here
stow zsh  # This will symlink my .zshrc file which is in ~/dotfiles/zsh to ~/.zshrc

By default stow will stow packages up one directory from the root directory. In this example the root directory is ~/dotfiles and the package is zsh. So the files in the zsh package will symlinked into ~/.

stow makes it easy to share dotfiles across machines, or safely experiment with config changes while always being protected by git since your dotfiles are in a git repo! …They are in a git repo… right?

Stow-Target

2022-03-04T00:00:00Z

Check out stow for a brief introduction to stow

What if I want to stow a package somewhere else? Boom, that’s where -t comes in…

Maybe I don’t like having my dotfiles repo at $HOME and instead I want it in ~/git or ~/personal just to stay organized… Well then I could have the same workflow except the stow command looks like this:

stow zsh -t ~/
#or
stow zsh -t $HOME

Git ammend to a commit

2022-03-04T00:00:00Z

After carefully staging only lines related to a specific change and comitting I suddenly realized I missed one… darn, what do I do?

Old me would have soft reset my branch to the previous commit and redone all my careful staging… what a PIA…

New me (credit: ThePrimeagen)…

# stage other changes I missed
git commit --amend --no-edit

Ubuntu-Static-Ip

2022-03-03T00:00:00Z

Sometimes I need to manually set a static IP of a Linux machine. I generally run the latest version of Ubuntu server in my VMs at home.

In Ubuntu 20 I’m able to change up /etc/netplan/<something>.yml

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

gateway4 is your router address nameservers is a list of desired DNS servers for that machine to use. I usually use my router which is configured to use my pi-hole as my primary DNS, then set 1.1.1.1 (CloudFlare) as a backup

Hit it with the sudo netplan apply and you should be good to go!