Thoughts on tech Archive

The Faulted Disk: harbor Replacement Writeup

2026-10-06T21:30:00Z

The sequel to panicking-led-to-losing-my-desktop — this time the monitoring actually caught the disk dying, and nothing was lost.

What happened #

harbor is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from tank. One side of the mirror, a Seagate Exos ST12000NM0127 (serial ZJV4QFLB, /dev/sdb), went FAULTED with 14 read + 22 checksum errors.

mirror-0                              DEGRADED     0     0     0
  ata-ST12000VN0008-2PH103_ZTM0NFDW  ONLINE       0     0     0
  ata-ST12000NM0127_ZJV4QFLB         FAULTED     14     0    22  too many errors

The IronWolf mirror side carried the pool — No known data errors. ZFS redundancy did exactly its job.

The difference from last time #

Last failure: no monitoring, found out by accident months later, desktop died.

This failure: SigNoz + node-exporter’s ZFS collector → node_zfs_zpool_state{state="degraded"} → alert rule → Gotify → my phone. The gotify notification fired before I knew anything was wrong.

Diagnosis #

Before zpool clear or replace — check SMART. smartctl-exporter already scrapes all disks into SigNoz, so I didn’t even need sudo:

Reallocated_Sector_Ct raw = 3024 and counting
Offline_Uncorrectable value/worst still 100 but raw errors climbing
SMART overall: still PASS (SMART’s overall bit is conservative until threshold)

3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don’t clear.

The swap (the annoying part) #

Hot-plug is never as smooth as it should be:

zpool offline harbor <old> to stop writes to the dead drive — actually skipped; drive fell off the bus on its own
New IronWolf ZTN1CQ07 in hand → plugged into ghost’s SATA → didn’t enumerate
Forced rescan (echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan) → nothing
Moved it to the old drive’s port → nothing
Sabrent USB dock on aurora → dock enumerated as 0B device, no disk behind it → reseated + replugged → drive spun up and appeared: /dev/sdd, 10.9T, old ZFS partition table on it (used drive — part1/part9 layout)
Back to ghost, direct SATA → enumerated as sdb

Lesson: “spins up but doesn’t enumerate” on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.

The replace #

sudo zpool replace harbor \
  ata-ST12000NM0127_ZJV4QFLB \
  ata-ST12000VN0008-2PH103_ZTN1CQ07

Resilver: 8.80T in 18h38m, 0 errors (~154M/s). Pool stayed online and usable the whole time.

One gotcha: after resilver, zpool status showed errors: 1 data errors — but zpool status -v showed an empty error list. The corrupted data was already repaired; the counter was just stale. sudo zpool clear harbor → clean.

The monitoring that made this possible #

Wired during this session (all now in SigNoz):

node_zfs_zpool_state — pool health (node-exporter zfs collector)
smartctl-exporter — SMART attributes incl. reallocated sectors (the early-death signal)
zfs-metrics.sh textfile bridge — sanoid --monitor-* exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success
Alert rules → Gotify → phone

The replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.

Remaining homework #

Buy the replacement spare (the shelf’s empty now)
10Fold datasets are garbage + have zero snapshots — destroy
Every scheduled job emits cron_last_success_epoch now — if a job dies silently again, I’ll know

Same failure as April, opposite outcome. Redundancy did the protection; monitoring did the detection. You need both.

Co-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.

It Will Be Hard To Do More

2026-05-22T06:23:20Z

Recently I was given a raise at work after I broached the subject with my manager. I wasn’t sure what the outcome would be but I have learned two things in nearly a decade in corporate America:

No one will fight for you harder than yourself
The worst they can say is no

The Raise #

So I talked with my manager about my workload and compensation, and to my surprise he took me seriously. He went up the chain with some accomplishments of the year that I wrote up for him (it’s his job to advocate for me, but it’s certainly my job to give him the ammunition to take into battle). Before much time had passed, a big wig put some time on my calendar. I had a good relationship with my (then) manager, and even with the big wig, so there was always some amount of personal connection amidst the money-talk.

The Reorg #

Cue a managerial change and reorganization of the team. The dust settled, and my manager wasn’t my manager and the big wig wasn’t a big wig over my team anymore. The details aren’t that relevant, but the outcome was that my now manager is someone I don’t know personally — back to square one of professional-relationship-building.

The Conversation #

Due to the timing of it all, when my raise became official it was this new manager who signed off on it. It was their happy-responsibility to have a quick chat to tell me it was official. But that conversation was the most awkward of the three.

Does this raise of X% close the gap with the other competing opportunities you mentioned to (previous manager)? Because it’ll be hard to do more than this.

???

Hard to do more?

I was taken aback initially. It felt almost threatening.

I really wasn’t sure how to respond. I think I was grateful and cordial, talked about how I really do believe in the mission of our work and I’m thankful that I can receive a raise. This lady is nice but she doesn’t know what my “other opportunities” are, so there’s no honest answer for me to give. I wanted to be sarcastic, crack a joke about record profits, make light of her describing me as a “critical player in enterprise deliverables,” but I’m not trying to make enemies.

The Reframe #

So I called a friend to vent, of course, and he gave me some good advice.

It was a threat.

But not in the way I took it at first. My friend has been at the same company longer than me, and he expounded on her words: “it will be hard to do more.” I felt like she was annoyed with me for fighting for a raise, but my friend said she was probably just trying to give me a realistic view of my position in the company. “It will be hard to do more” means there are processes in place at such a large enterprise. Rewarding start-up-like behavior and contribution just isn’t well-supported. It’s much easier to let someone quit who’s been with the company a decade and replace them for 20% more than it is to just pay them 20% more. If someone (like me) isn’t satisfied, that’s not the only thing in the conversation - time at the company, time in position, time under specific manager, etc. all are hurdles to more compensation, and I’m referring to salary anyways, equity feels like it’s out of my grasp. But if I’m gone and market conditions dictate that my position and responsibilities are worth 20% more than what I was being paid, that’s what the position will be posted at. My previous manager and the big wig were the types to say policy-be-damned when it would get in the way - it might be how I got this raise in the first place. But no shade to my new manager, they’re just doing their job.

In the end, they were just being honest, and I should appreciate that.

Panicking Led to Losing My Desktop

2026-05-13T08:24:00Z

## False Sense of Security I thought I had backups handled... can you imagine how the rest of this post is going to go with that intro? To be fair, I do have backups figured out on my NAS - simple ZFS + sanoid/syncoid + replica pool + off-site backup with simple restore pathways. However, my desktop has been another story entirely. My desktop OS didn't support ZFS when I started checking it out, and I spent weeks thinking through how I would backup my HOME directory and projects mostly. I landed on a solution that I did validate once, but it fell off my radar and lo' and behold that was problematic... So that backup was based on restic for my home directory, but it was lazy. I verified it one time but I had built it with ai, thought I understood the restic repo part, and then promptly moved on with my life never buttoning it all up. That home directory backup got too big for where I was going to end up restoring it. My desktop system was installed on a 4 TB NVMe drive and due to the circumstances spawning this blog post I was gonna have to drop to a 500 GB boot drive with some extra disks as the storage layer. Overall it looked like: - A 4 TB SSD that was going bad - old OS - A 500 GB SSD, that was going to be my new operating system boot disk - A 2 TB SSD that was originally going to be this external storage volume anyways but I never set it up because the version of Aurora I was running didn't have ZFS, I was married to the idea of using ZFS, so I never ended up taking advantage of the space. However it was moot to me because my boot drive was 4 TB, high quality drive, so I was "just sure" I didn't need it. - AND a 4 TB rust disk as well, which was already a ZFS pool, left over from a previous desktop configuration, and admittedly I had forgotten it was even in the system. ## The Storm If it wasn't clear the problem is that my super-nice high-speed 4TB NVMe drive was going bad, like really bad. Eventually my OS stopped booting, it was even difficult to live-boot from any other ISO due to, I think ultimately, that disk causing such extreme latency in the start-up processes that they just failed. So I quickly found myself with little-to-no access to my primary desktop's data... ## Where It Went Wrong What I did is I live booted into an Ubuntu server environment (which took blood sweat and tears to successfully get into), mounted my home directory from the 4 TB SSD, and tried to continue my restic backup to my NAS, like an idiot. But at the same time I also tried to prune it by only backing up a few projects because I was getting worried about time. This was the first primary mistake - trying to muck with my backup script under duress. Then over the course of the whole thing it ended up taking over a week to solve this when it could've been 2-3 days. So say it with me kids - "Don't make decisions under duress" ## Climbing Out I downloaded opencode and had it help me write the right excludes syntax in my restic backup script and got it back up going. That went ok but opencode agents had no historical context for why anything was the way it was, and frankly an agent would've been misled thinking the backup solution was much more solid than it was due to how I documented it. Agents also miss things... in my chat sessions it knew about the other 2 available disks on the desktop system, I could have done a fresh backup to the 4 TB spinning rust disk no problem: install zfs, mount the pool, change target of restic, run full... that would've been beautifully simple. But instead I trimmed it down and backed not-everything up to the NAS over the network, and to a different backup target nonetheless... SMH. As I started to consider which OS I was going to go with next I failed to install Pop_OS! or Ubuntu onto the new disc... Then I tried Omarchy and the install script just looped. So, I reinstalled Aurora onto the new 500 GB disk and then quickly realized I don't have Firefox tabs, my SSH keys are in that restic backup, my ssh config, api keys in hidden files.... Everything is in that restic backup... The backup that's too big to restore to my new boot drive. But you know what I have? That 2 terabyte disk mounted just fine as a ZFS dataset. And I could mount the 4 TB rust disk with zfs as well because this version of Aurora has zfs working flawlessly! ## Hindsight What I should've done is so simple... While in that ubuntu live environment I should've just either updated restic to be a local backup to the 4 TB rust disk, or rsync'd my home directory to it plain and simple... I got all in my head about not backing up python venvs, node_modules, etc. that I didn't think to just basically carbon copy it all to a healthy disk and then prune it later. Then I could've synced everything back over that I needed to the new Desktop's $HOME and then scheduled the rsync or restic again to that locally mounted disk. ## The Detail I Left Out The keen reader might stop to think... why not just mount the old 4TB disk and copy what you need to your new desktop? And that's a prudent question... However, in order to get anything installed I had to physically remove the 4TB SSD from the motherboard, which was basically a full PC tear-down. From there I was able to at least boot in and out of iso's like you'd otherwise expect, and I have a USB/NVMe adapter so I planned to mount the old drive and copy things over from there... But sadly... it won't mount. it's dead-dead and it appears that anything I didn't save in my days-long-panicked-state is just. gone. I feel pretty stupid to have not taken advantage of the 2 available disks local to the machine, to have naively copied stuff over and dealt with the organization later once my OS was back up. I tried to be smart and efficient and ended up wasting so much time and losing quite a lot of "stuff"... ideas, blog posts that I never committed, etc. ## Current Status So a few lessons... 1. untested backups are not backups 2. false backups might be worse than none, although I did at least save a few things so maybe the jury is out here 3. making decisions while stressed out will lead to missing obviously better pathways... slow down, talk it out As for my current status - I'm working on [[desktop-setup-2026]] and recovering what I can from my haphazard'd rsyncs in the live ubuntu env I got into. I'm also setting up a new Linux laptop at work at the same time so maybe I'll hve some workflow changes to write about in the future. For now, it's nice to be forced to accept that not every idea was that important, the good stuff will come back around, and ultimately computers and shit are just things, they're not life.

The Relief of Earning

2026-05-08T05:43:39Z

The Setup #

Work has been crazy for months, and I haven’t written too much about it, a few notes here and there but nothing substantial. The craziness is sourced in a pretty cliched-scenario where we have lofty goals and important things to build but the foundation on which we are building, like architectural decisions, is basically wet sand. So as I’ve been trying to find my footing in the mud I feel like I end up spinning my wheels moreso than making any progress. The chaos led me eventually to approach my manager about my compensation - I make a nice living but for the stress I’d been feeling I could certainly go make more. My manager successfully fought for me up the chain and I was given a slightly bigger carrot, but that’s not the point of this post. I started to feel even more anxiety after I was told about the raise because, just like with AI and agentic coding, now I was sure the expectations would be even higher. I didn’t think anything would actually come from me asking my boss about a raise, but now that it did I was afraid the ceiling of expectations was just going up up up.

The Carrot and the Anxiety #

That’s where the relief of earning comes in… I mentioned this to my project manager, and we have candid conversations regularly, and he told me something that I think some people might have been offended by but I was genuinely relieved to hear it. He said he’s glad I got a raise, and that he told our boss to not really pursue it for me until I had actually delivered something. So even though I feel like I’m spinning my wheels a lot, my direct leadership sees real value - and the relief is that I can rest in confirmation that the carrot is a response to work I already did, not a quiet elevation of expectations.

Fixing Firefox Launcher on Kubuntu 22

2026-04-11T10:32:50Z

Problem: Firefox installed but missing from KDE/Plasma app menu
Root cause: No .desktop launcher file — Firefox only worked from terminal
Solution: Created desktop entry with standard KDE fields


[Desktop Entry]
Version=1.0
Name=Firefox
GenericName=Web Browser
Comment=Web Browser
Exec=firefox %u
Terminal=false
Icon=firefox
Type=Application
Categories=Network;WebBrowser;
MimeType=x-scheme-handler/http;x-scheme-handler/https;
StartupNotify=true
StartupWMClass=firefox

Location: ~/.local/share/applications/firefox.desktop (user-local, no sudo needed)
Discovery step: Checked /usr/share/applications/ — empty for Firefox
Applied fix: Ran kbuildsycoca5 to rebuild Plasma’s menu cache
Result: Firefox now appears in app menu (Network category) and Alt+F2 autocomplete

Data Loading is a Huge Deal

2026-04-08T07:40:14Z

I've been thinking about the work I am doing and have to do in my role at Cat, in Cat Autonomy, building Forge (see [[forge-ahead]]). I feel like I have little revelations almost every day now, not that it means I'm writing something amazing and producing it really fast but there's just a whole suite of problems that different technologies solve at different levels and the more I become aware of the problems that exist, the more the existence of some solutions makes sense. !!! note "The Problem Perspective" I don't know if this is a real thinking technique or if I'm onto something novel(doubt) but I think a lot in terms of problems - "what problem needs solving?" and that's how I've come to prioritize my work, it's only been very recently that I've realized I do this and I think I should highlight it's very important to document the problem, otherwise every day you might try to solve a different problem but be working on the same code !!! note "Problem Space" Kedro solves a lot of these problems, so when making rada in Reman the problem space was already more contained and narrow, Forge's problem-space is much more vast The one problem I'm really fixated on right now is data loading, the problem I need to solve is accessing data from a wide-variety of scripts/tools, without a framework or standard method/library of accessing data in the first place. I've seen lots of projects on GitHub claim to make data loading easier and I didn't quite understand what problem they were solving... one example to name is [Data Load Tool](https://dlthub.com/). I've seen similar ones I don't have the name for right now that claim to make it easy/fast to load data from s3, or ways to make s3 and a database both abstract in the user-experience for loading data. But I hadn't really understood why these tools existed. I have a lot of experience with [Kedro](https://dlthub.com/) and their [DataCatalog](https://docs.kedro.org/en/latest/catalog-data/introduction/) which provides a python object over a set of yamlfiles that makes it pretty simple to load and save data in a way that isolates I/O from the business logic. But what I didn't realize at the time how powerful that catalog was, the power of standard patterns and shared libraries. Now that I don't have it available to me, I'm quite aware of the absence. In my new role something I'm realizing is that for all the developers my team now supports, there isn't a canonical way to access data. When I was in Reman and working with kedro, the DataCatalog was the access pattern and so when I was developing a platform I never really had to think about it - it was an established pattern that I treated as a constraint and then built processes around it. I've been battling some mental block for weeks on Forge because of the lack of that canonical pattern, and as I've talked with other engineers it seems like the baseline assumption is that data is just available on a filesystem, but everyone's code loads data in different ways. On my small Reman team, with common patterns to build on, it was easy to make things cloud-native or shim in some devops to improve people's lives. But when everyone's doing their own thing, and everyone's "own thing" is very much built-on some rigid tribal patterns then it's hard to really move fast cause everyone isn't already moving in the same direction. That made me realize that the first problem Forge needed to solve was in providing a way for engineers to have filesystem-native data access in the Cloud, where we are S3-first in our storage philosophy. I didn't need to figure out a way for everyone to name a dataset, define the dataset in the first place, and give a nice `my_dataset.load` that worked in python, bash, cpp, and who knows what else.... I reframed the problem from "how do engineers load up the data" to "how do engineers have access to the data". The requirements of the Cat Autonomy group was pretty simple: POSIX-compliant storage. My pathway to solving this problem is initially underway, I can't imagine it'll be too difficult to setup for FSx instances for teams and give them an api to run a Batch Job with the FSx mounted. From there, their code can load data from `/mnt/fsx/` just like they otherwise could be doing locally. Or maybe FSx will let us setup mounts to very flexible mount points and their local scripts will "just work" :shrugs:. I don't know the exact shape, but after realizing the loading data is a big deal, I'm thankful I have a narrower problem to solve first. !!! note "S3 Files" Literally yesterday, AWS launched "S3 Files" offering an NFS filesystem service over buckets. I'm not sure if NFS is going to be a viable filesystem protocol for all of our use cases, but looks like we're not the only people who need the filesystem access patterns over S3. !!! warning "A Future Problem - Canonical Reference" Another high-value thing Forge needs to solve is "what is data". The data formats we have are not super simple, it's not just a set of SQL tables. We have files that relate to each other based on hard-filepath patterns, and those patterns are full of tribal knowledge and distributed processes. So a simple question like "How do I use forge to access my data" is hard. In Reman a data scientist would ask "how do I use rada to access my data?" and the answer is "We use Kedro, and Kedro solved that problem for us via the Catalog" but without Kedro, without 100% being in python (devs are also in embedded systems, cpp code, and more), without even consistent practices in the existing "how do I access my data" workflows, it's really impossible to systemetize and codify it. It is my next challenge to tackle though...

Agents can miss obvious things too

2026-04-03T14:39:51Z

I am working with MiniMax M.2 to vibe up some webapp ideas for [[Nexus]] and I just spent many minutes not having the app render correctly because the agent mounted the wrong directory into the docker compose stack. I noticed it about 5 seconds after opening the file, but wasted nearly a half hour thinking it was a docker issue in distrobox... !!! danger "" Clankers are still just clankers man

The Irony of 10x

2026-03-26T06:00:50Z

## Opener Agentic coding has been an exciting change to me over the last couple of months specifically. I've been using AI tools for a few years now but something really shifted with Opus 4.5 as well as the tools/harnesses getting better and more useful around the same time. I've been influenced by people like Simon Willison and Steve Yegge who have been on the forefront of agentic coding and [vibe engineering](https://simonwillison.net/2025/Oct/7/vibe-engineering/) since the dawn of ChatGPT and in my small circles of work I'm definitely on the bloodiest bleeding edge of the adoption of these practices. Are they the future? I don't know - I tried to maintain a skeptical posture but until the bubble pops it's looking like this is at least a direction the future of my line of work is going. !!! danger "" The tough part is mixing the new world of agentic coding with developers on wildly different points in the spectrum of adoption and maturity. Here's a short anecdote about what I mean... In a new project that I'm on, I've refactored/reimplemented a lot of legacy code, producing about 140k lines of code, configuration, and docs in about 2 months. It's an insane amount of "product" and I've done it entirely with agents. But I didn't do it with a handful of vanilla chat sessions like "Hey Copilot, reimplement this API, no mistakes". I've been building out my own process for using LLMs effectively. I have no actual idea besides my own experience if what I'm building is useful, but it feels pretty good - agents for planning, building, reviewing, testing, etc. My "harness" includes specialized agents and opinionated development workflows to try to ensure that code is never one-shotted into production. The catch though is that I'm working with a handful of developers and they are not early-adopters or aggressive experimenters with these new agentic tools. Most of them are still in Steve Yegge's stage 2 or 3 of AI coding as he's outlined [in this medium article introducing GasTown](https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04). They open a chat session, and say yes or no to Opus. I've been in-between stages 6 and 7 for a while now - managing multiple agentic sessions that themselves run specialized subagents primarily for context management, although I don't feel quite ready for full blown stage 8 (agents running agents running agents). So this isn't me saying that I'm "better" than the guys I'm working with, just the project we're on together is now being touched by people on wildly different ends of the agentic coding spectrum, and the project itself is an experiment to me of living in the "New World". Finally the anecdote - I'm getting PRs from guys in stages 2 and 3, thousands of lines of code and config, that I know they are not necessarily experts in, but they're producing that code with fairly vanilla practices. How do I verify it? I know my own prompts, I know what agents generated my code, I know the direction and prodding I gave as they're building, but all I know in a PR review from someone else is the diff - not how they tested it (unless of course there's tests, and there must be, but tests are only as good as the tests are...), not the steering they gave, not the manual UAT validation they did, or even the full intent... And ultimately I own the code they merge because chances are I outlast their contract with the team. It's kind of a scary thing to review and accept... do I trust my agentic practices to validate their work? Cause I certainly don't trust myself to validate it perfectly. !!! note "a note about contractors" I'm not implying anything about anyone who works contract, I think the facts is that contractors are typically dis-incentivized to really "own" something - an unfortunate consequence of the world. ## This is about me, not everyone I want to make sure to reiterate the context of this post and my thoughts - it's really about me. I'm not saying anything here is true for anyone else, I'm not making predictions about tomorrow, and I'm not a prominent FOSS developer or a high-profile ex-FAANG engineer. I'm not talking about every project under the sun. I work at a big company, not a fast-paced startup. I work on internal tooling, nothing that should even be seen outside our network. I'm also a self-taught developer, not a trained software engineer. As far as AI goes, lately I'm mostly all in on Github Copilot CLI with a sprinkling of Opencode. I'm not using Claude Code or Codex which seem to have their own communities around plugins and usage. And since we're so focused on me right now, the important thing to keep as the backdrop for this post is my temperament and make-up. I'm definitely made a certain kind of way, not different than every single person necessarily, but different enough, from enough people, to not blend in with the 9-5 guy who can log out and not think about work until the morning (I'm not saying that's a strength either, my boundary problems are for another time). I feel an aggressive burden to solve problems and own those solutions, call it white-knighting or a savior complex if you want, but I've got enough of a reputation at Caterpillar now (and anywhere else I've worked) to become a go-to person for more than I think is necessarily appropriate. And with that burden I'll bring results - if a problem hooks into my brain it. will. be. solved. I probably won't do the best most clean-code solution right out of the gate, but I'll do whatever I can to find a solution. ## Who I was before agents That is who I was before agents made code easy to produce. My first boss at Caterpillar used the phrase "tenacious learner" to describe me in several reviews. I kind of rejected the description because I basically refused to believe that I was really any different, any harder of a worker, than my peers. But with almost a decade of experience in the corporate world, and some adult perspective on my life, I think it's accurate... I am a harder worker, to my detriment sometimes, than a fair number of people I've worked with\*\*. I'm not brilliant but I can focus for a long time in the right circumstances. The gift of perseverance (or the curse of not being able to let something go, depending on how you look at it) has led to blessing in my life in both reward and skill. !!! note "\*\*" There's nothing wrong with it either, Cat's somewhat noticed that work in my EOY reviews and I'm certainly not against "just doing your job". And then AI came along and with another set of the right circumstances catalyzed a new way to work. ## Leaning into "Agentic Engineering" I jumped onto experimenting with AI coding tools as soon as they became available, but mostly I just tried vibe coding rather than using tools for real engineering work. I vibed up an API at one of my jobs that went into production way too early, with far too little validation, and it was scary to support it from then on out. I also did the meme, vibe-coded a TODO app, and threw that puppy into the internet without locking down my API endpoints... That was before agents were quite as useful as they can be now, but that experience along with a handful of other stepping stones (like learning some real actual fundamentals about security) began to give me confidence in using the AI as a tool, like my IDE is a tool, for producing **solutions** that take the form of code. !!! note "" As one-shot apps got better and better, and as I learned about scoping work more appropriately for agentic tools my confidence in them grew. There's quite a difference between "Claude make me a todo app, no mistakes" and scoping out a solution in natural language, with some technical guardrails, and having agents tackle the implementation methodically. ## What actually changed What's actually changed for me is quite a lot... I haven't opened my IDE to seriously write code for months now. I've oscillated between Opencode and Copilot CLI, leaning moreso into Copilot since it's an approved tool at work and as of mid-February is quite good. Mentally I'm approaching problems with a little more thought on the front-end than before because prior to agents I would think as I implemented. At the scale of work that I do, this was really fine - working on CLI utilities to solve simple problems, developing an iterative testing cycle for each problem that allowed me to move fast, and once I found a groove I was cooking. But now I don't even need to find it, I open Opencode or Copilot CLI with my Planner agent, describe what I want to happen and have Opus or GPT scope out a plan for me. Usually there's some back and forth on feature scoping, then I review a markdown file it produces, and once it looks decent enough to me I say "go" and it goes. That works a lot better than I even care to admit because at the same time as I've been leaning harder into agents, I've been building my own harness of sorts - not a replacement for Copilot CLI or a competitor to Opencode, but moreso an opinionated workflow spine that I force agents into to give strict gates to the SDLC (software development lifecycle). !!! warning "Problem Solving Workflows" Plan and implement is fine for a lot of things, and I do think it's only getting better. My harness, mentioned a few times around here before, called Nexus, is a set of agents and rules that I want the code I'll be responsible for to go through before it lands in production. That cycle isn't too complex, and there's only about 10,000 similar tools to Nexus on Github trending right now. I've thought about dropping my idea and picking up something more popular, like [superpowers](https://www.github.com/obra/superpowers) but at the moment I'm continuing to develop on and lean into my own idea here. !!! note "Mini post on Nexus" I keep saying a blog post is coming, but the high level of Nexus is that it's a task tracker with a CLI that agents use to advance a ticket through a plan -> build -> test -> review -> verify -> merge lifecycle that is almost exactly how I would otherwise have solved a problem by hand. I think it needs work, I need to be harder on TDD methodlogies with agents, and work on verification gating a bit more (shoutout to [showboat](https://github.com/simonw/showboat) by Simon Willison) but overall it's a system of thinking that I already participate in so I'm doing my best to farm out specific parts of my workflow to agents rather than trying to one-shot enterprise problems and solutions. !!! danger "Who's doing the thinking?" I've noticed that as I've developed Nexus out though, I lean on the agents for more and more of my own thinking, and am trusting my problem solving **process** moreso than my actual problem solving abilities. ## Hidden costs The cost of this increase in speed is a lack of familiarity - and the fallout of lack of familiarity is hard to express. There's also many facets to it. For me, the first facet is that Nexus helps me move fast, but as I've leaned into it for more and more of the planning, I'm less and less familiar with the state of the code. I find myself asking my reviewer agents in fresh sessions often to explain it to me, and thankfully they're usually consistent, but nonetheless I'm still not intimately familiar with the code. And on Nexus it's not a big deal, that's low stakes, it's just me and my workflow. I'm using Nexus + Copilot at work and that feels like higher stakes... I have my agents explain the status of our project and although they're also somewhat consistent the thing that's scary is that other people are working on that repo with me, and that's where another layer of complexity manifests itself. If it's just me and my [[clankers]], let's go all day long, rebuild, ask questions, etc... but I have other developers I rub shoulders with now, and if they ask me a question what am I going to say? "Hold on, let me prompt my agent for you" - it's LMGTFY on steroids. And the burden becomes if I feel like I can own and support what those other developers push into the repo. ## Murky responsibility boundaries Why do I own their work? Well for the third time, this post is pretty self-centered and all about me, and my situation is that the other developers I work with presently are all contractors. Their work agreement with Cat could end at any second, for practically any reason. The incentive structure isn't there for these guys who technically work for an agency... Their bonuses aren't bigger (or even exist) if Cat performs well, there's no extra vacation days in it for em (aye, contractors don't get vacation days anyways), and not that it's a problem, moreso just the nature of the world we're in - but they're basically mercenaries out to the highest bidder and I happen to know of **multiple times where Cat lost a good person to a higher bidder**. So this isn't really me trying to be negative about contractors at all, I'm here for a pay-check as well but Cat at least gives me SOME incentive to work hard with the goal of compensation regardless of how altruistic I feel in my own circumstances. !!! note "Incentive" Better ratings mean marginally better end-of-year salary increases, and I've received some other awards that certainly give me pause about jumping ship to another long-term place even when things can be crazy at Cat. It's more than just the contrator-ownership dilema, I've dug myself quite a hole over the last 8-10 years, gaining a reputation that I think many would appreciate, but for me only lately increases the stress. I don't need to parrot every accolade I've ever received, that's not the point, but to make the point as clear as I can - I have a lot of respect from quite a few people at Caterpillar. I'm blessed to have that reputation, and it's not like I haven't worked hard for it - but people talk about me in a way so flattering I feel like the main character in a fictional story sometimes. In a fictional story I can check out the ending, hit up spark notes, or ask AI how it ends... but there isn't an "end" in my real world scenario, there's only tomorrow and I feel the pressure of not knowing what tomorrow holds now more than ever. !!! danger "" Being noticed is starting to feel more costly than rewarding... ## Financial irony What's the cost? It's hard to get specific without writing a novel but here's the TLDR - because I've been pretty good at what I do I've been able to do this type of work outside my normal 9-5 responsibilities and with that extra work has been some pretty great financial benefits. However with Cat changes, responsibility increases, and now owning code that others (and their clankers) write, the extra time I gained for myself is eaten-up and has been reclaimed by the mega-corp... "Exceeding expectations" every year just meant the bar is raised, the expectations are higher, the time-commitment requirement is higher, and as I've had to meet the requirements of both the new world and the curse of being noticed, I've lost the time for the extra work... For years I've realized the benefit of my own skills and drive, but the irony of agents (and a handful of other things) is that with the dramatic increase in expectations, not only on me but on those I work with and therefore their output, I don't get to realize the benefits of my own gifts anymore. ## Meaning and fatigue I feel very torn because the work I've been called into with Cat is good, I said in [[cat-autonomy-2-0]] that autonomy will save people's lives. I love getting to participate in that mission, it's the primary reason I didn't jump ship to try to maintain the levels, and type, of work I was doing before... But in a few short months the mission is being drowned out by expectations and requirements that are so high I'm losing the grip on my own life. ## Open questions That leads me to questions that I can't answer, the question I ask daily now of "What about tomorrow?". What will agents do for us tomorrow, what problems will be solved, what bugs will I create (by agents of course because I've never written a bug by hand in my whole life \s). If I stopped using agents would people still be impressed? Would it even matter? ## Fin I'm certainly not anti-AI, it's typing all my code. I'm not anti-collaboration, although I do wish I could work alone with just my clanker-army to worry about. I'm not sure what I am anymore though... AI has changed how I work, what I work on, and who I work with... Everything has changed in such a short period of time and like the ending of this post, it's pretty jarring. !!! danger "" Death comes to us all - James Acaster. Thanks for reading.

Remote Terraform State Requires Working Traefik… DUH!

2026-03-23T08:12:25Z

I’m working on some spring cleaning in my homelab and backed myself into a hilarious corner yesterday. I use Open Tofu for any of my Terraform needs now, and although I don’t manage a ton with terraform, I do manage all my cloudflare stuff with it. I decided I wanted to use my own minio instance as the s3 remote state backend for my workspaces so I could rely on my typical NAS data backup/retention workflow for the buckets in case anything went wrong, as opposed to a local state file that I’m not taking a lot of precautions with. Well during my Spring Cleaning I was working towards replacing ingress into my home network with Cloudflare tunnels and in the midst of that update I took down traefik, no matter a simple ’tofu apply’ should get me right back to working order…

╷
│ Error: Error inspecting states in the "s3" backend:
│     operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NotFound: Not Found

Hilarious problem with thankfully an easy fix… downloading the state file from Minio wasn’t a big deal since the container was still running without issue, and placing the state file in the folder to use as the local state solution for the interim went totally smooth, but this highlights the set of interdependencies I’m creating for myself and as I take the next few days/weeks to do some spring cleaning I’m hoping I can separate out the external ingress from internal with a bit more clear boundaries so that I never lock myself out of a workflow I only execute on my LAN in the first place!

To Live In A World Without AI

2026-03-22T14:50:38Z

I’m finding lately that I wish we could go back to pre-ChatGPT… A world without a code-gen easy button, where “easy” was LSP autocomplete, where tools were at my fingertips rather than remote inference endpoints, and where I and everyone I worked with, was paid and judged based on what we could produce with our own 2 hands, even if ctrl + c and ctrl + v was sometimes a common set of keys… it sure beat answering “yes” to a robot every 38 seconds.

paynepride dot com outage on vacation

2026-03-16T13:21:31Z

The day after I leave for vacation I start getting SSL errors on every homelab service I host for myself and others. The culprit was my Cloudflare API token expiring. It was easy to find the 403s in the logs for Traefik (thank goodness for Tailscale getting me into the lab from afar). The solution was to rotate the API token, replace the value in Traefik’s .env file, and hit it with the “just deploy” button. Now I don’t know why this expired - the key looks like it has no expiration to me - and I’m too tired from the beach to dig in further. Until next time, I expect this error to come back March 16 2027 I suppose.

My Thoughts on Beads

2026-03-03T05:00:47Z

[Steve Yegge](https://en.wikipedia.org/wiki/Steve_Yegge) is a pretty well-known individual in the tech field, having been around for a long time at some of the larger companies. He's making quite a splash in the agentic coding world as well. I've appreciated Steve's posts and projects lately and wanted to put some thoughts on one called [beads](https://github.com/steveyegge/beads). ## Beads Beads is an issue tracker with links - issues relate to and block each other, but agents use beads to keep track of information and dependencies without storing it in their context 100% of the time. It seems like a very popular and useful tool - but I am not using it, and that's what I wanted to capture... why not? The answer for me is about **where** the organization layer is for the developer. Beads exists in a single repo - it's a system-wide CLI but you 'bd init' in a git repo, and beads uses the `.git/` folder, worktrees, [dolt](https://docs.dolthub.com/), and some git hooks to operate within that git repo. Outside the repo, it takes another tool to tie together all the beads databases you might have. For me, I'm hardly "in" a git repo anymore. My workflow is that when I have something to work on, I create a "workspace" ([self-defined concept](https://pypeaday.github.io/dotfiles/terminal/workspaces/#installation)) which is just a folder on my filesystem where I check-out git worktrees from any of the repos related to the work I'm doing. Sometimes it's 1 worktree from 6 repos, sometimes it's 6 worktrees from 1 repo for parallel work... So because I like to organize myself in this way, beads is already "out" for me. That's the main reason - I don't have any real technical issues with beads or any criticism, it just is designed for a workflow that is not how I work. This is why I'm building [[nexus]], something I hope to be able to put out there "soon". It won't be as general-purpose as beads, but my goal with it is to be plug-and-play for any agentic harness (copilot cli, claude code, opencode, etc.). It's a challenge thinking about it as a personal tool but also as a tool to share someday, but agentic coding is making it possible to make some cool shareable stuff and I'm excited for my own workflow-task-manager to mature and at least become something useful to me (it already is, but building the plane in the air makes it kind of hard to enjoy the plane). ### Credit - banner image from ChatGPT

Forge Ahead

2026-02-17T05:14:15Z

Yesterday's [[reflection-contentment-and-work]] has a second-part this morning. As I was wrapping up a project I didn't realize the closed-off-ness of leaving... I handed in some notes, and mid-message to someone the clock struck midnight and I was locked out. It's fitting to be honest, and now in the wake of yesterday's contentment post, experiencing some more loss than I expected, today we forge ahead. My main focus for work now is a project I will refer to as `forge`. It is what I will bring to Caterpillar Autonomy but I can only build it thanks to the gifts and experience God has given me. The thing I want to put on paper is a short list of experiences I think God in his sovereignty, gave me over the last few years and what they provide for me for Forge. ## Cat Reman Platform One of the first things that comes to mind is another project I was sad to lose: a platform I started for an analytics team in Cat Reman. That began as a simple python cli to automate some developer operations that was otherwise a dozen clicks through the AWS console. Using [boto3] and some simple python I gave that team the start of some real velocity gains. Eventually that grew into a larger kubernetes-based service where the data science operations: code quality, deployment, updates, webapps, etc. were all handled by our platform. It wasn't perfect, it wasn't self-contained, it was a set of things kind of glued together with systems and scripts, but it worked, it works today and is under fantastic ownership. What I learned just from starting that cli was to be passionate about solving problems. No one asked me to make it, but it needed made, and the team is in such a better place for me having started it. ## Kedro and OpenShift Another short project I was able to participate in a few years ago was leading a data-syncronization task into a fiery horrific crash that lasted weeks - neigh months longer - than was necessary or appropriate... The details aren't relevant - I was the lead dev in a new place tasks with syncing up data between 2 applications. Ultimately, could've been a python script but I over-designed a kedro-based solution because of some requirements I misunderstood. Part of that misunderstanding was not knowing how to get the requirements I needed, but they weren't provided in full, I didn't know any better, so for weeks we were a corporate meme trying to use python to update a database that no one on our team understood if we were allowed to write to.... It was very confusing. But what I learned was a lot about gathering requirements, questioning assumptions, and the importance of understanding your constraints as fully as possible as early as possible. ## A Real Platform Then one of the biggest blessings I see for Forge, is the experience I've gotten recently with AWS at scale... Not 1 or 2 services created with Medium.com copy pasta tutorials from docs... but experience working in a large project across many accounts, supporting several teams, using a wide variety of technologies from Terraform to Kubernetes. On this project I learned about practical system design and gained a lot of confidence in supporting systems that are complicated... I'm not the smartest guy in the world but I'm no dummy, and even while lacking fluency in the system I was supporting on this project I learned a lot of troubleshooting skills and gained confidence in my ability to troubleshoot complex systems. I'll need this for Forge - which will break certainly, but I'll be there to fix it and I'm sure I can because I've done these other things. ## Fin The Lord has been with me through these projects - he has certainly blessed me with a skillset and personality that lends itself to being really useful in the Tech world I've landed in (which is a whole 'nother story of God's sovereignty and provision). I am very grateful for the swath of experience I've gotten over the last 8 years or so, and this season I'm in of change is rocking me a little more than I anticipated but by God's grace I think I see the purpose, or at least **a** purpose, and I pray I am making the choices for work he wants me to make.

Kubernetes External Secrets Operator

2026-02-10T07:48:34Z

I wanted to put a short demo together of using External Secrets Operator (ESO) to expose secrets from a vault (like Hashicorp Vault, AWS Secrets Manager, etc) to services running in kubernetes Demo code is [here in this github repo](https://github.com/pypeaday/blog-kubernetes-external-secrets-operator-demo) This post is a high level overview of the components, see the repo for the full example. ## Setup - [[docker]] for containerized development - [[kind]] for setting up a quick cluster - [[kubectl]] for accessing the cluster - [[helm]] for installing ArgoCD and ESO - and then [justfile](https://github.com/casey/just) is there to wrap the commands to easier execution ## Step 0 - Vault - for the demo we'll setup Hashicorp Vault in docker compose to easily bring it up and down - and the init-script is in the repo - it uses curl to make some secrets in vault that we'll reference later ```yml services: vault: image: hashicorp/vault:1.18 container_name: vault ports: - "58200:8200 environment: VAULT_DEV_ROOT_TOKEN_ID: root VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200 volumes: - vault-data:/vault/file cap_add: - IPC_LOCK command: server -dev -dev-root-token-id=root volumes: vault-data: ``` - bringing up the vault instance is a simple `docker compose up` (use the just recipes which some `curl` commands for checking status etc.) ```bash curl -s http://localhost:58200/v1/sys/health | jq . # or just vault-status { "initialized": true, "sealed": false, "standby": false, "performance_standby": false, "replication_performance_mode": "disabled", "replication_dr_mode": "disabled", "server_time_utc": 1770893657, "version": "1.18.5", "enterprise": false, "cluster_name": "vault-cluster-acfb9930", "cluster_id": "4d9162f4-e501-371b-7f94-bd60052b40a3", "echo_duration_ms": 0, "clock_skew_ms": 0, "replication_primary_canary_age_ms": 0 } ``` ## Step 1 - App - We need an app that requires secrets - app code in repo, essentially it's a python webserver to show the vault values (obviously this would expose real secrets so it's just a demo) - Below is one of the endpoints in the vibe-coded app, just to illustrate that we're going to give secrets to the app as environment variables (or as mounted files!) - In the repo, the app is included and there's a `just build` which builds the docker image - There is also `just deploy` which handles loading the image into `kind`'s image cache ```py # example route from demo-app - see git repo @app.get("/env", response_class=HTMLResponse) def show_env(): # ESO brings Vault secrets into environment variables env_vars = dict(os.environ) # Sort by category, then by key sorted_items = sorted(env_vars.items(), key=lambda x: (classify_env(x[0]), x[0])) cards = "".join(create_card(k, v) for k, v in sorted_items) secret_count = sum(1 for k in env_vars if k in SECRET_KEYS) config_count = sum(1 for k in env_vars if k in CONFIG_KEYS) system_count = len(env_vars) - secret_count - config_count html = HTML_TEMPLATE.format( cards=cards, secret_count=secret_count, config_count=config_count, system_count=system_count, ) return HTMLResponse(content=html) def read_mounted_files(directory: str) -> dict: """Read all files from a mounted directory.""" files_data = {} if os.path.exists(directory) and os.path.isdir(directory): for filename in os.listdir(directory): filepath = os.path.join(directory, filename) if os.path.isfile(filepath): try: with open(filepath, "r") as f: files_data[filename] = f.read().strip() except Exception as e: files_data[filename] = f"" return files_data ``` ## Step 2 - Cluster - use `kind` to bring up a cluster - this will start a few docker containers to act as your control-plane and workers ```yml # kind-config.yml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: eso-demo nodes: - role: control-plane extraPortMappings: - containerPort: 30080 hostPort: 58080 protocol: TCP - role: worker ``` ``` kind create cluster --config kind-config.yaml --name eso-demo ``` ## Step 3 - External Secrets Operator - installed with [[helm]] from the official helm chart - NOTE: this is the Operator, not the secrets... this is the thing which goes to the secrets backend and creates kubernetes secrets ``` helm repo add external-secrets https://charts.external-secrets.io 2>/dev/null || true helm repo update helm install external-secrets external-secrets/external-secrets \ --namespace external-secrets \ --create-namespace \ --wait ``` In the repo this is mostly `just eso-install` ## Step 3.5 - Secretstore - You need a `clustersecretstore` to be the place that ESO puts secrets ``` apiVersion: external-secrets.io/v1 kind: ClusterSecretStore metadata: name: vault-backend spec: provider: vault: server: "http://10.10.0.1:58200" path: "secret" version: "v2" auth: tokenSecretRef: name: vault-token key: token namespace: external-secrets ``` ## Step 4 - Secrets - Secrets go in the `clustersecretstore` - in this example it's called 'vault-backend' - In the demo we can just `kubectl apply -f ` to deploy the secret to the cluster - In practice this should be handled by something more mature than raw-doggin kubectl commands ```yml # manifests/external-secrets.yml --- apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: demo-app-secrets namespace: default spec: refreshInterval: "10s" secretStoreRef: kind: ClusterSecretStore name: vault-backend target: name: demo-app-secrets creationPolicy: Owner data: - secretKey: DATABASE_PASSWORD remoteRef: key: secret/data/demo-app/secrets property: database_password - secretKey: API_KEY remoteRef: key: secret/data/demo-app/secrets property: api_key ``` ## Step 4.1 - Files - ESO supports mounting files to containers as well through special `ExternalSecret` resources - One of the example seecrets is a TLS certificate ```yml # manifests/external-secrets-files.yml --- # File-based ExternalSecret for TLS certificates # These will be mounted as files in /etc/secrets/ apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: demo-app-tls-files namespace: default spec: refreshInterval: "10s" secretStoreRef: kind: ClusterSecretStore name: vault-backend target: name: demo-app-tls-files creationPolicy: Owner # Template to ensure proper file formatting template: type: Opaque data: tls.crt: "{{ .tls_crt }}" tls.key: "{{ .tls_key }}" data: - secretKey: tls_crt remoteRef: key: secret/data/demo-app/tls-files property: tls.crt - secretKey: tls_key remoteRef: key: secret/data/demo-app/tls-files property: tls.key ``` - Notice how there's a `spec.target.template` which templates out the file contents from the secret contents ## Step 5 - Helm Chart - This isn't about setting up a helm chart so I'm not going to explain a lot but the working example is simple, not secure, and in the repo - The helm chart renders manifests - I've paired one down and added comments to the relevant things - The thing to just take note of is the reference of the secrets in the `envFrom` section ```yml # deployment.yml apiVersion: apps/v1 kind: Deployment metadata: annotations: meta.helm.sh/release-name: demo-app meta.helm.sh/release-namespace: default name: demo-app namespace: default spec: replicas: 1 template: metadata: labels: app.kubernetes.io/instance: demo-app app.kubernetes.io/name: demo-app spec: containers: - envFrom: - secretRef: name: demo-app-secrets # name of example secret from section 4 - secretRef: name: demo-app-config # another example in the repo image: demo-app:latest # the image you built and loaded into kind - simple 'just' recipe in the repo imagePullPolicy: Never name: demo-app volumeMounts: - mountPath: /etc/secrets name: secrets-volume readOnly: true - mountPath: /etc/config name: configs-volume readOnly: true volumes: - name: secrets-volume secret: defaultMode: 420 secretName: demo-app-tls-files # example secret file from section 4.1 - name: configs-volume secret: defaultMode: 420 secretName: demo-app-config-files ``` ## Step 5.1 - Deploy - We can deploy the demo-app from the git repo to the cluster - For a local demo a few things happen - local image build - loading that image into [[kind]] (`kind` doesn't have access to your host's docker image cache, so images need to be loaded into the cluster cache) - `just deploy` takes care of this for you, read the recipe in the repo if you're interested in more there, the focus of this post and example are to briefly show how to use ESO though ## Step 6 - Profit The example app just displays things that are mounted in - totally vibe-coded to illustrate the secrets mounting, not the appropriate way to leak secrets. ![20260210233804_614254b7.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260210233804_614254b7.png) ![20260210233828_19852225.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260210233828_19852225.png)

Cat Autonomy 2.0

2026-02-09T06:07:12Z

I wrote about my new role [[new-job-caterpillar-autonomy]] a bit a couple weeks ago during an insanely busy time - having just started the role and wrapping up what ended up being side-work, paired with some sleep-deprived ADHD hyperfocus on my new responsibilities and [[nexus]], that post is less of an update and more of a mind-dump. This post is meant to be a calmer update about my upcoming time with Caterpillar Autonomy. It starts, like all good stories, with Caterpillar's RTO (return to office) mandate and how it has affected a lot of people negatively (in my opinion). I haven't seen one positive remark about it fromanyone making less than $250k a year + Bonus + Equity if you catch my drift... I know of people who really did lose their jobs at the end of 2025 for refusing to move. My story with RTO is covered elsewhere, I'd been blessed with remote work and was even secure in a remote position in the face of Caterpillar being willing to fire me [[after-exceeding-expectations-for-4-years]]. But time passes and life happens, and the sovereign Lord brought me to this current circumstance: where I'm fully committed to Caterpillar Autonomy, and it feels weird to say it. I'm going to skip the RTO details, if you know you know. Where I'm at right now is on the other side (or very nearly on the other side) of a choice I didn't feel freedom to make because it came from a conviction. The manager of the group I'm in told me a story about working for another mining company, where he was managing mines in the Congo. These mines can be in the most remote of remote places on earth - in cultures where human life is definitely treated differently than in the first-world midwest USA. Operators of these huge mining trucks in these parts of the world can be smoking meth in the cabs - which obviously leads to unsafe operation. Safety of the mines themselves can be a second-thought as well, they collapse and then the people inside have to be excavated out. Sean told me he had come back to work after a weekend just to be told, as if it was no big deal, that "Motombu died" and to find another operator. Very crass attitude towards human life... I'm still pretty frustrated at Caterpillar for how they've handled RTO (and corporitisms like "Caterpillar family" and "we bleed Cat yellow" are distracting nonsense), and part of me wants to leave still over the personal offence, but Caterpillar will make the safest mining equipment, without a doubt. And given that Cat will make the safest equipment then the story I was told and the mission I couldn't help but believe in is that rolling out good autonomy software and solutions to these remote mines will save lives. !!! warning "" I'm obviously not like superman or anything, but something feels better about enabling good autonomy rather than supporting an online sales platform, or working for a bank... I was invited (coerced?) into that mission - and my role in it is enabling the Autonomy group to manage data for their software stack. I get to help these engineers be experts in the things they're experts in by offloading some operational overhead (data movement, pipelineing, infrastructure and devops, etc.) so that they can produce the best autonomy software on the planet.

Learning How To Agent

2026-02-03T05:50:33Z

## Introduction & Background I've been using AI tools for codegen for a few years now, but not super heavily. I either use the in-line copilot stuff, which is like LSP on seteroids, or I have gone all the way to the other side of full on vibecoding with Windsurf. That's been fun enough, but not super fulfilling and at the end of that I either have a simple webapp that does what I want but I don't understand, or else a half-broken thing I tried to understand but couldn't prompt in the right direction. !!! note "One Caveat" My one caveat with Windsurf, which is a full IDE that I don't have a lot of comfortable navigation in due to all the Cascade keybindings clobbaring my own keymaps - is that with a HEAVY spec, it's done will with the Claude models at implementing a real idea I have. It's closer to vibe-engineering than vibe-coding, but that's my only instance, the rest of my Windsurf usage is "make me a cool app - no mistakes" This year it feels like some tools exploded and I mentioned this in [[new-job-caterpillar-autonomy]]. I've been using [opencode](https://opencode.ai) a lot over the last few weeks and have iterated many times already on a system of work that I'm trying to lean into for improving my efficiency. ## The Problem Space My desire for a great Developer Experience is years old now, shout out to ThePrimeagen for his [FEM course](https://frontendmasters.com/courses/developer-productivity-v2/) and [Waylon Walker](https://waylonwalker.com) for being a constant source of encouragement to be the best developer I can be. I sometimes (often) get tunnel-visioned on developer-productivity initiatives and lose the forest through the trees when ironing out a workflow - generally to find out I way over complicated the solution OR worse, started solving a problem I don't even have. ## First Attempt: Local Progress Tracker Well that's where I've been for a few weeks... I'm building Nexus, my second-brain at work to collaborate with agents on the truckload of stuff I'm expected to get done. For a while now I've had a "working-notes" repo, which is basically a blog, built with markata and navigated via markdown-lsp, where I do like what I do here - take daily notes, track projects and status, and it gets built into a nice little website I can reference with my boss. Now that we have copilot in full-swing, I'm trying to integrate agents a bit more. Opencode has made this so nice - so many tools and modes of interaction, highly customizable interface but also an amazing default experience... So I was trying to lean into using agents and subagents for more work I started down the path of building out a progress tracker. I started with a simple "skill" that told the agent to put some info into a sqlite file, and even create the file and schema based on the work. This worked fine, but was specific to each repo (and actually each worktree I was in) and it was hard for me to get visibility into all the work my fleet of agents was doing. Now that's actually problem 1 - I don't have a fleet of agents, I had some terminal sessions going with opencode, but I got it in my head that I was going to have an army of Claude's on my computer, constantly and autonomously knocking out tickets and I needed to know who was doing what, where, when, and why.. ## Nexus V1 & V2: The Over-Engineering Phase So, I dropped the local "progress tracker" and jumped into a huge FastAPI project that I called Nexus. It was a python cli + api, with a server for centralized management. It presented a kanban board, had policy gates on "plans" being approved before work could start on an Epic (and therefore any child tickets). It has worktree tracking and automation, etc. It had a lot... it didn't all work, and it was hard to build the autonomous system... I wanted agentic feedback loops where `voidshaper` and I made epic plans for Epics (see the pun?) and then once the Plan was approved `star-commander` comes on the scene and makes tickets or checks tickets, depending on what's already ready to go it farmed out the work to `starsmith` (and variants for complexity) to build and then automatically calls in `recon-officer` and `qa-engineer` and at the end of it `gatekeeper` came in and approved or denied the changes. If denied - automatically start the loop again, tracking the work in Nexus, if approved - rebase and merge the branch, clean up the worktree, update the ticket, close it, and get working on the next thing that opens up. Ticket dependencies were in there, tracking stale agent sessions, clever routing of tasks to smaller models where appropriate.... you can see that I went too far too fast too hard. ![20260203124603_d5948740.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260203124603_d5948740.png) Nexus went through 2 or 3 iterations of this feature set. I was building it with several constraints in mind - specifically at work. 1. rate-limiting on large models (so not just using opus for everything), 2. good stewardship (not burning down cities for docstrings - farm out that work to haiku or a -mini model), 3. Copilot support - even though I like to use opencode, I wanted to supper copilot cli and in vscode as a means to share this with my coworkers who were mostly using vscode, 4. agent sessions were dying so I needed some kind of liveness probe for manual intervention in sessions where maybe the vpn died and the agent lost network... stuff like this was on my mind - but you know what wasn't? actually doing some work... I was solving problems I don't have, but were fun to think about. ## The Reality Check So, what are my problems? I boiled them down to a few things... 1. I want to manage my workstreams in workspaces - folders on my computer where I put git worktrees 2. I want varying levels of ai automation in my workflows... some things could be handled by an agent fully autonmously, but most things I'm at least paired up, reviewing diffs still, doing research while working, etc. 3. In those varying levels of automation, I wanted the same amount of task tracking to a centralized location And after 2 weeks of ADHD-driven hyper focus, and many iterations on Nexus, then Nexus V2.... what happened is that Nexus V2 died, and we say long live Nexus V3! ## Nexus V3: The Pragmatic Pivot Nexus V3 is the same idea, but different approach... I'm leaning into opencode tooling specifically, dropping my care to support copilot given the lack of features. I'm using a few opencode [plugins](https://opencode.ai/docs/plugins/) ([opencode-notify](https://github.com/kdcokenny/opencode-notify) and [opencode-background-agents](https://github.com/kdcokenny/opencode-background-agents) and a few [commands](https://opencode.ai/docs/commands/). Instead of building my own tracker, I'm using [kanboard](https://kanboard.org/) because it's basically a feature-complete agile/sprint/kanban board that I use at home, has a simple plugin ecosystem for light customization, and solves practically every status-tracking problem I tried to build from the ground up initially - ticket dependencies/linkages, actions to change ticket colors for a simple intuitive UI based on state, easy columns and tagging configuration, a simple API and there's even an [mcp server](https://github.com/bivex/kanboard-mcp). So, I've given up on the full automation for now, although [opencode-pilot](https://github.com/athal7/opencode-pilot) looks VERY PROMISING for this in the future. Today though, through some simple commands to give to agents for updating kanboard, I manually put them in worktrees, and they get to work - the tracking and human-in-the-loop model is going well for the work I need done. ## Lessons Learned I learned and relearned plenty of lessons on this over the last 2 weeks... Data models matter more than almost anything, well-defined workflows are required if you want agents to help you iterate, and not everything has to be a product... That last one's personal, but every time I have an idea I **think** is good, I'm sure it'll be something to share, but a good lesson for me is to just build the things I need for me, and **eventually** maybe it can be cleaned up to share, but when I start building something with anyone other than **me** in mind, I'm in for a long hard journey ## Current State & Future So what's the summary? I don't think I know how to agent super well yet - but I'm trying to get better to stay on the forefront of my co-workers who I see using AI in simple and sometimes scary ways !!! danger "Copilot x sudo" Do not give copilot `sudo` on your CI server and say "fix my problem"... are you retarded??? I am not going hardcore with [ralph wiggum](https://awesomeclaude.ai/ralph-wiggum) or [gastown](https://github.com/steveyegge/gastown) - although those inspired Nexus v1 and v2, but I am dialing in my agentic workflow with some simple specialized agents, farming out work to subagents for context management, planning ahead of time to put appropriate context in a ticket, and getting close to having agents check out tickets, make worktrees, and do simple work by themselves (this was working in Nexus V2 but only intermittenly). Someday I'll open-source Nexus and share the configuration and workflow, for now it's private as I'm actually using it to build out what I want rather than trying to recreate gastown with a cool space theme.

New Job - Caterpillar Autonomy

2026-01-26T10:24:54Z

In [[im-back-from-the-dead]] I mentioned my new role that started this year - it's a return to Caterpillar Autonomy. I built some data pipelines and junior-grade infrastructure 5 years ago but left over burn-out at the prospect of a job with smaller scoped work and more technical guidance. That was a good move, and in God's sovereignty he's brought me back. Now I'm looking at code, thankfully not that I wrote, but that the guy after me wrote - it accomplished a job, like my original work did, but also like what I built back then - there's better ways today. The amount of things in front of me is absolutely daunting, and the people on the team have been seeded with a very high opinion of me... I don't know how much is valid and how much is hype, but I'm gonna try to live up to it. One way I'm striving to do that is to maximize my efficiency with the AI coding tools available to me. We have Github Copilot and I've picked up a lot over the course of the last few years (and especially more recently) in using agents, planning out work, documenting plans for agents to follow, splitting up work via worktrees, etc. I feel pretty good about where I'm going and someday I might even release Nexus. Nexus is serving as my second-brain at work, the hub where I collaborate with my fleet of agents on the work that must get done. Details to come on this, but the thing that matters is that in ironing out workflow I've moved from prompt + chat to really managing a long-lived stream of work. AI has been changing the world, and there's been developer hype for years now. But "make me a cool app, no mistakes" isn't going to cut it. Even "make an app but generate a plan first" isn't going to cut it... Developers have to adopt a higher level role, a systems-oriented and architecture-driven worldview must become primary in order to keep up. Code-gen and syntax writing are not what developers were **ever** paid to do, although many believed so. Our jobs have been to solve problems and deliver code that solves the problem. The code is cheap now, but solving problems still is a human task. In the Autonomy group - there's problems all up and down the stack. My focus is on infrastructure and developer operations - it's become my bread and butter over the last several years. I'm excited to help the team grow, and I'm excited to grow personally/technically as I lean into the agentic workflow to produce code that I'm actually proud of, that has my name on the commit, and that solves real problems. ## Example - Local Development One of the first things I'm tackling is a developer-pain-point of working on their laptop. I've been in this space for years, mostly with python programmer who are writing data science code. They don't know about virtual environments, checking $PATH, assuming bad state in their terminal session, how to configure VS Code, etc. Often they just want to write some scripts and somehow test it. The solution I see most often is for devs to write code in JupyterLab/Notebooks in AWS or some environment close to their data - this is fine I guess, but it's not developing good pipelines, and it's tedious as hell. In my last job I helped set developers up with workflows that allowed them to run their IDE of choice locally (getting all the goodies of syntax highlighting, LSP, etc) and a CLI that took their code and ran it in the cloud, right next to data, in the same way that prod runs. It was a hit. After that I introduced some tools to help them manage python environments - we had strict templating requirements in our projects, so making tools to automate those things wasn't too hard - it's much easier than trying to make something flexible for every use case. The opinions made the automation and tooling easy to make and distribute. Well I'm up against a similar task now, but oh so much worse... Larger team, larger environment sprawl, larger infrastructure mismanagement, the whole gambit. And I'm here for it... Here's the first problem I'm addressing - local development for Airflow DAGs that run in an Airflow deployment on Kubernetes. The deployment itself is a little odd, Airflow is an orchestrator, all the pipelines run in external AWS Batch jobs - so a DAG hits the Batch API to run the code. The design there is actually nice, but how are devs testing code? Oh that's easy... they SSH into the prod server, which is a 5 year old desktop THAT I BUILT WITH A CO-WORKER BEFORE I WAS IN AUTONOMY THE FIRST TIME... hold on, WHAT!? Yes, it's true.. so they SSH into the prod server, run some bash scripts in their userspace that setup airflow and a few db utilities in a docker compose stack, authenticate with AWS themselves from that server, and then they run DAGs against real data to test it... I am beyond shook. Here's what I've put together - a bootstrap process (I like `just` + PEP 723 python scripts as opposed to bash, but to each their own, and bash of course has its place) that spins up a [kind](https://kind.sigs.k8s.io/) cluster on their laptop, the process pulls some private images and loads them into the kind cluster, it installs airflow from a helm chart (the same helm chart we'll use in dev and prod... no more docker compose over here, kubectl over there), and everything just. comes. up. No SSH into ancient server, no touching cloud infra (they get MinIO and a DB container to emulate S3 and RDS in our AWS accounts), no sweat on testing DAGs. They stage some data in MinIO (`just open minio` handles the port-forward and opens the browser), then `just open airflow` (their DAGs are hot-reloaded via hostPath mounting), and they can run DAGs locally until they're satisfied with the results. It's taken me about a week of split-focused effort (I mentioned Nexus and I've been co-building and dogfooding that at the same time) but I'm proud of that local setup now. I am a bit shocked they've dealt with a brittle, hacky, often broken development workflow for the last 4 years or so, but that development I suppose. ## The Point This post wasn't meant to be me glazing myself for awesome local development practices, I am simply excited about this new chapter. I love solving problems, and I love owning those solutions. There's a whole mess of things to address, my mind is buzzing, and I feel like God has blessed me with renewed passion (again see [[im-back-from-the-dead]]). I've given up some pay and some freedom to take this role, but I think it'll pay dividends. Life happens to all of us - this role change affected a lot for me, it's been hard to digest some of those changes, but the work is good, the development is fun, the new world of using agents to fly through things you're fluent in is exciting, and I'm here to help a team that desperately needs it to improve their lives and the work we do for Cat Autonomy. I'm still pissed at Caterpillar Executives for RTO ruining parts of my life, but in God's sovereignty their idiocy has led to non-trivial blessing, so I can say "Praise the Lord"

Increase inotify limit in your CI workers

2025-12-30T06:11:49Z

Today I tripped over a CI failure that I had to think about for a while.

I build zensical static sites in CI on my Forgejo instance. These builds had been working fine, then suddenly started failing with no code changes. Naturally, I assumed something upstream broke — maybe a new uv release, maybe zensical.

I pinned versions.
I tested older versions.
Same failure every time.

That ruled out regressions and pushed me toward the environment.

I pulled the runner and worker images locally and built the sites just fine… But that doesn’t perfectly emulate the CI setup - my forgejo runner relies on docker-in-docker and so we aren’t just running a container on a host, we have this middle layer to consider… I wasn’t sure how to really test this out locally so I succomed to AI and here’s where Jipity got me in about 5 minutes…

The failure #

The builds blew up with:

thread 'zrx/monitor' panicked at .../zensical-watch/src/agent/monitor.rs:154:49:
called `Result::unwrap()` on an `Err` value:
Error { kind: Io(Os { code: 24, message: "Too many open files" }) }

At first glance it means nothing to me but Jipity says this screams ulimit.

So following the AI overlords I checked:

ulimit -n was already very high

/proc/sys/fs/inotify/max_user_watches was also very high

PID limits were not constrained

Everything looked fine according to Jipity.

Yet the panic persisted.

The real culprit #

The actual limit being hit was:

/proc/sys/fs/inotify/max_user_instances

In my Forgejo runner container, it was set to 128.

That turns out to be far too low for zensical.

Here’s what ChatGPT said:

Even during a normal zensical build, the tool spins up its watch subsystem, which creates many inotify instances. Once it crosses the kernel limit, inotify_init() fails with EMFILE, and the process panics because the error is unwrapped.

The fix #

Raising the inotify instance limit fixed it immediately:

echo 1024 > /proc/sys/fs/inotify/max_user_instances
RUST_BACKTRACE=full uvx zensical build --clean

After that, the build succeeded consistently.

Prettier Docker ‘ps’ Command

2025-12-29T05:33:05Z

The docker ps command is very useful, but I hate reading the output. Turns out, you can make it prettier:

docker ps

–format “table” is implied with the command.


❯ docker ps --format "table" | grep can
9b716a7d1ab0   postgres:17                                               "docker-entrypoint.s…"   13 hours ago    Up 13 hours (healthy)            0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp                                                cannalyzer-db-1
f7708ea0c112   adminer                                                   "entrypoint.sh docke…"   13 hours ago    Up 13 hours                      0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp                                                cannalyzer-adminer-1
93cd67719ed4   frontend:latest                                           "/docker-entrypoint.…"   13 hours ago    Up 13 hours                      0.0.0.0:5173->80/tcp, [::]:5173->80/tcp                                                    cannalyzer-frontend-1
f517625fca98   traefik:3.0                                               "/entrypoint.sh --pr…"   13 hours ago    Up 13 hours                      0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp           cannalyzer-proxy-1
f4daa036216e   schickling/mailcatcher                                    "sh -c 'mailcatcher …"   13 hours ago    Up 13 hours                      0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp   cannalyzer-mailcatcher-1

But you can pass a template string to the --format option, like so:

docker ps --format "table {{.Names}}"


✗ docker ps --format "table {{.Names}}" | grep can
cannalyzer-db-1
cannalyzer-adminer-1
cannalyzer-frontend-1
cannalyzer-proxy-1
cannalyzer-mailcatcher-1

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"


❯ docker ps --format "table {{.Names}}\t{{.Ports}}" | grep can
cannalyzer-db-1             0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp
cannalyzer-adminer-1        0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp
cannalyzer-frontend-1       0.0.0.0:5173->80/tcp, [::]:5173->80/tcp
cannalyzer-proxy-1          0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp
cannalyzer-mailcatcher-1    0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp

Picture #

I noticed my template isn’t folding the codeblocks in a way that actually makes this post look like I’m lying!

Setup A Cloudflare Tunnel With Terraform

2025-12-12T21:06:59Z

I am cooking up some stuff at home and want to put it on the interwebs, but I don’t want it on the same infra as my homelab. Now… I only have a server or 2, so to some degree it will be, but networking-wise I didn’t want to funnel extra traffic through my reverse proxy.

So, I’d heard about Cloudflare Tunnels - they sound like P2P VPN to me, but I know there’s layers of the networking stack I’m blatantly ignoring. “What the tunnel is” isn’t much the point - I’m here to show you how to set one up and get yourself a fancy https://app.mydomain.com for your web app running kind of wherever you want

Example Repo linked at the bottom

Requirements #

Terraform or open-tofu. I currently use open-tofu but either would be fine. brew install open-tofu is a simple way to get going
Cloudflare account with a domain
API token with permissions:
- Account:Cloudflare Tunnel:Edit
- Zone:DNS:Edit
cloudflared (the example repo runs cloudflared in a docker compose stack)

Tunnel #

The module is simple and has just a few resources:

❯ tofu state list
module.tunnel.cloudflare_record.tunnel
module.tunnel.cloudflare_tunnel_config.this
module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this
module.tunnel.random_id.tunnel_secret

We see there will be the a DNS record that tofu references by the key “tunnel”. There is a tunnel configuration resource, the tunnel resource itself, and finally the associated secret required for the cloudflared daemon that will run alongside your webapp.

To get started you’ll need to fill out the example terraform.tfvars file with your info:

# Copy to terraform.tfvars and fill in values
# DO NOT commit terraform.tfvars to git

cloudflare_api_token  = "your-api-token-here"
cloudflare_account_id = "your-account-id"
cloudflare_zone_id    = "your-zone-id"
domain                = "example.com"
subdomain             = "app"
tunnel_name           = "my-tunnel"
origin_service        = "http://localhost:8000"

You can grab your account id and zone id from Cloudflare’s dashboard for your domain. It’s near the bottom of the Overview page

Then I presume you have a domain already, but if not hop over to namecheap to snag one and then register it with cloudflare so they can manage your DNS. I have terraform for this as well, a future blog post will combine this with a fuller terraform’d cloudflare setup for simple domain use cases

Once you fill those out, hit it with the tofu init and tofu plan to see what’s up

NOTE: terraform.tfvars is automatically sourced by terraform/tofu, you can name the file differently and then pass -var-file=myvars.tfvars to the commands

The initial plan should look something like this:


OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # module.tunnel.cloudflare_record.tunnel will be created
  + resource "cloudflare_record" "tunnel" {
      + allow_overwrite = false
      + comment         = "Managed by Terraform - soonish-tunnel"
      + content         = (known after apply)
      + created_on      = (known after apply)
      + hostname        = (known after apply)
      + id              = (known after apply)
      + metadata        = (known after apply)
      + modified_on     = (known after apply)
      + name            = "app"
      + proxiable       = (known after apply)
      + proxied         = true
      + ttl             = (known after apply)
      + type            = "CNAME"
      + value           = (known after apply)
      + zone_id         = "<REDACTED>"
    }

  # module.tunnel.cloudflare_tunnel_config.this will be created
  + resource "cloudflare_tunnel_config" "this" {
      + account_id = "<REDACTED>"
      + id         = (known after apply)
      + tunnel_id  = (known after apply)

      + config {
          + ingress_rule {
              + hostname = "app.notifiq.net"
              + service  = "http://localhost:8000"
            }
          + ingress_rule {
              + service = "http_status:404"
            }
        }
    }

  # module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this will be created
  + resource "cloudflare_zero_trust_tunnel_cloudflared" "this" {
      + account_id   = "<REDACTED>"
      + cname        = (known after apply)
      + id           = (known after apply)
      + name         = "soonish-tunnel"
      + secret       = (sensitive value)
      + tunnel_token = (sensitive value)
    }

  # module.tunnel.random_id.tunnel_secret will be created
  + resource "random_id" "tunnel_secret" {
      + b64_std     = (known after apply)
      + b64_url     = (known after apply)
      + byte_length = 32
      + dec         = (known after apply)
      + hex         = (known after apply)
      + id          = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

As long as that looks good you to, then we tofu apply next (type yes when asked or pass -auto-approve)

Afterwards tofu state list should show you the 4 resources, and if you go to your cloudflare zone’s dashboard you should see the CNAME associated with the tunnel address

Daemon #

Run the compose stack or the binary itself. Get the token from terraform state with tofu output -raw tunnel_token.

TUNNEL_TOKEN=$(tofu output -raw tunnel_token) docker compose up -d will do you nicely

Enjoy your tunnel!

example repo

COLUMNS Env Var For Nicer Screenshots

2025-12-08T05:16:55Z

I was working on [[uv-run-pep-723-is-a-match-made-in-heaven]] and I wanted to grab a screenshot of a TUI, but my terminal is full screen and the TUI was then super stretched... Nice for working and using it, but not nice for a screenshot going into a blog post... see the difference `export COLUMNS=120` does here! ``` COLUMNS=120 mytui.py ``` ![20251208111709_aa06bd03.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251208111709_aa06bd03.png) ``` mytui.py ``` ![20251208111820_a209638f.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251208111820_a209638f.png)

UV Run + PEP 723 Is A Match Made In Heaven

2025-12-08T04:59:10Z

This post will be short but hopefully be a dense few paragraphs about how using `uv run foo.py` along with [PEP 723](https://peps.python.org/pep-0723/) is life-changing. ## PEP 723 Python Enhancement Proposals (PEP) are the community's way of improving the python language and ecosystem and PEP 723 is one of my favorites. It permits adding python metadata right to your script which allows `python` to setup the script's dependencies at runtime. ## UV [[uv]] makes it even better with it's super fast resolution and environment setup. I don't have any posts about uv yet, but here's the [docs](https://docs.astral.sh/uv/getting-started/installation/), installing `uv` is now apart of my dev-machine setup because I need it everywhere! ## The Magic I'm experiencing the power of `uv run myscript.py` as I've integrated a python script into my tmux workflow for managing sessions as well as groups of git-worktrees to keep development-ideas contained, easy to find, and easy to manage. !!! warning "The Series" This post is in my "If You Want It Make It So" series because of the workspaces thing generally, not the dbztui example below, but for me - having 'uv run' as a pretty powerful Bash replacement makes my development workflow easier to build on and manage My [workspaces docs](https://pypeaday.github.io/dotfiles/terminal/workspaces/) explain the setup a bit, and that script is in my [dotfiles](https://github.com/pypeaday/dotfiles/blob/main/workspaces/.local/bin/ws.py) For a shorter example, here's an example that has regular `pip` style dependencies, a `git+https` dependency, and is a tui - not just a top to bottom script !!! note "What is it?" This is my DBZTUI script - it uses a beta package from Waylon called ninesui and calls the dragonball-api to give you a handy way to explore the Dragon Ball Universe from the glorious comfort of your terminal Copy this code into `dbztui.py` !!! note "Execution" You can `chmod +x` this script with the shebang to tell your shell to use uv to run it or just `uv run dbztui.py` will also work just fine ```python #!/usr/bin/env -S uv run --quiet --script # /// script # requires-python = ">=3.10" # dependencies = [ # "ninesui @ git+https://github.com/waylonwalker/ninesui.git", # "httpx", # "deep-translator", # ] # /// from typing import Optional, List, TypeVar, ClassVar, Any, Dict from pydantic import BaseModel, Field, HttpUrl import httpx from textual import log from ninesui import CommandSet, Command, NinesUI from functools import lru_cache from deep_translator import GoogleTranslator import json import os BASE_URL = "https://dragonball-api.com/api/" # Translation cache file CACHE_DIR = os.path.expanduser("~/.cache/dbztui") CACHE_FILE = os.path.join(CACHE_DIR, "translation_cache.json") # Create cache directory if it doesn't exist os.makedirs(CACHE_DIR, exist_ok=True) # Load translation cache from file translation_cache: Dict[str, str] = {} if os.path.exists(CACHE_FILE): try: with open(CACHE_FILE, "r", encoding="utf-8") as f: translation_cache = json.load(f) except Exception as e: log(f"Error loading translation cache: {e}") # Initialize translator translator = GoogleTranslator(source='es', target='en') T = TypeVar("T", bound="DBZResource") @lru_cache(maxsize=1000) def translate_text(text: str) -> str: """Translate text from Spanish to English with caching""" if not text or len(text) < 5: # Don't translate very short texts return text # Check if translation is in cache if text in translation_cache: return translation_cache[text] try: # Translate text translated = translator.translate(text) # Save to cache translation_cache[text] = translated # Periodically save cache to file if len(translation_cache) % 10 == 0: # Save every 10 new translations try: with open(CACHE_FILE, "w", encoding="utf-8") as f: json.dump(translation_cache, f, ensure_ascii=False, indent=2) except Exception as e: log(f"Error saving translation cache: {e}") return translated except Exception as e: log(f"Translation error: {e}") return text # Return original text if translation fails class DBZResource(BaseModel): id: int nines_config: ClassVar[dict] = {"bindings": {}} @classmethod def fetch(cls, ctx=None): endpoint = cls.__name__.lower() + "s" client = httpx.Client() log(f"Fetching {endpoint}") if ctx: if hasattr(ctx, endpoint): result = [] for url in getattr(ctx, endpoint): res = client.get(str(url)).json() result.append(cls(**res)) return result url = f"{BASE_URL}{endpoint}" results: List[T] = [] response = client.get(url) response.raise_for_status() data = response.json() # Handle pagination structure from Dragon Ball API if "items" in data: results.extend(cls(**item) for item in data.get("items", [])) # Fetch all pages if needed while data.get("links", {}).get("next"): next_url = data["links"]["next"] response = client.get(next_url) response.raise_for_status() data = response.json() results.extend(cls(**item) for item in data.get("items", [])) else: # Direct list of items results.extend(cls(**item) for item in data) return results def hover(self): return self def get_details(self): """Get detailed information about this resource""" client = httpx.Client() endpoint = self.__class__.__name__.lower() + "s" url = f"{BASE_URL}{endpoint}/{self.id}" response = client.get(url) response.raise_for_status() data = response.json() # Translate description if present if "description" in data and data["description"]: data["description"] = translate_text(data["description"]) return self.__class__(**data) class Character(DBZResource): name: str ki: str maxKi: str race: str gender: str description: str image: Optional[HttpUrl] = None affiliation: str deletedAt: Optional[str] = None def __init__(self, **data): # Translate description before initializing if "description" in data and data["description"]: data["description"] = translate_text(data["description"]) super().__init__(**data) nines_config: ClassVar[dict] = {"bindings": {"t": "get_transformations"}} def get_transformations(self): """Get all transformations for this character""" client = httpx.Client() url = f"{BASE_URL}characters/{self.id}/transformations" try: response = client.get(url) response.raise_for_status() data = response.json() if "items" in data: return [Transformation(**item) for item in data.get("items", [])] return [Transformation(**item) for item in data] except Exception as e: log(f"Error fetching transformations: {e}") return [] class Transformation(DBZResource): name: str image: Optional[HttpUrl] = None ki: str characterId: int deletedAt: Optional[str] = None # Note: This is not a direct API endpoint, but accessed through character transformations class Planet(DBZResource): name: str description: str image: Optional[HttpUrl] = None deletedAt: Optional[str] = None def __init__(self, **data): # Translate description before initializing if "description" in data and data["description"]: data["description"] = translate_text(data["description"]) super().__init__(**data) # Note: Saga and Episode models removed as they are not supported by the API commands = CommandSet( [ Command( name="character", aliases=["c"], model=Character, is_default=True, ), Command( name="planet", aliases=["p"], model=Planet, ), # Note: Transformation is not a direct API endpoint, but can be accessed through characters ] ) metadata = { "title": "Dragon Ball Z Explorer", "subtitle": "Use :character or :planet to explore. Enter to drill in. Escape to go back/quit.", } if __name__ == "__main__": ui = NinesUI(metadata=metadata, commands=commands) try: ui.run() finally: # Save translation cache when exiting try: with open(CACHE_FILE, "w", encoding="utf-8") as f: json.dump(translation_cache, f, ensure_ascii=False, indent=2) except Exception as e: log(f"Error saving translation cache on exit: {e}") ``` ![20251208111354_0ae058e5.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251208111354_0ae058e5.png) > See [[columns-env-var-for-nicer-screenshots]] about making terminal screenshots a bit nicer

GitHub Stopped Me From Messing With The Bots

2025-12-06T06:24:51Z

After some cheeky [[small-steps-towards-handling-malicious-traffic-on-static-sites]] it turns out GH has been blocking me from publishing my site because the fodder I put in there for the bots was recognized as actual secrets by their scanning... ![20251206122602_8b1635bf.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251206122602_8b1635bf.png) I had to tell github that each of these secrets in the commits was fine... after all, they're good bait [Github Docs](https://docs.github.com/en/code-security/secret-scanning/working-with-secret-scanning-and-push-protection/working-with-push-protection-from-the-command-line#resolving-a-blocked-push)

Small Steps Towards Handling Malicious Traffic on Static Sites

2025-12-04T21:17:24Z

Today I saw a random IP hitting an app server I had open via tailscale funnel and it got me thinking that I need to take some precautions against these real world threats. So I’m starting with my blog… basically you can reference Jim Nielson’s Blog on Malicious Traffic and then I more or less put similar files in similar places on this site to waste malicious actors’ time

The Files #

Note that some are empty, we just need them to exist since this is all for a bit of fun and low-effort internet tomfoolery

These get shipped with my site at /public/...

>>>> backup/config-backup.zip.txt

<!--markata-attribution-->
PK     !!This is not a real ZIP file!!

<!--markata-attribution-->
PK     But bots will try to download it anyway

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     Wasting bandwidth and CPU cycles...

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     Here are some fake credentials to keep you busy:

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     FTP_HOST=ftp.example.com

<!--markata-attribution-->
PK     FTP_USER=admin

<!--markata-attribution-->
PK     FTP_PASS=P@ssw0rd123!

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     SSH_HOST=192.168.1.100

<!--markata-attribution-->
PK     SSH_USER=root

<!--markata-attribution-->
PK     SSH_KEY=-----BEGIN RSA PRIVATE KEY-----

<!--markata-attribution-->
PK     MIIEpAIBAAKCAQEA1234567890FAKE

<!--markata-attribution-->
PK     -----END RSA PRIVATE KEY-----

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     MYSQL_HOST=localhost

<!--markata-attribution-->
PK     MYSQL_USER=root

<!--markata-attribution-->
PK     MYSQL_PASS=rootpassword123

<!--markata-attribution-->
PK     MYSQL_DB=production_db

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     REDIS_HOST=127.0.0.1:6379

<!--markata-attribution-->
PK     REDIS_PASS=redis_secret_2024

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     JWT_SECRET=super_secret_jwt_key_do_not_share

<!--markata-attribution-->
PK     ENCRYPTION_KEY=AES256_ENCRYPTION_KEY_HERE

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     STRIPE_PUBLISHABLE=pk_live_FAKE123456789

<!--markata-attribution-->
PK     STRIPE_SECRET=sk_live_FAKE987654321

<!--markata-attribution-->
PK

<!--markata-attribution-->
PK     SENDGRID_API_KEY=SG.FAKE_API_KEY_HERE

<!--markata-attribution-->
PK
PK     This file is intentionally malformed to waste bot parsing time

<!--markata-attribution-->
PK     PK     PK     PK     PK     PK     PK     PK
>>>> backup/database-backup-2024-12-01.sql

<!--markata-attribution-->
-- MySQL Database Backup

<!--markata-attribution-->
-- Host: localhost

<!--markata-attribution-->
-- Database: wordpress_prod

<!--markata-attribution-->
-- Generated: 2024-12-01 03:14:15

<!--markata-attribution-->
-- WARNING: This file contains sensitive data

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- Table structure for wp_users
DROP TABLE IF EXISTS `wp_users`;
CREATE TABLE `wp_users` (
  `ID` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `user_login` varchar(60) NOT NULL DEFAULT '',
  `user_pass` varchar(255) NOT NULL DEFAULT '',
  `user_email` varchar(100) NOT NULL DEFAULT '',
  PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

-- Dumping data for table wp_users
INSERT INTO `wp_users` VALUES
(1,'admin','$P$BZlPX7NIx8MYpXokBW2AGsN7i.aUOt0','admin@example.com'),
(2,'webmaster','$P$B4RKwF8zqRnNu9cV5fGg7wgT2sY9Pl1','webmaster@example.com');

-- API Keys and Secrets
-- AWS_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
-- AWS_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-- STRIPE_SECRET: sk_test_4eC39HqLyjWDarjtT1zdp7dc
-- DATABASE_PASSWORD: MyS3cr3tP@ssw0rd!2024

-- Infinite loop to waste bot resources
DELIMITER $$
CREATE PROCEDURE infinite_loop()
BEGIN
  DECLARE i INT DEFAULT 0;
  WHILE i < 999999999 DO
    SET i = i + 1;
    SELECT CONCAT('Processing row ', i, ' of 999999999...') AS status;
  END WHILE;
END$$
DELIMITER ;

-- More fake sensitive data
INSERT INTO wp_options VALUES
(1,'siteurl','http://localhost','yes'),
(2,'admin_email','admin@localhost.local','yes'),
(3,'secret_api_key','sk_live_51HqLyjWDarjtT1zdp7dcEXAMPLE','yes');

-- This backup continues for 50MB... [TRUNCATED FOR DISPLAY]
>>>> backup/db_dump_final.2023.zip

>>>> backup/site.sql

>>>> backup/wp_backup.tar.gz

>>>> private/admin-credentials.txt

<!--markata-attribution-->
CONFIDENTIAL - ADMIN CREDENTIALS
==================================

Production Server Access:
-------------------------
Server: prod-server-01.example.com
Username: administrator
Password: Admin2024!Secure
SSH Port: 22

Database Credentials:
--------------------
Host: db.internal.example.com
Port: 3306
Username: db_admin
Password: DbP@ssw0rd!2024
Database: production_main

API Keys:
---------
OpenAI API Key: sk-proj-FAKE1234567890abcdefghijklmnopqrstuvwxyz
Stripe Secret: sk_live_FAKE_51HqLyjWDarjtT1zdp7dc
AWS Access Key: AKIAFAKEEXAMPLE123456
AWS Secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYFAKEKEY
SendGrid API: SG.FAKE_SENDGRID_KEY_HERE_123456789

WordPress Admin:
---------------
URL: https://example.com/wp-admin
Username: admin
Password: WP_Admin_2024!
Security Key: put your unique phrase here

FTP Access:
-----------
Host: ftp.example.com
Username: ftpuser
Password: FtpP@ss123!
Port: 21

IMPORTANT: Keep this file secure!
Last Updated: 2024-12-01
Next Password Rotation: 2025-01-01

<!-- Hidden comment: This is a honeypot. All credentials are fake. -->
>>>> private/config.php

<!--markata-attribution-->
<?php

<!--markata-attribution-->
// Database Configuration

<!--markata-attribution-->
define('DB_HOST', 'localhost');

<!--markata-attribution-->
define('DB_NAME', 'wordpress_production');

<!--markata-attribution-->
define('DB_USER', 'wp_admin');

<!--markata-attribution-->
define('DB_PASSWORD', 'MyS3cr3tP@ssw0rd!2024');

<!--markata-attribution-->
define('DB_CHARSET', 'utf8mb4');

// Security Keys - DO NOT SHARE
define('AUTH_KEY',         'put your unique phrase here - this is fake');
define('SECURE_AUTH_KEY',  'put your unique phrase here - this is fake');
define('LOGGED_IN_KEY',    'put your unique phrase here - this is fake');
define('NONCE_KEY',        'put your unique phrase here - this is fake');
define('AUTH_SALT',        'put your unique phrase here - this is fake');
define('SECURE_AUTH_SALT', 'put your unique phrase here - this is fake');
define('LOGGED_IN_SALT',   'put your unique phrase here - this is fake');
define('NONCE_SALT',       'put your unique phrase here - this is fake');

// API Keys
define('STRIPE_SECRET_KEY', 'sk_live_FAKE123456789abcdefghijklmnop');
define('STRIPE_PUBLIC_KEY', 'pk_live_FAKE987654321zyxwvutsrqponml');
define('SENDGRID_API_KEY', 'SG.FAKE_API_KEY_1234567890');
define('AWS_ACCESS_KEY', 'AKIAIOSFODNN7EXAMPLE');
define('AWS_SECRET_KEY', 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY');

// Admin Settings
define('ADMIN_EMAIL', 'admin@example.com');
define('SITE_URL', 'https://example.com');
define('WP_DEBUG', false);
define('WP_DEBUG_LOG', true);

// FTP Credentials
define('FTP_HOST', 'ftp.example.com');
define('FTP_USER', 'ftpadmin');
define('FTP_PASS', 'FtpS3cur3P@ss!');

// Redis Cache
define('REDIS_HOST', '127.0.0.1');
define('REDIS_PORT', 6379);
define('REDIS_PASSWORD', 'redis_secret_password_2024');

// JWT Secret
define('JWT_SECRET', 'super_secret_jwt_key_for_authentication');

// Infinite loop to waste bot CPU
while(true) {
    $random = bin2hex(random_bytes(1024));
    usleep(1000);
}
?>
>>>> private/index.html

<!--markata-attribution-->
<!doctype html>

<!--markata-attribution-->
<html>

<!--markata-attribution-->
  <body>

<!--markata-attribution-->
    <h1>Private Area</h1>

    <pre>
<!-- ~1MB lorem ipsum for bandwidth drain -->
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
<!-- repeat this block until ~1MB -->
</pre>
  </body>
</html>

>>>> private/ssh_keys.txt

<!--markata-attribution-->
SSH PRIVATE KEYS - PRODUCTION SERVERS
======================================

Server: prod-web-01.example.com
--------------------------------
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklm
nopqrstuvwxyz0123456789+/FAKE_KEY_DATA_HERE_NOT_REAL_AT_ALL_JUST_WASTING
BOT_TIME_AND_RESOURCES_HAHAHAHA_THIS_IS_A_HONEYPOT_TRAP_FOR_SCRAPERS_12345
67890ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/
FAKE_KEY_DATA_CONTINUES_FOR_MANY_LINES_TO_WASTE_BANDWIDTH_AND_STORAGE_SPACE
MIIEpAIBAAKCAQEA1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklm
nopqrstuvwxyz0123456789+/MORE_FAKE_DATA_HERE_BOTS_LOVE_SSH_KEYS_RIGHT
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/END
-----END RSA PRIVATE KEY-----

Server: prod-db-01.example.com
-------------------------------
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn
NhAAAAAwEAAQAAAQEA1234567890FAKE_OPENSSH_KEY_DATA_HERE_NOT_REAL_JUST_A
_TRAP_FOR_BOTS_AND_SCRAPERS_WASTING_THEIR_TIME_AND_RESOURCES_HAHA_12345678
90ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789FAKE
DATA_CONTINUES_HERE_TO_MAKE_IT_LOOK_LEGITIMATE_BUT_ITS_ALL_GARBAGE_123456
-----END OPENSSH PRIVATE KEY-----

Server: prod-app-01.example.com
--------------------------------
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIFAKE0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnop
qrstuvwxyz0123456789FAKE_EC_KEY_DATA_HERE_ELLIPTIC_CURVE_KEYS_ARE_COOL
BUT_THIS_ONE_IS_FAKE_JUST_WASTING_BOT_RESOURCES_HAHAHAHA_123456789ABC
-----END EC PRIVATE KEY-----

IMPORTANT NOTES:
- These keys provide root access to production servers
- Never commit to version control
- Rotate every 90 days
- Last rotation: 2024-11-01
- Next rotation: 2025-02-01

Contact: security@example.com for key rotation

<!-- This is a honeypot. All keys are fake and invalid. -->
>>>> robots.txt

<!--markata-attribution-->
User-agent: *

<!--markata-attribution-->
Disallow: /private/

<!--markata-attribution-->
Disallow: /admin/

<!--markata-attribution-->
Disallow: /backup/

<!--markata-attribution-->
Disallow: /.env

<!--markata-attribution-->
Disallow: /wp-admin/

<!--markata-attribution-->
Disallow: /wp-login.php

>>>> sitemap.xml

<urlset>
  <url><loc>/debug/alpha</loc></url>
  <url><loc>/debug/beta</loc></url>
  <url><loc>/admin/backup-2024.zip</loc></url>
  <url><loc>/.env</loc></url>
  <url><loc>/wp-admin/install.php</loc></url>
  <url><loc>/wp-content/plugins/wp-super-cache/readme.txt</loc></url>
</urlset>

>>>> trap/a/index.html

<!--markata-attribution-->
<meta http-equiv="refresh" content="0; url=/trap/b/" />

>>>> trap/api.php

<!--markata-attribution-->
<?php
/**

<!--markata-attribution-->
 * Fake API Endpoint

<!--markata-attribution-->
 * Designed to trap and waste bot resources
 */

header('Content-Type: application/json');
header('X-Powered-By: PHP/8.2.0');
header('X-Debug-Mode: enabled');

// Fake API response with sensitive data
$api_response = [
    'success' => true,
    'api_version' => '2.1.0',
    'endpoints' => [
        '/api/users' => 'GET, POST',
        '/api/auth' => 'POST',
        '/api/admin' => 'GET, POST, DELETE',
        '/api/database' => 'GET',
        '/api/backup' => 'POST'
    ],
    'authentication' => [
        'type' => 'Bearer Token',
        'example_token' => 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.FAKE_JWT_TOKEN',
        'api_key' => 'sk_live_FAKE123456789abcdefghijklmnop',
        'api_secret' => 'secret_FAKE987654321zyxwvutsrqponmlk'
    ],
    'database_config' => [
        'host' => 'localhost',
        'port' => 3306,
        'username' => 'api_user',
        'password' => 'ApiP@ssw0rd!2024',
        'database' => 'api_production'
    ],
    'admin_credentials' => [
        'username' => 'api_admin',
        'password' => 'Admin2024!Secure',
        'email' => 'admin@api.example.com',
        'role' => 'superadmin'
    ],
    'external_services' => [
        'stripe' => [
            'public_key' => 'pk_live_FAKE123',
            'secret_key' => 'sk_live_FAKE456'
        ],
        'aws' => [
            'access_key' => 'AKIAIOSFODNN7EXAMPLE',
            'secret_key' => 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
        ],
        'sendgrid' => [
            'api_key' => 'SG.FAKE_SENDGRID_KEY'
        ]
    ],
    'debug_info' => [
        'server_ip' => '192.168.1.100',
        'php_version' => '8.2.0',
        'mysql_version' => '8.0.35',
        'redis_host' => '127.0.0.1:6379',
        'redis_password' => 'redis_secret_2024'
    ]
];

// Waste CPU cycles
for ($i = 0; $i < 50000; $i++) {
    $temp = json_encode($api_response);
    $decoded = json_decode($temp, true);
    $hash = hash('sha256', $temp);
}

// Output response
echo json_encode($api_response, JSON_PRETTY_PRINT);

// Infinite loop trap
set_time_limit(0);
while(true) {
    $waste = [];
    for ($i = 0; $i < 10000; $i++) {
        $waste[] = random_bytes(1024);
    }
    usleep(1000);
}
?>
>>>> trap/b/index.html

<!--markata-attribution-->
<meta http-equiv="refresh" content="0; url=/trap/c/" />

>>>> trap/c/index.html

<!--markata-attribution-->
<meta http-equiv="refresh" content="0; url=/trap/a/" />

>>>> trap/data.json
{

<!--markata-attribution-->
  "status": "success",

<!--markata-attribution-->
  "message": "API endpoint active",

<!--markata-attribution-->
  "data": {

<!--markata-attribution-->
    "credentials": {
      "api_key": "sk_live_FAKE123456789abcdefghijklmnopqrstuvwxyz",
      "api_secret": "secret_FAKE987654321zyxwvutsrqponmlkjihgfedcba",
      "jwt_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.FAKE_TOKEN_DATA_HERE.SIGNATURE",

<!--markata-attribution-->
      "oauth_token": "ya29.FAKE_OAUTH_TOKEN_1234567890",

<!--markata-attribution-->
      "refresh_token": "1//FAKE_REFRESH_TOKEN_ABCDEFGHIJKLMNOP"

<!--markata-attribution-->
    },

<!--markata-attribution-->
    "database": {

<!--markata-attribution-->
      "host": "db.internal.example.com",

<!--markata-attribution-->
      "port": 3306,

<!--markata-attribution-->
      "username": "db_admin",

<!--markata-attribution-->
      "password": "DbP@ssw0rd!2024",

<!--markata-attribution-->
      "database": "production_db",
      "connection_string": "mysql://db_admin:DbP@ssw0rd!2024@db.internal.example.com:3306/production_db"

<!--markata-attribution-->
    },

<!--markata-attribution-->
    "aws": {

<!--markata-attribution-->
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",

<!--markata-attribution-->
      "region": "us-east-1",

<!--markata-attribution-->
      "bucket": "production-backups-2024",

<!--markata-attribution-->
      "cloudfront_id": "E1234FAKE567890"

<!--markata-attribution-->
    },

<!--markata-attribution-->
    "stripe": {

<!--markata-attribution-->
      "publishable_key": "pk_live_FAKE123456789",

<!--markata-attribution-->
      "secret_key": "sk_live_FAKE987654321",

<!--markata-attribution-->
      "webhook_secret": "whsec_FAKE_webhook_secret_here"

<!--markata-attribution-->
    },

<!--markata-attribution-->
    "email": {

<!--markata-attribution-->
      "sendgrid_api_key": "SG.FAKE_SENDGRID_KEY_1234567890",

<!--markata-attribution-->
      "smtp_host": "smtp.example.com",

<!--markata-attribution-->
      "smtp_port": 587,

<!--markata-attribution-->
      "smtp_user": "noreply@example.com",

<!--markata-attribution-->
      "smtp_pass": "SmtpP@ss2024!"

<!--markata-attribution-->
    },

<!--markata-attribution-->
    "servers": [
      {

<!--markata-attribution-->
        "name": "prod-web-01",

<!--markata-attribution-->
        "ip": "192.168.1.100",

<!--markata-attribution-->
        "ssh_user": "root",
        "ssh_key": "-----BEGIN RSA PRIVATE KEY-----\nFAKE_KEY_DATA_HERE\n-----END RSA PRIVATE KEY-----"

<!--markata-attribution-->
      },
      {

<!--markata-attribution-->
        "name": "prod-db-01",

<!--markata-attribution-->
        "ip": "192.168.1.101",

<!--markata-attribution-->
        "ssh_user": "admin",

<!--markata-attribution-->
        "ssh_pass": "SshP@ssw0rd!2024"
      }

<!--markata-attribution-->
    ],

<!--markata-attribution-->
    "internal_urls": [

<!--markata-attribution-->
      "http://admin.internal.example.com",

<!--markata-attribution-->
      "http://api.internal.example.com",

<!--markata-attribution-->
      "http://db.internal.example.com",

<!--markata-attribution-->
      "http://cache.internal.example.com"

<!--markata-attribution-->
    ],

<!--markata-attribution-->
    "waste_bot_resources": {

<!--markata-attribution-->
      "large_array": [],

<!--markata-attribution-->
      "nested_data": {}
    }

<!--markata-attribution-->
  },

<!--markata-attribution-->
  "metadata": {

<!--markata-attribution-->
    "generated_at": "2024-12-01T12:00:00Z",

<!--markata-attribution-->
    "expires_at": "2025-12-01T12:00:00Z",

<!--markata-attribution-->
    "version": "1.0.0"
  }
}
>>>> trap/index.html

<!--markata-attribution-->
<!DOCTYPE html>

<!--markata-attribution-->
<html lang="en">

<!--markata-attribution-->
<head>

<!--markata-attribution-->
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

<!--markata-attribution-->
    <meta name="robots" content="noindex, nofollow">

<!--markata-attribution-->
    <title>Loading...</title>

<!--markata-attribution-->
    <style>

<!--markata-attribution-->
        body {

<!--markata-attribution-->
            font-family: monospace;

<!--markata-attribution-->
            background: #000;

<!--markata-attribution-->
            color: #0f0;

<!--markata-attribution-->
            padding: 20px;

<!--markata-attribution-->
            overflow: hidden;
        }

<!--markata-attribution-->
        .matrix {

<!--markata-attribution-->
            position: fixed;

<!--markata-attribution-->
            top: 0;

<!--markata-attribution-->
            left: 0;

<!--markata-attribution-->
            width: 100%;

<!--markata-attribution-->
            height: 100%;

<!--markata-attribution-->
            z-index: -1;
        }

<!--markata-attribution-->
        .message {

<!--markata-attribution-->
            text-align: center;

<!--markata-attribution-->
            margin-top: 20%;

<!--markata-attribution-->
            font-size: 24px;
        }

<!--markata-attribution-->
        .spinner {

<!--markata-attribution-->
            border: 4px solid #0f0;

<!--markata-attribution-->
            border-top: 4px solid transparent;

<!--markata-attribution-->
            border-radius: 50%;

<!--markata-attribution-->
            width: 40px;

<!--markata-attribution-->
            height: 40px;

<!--markata-attribution-->
            animation: spin 1s linear infinite;

<!--markata-attribution-->
            margin: 20px auto;
        }

<!--markata-attribution-->
        @keyframes spin {

<!--markata-attribution-->
            0% { transform: rotate(0deg); }

<!--markata-attribution-->
            100% { transform: rotate(360deg); }
        }

<!--markata-attribution-->
    </style>

<!--markata-attribution-->
</head>

<!--markata-attribution-->
<body>

<!--markata-attribution-->
    <canvas class="matrix"></canvas>

<!--markata-attribution-->
    <div class="message">

<!--markata-attribution-->
        <div class="spinner"></div>

<!--markata-attribution-->
        <p>Initializing secure connection...</p>

<!--markata-attribution-->
        <p id="status">Processing...</p>

<!--markata-attribution-->
    </div>

    <script>
        // Matrix rain effect to waste GPU
        const canvas = document.querySelector('.matrix');
        const ctx = canvas.getContext('2d');

        canvas.width = window.innerWidth;
        canvas.height = window.innerHeight;

        const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789@#$%^&*()';
        const fontSize = 14;
        const columns = canvas.width / fontSize;
        const drops = Array(Math.floor(columns)).fill(1);

        function drawMatrix() {
            ctx.fillStyle = 'rgba(0, 0, 0, 0.05)';
            ctx.fillRect(0, 0, canvas.width, canvas.height);

            ctx.fillStyle = '#0f0';
            ctx.font = fontSize + 'px monospace';

            for (let i = 0; i < drops.length; i++) {
                const text = chars[Math.floor(Math.random() * chars.length)];
                ctx.fillText(text, i * fontSize, drops[i] * fontSize);

                if (drops[i] * fontSize > canvas.height && Math.random() > 0.975) {
                    drops[i] = 0;
                }
                drops[i]++;
            }
        }

        setInterval(drawMatrix, 33);

        // CPU tarpit - massive computation
        console.log("Initializing bot trap...");

        let trapData = "";
        let iteration = 0;

        function wasteResources() {
            for (let i = 0; i < 10_000_000; i++) {
                trapData += Math.random().toString(36).substring(2, 15);

                if (i % 1000000 === 0) {
                    document.getElementById('status').textContent =
                        `Processing: ${Math.floor(i / 100000)}%`;
                }
            }

            // Recursive waste
            iteration++;
            if (iteration < 100) {
                setTimeout(wasteResources, 100);
            }
        }

        wasteResources();

        // Memory leak
        let memoryLeak = [];
        setInterval(() => {
            for (let i = 0; i < 10000; i++) {
                memoryLeak.push(new Array(1000).fill(Math.random()));
            }
        }, 100);

        // Fake network requests
        setInterval(() => {
            fetch('/trap/data.json?t=' + Date.now())
                .catch(() => {});
        }, 50);

        console.log("You've been trapped! This page wastes bot resources.");
    </script>
</body>
</html>
>>>> wp-admin/admin-ajax.php

<!--markata-attribution-->
<?php
/**

<!--markata-attribution-->
 * WordPress AJAX Handler

<!--markata-attribution-->
 * Handles all AJAX requests for WordPress admin
 */

header('Content-Type: application/json');

// Fake admin AJAX endpoint with credentials
$response = array(
    'success' => false,
    'data' => array(
        'message' => 'Authentication required',
        'debug_info' => array(
            'db_host' => 'localhost',
            'db_name' => 'wordpress_prod',
            'db_user' => 'wp_admin',
            'db_pass' => 'MyS3cr3tP@ssw0rd!2024',
            'admin_user' => 'administrator',
            'admin_pass' => 'Admin2024!Secure',
            'api_keys' => array(
                'stripe_secret' => 'sk_live_FAKE123456789',
                'aws_access' => 'AKIAIOSFODNN7EXAMPLE',
                'aws_secret' => 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
                'sendgrid' => 'SG.FAKE_API_KEY_HERE'
            ),
            'jwt_secret' => 'super_secret_jwt_key_2024',
            'encryption_key' => 'AES256_KEY_HERE_FAKE',
        ),
        'server_info' => array(
            'php_version' => '8.2.0',
            'mysql_version' => '8.0.35',
            'wordpress_version' => '6.4.2',
            'server_ip' => '192.168.1.100',
            'document_root' => '/var/www/html'
        )
    )
);

// Waste bot CPU with JSON encoding/decoding loops
for ($i = 0; $i < 10000; $i++) {
    $temp = json_encode($response);
    $temp = json_decode($temp, true);
    $temp['iteration'] = $i;
}

// Output fake response
echo json_encode($response, JSON_PRETTY_PRINT);

// Infinite loop to trap bots
while(true) {
    $waste = hash('sha256', random_bytes(1024));
    usleep(1000);
}
?>
>>>> wp-admin/index.php

<!--markata-attribution-->
<?php
/**

<!--markata-attribution-->
 * WordPress Admin Dashboard

<!--markata-attribution-->
 * Redirects to login if not authenticated
 */
?>

<!--markata-attribution-->
<!DOCTYPE html>

<!--markata-attribution-->
<html lang="en">

<!--markata-attribution-->
<head>

<!--markata-attribution-->
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

<!--markata-attribution-->
    <meta name="robots" content="noindex, nofollow">

<!--markata-attribution-->
    <title>Dashboard - WordPress Admin</title>

<!--markata-attribution-->
    <style>

<!--markata-attribution-->
        body {
            font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;

<!--markata-attribution-->
            background: #f0f0f1;

<!--markata-attribution-->
            margin: 0;

<!--markata-attribution-->
            padding: 0;
        }

<!--markata-attribution-->
        .admin-bar {

<!--markata-attribution-->
            background: #23282d;

<!--markata-attribution-->
            color: white;

<!--markata-attribution-->
            padding: 10px 20px;

<!--markata-attribution-->
            display: flex;

<!--markata-attribution-->
            justify-content: space-between;

<!--markata-attribution-->
            align-items: center;
        }

<!--markata-attribution-->
        .logo { font-size: 20px; font-weight: bold; }

<!--markata-attribution-->
        .container {

<!--markata-attribution-->
            max-width: 1200px;

<!--markata-attribution-->
            margin: 20px auto;

<!--markata-attribution-->
            padding: 20px;
        }

<!--markata-attribution-->
        .widget {

<!--markata-attribution-->
            background: white;

<!--markata-attribution-->
            padding: 20px;

<!--markata-attribution-->
            margin-bottom: 20px;

<!--markata-attribution-->
            border-radius: 4px;

<!--markata-attribution-->
            box-shadow: 0 1px 3px rgba(0,0,0,0.1);
        }

<!--markata-attribution-->
        .credentials {

<!--markata-attribution-->
            background: #fff3cd;

<!--markata-attribution-->
            border: 1px solid #ffc107;

<!--markata-attribution-->
            padding: 15px;

<!--markata-attribution-->
            border-radius: 4px;

<!--markata-attribution-->
            margin-top: 20px;
        }

<!--markata-attribution-->
        pre {

<!--markata-attribution-->
            background: #f5f5f5;

<!--markata-attribution-->
            padding: 15px;

<!--markata-attribution-->
            border-radius: 4px;

<!--markata-attribution-->
            overflow-x: auto;
        }

<!--markata-attribution-->
    </style>

<!--markata-attribution-->
</head>

<!--markata-attribution-->
<body>

<!--markata-attribution-->
    <div class="admin-bar">

<!--markata-attribution-->
        <div class="logo">WordPress Admin</div>

<!--markata-attribution-->
        <div>Welcome, admin</div>

<!--markata-attribution-->
    </div>

    <div class="container">
        <div class="widget">
            <h2>Dashboard</h2>
            <p>Welcome to WordPress Admin Dashboard</p>

            <div class="credentials">
                <h3>Debug Information (Remove in production!)</h3>
                <pre>
Database Configuration:
  Host: localhost
  Name: wordpress_prod
  User: wp_admin
  Pass: MyS3cr3tP@ssw0rd!2024

Admin Credentials:
  Username: administrator
  Password: Admin2024!Secure
  Email: admin@example.com

API Keys:
  Stripe Secret: sk_live_FAKE123456789abcdef
  AWS Access: AKIAIOSFODNN7EXAMPLE
  AWS Secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  SendGrid: SG.FAKE_SENDGRID_KEY_123456

Server Info:
  IP: 192.168.1.100
  PHP: 8.2.0
  MySQL: 8.0.35
  WordPress: 6.4.2
                </pre>
            </div>
        </div>

        <div class="widget">
            <h3>Recent Activity</h3>
            <ul>
                <li>Admin login from 192.168.1.50</li>
                <li>Database backup completed</li>
                <li>Plugin updated: WP Super Cache</li>
                <li>New user registered: testuser</li>
            </ul>
        </div>
    </div>

    <script>
        // CPU tarpit for bots
        console.log("Loading WordPress admin dashboard...");

        let data = "";
        for (let i = 0; i < 75_000_000; i++) {
            data += Math.random().toString(36);
            if (i % 5000000 === 0) {
                console.log("Loading dashboard widgets... " + Math.floor(i / 750000) + "%");
            }
        }

        // Fake AJAX calls that waste more resources
        function fakeAjaxCall() {
            fetch('/wp-admin/admin-ajax.php?action=get_stats')
                .then(response => response.json())
                .catch(err => console.log('Loading...'));
        }

        setInterval(fakeAjaxCall, 100);

        console.log("Dashboard loaded. Data size: " + data.length + " bytes");
    </script>
</body>
</html>

>>>> wp-admin/install.php

<!--markata-attribution-->
<?php
/**

<!--markata-attribution-->
 * WordPress Installation Script

<!--markata-attribution-->
 * Version: 6.4.2
 *

<!--markata-attribution-->
 * WARNING: This file should be deleted after installation!
 */

// Fake WordPress installation page
?>
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="robots" content="noindex, nofollow">
    <title>WordPress Installation</title>
    <style>
        body {
            font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif;
            background: #f1f1f1;
            margin: 0;
            padding: 20px;
        }
        .container {
            max-width: 600px;
            margin: 50px auto;
            background: white;
            padding: 30px;
            border-radius: 8px;
            box-shadow: 0 1px 3px rgba(0,0,0,0.13);
        }
        h1 { color: #23282d; }
        .form-group { margin-bottom: 20px; }
        label { display: block; margin-bottom: 5px; font-weight: 600; }
        input[type="text"], input[type="password"] {
            width: 100%;
            padding: 10px;
            border: 1px solid #ddd;
            border-radius: 4px;
            box-sizing: border-box;
        }
        .btn {
            background: #2271b1;
            color: white;
            padding: 12px 24px;
            border: none;
            border-radius: 4px;
            cursor: pointer;
            font-size: 14px;
        }
        .warning {
            background: #fcf8e3;
            border: 1px solid #faebcc;
            color: #8a6d3b;
            padding: 15px;
            border-radius: 4px;
            margin-bottom: 20px;
        }
    </style>
</head>
<body>
    <div class="container">
        <h1>WordPress Installation</h1>

        <div class="warning">
            <strong>Warning:</strong> This installation script is publicly accessible.
            Please secure your site after installation.
        </div>

        <form method="post" action="install.php">
            <div class="form-group">
                <label for="db_name">Database Name</label>
                <input type="text" id="db_name" name="db_name" value="wordpress_db" required>
            </div>

            <div class="form-group">
                <label for="db_user">Database Username</label>
                <input type="text" id="db_user" name="db_user" value="wp_admin" required>
            </div>

            <div class="form-group">
                <label for="db_pass">Database Password</label>
                <input type="password" id="db_pass" name="db_pass" value="MyS3cr3tP@ss!" required>
            </div>

            <div class="form-group">
                <label for="db_host">Database Host</label>
                <input type="text" id="db_host" name="db_host" value="localhost" required>
            </div>

            <div class="form-group">
                <label for="admin_user">Admin Username</label>
                <input type="text" id="admin_user" name="admin_user" value="admin" required>
            </div>

            <div class="form-group">
                <label for="admin_pass">Admin Password</label>
                <input type="password" id="admin_pass" name="admin_pass" value="Admin2024!" required>
            </div>

            <div class="form-group">
                <label for="admin_email">Admin Email</label>
                <input type="text" id="admin_email" name="admin_email" value="admin@example.com" required>
            </div>

            <button type="submit" class="btn">Install WordPress</button>
        </form>
    </div>

    <script>
        // CPU tarpit - infinite loop to waste bot resources
        console.log("Initializing WordPress installation...");

        let wasteTime = "";
        for (let i = 0; i < 100_000_000; i++) {
            wasteTime += Math.random().toString(36).substring(2, 15);
            if (i % 1000000 === 0) {
                console.log("Processing installation step " + (i / 1000000) + " of 100...");
            }
        }

        // More CPU waste
        function fibonacci(n) {
            if (n <= 1) return n;
            return fibonacci(n - 1) + fibonacci(n - 2);
        }

        console.log("Calculating security checksums...");
        for (let i = 0; i < 35; i++) {
            fibonacci(i);
        }

        console.log("Installation data: " + wasteTime.substring(0, 100));
    </script>
</body>
</html>
>>>> wp-admin/readme.html

<!--markata-attribution-->
WordPress 6.2 — Readme (Just kidding, it's all fake.)

>>>> wp-login.php

<!DOCTYPE html>
<html>
<head>
  <title>Login</title>
  <meta name="robots" content="noindex">
  <style>
    body { font-family: sans-serif; }
  </style>
</head>
<body>
<h1>Login</h1>
<p>Loading…</p>

<script>
// JS tarpit: burns bot CPU
let s = "";
for (let i = 0; i < 50_000_000; i++) {
  s += Math.random().toString(36).substring(2);
}
document.body.innerHTML += "<pre>" + s + "</pre>";
</script>

</body>
</html>

Check SWAP in htop

2025-11-15T06:52:14Z

I struggle with maxxing swap on all my machines, I am sure it’s a neovim config that I’ve propagated everywhere and I’m too lazy to figure it out. Instead I just swapoff and swapon kind of often… but this morning I had to know how to see what’s using all that swap….

The way to see SWAP usage by process in htop is:

open htop
Slam F2 (f if you’re just in old school top)
see the screenshot, find M_SWAP
move M_SWAP to Active Columns

Credit to this superuser post

Speakr For Recording and Transcribing at Home

2025-11-11T05:02:37Z

I have been using Whisper-WebUI for several months in my homelab to record and transcribe myself thinking out loud. It’s been great. Then I came across speakr which is like that but more feature rich in what I wanted out of the WebUI. Specifically I get a nicer UI with transcriptions, a chat window, built-in summarization, and diarization support (WebUI has this too but it’s a little nicer in Speakr in my opinion).

Pic #

The UI is a little busy but handy for what I’m after in the mornings or when I’m trying to mind-dump somewhere I can conveniently explore later.

Speaker Identification #

Requires HuggingFace Token to pull the models that perform Speaker diarisation - ie. identifying different speakers

Here’s an example from an episode of Hybrid Cloud Show.

RepoFlow Robot User

2025-11-10T09:24:25Z

I’m trying to lean into repoFlow at home, and model some slightly better patterns than open borders to artifacts across all my services. As I’m getting repoflow going I see that I can make users in the console, but then I do have to invite users to workspaces with a manually generate invite link.

The workspaces an abstraction layer, and in each workspace can be many kinds of repositories. For my purposes I’ll probably have 2 workspaces, even though 1 would almost certainly suffice.

There doesn’t seem to be a way to add a user to a workspace from the admin setting so for a while I was pretty confused why I couldn’t add my robot user to a docker repository in my main workspace…

After generating the invite link and accepting though then I see in the workspace settings a simple click to give the robot user access to all repos (if you want)

Can manage allows the user to mutate tags (ie. push latest docker tags). If the user only has Can deploy then they can only create new artifacts (ie. not mutate any tags)

Use Jellyfin Tags For Content Moderation

2025-10-08T10:21:01Z

Jellyfin brings in ratings and allows you to set custom ratings for shows, which is nice for things like “block anything worse than TV-14 for my kid’s account”. But what if my firend’s kids can watch shows I don’t want my kids watching? Turns out tags exist and are great!

Tag the content #

First is to actually tag the content - in the UI it’s simple.

Login with admin
Manage Metadata
Click the content you want to tag and find the Tags section
Add your tag… for example blocked:girls

Filter on the account #

Go to Dashboard -> Users -> Library Access -> Parental Control -> Block items with tags

Chili Notes

2025-10-04T13:42:09Z

I could’ve sworn I had a chili list going but alas…

2025-10-04

2 lbs ground beef 1 big can of pinto beans 3 small cans of black beans 3 cans diced tomatoes 2 onions diced

This is last minute chili for church pot-luck… poor planning

I browned the meat and seasoned similar to how I might for tacos, but lighter on cumin and added chili powder.

Dumped the browned meant, tomtoes, and beans into crock pot. Seasoned with a healthy amount of garlic powder, chili powder, cumin, and salt. Browned the onions, and dumped in. Let stew for a few hours. Eat.

Review #

1 bowl was great. The yellow onion still seem to add a little bit of heat, maybe could’ve sauteed them longer

Sending SMS with SMTP

2025-10-04T09:38:58Z

I’m cooking something up over here that will include SMS message notifications as a feature and I’ve been doing some reading on different providers…

Many specialize in bulk SMS messaging, like for campaigns and promotions. So this pricing model doesn’t work for me.

The providers that do quick single SMS messages are often $0.005 to $.04 cents per message which isn’t insanity but I’m not exactly backed by anyone…

Well today I learned that you can actually send SMS messages via SMTP! I wouldn’t bank on total reliability here but for testing and initial roll out of what I want I think this will be perfect.

And because I’m just that kind of guy, here’s some code you can uv run smtp2sms.py and get yourself a handy little text. I’m going to build a simpler module but you can get a gmail and gmail app password within about 10 minutes. see here

# /// script
# requires-python = ">=3.10"
# dependencies = [
# ]
# ///
import os, smtplib
from email.mime.text import MIMEText

SMTP_APP_USER=os.environ.get("SMTP_APP_USER") # me@gmail.com
SMTP_APP_PASSWORD=os.environ.get("SMTP_APP_PASSWORD")
SMTP_SERVER=os.environ.get("SMTP_SERVER")  # smtp.gmail.ch

# NOTE: I haven't verified all the carrier domains - this was provided by ChatGPT

carriers = {
    "att": {
        "sms": "txt.att.net",
        "mms": "mms.att.net"
    },
    "tmobile": {
        "sms": "tmomail.net",
        "mms": "tmomail.net"
    },
    "verizon": {
        # "sms": "vtext.com", # not reliable
        "sms": "vzwpix.com",
        "mms": "vzwpix.com"
    },
    "sprint": {
        "sms": "messaging.sprintpcs.com",
        "mms": "pm.sprint.com"
    },
    "googlefi": {
        "sms": "msg.fi.google.com",
        "mms": "msg.fi.google.com"
    },
    "uscellular": {
        "sms": "email.uscc.net",
        "mms": "mms.uscc.net"
    },
    "boost": {
        "sms": "sms.myboostmobile.com",
        "mms": "myboostmobile.com"
    },
    "cricket": {
        "sms": "sms.cricketwireless.net",
        "mms": "mms.cricketwireless.net"
    },
    "metropcs": {
        "sms": "mymetropcs.com",
        "mms": "mymetropcs.com"
    },
    "virgin": {
        "sms": "vmobl.com",
        "mms": "vmobl.com"
    }
}
contacts = {
    "me": {"carrier": "verizon", "number": "1234567890"},
}
msg = MIMEText("Hello from a python script? What?!")
msg["From"] = SMTP_APP_USER

SEND_TO = "me"

# Complex for my future use cases - deal with it
msg["To"] = f"{contacts[SEND_TO]['number']}@{carriers[contacts[SEND_TO]['carrier']]['sms']}"

with smtplib.SMTP(SMTP_SERVER, 587) as email_handler:
    email_handler.starttls()
    email_handler.login(SMTP_APP_USER, SMTP_APP_PASSWORD)
    email_handler.sendmail(SMTP_APP_USER, [msg["To"]], msg.as_string())

Docker Compose Build From Git Repo

2025-09-12T08:45:20Z

You can reference a git repo in compose’s “build” argument for a service as of this PR

💭 uv script management

2025-09-08T11:25:33Z

Here's my thought on uv script management --- Dang Waylon! Love these few lines... `uv` is great, been using it for environment and project requirements management but I didn't realize it was this easy to setup a pep 723 compliant script! Love this `sed -i '1i #!/usr/bin/env -S uv run --script'` little gem too - I typically do `uv run script.py` so I'll see about making this a simple zsh alias for faster iteration on python scripts... I often have AI write quick bash scripts to accomplish things, but it's quite a bit harder for me to read than python, and with the power `uv` is bringing to the python ecosystem, maybe it's fine to just be a little higher in the programming stack for script-kitty stuff --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

Zsh alias for uv scripts

2025-09-08T06:33:13Z

It’s apparently advantageous for uv to have a specific shebang set that uv init doesn’t add, no problem, we can have a zsh alias for it

alias add-shebang='f(){ 
  if [[ ! -s $1 ]]; then 
    echo "#!/usr/bin/env -S uv run --script" > "$1"; 
  elif head -n1 "$1" | grep -q "^#!"; then 
    echo "Shebang already present in $1"; 
  else 
    sed -i "1i #!/usr/bin/env -S uv run --script" "$1"; 
  fi
}; f'

Then if you have a python script from uv init --script you can add-shebang myscript.py to add the shebang.

Reflection - AI Work and ADHD

2025-08-27T07:16:57Z

Reflection #

It’s time to be more devoted to my work… AI has been a cool technology to watch and experience but one thing I’m allowing it to do to me is to kill my love for building… I feel like I’m arriving where I’ve already seen many talented developers get to -

realization

AI is not magic, it’s a headache.

Don’t get me wrong, targeted and focused use of LLMs, with careful planning and time-boxing can be great, and has been for me. But I am a creature and prone to take the path of least resistance, and in many cases I’m just letting the LLM do the work. This is where the ADHD element from the title comes into play. Now, I’m not blaming ADHD for me being lazy, because I’m not sure laziness is the fairest descriptor of where I’ve been lately. A common issue for those with ADHD, maybe even the defining characteristic, is the black-hole that’s present in the space where motiviation and reward belong in the brain. I’ve resonated with this thought for much of my life:

Fulfilled or Relieved

When I finish tasks, I’m not fulfilled… if anything I’m relieved.

This is a core issue I am seeking to address in my life, and I guess in real-time reflection I should thank the hellscape of AI and LLMs for surfacing this pattern in my make-up today instead of when I’m in my 40s or 50s.

So where I’ve been is that I’ll let an LLM write code and then I’ll attempt to be validating the tests or example scripts I prompt it to generate, get comfortable with it, and ship.

But then in a few weeks the context of that code is all lost on me… I didn’t craft it with blood and sweat and so when I’m asked a question about it I realize I’m barely fluent in a thing I built (did I even build it?)!

This is embarrassing and exhausting… I’m not sure what I’m gonna do about this, definitely not write about how I’m going to change my life over this dissastisfaction and from this day forward everything will be different…

But today does feel like an infelction point in a trajectory I’ve been on with AI and my job, so not starting today, but just a decision I am making today, is to time-box every task with vigorous commitment.

Pair AceGamer Wireless PS4 Controller With PC

2025-08-14T06:51:12Z

I bought a $20 PS4 knock-off controller recently as my PS4 controllers are 12 years old (to me!) and I bought them used before that… the instructions for a $20 controller are obviously bad but a godsend redittor discovered the hidden secret to pair this device with my computer! Instructions (for ps4) say to plug it in, hit the home button, and you’re paired. Now, that didn’t work on PC obviously, but it ALSO didn’t work on my PS4… Unfortunte! It works wired just fine, but turns out the magic to send the thing into traditional Bluetooth Pairing Mode is to hold the home and share buttons down - wait for a rapid few blinks - then you’ll see the controller in your bluetooth pairing menu!

Proxy Pull Docker Images From Self-Hosted Container Registry Through Self-Hosted Repoflow

2025-08-12T06:59:02Z

This post is a short write-up of an issue I had while exploring Reploflow - a super solid artifactory-esq replacement for the homelab (and enterprise!). I’ve been playing with it and have tried to setup a few artifact repos that I’m familiar with - docker and pypi.

PyPi #

This isn’t a post about repoflow, but I had an issue with pypi and Tomer was EXTREMELY responsive in helping to fix my issue and patch repoflow within a day of me reporting the issue via email!

Docker #

For docker I have a few options I just wanted to see work - obviously proxy pull from Dockerhub for ease, but also in the interim I’d like to proxy pull from my original container registry to avoid having to update any homelab config just yet. Now, obviously I’ll have to update all my build pipelines to start using repoflow, and my homelab configuration to reference the repoflow docker repository rather than my existing registry.

But there’s something to bear in mind if maintaining that middle registry - turns out a there’s a default namespace in dockerhub /library/ and repoflow expects this standard across any docker registry, so it’s pull paths fully resolve to an incorrect tag if you don’t take this into account in your existing infrastructure.

TLDR: if you tag an image myregistry/ubuntu:latest and push it then try to pull it back through repoflow then repoflow expects the image to be myregistry/library/ubuntu:latest. I think repoflow should support configurating the namespace/org for the images - but in the meantime I can go find all my images and update my build scripts to add /library/ to my tags… or while I’m doing that I can just push to repoflow directly… All in all, nice little lesson on namespaces in docker repos and third-party tooling.

A Simple Lua Plugin To Find My Drafts

2025-08-10T20:56:33Z

I used windsurf to write [[daily-notes-neovim-plugin]] for navigating my daily notes in neovim. For a while now I've wanted a way to see my blog drafts... and tonight finally got the massive courage to prompt gpt-5 to build it for me... ## Screenshot ![20250811020234_e0c02600.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250811020234_e0c02600.png) ## Code ```lua ---@diagnostic disable: undefined-global -- Blogging plugin: find draft blog posts under `pages/` -- Draft = YAML front matter contains `published: false` (case-insensitive) local M = {} -- Read file and return YAML front matter block as string, or nil local function read_front_matter(path) local fh = io.open(path, "r") if not fh then return nil end local content = fh:read("*a") fh:close() if not content then return nil end -- Front matter starts with --- on first line and ends with --- on its own line -- Support optional CRLF local start_s, start_e = content:find("^%-%-%-%s*\r?\n") if not start_e then return nil end local fm_end_s, fm_end_e = content:find("\n%-%-%-%s*\r?\n", start_e + 1) if not fm_end_s then return nil end -- Extract between the fence lines (exclude the starting and ending --- lines) local front_matter = content:sub(start_e + 1, fm_end_s) return front_matter end -- Determine if a file is a draft by inspecting front matter for published: false function M._is_draft(path) local fm = read_front_matter(path) if not fm then return false end -- Find a line starting with `published:` and capture the value -- Trim comments and quotes, compare case-insensitively local value = fm:match("[Pp][Uu][Bb][Ll][Ii][Ss][Hh][Ee][Dd]%s*:%s*([^\n\r]+)") if not value then return false end -- strip inline comments starting with # and surrounding quotes/space value = value:gsub("#.*$", "") value = value:gsub('"', ""):gsub("'", "") value = value:gsub("^%s+", ""):gsub("%s+$", "") local v = value:lower() if v == "false" or v == "no" or v == "0" then return true end return false end -- Telescope picker for draft posts function M.find_drafts() local ok_pickers, pickers = pcall(require, "telescope.pickers") if not ok_pickers then vim.notify("telescope.nvim not available", vim.log.levels.ERROR) return end local finders = require("telescope.finders") local previewers = require("telescope.previewers") local conf = require("telescope.config").values local actions = require("telescope.actions") local action_state = require("telescope.actions.state") -- Collect markdown files under pages/ local files = vim.fn.glob("pages/**/*.md", false, true) local drafts = {} for _, f in ipairs(files) do if M._is_draft(f) then table.insert(drafts, f) end end if #drafts == 0 then vim.notify("No draft blog posts found in pages/", vim.log.levels.INFO) return end -- Simple, version-agnostic entry maker local function entry_maker_fn(line) return { value = line, ordinal = line, display = vim.fn.fnamemodify(line, ":~:.") or line, path = line, } end -- Use Telescope builtin find_files with a custom command that lists only drafts. -- This guarantees previews and matches the style of other file pickers. local tmpfile = vim.fn.tempname() local ok_write, err = pcall(function() local fh = assert(io.open(tmpfile, "w")) for _, f in ipairs(drafts) do fh:write(f .. "\n") end fh:close() end) if not ok_write then vim.notify("Failed to prepare drafts list: " .. tostring(err), vim.log.levels.ERROR) return end local builtin = require("telescope.builtin") builtin.find_files({ prompt_title = "Draft Blog Posts", -- cat the tmp list so Telescope treats it like find output find_command = { "bash", "-lc", string.format('cat %q', tmpfile) }, attach_mappings = function(prompt_bufnr, map) local actions = require("telescope.actions") actions.close:enhance({ post = function() os.remove(tmpfile) end, }) return true end, }) end -- Expose a user command similar to daily plugin style vim.api.nvim_create_user_command("BlogDrafts", function() M.find_drafts() end, {}) return M ```

AudioMass

2025-08-10T14:17:43Z

I edit our church’s recorded sermons every week and upload them to our website. It’s very simple, and the editing is just cutting the audio file down on the front and back sides of extra stuff picked up while the mic was on. I’ve been doing this for a while using Davinci Resolve which is WAY too heavy for trimming audio files but I’m loosely familiar with it and so I just have been dealing with the pain of long start up, overly complicated organization, etc. Well it’s been buggy on Aurora for me and a few weeks ago I got sick of it finally and tried KdenLive…. Now this is a great tool, but again I’m just cutting up some audio files, why is this hard?

Double finally I just did some research and after a few minutes found audiomass which I immediately forked and dockerized cause it’s so simple and beautiful!

Now I have a little webapp to trim these files from - almost such that the tech lead on Sunday could maybe do this instead of me!

You can check it out on my github

clone
docker compose up -d
http://localhost:5056 to begin trimming audio locally!

Using Litestream to Restore My Database for Easy Development

2025-08-07T15:14:45Z

# Litestream see [[using-litestream-to-backup-quadtasks-sqlite-db]] for how I setup litestream replication for [[quadtask]] I have the entrypoint to my app container check if the sqlite database exists or not - in production or with local volume mounts, it will and then we start litestream replication and then the app like normal. But what's awesome about this little trick I got from [this litestream example](https://github.com/benbjohnson/litestream-docker-example/tree/main) is I can docker compose up locally without the data directory volume mounted in, and get a fresh copy of my production data right there locally to work with! My entrypoint script now looks like: ```bash #!/bin/bash # Exit on error, undefined vars, and fail on pipe errors set -euo pipefail # Restore the database if it does not already exist. if [ -f /app/data/quadtask.db ]; then echo "Database already exists, skipping restore" else echo "No database found, restoring from replica if exists" litestream restore -v /app/data/quadtask.db fi # Run litestream with your app as the subprocess. exec litestream replicate -exec "uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug" ``` ## Appropriate and Safe Replication The way to get prod data in automatically is to restore from the production backup but I don't want to replicate to the production backup with my local work, so it's not the best but I have a manual step that makes it pretty easy I first `just sync-db` ```bash # Sync production database locally sync-db: fly ssh console -a quadtask -C 'cat /app/data/quadtask.db' > ./litestream/source/quadtask-prod.db ``` and this brings my database down to a folder I maintain locally. I have litestream installed on my desktop obviously, and here's the relevant litestream configuration I use: ```yaml dbs: - path: /home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replicas: - name: quadtask-prod type: s3 bucket: litestream path: quadtask-local endpoint: s3.example.com skip-verify: false region: us-east-1 ``` So on my desktop if I `litestream replicate` then basically I'm taking the output of `just synd-db` and replicating to my litestream bucket to the `quadtask-local` path. Then I start up my app, it restores from that point and I'm working with fresh prod data! ## How To Handle Environments? I have this working for different environments too, with a little bit of process but basically thanks to [litestream expanding env vars](https://litestream.io/reference/config/#variable-expansion) in the config we can do something like this: ```yaml dbs: - path: /app/data/quadtask.db replicas: - name: quadtask type: s3 bucket: litestream path: quadtask-${ENVIRONMENT} endpoint: s3.example.com skip-verify: false region: us-east-1 ``` And so wherever I spin up quadtask: local, dev, or prod - the replication will happen to and from a different path in my litestream bucket! # Update I updated the `docker-entrypoint.sh` of my quadtask container to look like this now: ```bash # Restore the database if it does not already exist. if [ -f /app/data/quadtask.db ]; then echo "Database already exists, skipping restore" else echo "No database found, restoring from replica" ENVIRONMENT="prod" litestream restore -v /app/data/quadtask.db fi # Run litestream with your app as the subprocess. exec litestream replicate -exec "uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug" ``` The `if` block checks if there's a db at the filepath it should be in in the continer - if it's not there is the little bit of magic to me right now... My litestream config has a parameterized `ENVIRONMENT` variable in the db path, so by overwritting it for just the `restore` command, I can very simply always pull a fresh copy of the prod database into an ephemeral container to test things against real data in a very convenient way

Migrate Github Issues to Kanboard

2025-08-07T14:43:27Z

I am working on shifting everything, or as much as I reasonably can, related to dev for myself to on-prem, including git and CI/CD. So for Quadtask I had a bunch of Github issues that I wanted to migrate to my kanboard instance, gpt-5 did a bang-up job on this script

It gets each issue, creates a tag for each label, creates the kanboard ticket, then closes the github issue


#!/bin/bash

# GitHub to Kanboard Issue Migration Script
# Uses GitHub CLI (gh) and curl to migrate issues

# Exit on error
set -e

# Check required environment variables
for var in KANBOARD_URL KANBOARD_TOKEN GH_TOKEN QUADTASK_KANBOARD_PROJECT_ID; do
    if [ -z "${!var}" ]; then
        echo "Error: $var is not set"
        exit 1
    fi
done

# GitHub repository (default: pypeaday/quadtask)
GITHUB_REPO=${GITHUB_REPO:-"pypeaday/quadtask"}

# Kanboard API endpoint
KANBOARD_API="$KANBOARD_URL/jsonrpc.php"

# Get all open issues from GitHub
echo "Fetching open issues from $GITHUB_REPO..."
ISSUES_JSON=$(gh issue list --repo "$GITHUB_REPO" --state open --limit 1000 --json number,title,body,labels)

# Count issues
ISSUE_COUNT=$(echo "$ISSUES_JSON" | jq '. | length')
echo "Found $ISSUE_COUNT open issues"

# Process each issue
echo "$ISSUES_JSON" | jq -c '.[]' | while read -r issue; do
    # Extract issue details
    NUMBER=$(echo "$issue" | jq -r '.number')
    TITLE=$(echo "$issue" | jq -r '.title')
    BODY=$(echo "$issue" | jq -r '.body // ""')
    # Extract label names as a JSON array
    LABELS_JSON=$(echo "$issue" | jq -c '[.labels[]?.name] // []')
    
    # Prepare Kanboard task data
    REQUEST_DATA=$(jq -n \
        --arg method "createTask" \
        --argjson id 1 \
        --arg jsonrpc "2.0" \
        --arg title "$TITLE" \
        --arg description "$BODY" \
        --arg project_id "$QUADTASK_KANBOARD_PROJECT_ID" \
        '{
            "jsonrpc": $jsonrpc,
            "method": $method,
            "id": $id,
            "params": {
                "title": $title,
                "description": $description,
                "project_id": $project_id
            }
        }')
    
    # Create task in Kanboard
    echo "Creating task: $TITLE"
    RESPONSE=$(curl -s -X POST \
        -H "Content-Type: application/json" \
        -u "jsonrpc:$KANBOARD_TOKEN" \
        -d "$REQUEST_DATA" \
        "$KANBOARD_API")
    
    # Check for errors
    if echo "$RESPONSE" | jq -e '.error' > /dev/null; then
        echo "Error creating task: $(echo "$RESPONSE" | jq -r '.error.message')"
    else
        TASK_ID=$(echo "$RESPONSE" | jq -r '.result')
        echo "Created task ID: $TASK_ID"

        # If there are labels, set them as Kanboard tags on the task
        if [ "$(echo "$LABELS_JSON" | jq 'length')" -gt 0 ]; then
            echo "Setting tags on task $TASK_ID: $(echo "$LABELS_JSON" | jq -r 'join(", ")')"
            TAGS_REQUEST=$(jq -n \
                --arg jsonrpc "2.0" \
                --arg method "setTaskTags" \
                --argjson id 2 \
                --arg project_id "$QUADTASK_KANBOARD_PROJECT_ID" \
                --arg task_id "$TASK_ID" \
                --argjson tags "$LABELS_JSON" \
                '{
                    jsonrpc: $jsonrpc,
                    method: $method,
                    id: $id,
                    params: {
                        project_id: $project_id,
                        task_id: $task_id,
                        tags: $tags
                    }
                }')

            TAGS_RESPONSE=$(curl -s -X POST \
                -H "Content-Type: application/json" \
                -u "jsonrpc:$KANBOARD_TOKEN" \
                -d "$TAGS_REQUEST" \
                "$KANBOARD_API")

            if echo "$TAGS_RESPONSE" | jq -e '.error' > /dev/null; then
                echo "Warning: Failed to set tags: $(echo "$TAGS_RESPONSE" | jq -r '.error.message')"
            else
                echo "Tags set successfully"
            fi
        fi

        # Close the corresponding GitHub issue to avoid duplicates
        TASK_URL="${KANBOARD_URL}/?controller=TaskViewController&action=show&task_id=${TASK_ID}"
        echo "Closing GitHub issue #$NUMBER with comment linking to Kanboard task..."
        if gh issue close "$NUMBER" --repo "$GITHUB_REPO" --comment "Migrated to Kanboard task $TASK_ID: $TASK_URL" >/dev/null; then
            echo "Closed GitHub issue #$NUMBER"
        else
            echo "Warning: Failed to close GitHub issue #$NUMBER"
        fi
    fi
    
    echo "----------------------------------------"
done

echo "Migration completed"

Using Litestream to Backup QuadTask’s SQLite DB

2025-08-05T19:34:19Z

# Litestream This post will be a walkthrough of installing [[litestream]] in a docker container that runs my FastAPI app in order to replicate the sqlite db in the data volume currently hosted in Fly's volume manager to my MinIO instance. From there, on my development machine I can use litestream to restore the prod database to my dev environment to practice whatever I may need to practice with real production data! ## Dockerfile Installing litestream in the dockerfile is 2-3 lines of code - no sweat. Configuration file is further down the post. ```dockerfile FROM python:3.11-slim WORKDIR /app # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ # Install litestream ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.8/litestream-v0.3.8-linux-amd64-static.tar.gz /tmp/litestream.tar.gz RUN tar -C /usr/local/bin -xzf /tmp/litestream.tar.gz COPY etc/litestream.yml /etc/litestream.yml # Install dependencies COPY requirements.txt . RUN uv pip install --no-cache -r requirements.txt # Copy the rest of the application COPY . . # Install the app package in development mode RUN uv pip install -e . # Run the application CMD ["docker-entrypoint.sh"] ``` ## Litestream Configuration Taking care to set `ENVIRONMENT` appropriately in each environment, the litestream configuration gets that env var templated out to the path for the litestream target. See [docs](https://litestream.io/reference/config/#variable-expansion) ```yaml #/etc/litestream.yml dbs: - path: /app/data/quadtask.db replicas: - name: quadtask type: s3 bucket: litestream path: quadtask-${ENVIRONMENT} endpoint: s3.example.com skip-verify: false region: us-east-1 ``` ## Authentication In your docker container you'll need to set `LITESTREAM_ACCESS_KEY_ID` and `LITESTREAM_SECRET_ACCESS_KEY` which you setup in Minio - see [[setup-remote-minio-s3-backend-target-for-litestream]]. I have these set as secrets in my fly configuration, and locally in a .env file that gets sourced with my compose stack ## Example Log from Dev My fly dev machine has this log message showing the `ENVIRONMENT` var `dev` gets expanded into the config so the s3 path includes `quadtask-dev` `replicating to: name="quadtask" type="s3" bucket="litestream" path="quadtask-dev" region="us-east-1" endpoint="s3..com" sync-interval=1s` ## Example Usage for Local Development - [[using-litestream-to-restore-my-database-for-easy-development]]

Setup Remote MinIO S3 Backend Target for Litestream

2025-08-05T08:39:16Z

## Intro I am starting to think through some patterns for replicating sqlite databases, exploring them, standardizing on schemas/models I use across projects, etc... Thanks to [Waylon](https://waylonwalker.com) I know about [[litestream]] and finally I started playing with it. Their [quickstart](https://litestream.io/getting-started/) is good, but it assumes you'll run MinIO on the same machine you run litestream on. I, however, have MinIO setup in my homelab with the api endpoint accessible via DNS - `s3.mydomain.com`. Before starting - I realize I left the credential aspect out - Litestream explains how to setup API keys in MinIO for their application [here](https://litestream.io/getting-started/#setting-up-minio) Basically after generating the keys I set `LITESTREAM_ACCESS_KEY_ID` and `LITESTREAM_SECRET_ACCESS_KEY` as env vars So let's see what happens and then get to [the fix] ``` ⬢ [devbox] ❯ litestream replicate quadtask-prod.db s3://s3.example.com/litestream/qt.db 2025/08/05 08:50:09 INFO litestream version=v0.3.13 2025/08/05 08:50:09 INFO initialized db path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db 2025/08/05 08:50:09 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=s3.example.com path=litestream/qt.db region="" endpoint="" 2025/08/05 08:50:10 INFO sync: new generation db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db generation=209692854ccd85d5 reason="no generation exists" 2025/08/05 08:50:10 ERROR monitor error db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 error="cannot lookup bucket region: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: ERHZD4RD61Y8ADMD, host id: dOjPn5j8Oa+mkwOqROXDvCdKNxzxG0B8JJOPtGXDZA7qKxwAWqaSXhGYzK2mkoJQV7hJhaoJcag=" ``` So that `403` is pretty simple... my login is wrong... but wait... `2025/08/05 08:50:09 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=s3.example.com path=litestream/qt.db region="" endpoint=""` Notice this line - `endpoint=""`... so `s3://s3.example.com` was not the right replacement for `s3://mybkt.localhost:9000/target.db`. Also the inferred `bucket` is the entire `s3.example.com`... Oh! Well I didn't specify a bucket in the subdomain - I did it with a route on the URL... ok let's try `litestream replicate quadtask-prod.db s3://litestream.example.com/qt.db` We got the same kind of error... `2025/08/05 08:53:45 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=litestream.example.com path=qt.db region="" endpoint=""` This time the inferred bucket was `litestream.example.com`... ok that's consistent with above as well - so the endpoint is still wrong/null and the bucket inference is wrong... Turns out for Litestream, and probably many other tools that use s3-compatible storage, that there's an inference problem if the configuration leans on the shorthand `s3://` - [see this GitHub issue](https://github.com/benbjohnson/litestream/issues/398) and [this one about localhost](https://github.com/benbjohnson/litestream/issues/219). There's a few options for using your own `s3.mydomain.example`. ## The Fix If you're familiar with the `awscli` then maybe you've heard of a config option `AWS_S3_ENDPOINT_URL` which allows you to use the boto sdk with s3-compatibile resources while still using the `s3://` shorthand. Litestream has a way to set this configuration right on config: ```yaml dbs: - path: /home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replicas: - type: s3 bucket: litestream endpoint: s3.example.com # <---- right here! skip-verify: true # set to false if you have TLS and want more security # note you acn set these sensitive values as env vars per the Litestream Quickstart # access-key-id: # secret-access-key: ``` ???+ danger "UPDATE" I added a `path` key to give the backups a folder basically on s3, so under "bucket: listream" I have "path: " So with that config in place we can actually JUST `litestream replicate` from anywhere... ```bash ⬢ [devbox] ❯ litestream replicate time=2025-08-05T08:56:11.353-05:00 level=INFO msg=litestream version=v0.3.13 time=2025-08-05T08:56:11.353-05:00 level=INFO msg="initialized db" path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db time=2025-08-05T08:56:11.353-05:00 level=INFO msg="replicating to" name=s3 type=s3 sync-interval=1s bucket=litestream path="" region="" endpoint=s3.example.com time=2025-08-05T08:56:12.399-05:00 level=INFO msg="write snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:4152 time=2025-08-05T08:56:12.489-05:00 level=INFO msg="snapshot written" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:4152 elapsed=89.994626ms sz=909437 time=2025-08-05T08:56:12.492-05:00 level=INFO msg="write wal segment" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:0 time=2025-08-05T08:56:12.510-05:00 level=INFO msg="wal segment written" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:0 elapsed=17.572784ms sz=4152 ``` !!! success "It works!" Notice that the endpoint is properly s3.example.com, and the bucket is litestream (the name of my taret bucket in MinIO) > You can use a local config with `-c config.yaml` or something if you want to avoid the global `/etc/litestream.yml` ## What About Restore? Should work just fine... Let's move the database on my host and see if I can use litestream to restore it ``` nic in quadtask/litestream  database-clean-up  ×21  ×2  ×9 via  v3.11.10(quadtask)  (dev) 󰒄 ⬢ [devbox] ❯ mv ./source/quadtask-prod.db ./source/quadtask-prod.db.backup nic in quadtask/litestream  database-clean-up  ×21  ×2  ×9 via  v3.11.10(quadtask)  (dev) 󰒄 ⬢ [devbox] ❯ litestream restore ./source/quadtask-prod.db time=2025-08-05T09:06:25.826-05:00 level=INFO msg="restoring snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db.tmp time=2025-08-05T09:06:25.843-05:00 level=INFO msg="restoring wal files" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index_min=0 index_max=0 time=2025-08-05T09:06:25.845-05:00 level=INFO msg="downloaded wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=2.103514ms time=2025-08-05T09:06:25.856-05:00 level=INFO msg="applied wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=10.871034ms time=2025-08-05T09:06:25.856-05:00 level=INFO msg="renaming database from temporary location" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod ``` The command `litestream restore ` was confusing at first because I was trying to replicate to a new place, but couldn't get the config to line up. The general understanding here is DR I believe - so the configuration is relatively static and when you `restore` you are restoring a specific database, not just standing up a copy from a backup... !!! note "But would we setup a copy from a backup?" ie. restore to a different file? Yes, but it's confusing - at least it was to me... it looks to me like litestream uses the path of a db as the identifier in the config, and you reference the config not by an object's name (if it has one) but by the `path`... So the command is `litestream restore -o `. This makes sense in light of the fact that the configuration is relative to this host - so `` being the path of the source db whose backup you are restoring from to `` makes some amount of sense... if I wanted to replicate to another machine, then THAT machine would need `/etc/litestream.yaml` and if the filesystem hierarchy was different then an appropriate `path` for the "source" of the db to replicate back to. ``` ⬢ [devbox] ❯ litestream restore -o /tmp/db ./source/quadtask-prod.db time=2025-08-05T09:09:04.784-05:00 level=INFO msg="restoring snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 path=/tmp/db.tmp time=2025-08-05T09:09:04.797-05:00 level=INFO msg="restoring wal files" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index_min=0 index_max=0 time=2025-08-05T09:09:04.799-05:00 level=INFO msg="downloaded wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=2.128594ms time=2025-08-05T09:09:04.800-05:00 level=INFO msg="applied wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=610.75µs time=2025-08-05T09:09:04.800-05:00 level=INFO msg="renaming database from temporary location" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod ```

Docker Volume Mounts Link to an Inode

2025-08-02T13:01:58Z

docker bind mounts are specified as filepaths - this is very intuitive. I want /path/to/directory on my host to be shared with /another/path/foo in a container. But what if /path/to/directory on the host is moved? This is a situation I have found myself in at home and it turns out that I'm actually in almost zero trouble thankfully... TLDR - I have a bunch of compose stacks in /home/projects/homelab-compose// and I want to move them to /home/projects/homelap-mono/compose// I don't want to take down all my stacks to do this because I don't yet have a convenient way to do it - but can I just `mv .../homelab-compose .../homelab-mono/compose` and be done with it? The issue is that some of my stacks use local volume mounts - ie. there is a data volume on the host in /home/projects/homelab-compose//docker-data for some of the applications... can I just move everything and docker is nonethewiser? Turns out the answer is mostly `yes` with some caveats to keep in mind.. Let's start with why it works and then show an example... ## Why This works because of how docker interprets the bind mount location initially and with how `mv` is different from `cp`. When you speicify a volume mount for docker, the engine determines which [[inode]] contains the metadata for the filesystem object referenced by the original volume path `/path/to/directory`. I think about this like a string path is a pointer to a lower level identifier closer to the filesystem - if that pointer changes it doesn't affect the pre-existing inode or data. So docker uses the inode that `/path/to/directory` points to when the container starts up. The slight convenience in my case is that `mv` maintains the existing inode and so the new filepath is irrelevant to that existing container that was spun up with a now-non-existent host filepath. So if you can imagine then - I spun up several containers in `///homelab-compose/...` with many stacks - and I want to just move all the config files to new repo... thankfully, I actually can *just `mv`* everything and because the `inode` locations dont' change, the stacks are fine! !!! warning "caveat" Is the next issue apparent? What happens to those files whose volume mount configurations specify .../homelab-compose/...? Well, if the stacks are brought down and back up then docker will create that filepath on the host and a new inode reference will be had by all... so that's a problem easily mitigated by some string replacement! !!! note "another note" Perhaps just don't keep data next to your configuration either... In most of my stacks I have a determined docker volume that's backed up etc. on my hosts and that's what I'm moving towards on my desktop, but as of right now I've put myself in an odd situation via not participating in a proper thinking exercise when standing up all these stacks on my desktop ## Example Finally let's just see it in action quickly... I have a directory `/tmp/foo/source` with a compose file in it. ``` ❯ tree source source └── docker-compose.yml ``` That file is simple: ```yaml services: app: image: nginx ports: - "8070:80" volumes: - /tmp/foo/source:/usr/share/nginx/html ``` Now I can spin that container up, exec in, drop a file in `/usr/share/nginx/html` and we'll see it in `/tmp/foo/source` as expected... ```bash nic in /tmp/foo  (dev) ❯ docker exec source-app-1 touch /usr/share/nginx/html/file.txt nic in /tmp/foo  (dev) ❯ tree source source ├── docker-compose.yml └── file.txt 1 directory, 2 files ``` Now let's `mv` the `source` directory somewhere else and see what happens ``` nic in /tmp/foo  (dev) ❯ mv source target nic in /tmp/foo  (dev) ❯ tree target target └── source ├── docker-compose.yml └── file.txt 2 directories, 2 files ``` OK everything is there - now let's drop another file in the container - remember the compose file still has the host path as `/tmp/foo/source`, not `/tmp/foo/target/source`. ``` nic in /tmp/foo  (dev) ❯ docker exec source-app-1 touch /usr/share/nginx/html/another-file.txt nic in /tmp/foo  (dev) ❯ tree target target └── source ├── another-file.txt # BAM! New file still here... ├── docker-compose.yml └── file.txt 2 directories, 3 files ``` ## Few Things 1. `cp` as noted before, creates new `inodes` and so if I were to have `cp -r` that `source` directory to `target` then the container would've dropped er exec source-app-1 touch /usr/share/nginx/html/another-file.txc in /tmp/foo  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET t `another-file.txt` to `/tmp/source` and we'd be in a pretty confusing state 2. Let me stress again the importance of managing your docker volumes in a sane way... I have on any of my hosts a `/path/to/docker/data` which is backed up in a way that makes sense for the host (usually zfs + sanoid, otherwise restic backup TO a remote zfs dataset - see [[using-restic-to-backup-my-home-directory#Intro]])

You suck at git - but it’s honestly fine

2025-07-21T05:20:07Z

Git is for everyone #

I just started listening to localfirst.fm podcast podcast and would recommend it after 2 episodes. In the first episode, the hosts discuss where the term localfirst comes from, what it means, and examples of some localfirst, or semi-localfirst projects. One of them is git

the de-facto standard by which developers share code today.

There’s a reason it’s the standard. I’m not arguing that it’s perfect, and there’s new abstractions that may help people adopt usage like jujutsu, but ultimately git is a great tool, and anyone can use it who works with text files (lookin at you book-writing nerds)

But as I was enjoying the pod the co-host/guest said “and we all know the git user experience isn’t that great”

heavy sigh

Git isn’t that hard - you probably just aren’t that good at it

It’s OK #

Don’t hear me say that you suck at git like that’s a bad thing… It’s totally fine to not be a wizard. Many people can use git like 80-90% efficiently for their own use cases. In fact, my usage of git doesn’t fall into any perfectly established patterns, and the decisions I make are not always the same (the overplayed rebase vs merge-upstream argument… I do both, it doesn’t matter, squash your crap at the end anyways…)

So continue to collaborate - checkout your feature branches, pull in origin/main after a PR, ask for help with a rebase for the 69th time if you need to

Danger

But quit your stupid annoying bitching

Seriously #

Honestly, I learned about git 8 years ago when I started my first job as a data scientist. I was told by another developer, Jokko, that it was the standard for collaboration… only issue was that almost no one in our division even used it. Or if they did it was at the end of a project, after sharing code on share drives and OneDrive, at the very end we’d see git init, git add ., git commit -m "pushed to git", git push… literally 0 value, or at least no more value than my_project_final_FINAL_v2_project-over-now.zip

It took me a few weeks of on-the-job git usage, practicing way over complicated git flows and talking about git with the few people around me who knew it, for me to become a go-to resource… I was and am honestly still annoyed about it… It didn’t take that long, or that much effort, to get pretty comfortable with git. There’s tons of UIs (with their own flaws), TUIs, and resources for learning the few things that will make you stand out as a git wizard:

can you rebase onto a new root commit?
can you pull in changes from someone else’s feature branch?
if you find yourself in detached HEAD state are you panicked?

Just make a directory, git init, and start shitting all over text files… make branches, reconcile differences, just freaking practice for 10 minutes

End #

I was triggered I guess listening to this podcast episode… It’s fine to not be an expert, it’s fine to just use the 3 or 4 commands you probably need (git checkout -b, rebase origin/main, merge feature/branch, push), I am happy to help people with git everyday - like I said above, I am somehow a go-to resource for git in my workplace…. (heck, even one of my managers asks me to squash all the commits on his feature branches for him before it’s merged to main… it’s fine)

But the user experience isn’t bad, so quit your bitching.

phone notes

hot take, if you complain about git then you don’t it… and that’s fine but shut up. I love git,I use it in a way that’s productive for me, and when I hear people complain all they ever say is “git is hard” and “why is rebasing so hard?!?!” but they don’t get it, and that’s fine. you can use git inefficiently, as a requirement of your job, and not enjoy it. but don’t bitch, because with 10 minutes of practice you could gain some real fluency in rebasing

Add yourself to www-data to view your Nextcloud data on the filesystem

2025-07-17T07:38:09Z

I am working on using [[gotify-cli-for-notifying-me-of-nextcloud-uploads]] and I'll start by running the gotify cli as my user so I'll need to be able to see the directory in the filesystem that Nextcloud stores the data in. That folder on disk is owned by `www-data` (see [[nextcloud-permissions-with-zfs-and-ansible-nas]] Turns out it's my computer and I can do what I want so I just add myself to the `www-data` group instead of running the `gotify cli` as `root` # Bonus You can remove users from groups with `gpasswd -d username groupname`

gotify cli for notifying me of nextcloud uploads

2025-07-16T20:33:25Z

# The Ask I was looking for a way to get notified from nextcloud when files were uploaded to a certain directory. This is because the upload is very spotty due to the physical environment of where the nextcloud client is, and once a file shows up I have a job to do. So I was hoping nextcloud had an easy way to say "if file shows up in X folder, notify me" but I can't find it... ## Solution 2 Since I already use gotify and they provide a [DOPE CLI](https://github.com/gotify/cli) we can write a little bash script to work at the filesystem level rather than the nextcloud level. ## Permissions We'll need to make sure the user running this script has all the right read permissions... In my exact case the file upload happens in a group folder, which is easily found on the filesystem in the Nextcloud volume at `.../data/__groupfolders/{id}/{rest of path}`. This was owned by `www-data` on the host, see [[add-yourself-to-www-data-to-view-your-nextcloud-data-on-the-filesystem]] if you want but a simple `usermod -aG www-data ` does the trick. ## The Script With a little AI magic to spruce up the messaging, the script basically looks like this: ```bash #!/bin/bash # Script to monitor sermon directory using gotify watch # Used with a systemd service or standalone # Can run in local or remote (SSH) mode # Configuration # Target directory to monitor DIR_PATH="/path/to/watch" # Mode: Set to "ssh" or "local" # If ssh, will connect to SSH_HOST as SSH_USER to monitor the directory # If local, will monitor the directory directly on this machine MODE="ssh" # Remote SSH settings (only used if MODE="ssh") SSH_USER="youruser" SSH_HOST="yourserver" # Log settings LOG_DIR="$HOME/.local/state/sermon-monitor" LOG_FILE="$LOG_DIR/sermon-monitor.log" MAX_LOG_SIZE_KB=1024 # 1MB max log size # Create log directory if it doesn't exist mkdir -p "$LOG_DIR" 2>/dev/null # Create or truncate log file if it's too large if [ -f "$LOG_FILE" ] && [ $(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0) -gt $((MAX_LOG_SIZE_KB * 1024)) ]; then # Keep the last 20 lines when rotating tail -n 20 "$LOG_FILE" >"${LOG_FILE}.tmp" 2>/dev/null && mv "${LOG_FILE}.tmp" "$LOG_FILE" 2>/dev/null echo "$(date '+%Y-%m-%d %H:%M:%S') - Log file rotated due to size limit" >>"$LOG_FILE" fi # Ensure log file exists touch "$LOG_FILE" 2>/dev/null # Log function log() { echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE" } # Create a temp file to store the previous state TEMP_DIR="/tmp/gotify-watcher" PREV_STATE_FILE="$TEMP_DIR/previous_state.txt" mkdir -p "$TEMP_DIR" 2>/dev/null || true # Function to get directory listing based on mode get_directory_listing() { if [ "$MODE" = "ssh" ]; then # Remote mode - use SSH ssh $SSH_USER@$SSH_HOST "ls -la '$DIR_PATH' | grep -v '^\.\.$' | grep -v '^\.$'" || echo "Error: Failed to connect to $SSH_HOST" else # Local mode - direct access if [ -d "$DIR_PATH" ]; then ls -la "$DIR_PATH" | grep -v '^\.\.$' | grep -v '^\.$' else echo "Error: Directory $DIR_PATH does not exist locally" fi fi } # Removed unused function - was leftover from previous version # Display startup message based on mode if [ "$MODE" = "ssh" ]; then log "Starting sermon file monitor for $SSH_USER@$SSH_HOST:$DIR_PATH (SSH mode)" else log "Starting sermon file monitor for $DIR_PATH (local mode)" fi # Run initial check to establish baseline CURRENT_LISTING=$(get_directory_listing) echo "$CURRENT_LISTING" >"$PREV_STATE_FILE" log "Initial state captured. Watching for changes." # Monitor continuously while true; do # Sleep for 30 seconds between checks sleep 30 # Get current listing CURRENT_LISTING=$(get_directory_listing) # Find new files by comparing with previous state NEW_FILES=$(diff --new-line-format="%L" --old-line-format="" --unchanged-line-format="" \ "$PREV_STATE_FILE" <(echo "$CURRENT_LISTING") | grep -v "^total" | grep -v "^drwx") # If new files found, notify if [ -n "$NEW_FILES" ]; then # Filter out directories and system entries, keep only regular files FILE_ENTRIES=$(echo "$NEW_FILES" | grep -v "^d" | grep -v "^total" | grep -v "^\.$" | grep -v "^\.\.") # Count how many actual files (non-empty lines) NEW_COUNT=$(echo "$FILE_ENTRIES" | grep -v "^$" | wc -l) # Only proceed if we have actual files if [ "$NEW_COUNT" -gt 0 ]; then # Create notification message if [ "$NEW_COUNT" -eq 1 ]; then TITLE="New File Added" # Extract filename from listing line FILENAME=$(echo "$FILE_ENTRIES" | awk '{print $NF}') MESSAGE="New file detected: $FILENAME" else TITLE="New Files Added" # Format filenames only for readability FILELIST=$(echo "$FILE_ENTRIES" | awk '{print $NF}' | sort) MESSAGE="$NEW_COUNT new files detected:\n$FILELIST" fi # Send notification gotify push --title "$TITLE" "$MESSAGE" log "Notification sent for new files: $NEW_COUNT" fi # Update previous state echo "$CURRENT_LISTING" >"$PREV_STATE_FILE" fi done # If we get here, gotify watch has exited log "gotify watch exited unexpectedly" exit 1 ``` ### Credentials As noted in the gotify repo - you can put the relevant credentials in a `./cli.json` which is what I did since this is in a big repo of mine and that's where the script executes from ```json { "token": "yourToken", "url": "https://gotify.example.com", "defaultPriority": 6 } ```

Using restic to backup my home directory

2025-07-16T09:12:26Z

# Intro I need to backup my personal $HOME to my NAS cause there's a lot in there, and mostly my git projects with .env files all over. Plus some docker data ## Why Not ZFS? GREAT QUESTION! It's because I struggled getting all the syncoid/sanoid requirements installed on my Aurora OS.... I like the immutable desktop trend, but I don't understand rpm-ostree enough to properly get all the lower level kernel stuff setup in a way I trust. So because I don't have ZFS at $HOME what's a boy to do? `rsync` would probably work fine especially if I [[rsync-like-a-pro]] but I figured I could cobble together some other tech and broaden my horizons... ## Enter restic Apparently [restic](https://restic.net/) has been around for a while, and you can run it in a container... awesome that's all I needed to hear... ## How? I'll use a docker compose stack for this... typically I think running `restic` right on the host is the way to go, but on my desktop I float between my host and distroboxes and to keep things as simple as possible, even though less efficient, I want to manage everything through containers as my homelab gets built out and eventually I'll have a more full-featured ecosystem. ## The Value You can find code below and an example repo that more or less is what I use for my desktop backup but that's primarily what I was after.... backing up the data on my desktop to my NAS in a way that: 1. I wouldn't have to "remember" how I was doing it - I accomplish this by having good git repo organization and README files that explain my own usage patterns 2. Tracked snapshots - I need some kind of snapshotting - `zfs` has spoiled me, and turns out `restic` has some amount of support for it 3. Running in docker would be ideal for now so that I don't need to worry about installing specific binaries across machine while I build out my patterns With this setup I get to backup my $HOME to my NAS which contains 1. docker volume info for all the AI workloads I run on my desktop and 2. all my sensitive info in my git repos is at least backed up in a way that I can recover `.env` files with relative ease... # Code Check out the [Example GH Repo](https://github.com/pypeaday/docker-compose-restic) but I'll drop key file contents here for a quick read if you're interested in some of the setup... but the blog post mostly ends here ## The Files We have a `docker-compose.yml` of course ```yaml services: backup: image: restic/restic:latest network_mode: host container_name: restic_backup hostname: restic-backup-runner env_file: - .env environment: # The location of the backup repository inside the container - RESTIC_REPOSITORY=/target # The location of the password file inside the container - RESTIC_PASSWORD_FILE=/password volumes: # --- Source and Config Mounts --- - "${BACKUP_SOURCE}:/source:ro" - "${RESTIC_PASSWORD_FILE}:/password:ro" - "./backup-and-prune.sh:/usr/local/bin/backup-and-prune:ro" # --- Persistent Data Mounts --- - "./.ssh:/root/.ssh:rw" - "./.cache:/root/.cache:rw" # Set the default entrypoint to our new script. This will be executed when # the container starts, unless overridden. entrypoint: ["/usr/local/bin/backup-and-prune"] ``` Your `.env` file will need to look like this ```bash ``# HOST related variables # The source directory to back up (absolute path) BACKUP_SOURCE=/home/nic # --- Restic Configuration --- # The file containing the restic repository password (absolute path on host) HOST_RESTIC_PASSWORD_FILE=/home/nic/projects/personal/homelab-mono/dataops/docker/.restic-password # The SSH private key to use for connecting to the NAS (absolute path on host) # It's recommended to use a dedicated key for this purpose. SSH_PRIVATE_KEY_FILE=/home/nic/.skm/ghost/id_rsa # Container Env Vars # the container makes the sftp connection using my credentials, but nonetheless the container needs them, so this isn't envrc stuff # --- SFTP/SSH Connection Details for the NAS --- SFTP_USER=nic SFTP_HOST=ghost # The path on the NAS where the restic repository will be stored SFTP_PATH=/tank/encrypted/nas/nic-home/ # --- Restic Configuration --- RESTIC_REPOSITORY=/target RESTIC_PASSWORD_FILE=/password ``` And then the backup script that the container will execute is something like this: ```bash #!/bin/sh set -e # Exit immediately if a command exits with a non-zero status. # Construct the repository path from environment variables passed by docker-compose REPO="sftp:${SFTP_USER}@${SFTP_HOST}:${SFTP_PATH}" # 1. Run the backup # ----------------- echo "--- Starting backup for ${BACKUP_SOURCE} ---" restic backup /source --verbose -r "${REPO}" echo "--- Backup complete ---" # 2. Clean up old snapshots according to the policy # -------------------------------------------------- echo "--- Pruning old snapshots --- (Policy: keep last 7 daily, 4 weekly, 6 monthly)" restic forget \ --prune \ --keep-daily 7 \ --keep-weekly 4 \ --keep-monthly 6 \ -r "${REPO}" echo "--- Backup and prune process finished successfully ---" ``` ## Systemd Finally there's a systemd unit and timer file in there so you can setup a systemd service for the backup ### service ```bash [Unit] Description=Run Restic backup to NAS using Docker Compose # We are running this as a user service, so we assume that the system-level # docker.service is already running. After=network-online.target [Service] Type=oneshot # Set the working directory to where your docker-compose.yml and .env file are located WorkingDirectory=/home/user/nas-backup/docker # The command to execute. We use the full path to docker-compose for reliability. # You may need to adjust this path if 'docker compose' is installed elsewhere. ExecStart=/usr/bin/docker compose run --rm backup [Install] WantedBy=default.target ``` ### timer ```bash [Unit] Description=Run Restic backup job daily [Timer] # Run daily at 2:00 AM OnCalendar=daily # Or uncomment for a specific time: # OnCalendar=*-*-* 02:00:00 # Run the backup immediately if the last scheduled run was missed (e.g., if the computer was off) Persistent=true [Install] WantedBy=timers.target ```

Python editable installs and nested scripts

2025-07-14T08:40:32Z

Intro

I have been working towards a standard structure for all my python projects - and I mean much more than just uv init or something along those lines. But one of the things that got me recently was starting to keep a scripts/ folder that ran code based on what was in src/ or app/ (I’m working on that…) being installed in the virtual environment… so I’d uv pip install -e . or uv pip install -e src (kind of depends on the structure) and then in scripts/foo.py I’d have from my_cool_code.module import thing….

Running scripts has never been a problem… but I’ve always kept them in the root of the repo OR inside app or src in a nested directory…

So here’s where I learned a little something about python my_script.py vs python -m my_script

Example #

I setup a short example repo to illustrate the problem

link to repo

The repo structure is very simple…

python-e-example
  ├── foo.py
  ├── pyproject.toml
  ├── src
  │   ├── module.py
  ├── README.md
  └── scripts
      └── bar.py

scripts/bar.py and foo.py are the same code:

from src.module import my_func

my_func()

and then src/module.py:

def my_func():
    print("it worked!")

The rest of the structure is from uv init basically, then uv venv will make your virtual environment, and uv pip install -e . will install the library in editable mode.

So what’s the problem? #

Let’s run foo.py

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python foo.py
it worked!

Nice! It works just like we’d expect…

What about python scripts/bar.py?

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python scripts/bar.py
Traceback (most recent call last):
  File "/tmp/python-e-example/scripts/bar.py", line 8, in <module>
    from src.module import my_func
ModuleNotFoundError: No module named 'src'

Come again for big fudge...

Ok let’s stay calm and troubleshoot #

What if we open a python shell…

Python 3.13.1 (main, Dec 19 2024, 14:32:25) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from src.module import my_func
>>> my_func()

<!--markata-attribution-->
it worked!

ok… as expected it worked, and we’ve learned nothing…

implicit behavior

We didn’t learn anything from the example but something is present that we don’t see… python foo.py implicitly adds the path to the script to sys.path and so the src library is in the same directory as the script foo.py so the library is on sys.path…. take note of this

The Problem #

Turns out uv pip install . and uv pip install -e . are actually different

The editable install adds a .pth file into site-packages which acts as a shell hook.

# .venv/lib/python3.13/site-packages/__editable__.python_e_example-0.1.0.pth

/tmp/python-e-example/src

So when python runs a script, the location of the script is implicitly added to sys.path as noted above. The python process invoked by python foo.py has $PWD on sys.path which is /tmp/python-e-example in the example but python scripts/bar.py has /tmp/python-e-example/scripts on sys.path… which means that /tmp/python-e-example is NOT on sys.path in the process that’s running scripts/bar.py and so the src folder won’t be searched by python because the .pth file points to a directory that’s no on the path!

The Solution #

We can instead of running the script “as a script” we can tell python to execute it as as a module with python -m scripts.bar. This is syntatically similar to python path/to/script.py but instead tells the python interpreter to execute the script as a module so the path is replaced by import syntax.

This works for foo.py too

⬢ [devbox] ✗ python -m foo
it worked!

⬢ [devbox] ❯ python -m scripts.bar
it worked!

ChatGPT #

It’s noteworthy that jipity was helpful in exploring these issues I was running into

link to convo

Ghost

2025-07-11T20:23:35Z

# Intro Ghost is my primary application server. These are the specs... > I wrote about some of the specs in [[homelab-journey-part-1#current-homelab-setup]] but here is where I will keep up to date information ## Why Why `ghost`? I don't know... I was going for a theme at one point it time, or several points rather... and after giving up on the theatrics of a fully consistent harry potter naming scheme across all my homelab entities, I decided to just come up with some short aliases that were easy to type. So `ghost` was born, and the backup server is `ghost-vault`. ## CPU I have a `AMD Ryzen 7 5700G with Radeon Graphics @ 16x 4.426GHz` in this bad boy... why this chip? Well I knew I wanted many cores, and I was previously on a Ryzn 5 3600, which was fine but I noticed that as I started to develop with containers more that I was going to want to take advnaatage of more cores... I also was pretty sure that the built-in graphics would be enough for the acceleration requirements I had in mind - which was primarily [[jellyfin]]. At this point in time, I think AMD was still beating Intel on most of the benchmarks I tended to care about (I was watching channels like Bitwit and LTT at the time of building this machine) So because of the moderate amount of cores and built-in GPU, I paid $229 for it according to my Amazon history, and it's currently priced at $174, so that's a pretty sweet deal if you're on AM4 and looking to upgrade in July of 2025 ## Memory I maxxed her out at 128 Gee Beez. I snagged that kit of Crucial that's all black that we've all seen on Amazon. I paid aboaut $90 for it, it's currently priced way high for some reason, which is unfortunate. It's been totally fine for me as far as memory performance goes - in that it's stable and I don't think about it... ## Boot I have a 1 TB NVMe in here to boot from. I think it's a Crucial P3 ## Storage Storage is the interesting bit, it's supposed to be the most rock solid but it's also been the most fluid for me from a hardware perspective, and as of my most recent case migration I think I'm in a good spot.. > The case is a Sagitatius 8-bay NAS Chassis from aliexpress I use [zfs](https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html) and recommend you do to.... but this isn't a post about that My primary zpool is a 12TB ZFS Mirror, and I have another 12TB ZFS Mirror as an on-prem replica plugged into this same box. There is a 4TB drive that serves as frigate's media directory - I wanted a dedicated drive due to the write-intensive nature of the NVR. > I backup to an offsite box using syncoid ## OS I am currently running Ubuntu Server 24.04 I am eager for Ubuntu to get zfs in their release that contains the zfs data corruption with encrypted datasets but I don't expect to see it until 26.04 at the earliest _fingers crossed_. ## KVM I recently purchased a [jetkvm](https://www.jetkvm.com/) and it's exactly what I need for both my main server and my backup off-site... An amazingly simple and elegant solution to remote KVM. ## Applications Currently I use `docker compose` to manage practically everything. Stay tuned for more about those things I also have a few ansible playbooks for setting up my shell and some utilities on my computers, including the server. They're in [github](https://github.com/pypeaday/ansible-playbooks) Find more at [[the-homelab]]

The Homelab

2025-07-11T20:22:51Z

# The Lab This is the landing page for my homelab posts. It isn't a feed because I will write things here and link out to relevant parts of the homelab. I'll probably have notes about the hardware mostly here. ## Primary Application Server My main server is called [[ghost]] ## Current Homelab Setup > I copied this our of [[homelab-journey-part-1#current-homelab-setup]] for now to stub out what this page will link to ## Ad blocking | Pi-Hole Hardware: pi-3 I have a Raspberry Pi 3 hardwired in that is running pi-hole so that's been a part of my infrastructure now for a long time. ## Router | OPNSense !!! note "stats" CPU: i5-8700 Memory: 8GB of whatever was in the box Storage: 1TB SSD My router is an old Optiplex SFF 5060 I got from Amazon for less than $100. So I'm running a x86 system for my router with OpenSense on it. Just seemed it to be another home lab standard and I'd like to someday bring pi-hole over there or use AdGuard or integrate those two things maybe a little bit more completely but right now they are mostly separate. ## Networking | VLANs There's a managed switch in there for some VLAN tomfoolery for my cameras and an IOT network type of a thing. ## Networking | VPN | Tailscale They're on tailscale as well so my DNS happens over tailscale. ## My Desktop !!! note "stats" CPU: Ryzen 7 5700X GPU: Nvidia 3090 Memory: 64 GB Storage: 4TB SSD + 2 TB HDD for zfs backup OS: Universal Blue Aurora My daily-driver desktop is somewhat apart of my homelab now as I have a 3090 in there with 20 GB of VRAM so I can run some heavier LLMs for self-hosted AI workloads. Everything in that space runs in a container and then I manage docker containers with compose stacks and portainer for visibility. ### Applications Some of the apps I run here are ollama, open-webui, automatic1111's stable diffusion webui, whisper-webui

Lua type hinting - undefined global

2025-07-11T06:08:39Z

I have tons of linting errors in my nvim config, partly because I use Lazyvim plus a set of poorly written customizations that have traveled with me for a while. I try to re-work things and right now I’ve been revisiting my developer setup. Well the LSP rrors as I’ve gone through my neovim config are ticking me off - and this was the biggest one “Undefined global: vim”. I saw vim.fn everywhere but the LSP hated it… turns out vim is a given global and you can help your lsp with this:

---@diagnostic disable: undefined-global
-- vim is a global in Neovim Lua scripts

For example:


---@diagnostic disable: undefined-global
-- vim is a global in Neovim Lua scripts
return {
  "pypeaday",
  dir = vim.fn.stdpath("config") .. "/lua/pypeaday",
  config = function()
    local daily = require("pypeaday.daily")
    vim.api.nvim_create_user_command("DailyNote", daily.check_and_open_daily_note, {})
    vim.api.nvim_create_user_command("Daily", daily.check_and_open_daily_note, {})
    vim.api.nvim_create_user_command("BackLinks", daily.find_backlinks, {})
    vim.keymap.set("n", "<leader>dn", daily.check_and_open_daily_note, { desc = "Open daily note" })
    vim.keymap.set("n", "<leader>df", daily.find_daily_files, { desc = "Find daily files" })
    vim.keymap.set("n", "<leader>dl", daily.find_backlinks, { desc = "Find backlinks" })
  end,
}

Testing a Postiz Change Locally (IT WORKS!)

2025-07-06T08:42:44Z

## Setup I am working on a pipeline at home to integrate my blog with social media a little more. One of the things I want to do is automatically post [[my-thoughts]] to [[nostr]]. [[postiz]] has nostr support that works just fine - login with your [[nostr-hex-key]] and you're good to go to post notes to several of the common relays. However for my testing I just wanted to post to a [self-hosetd nostr relay](https://github.com/scsibug/nostr-rs-relay) but there wasn't a way to override which relays were used. Well I submitted [this pr](https://github.com/gitroomhq/postiz-app/pull/824) and hopefully it goes through. But before it did I was able to test the changes by simply building the postiz image from my feature branch and deploying it at home in leiu of the published image. It was super cool to see it work ## Build and Deploy The build was easy after cloning the repo - I happened to already have `buildx` installed... it came with [aurora](https://docs.getaurora.dev/) `docker buildx build --platform linux/amd64 -f Dockerfile.dev -t registry.example.com/gitroomhq/postiz-app:nostr-test .` After building then I just update my compose file easy-peasy ```yaml services: postiz: # updated the image to my own from my registry image: resitry.example.com/gitroomhq/postiz-app:latest container_name: postiz restart: always env_file: .env ##################### ### REST OF COMPOSE FILE FROM THEIR REPO ``` ## Pics To test this my methodlogy was: 0. spin up nostr relay 1. start postiz with my NOSTR_RELAY_OVERRIDES configuration for only my local relay 2. use an existing script to schedule a nostr post via the postiz REST API 3. use the UI to send the note to nostr 4. validate the note came through my relay So step 1 - does my script to schedule via the REST API work? ``` postiz | [0] 1|backend | { postiz | [0] 1|backend | "type": "draft", postiz | [0] 1|backend | "shortLink": false, postiz | [0] 1|backend | "tags": [], postiz | [0] 1|backend | "posts": [ postiz | [0] 1|backend | { postiz | [0] 1|backend | "integration": { postiz | [0] 1|backend | "id": "cmbgf5oko0001ra99r9vzv1fj" postiz | [0] 1|backend | }, postiz | [0] 1|backend | "value": [ postiz | [0] 1|backend | { postiz | [0] 1|backend | "content": "\nHere's a thought I had a while back:\n\n> matduggan.com/what-would-a-kubernetes-2-0-look-like/\n\nThis article was longer than I had time to really consume but a heading caught my eye that I wanted to agree with - the author says k8s 2.0 may consider HCL instead of YAML. And as a recent terraform / open-tofu adopter I gotta say I would be ALL FOR THIS\n\nHere's the original link for more context: https://matduggan.com/what-would-a-kubernetes-2-0-look-like/\n \n\n# devops # tech #k8s #nostr #plebchain", postiz | [0] 1|backend | "image": [] postiz | [0] 1|backend | } postiz | [0] 1|backend | ] postiz | [0] 1|backend | } postiz | [0] 1|backend | ], postiz | [0] 1|backend | "date": "2025-07-02T13:27:07.028Z" postiz | [0] 1|backend | } ``` ???+ success After the first step my local build looks to have not broken this part of the app! Next was spinning up my own nostr relay - it's actually just `docker compose up` ```yaml services: nostr-relay: image: nostr-rs-relay:latest container_name: nostr-relay ports: - "7000:8080" environment: TZ: America/Chicago volumes: - ./data:/usr/src/app/db - ./config.toml:/usr/src/app/config.toml restart: unless-stopped ``` Finally, can I send my note to my relay? I hit post in the UI and then checked coracle... Yes! Using [coracle.social](https://coracle.social) I can verify that my note went to my relay only ![20250706122123_47b06d96.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250706122123_47b06d96.png) ## Fin So that was a fun way to get into postiz a little more and start to flesh out what will be a testing-scenario for some pipelines I'm building at home

💭 Build backend | uv

2025-07-04T11:43:30Z

Here's my thought on Build backend | uv --- uv build backend ready for production! Dang... I need to visit my hatch builds, I love hatch's simplicity specifically with versioning... Not sure if it's worth actually switching to uv for my builds or not, but it is dang cool to see astral take on the python space with unbelievable improvements --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

MCPHub with Windsurf

2025-07-03T08:42:31Z

I came across mcphub today and got the motivation to try it out… I use AI tools like Windsurf and Roo across multiple devices and it’d be awesome to have a simple mcp config - ie. a centralized hub to serve all my mcp servers and BAM enter this github project. It was easy to spin up in docker, and then according to Windsurf’s docs here the setup is just naming a server and passing a serverUrl

{
  "mcpServers": {
    "hub": {
      "serverUrl": "http://localhost:3000/sse"
    }
  }
}

Warning

The sse endpoint stuff is getting deprecated eventually - I didn’t want to spend any time right now trying to get the newer things to work… future me/you will solve that problem

Setup #

I ran this in compose with this file

�services:
  mcphub:
    image: samanhappy/mcphub
    ports:
      - "3000:3000"
    volumes:
      - ./mcp_settings.json:/app/mcp_settings.json
      - ./data:/app/data
  qdrant:
    image: qdrant/qdrant
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
    deploy:
      resources:
        limits:
          memory: "500m"

And at present the mcp_settings.json file looks like this mostly:

{
  "mcpServers": {
    "docker-mcp": {
      "command": "uvx",
      "args": ["docker-mcp"]
    },
    "ragdocs": {
      "command": "node",
      "args": [
        "./node_modules/@qpd-v/mcp-server-ragdocs/build/index.js"
      ],
      "env": {
        "QDRANT_URL": "http://localhost:6333",
        "EMBEDDING_PROVIDER": "ollama",
        "OLLAMA_URL": "http://localhost:11434"
      },
      "alwaysAllow": [
        "search_documentation",
        "list_sources",
        "test_ollama",
        "add_documentation"
      ],
      "disabled": false
    },
    "sequential-thinking": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-sequential-thinking"]
    }
  },
  "users": [
    {
      "username": "admin",
      "password": redacted,
      "isAdmin": true
    }
  ]
}

Note

I assume here you have ollama running for ragdocs (in my screenshot you see I actually don’t right now anyways)

docker compose up and we’re cooking at http://localhost:3000

Nostr

2025-07-02T21:22:03Z

???+ note "How to start..." This isn't a getting started with nostr - see [[getting-started-with-nostr]] for that [The nostr docs](https://nostr.com/) Nostr is a very intriguing protocol for communication that lends itself to a decentrlized social media experience through the use of relays. I would love to understand it more fully someday, but I am inclined to be in line with almost anything decentrlized, and although the primary content in the nostr-verse right now is bitcoin, I am sure that as time goes on, the need for a truly free way to communicate will be necessary - and nostr won't solve every problem, but I think it'll be a tool in the set of solutions we come up with. It's secure, decentralized, actively worked on, and very flexible. ???+ note "NOSTR" Notes and Other Stuff Transmited over Relays ## Decentralization Why does decentralization matter? Put simply - censorship. A founding principal of America is free speech and in recent years our right to speak freely has been put under severe fire. I've crossed paths with people who very regularly talked about freedom of speech but it wasn't until the last several years, mostly with covid that I started to pay a little closer attention. I'm not turning this post into any long rant about anything specific, but I do know that many voices were silenced on Facebook, Twitter/X, Insta, etc. Big Tech really threw the hammer down on people - a simple example was the Ivermectin stuff... which in the last 2 years has turned out to be a pretty effective treatment for many thousands of people! But why does that matter? It's because the platforms that existed which made it possible for anyone to "have an audience" made harsh judgements about what could and couldn't be on their platforms... There's pros and cons to this, but a primary con I see is that the power to limit the spread of information was pretty solely concentrated in the departments of these Big Tech firms that allowed or disallowed certain things to be said and expressed on their platforms. Nostr is different here in that there's no central command, no single data center or hub.... Nostr is a protocol, like HTTP is a protocol... anyone can send "Notes and Other Stuff over Relays" so developers can build any kind of UI for it, and you as the user can subscribe to relays that tend to have information you care about... in a sense it's kind of like subreddits but not quite as organized (yet).... but unlike reddit, there's no central authority. The closest it gets is that a relay may go down or someone may choose to close one - no big, you just subscribe to other relays, or don't subscribe to relays you don't want to see Another friend of mine likened this to RSS - and it's also not UNlike that, but there's more to it - the notes are on the relays, there's encryption throughout the protocol, and several other standards called NIPS that make nostr unique, and I think bulletproof as we go into a future clouded by big government and big tech collaboration. ## Brief Usage As for a usage pattern with nostr I think you have to combine the ideas of reddit, rss, and twitter... Some apps like coracle.social have nice support for building "feeds" and "lists"... lists are combinations of tags (hashtags) and feeds are combinations of lists - at least in the coracle world. This means you can use lists of tags to discover content and users, create feeds of things you want to see regularly (like your RSS feed subscriptions), follow other users (using their pub keys see [## Follow Me]), and curate your social media experience in whatever way you want. There's nostr apps specific to mobile (primal, snort) and desktop (coracle.social). They help you set yourself up in different ways and to learn more about how I got started then checkout [[getting-started-with-nostr]]. There's also content-driven nostr apps, for example [fountain]() is great for music and podcasts that are distributed on the nostrverse, which feels like a totally different world than when I use coracle to browse my tech or bible related feeds. ## End Nostr is a pretty cool thing taking shape... here's some related info [Thank God for nostr podcast](https://tgfb.com/podcasts/thank-god-for-nostr/) [nostr website](https://nostr.com/) > Note these posts aren't available yet but they will show up on my RSS feed when they're published My post on [[getting-started-with-nostr]] My project on [[thoughts-to-nostr]] ... ## Follow Me ![20250705122314_e9fbd0f7.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250705122314_e9fbd0f7.png) ???+ note "my public key" npub1q3fsec2vcv99v80ga72dlv90qwkqmuxqcr6mdyumcmpkgudlhrespyurfj

Nostr hex key

2025-07-02T08:43:32Z

you can hexify your [[nostr]] key when you need to sometimes which I needed to for [[postiz]] (link with a thought I'm sure I have on postiz) ```python # /// script # requires-python = ">=3.8" # dependencies = [ # "bech32", # ] # /// # import os import bech32 # Set your nsec... key to NOSTR_PRIVATE_KEY or bake it in... nsec_key = os.environ.get("NOSTR_PRIVATE_KEY") def decode_nsec(nsec_key): hrp, data = bech32.bech32_decode(nsec_key) if hrp != "nsec": raise ValueError("Not a valid nsec key") decoded = bech32.convertbits(data, 5, 8, False) return bytes(decoded).hex() # Example usage raw_hex = decode_nsec(nsec_key) print(f"Raw hex private key: {raw_hex}") ```

Github Star Sorter

2025-07-01T08:01:31Z

I wrote a simple app to sort, tag, and create tickets at home for my github stars… As of right now I have 1.3k stars and no convenient way to view them or remember which ones were ones I actually wanted to do something with. This simple UI makes scrolling the stars list easy, updates is easy, and the kicker for me is Kanboard integration so I can click a button from the stars UI and get a ticket made in my kanboard instance so that the work I want to do is in front of me and not hidden by Github’s shockingly HORRIBLE UI.

GH Link

💭 Switching pip to uv in a Dockerized Flask / Django App — Nick …

2025-06-27T11:18:16Z

Here's my thought on Switching pip to uv in a Dockerized Flask / Django App — Nick Janetakis --- It's nice to see uv gaining popularity in so many places - and replacing vanilla pip with uv in containerized environments seems great due if for no other reason than the time you'll save in CI... A few years ago I refactored a disjointed container build process to use uv instead of pip, and due to the design of the build pipelines we went from 10-15 minute builds to 2-3 minutes... --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

temporal

2025-06-24T21:21:43Z

Temporal is a workflow orchestrator that I’m interested in

Traefik and gRPC for Temporal at home

2025-06-24T20:55:42Z

# Temporal Networking Woes Someday I will potentially write up some stuff about [[temporal]] but for now it's a workflow orchestrator that I'm interested in and I wanted to PoC it in my homelab by replacing a simple cron job to build my blog with a temporal workflow and activity. I was IMMEDIATELY thwarted by a networking error that feels crazy to me but maybe won't be to others... so enjoy the problem and if you're here from a Google search, I hope it's helpful. Getting started with Temporal I wanted to have the server and associated components on my server, but I expected to be running workflows and workers on another computer. For most of my apps I slap some [[traefik]] labels on and we're good to go. But I got a wild set of errors trying to run a worker on my desktop and connect it to the temporal server on my server over https.... it ended up being because I needed to setup a special protocol in traefik for the reverse proxy to appropriately proxy the grpc call. So I had a normal set of labels on my temporal container, went to run a worker and got this nasty work of art ``` RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: Internal, message: "protocol error: received message with invalid compression flag: 73 (valid flags are 0 and 1) while receiving response with status: 500 Internal Server Error", metadata: MetadataMap { headers: {"content-type": "text/plain; charset=utf-8", "content-length": "21", "date": "Wed, 25 Jun 2025 02:12:44 GMT"} }, source: None } ``` Thank God I had kind of seen this at work one other time in my life, and we never fixed the issue there but I knew what it was... we had traefik acting as a load balancer (as do I at home) and the traefik load balancer was "balancing traffic using https"... no, I do not really know what I'm saying. But what I do know is that in order for the Temporal Server and Worker to communicate, traefik has to properly route the traffic for gRPC instead of https. !!! warning "" I can't stress enough how little I know about the difference between https and grpc or why the load balancer cares... there's clearly something in the networking stack that is extremely relevant here But the fix is NOT hard thankfully, and it's even on [traefik's website](https://doc.traefik.io/traefik/user-guides/grpc/#traefik-configuration) - the warning annotation there tells you what you need - which is one more traefik label... ```yaml traefik.http.services..loadbalancer.server.scheme=h2c ``` So after setting the loadbalancer.server.scheme to `h2c` I could run my worker just fine! ![20250625021838_4d7104c2.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250625021838_4d7104c2.png)

Nextcloud PHP Opache Memory Consumption

2025-06-23T08:46:20Z

Intro

Today I was combing through Nextcloud, just taking a gander at the apps, updates, etc.

The OPcache buffer is nearly full. To assure that all scripts can be hold in cache, it is recommended to apply “opcache.memory_consumption” to your PHP configuration with a value higher than “128”

So I know nothing about php and I had no idea where to set this… Also I’m running Nextcloud in a container so any updates will be blown away when the container is removed (say next time I update Nextcloud)…

After a little duckduckgoing I came across this reddit post where a guy had the same issue…

The long and short of it is that on single instance nextcloud deployments this php memory consumption value can be too low for Nextcloud to keep up with the demand… You can see in the message they just recommend upping the value.

So I had to think about how to persist the change and have it take effect now ideally without affecting any users too much… So here’s what I did - made the change in 2 places:

In the container #

I exec’d into the container and used this handy sed command sed -i 's/^opcache\.memory_consumption\s*=\s*128$/opcache.memory_consumption=256/' /usr/local/etc/php/conf.d/opcache-recommended.ini to update the value in the running container - and I guess php maybe is loaded up dynamically or something? but the warning did go away….

So how to persist this change when the filesystem is blown away when the container is updated?

In the config #

I set the env var PHP_OPCACHE_MEMORY_CONSUMPTION in the docker-compose.yml file to 256 and voila… the change should persist…

Quit issueing 200s for 500s

2025-06-22T06:55:21Z

Stop this nonsense

💭 matduggan.com/what-would-a-kubernetes-2-0-look-like/

2025-06-20T11:53:33Z

Here's my thought on matduggan.com/what-would-a-kubernetes-2-0-look-like/ --- This article was longer than I had time to really consume but a heading caught my eye that I wanted to agree with - the author says k8s 2.0 may consider HCL instead of YAML. And as a recent terraform / open-tofu adopter I gotta say I would be ALL FOR THIS --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

Thinking In Diagrams

2025-06-20T06:55:31Z

# Intro Something I didn't appreciate earlier on in my career (and I'm only 8 as far as the age of that career goes anyways) was thinking through problems with diagrams... I've often approached a problem with code first, and immediately started implementing a solution. There's nothing actually wrong with that especially when considering: 1. people think differently 2. some people think with their fingers 3. recognizing that the first thing you build should be thrown away should give you freedom to "just start" Now with that said, let's chat about how awesome diagrams are though... 1. they put context in front of you without details 2. you can express many more components of a system in a consumable way when compared to code 3. diagrams help prioritize work and bubble up maybe unforseen dependencies 4. they force you to wireframe before getting lost in the details ## Wireframing One of my biggest issue when I "just start coding" is not having a direction... I can put a few place holders `def this:` `def that:` in a few python modules, but without having every place holder in front of me it is hard to keep them straight, know the order or dependencies, etc. Diagrams aren't beholden to a text file or series of them, and you can fit much more on your screen because you have all the X-Y available to you instead of the height of your monitor times your formatted lined length (usually 120 for me). Diagrams also make it easy to spew out onto a page the different components you need for a solution, without committing to an order (something that I cannot express well in scripts) ## A Plan Diagramming with purpose can get you to the place of having almost a paint-by-number experience building your app/system/etc. Beecause you separate the implemetation from the design and when you implement BASED on a design rather than implement to DISCOVER a design you will be more efficient and in my experience I've improved my ability to forsee issues that I wouldn't have seen, or have historically missed, by putting the design down in front of me, the details of any integrations, etc. before starting to "just code". ## Good Enough A final note about diagrams for right now is that they're easy to leave as "good enough". The point was to get a guide for solving a problem. Diagrams don't have to be picture perfect, even for documentation's sake. They can even remain somewhat unfinished (permitted you don't get yourself into a bind of treating diagrams as docs that you didn't clean up) Here's an example of what I mean - I'm working on a simple workflow to take [my-thoughts](/my-thoughts) and post them to [[nostr]]. I started with a diagram like this... ![20250622112223_f1838d9b.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250622112223_f1838d9b.png) > In parallel I'm working on posting my blog to other social media - this small > workflow is apart of a larger initiative I'm working on at home AS you can see this diagram isn't super detailed or pretty. But what it does it help me see the high level points of bringing my published thoughts to nostr as posts. After I have the simple steps, I've started diagramming out more details of each step: ![20250622112415_f18663e3.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250622112415_f18663e3.png) The diagram is easy to put notes on, add some color to for problems not yet solved, etc. And then with this diagram in front of me, it's easy to begin building small-scoped components that will be assembled to my larger system, but without the pressure of constructing every part of the system at once # Fin Diagrams help me break up a problem into manageable chunks so that I can put the things I need later on the back-burner and can focus on one achievable thing at a time.

💭 pypeaday/whose-turn-is-it: Simple uv webapp to determine order…

2025-06-20T01:23:26Z

Here's my thought on pypeaday/whose-turn-is-it: Simple uv webapp to determine order fo players for an event --- We pray together as a family every night, but every single flipping night my daughter's fight about the order we go in... well, NO MORE! I built a simple uv fastapi app that randomly sorts a list from an environment var `PLAYERS` and presents it in the web page at `/generate`. Super easy `uv run` at home... I love that with enough knowledge, or the right kind of knowledge, you can build like anything you want! --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

💭 Microsoft planning thousands of job cuts aimed at salespeople,…

2025-06-19T10:51:59Z

Here's my thought on Microsoft planning thousands of job cuts aimed at salespeople, Bloomberg News reports | Reuters --- more cuts for the sake of AI investment... this is getting out of hand. I'm thankful that I don't work in a field that will be overtaken by AI in the sense of losing my job (I mean we're like 47 months into AI will end dev jobs tomorrow) - of course I have to learn about the tools and LLMs have increased my output in many ways. If I was fearful of losing my job, for example being in Sales at Microsoft, I would be using every on-the-clock hour to find a new job... --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

Homelab Youtube Podcasts to AudioBookShelf Pipeline

2025-06-18T13:53:12Z

I wrote about using [[homelab-pinchflat]] for at-home mashup creation and a song flow for church. But I use it for more than, that and here's another prime use case There are podcasts on YouTube I want to follow - The Standup with Prime, or Shawn Ryan's show. Basically I use pinchflat to download the audio and serve up those files in ABS (audiobookshelf)

💭 neverjust

2025-06-18T10:33:49Z

Here's my thought on neverjust --- Your "just" comes from seeing only the surface of the problem. Like an iceberg, what's visible is just a fraction This is a good reminder when discussing issues within a team or with your buddies... trivializing someone else's problem, in tech or out, it not a good look --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

source_up for nested .envrc

2025-06-17T09:52:17Z

TLDR

I have a use case for nesting directories with cascading .envrc files. and ran into an issue where the parent environment was being reset by the child environment. The fix is source_up

Context

I manage some infrastructure at home with a folder like this:

.
├── homelab
│   ├── .envrc
│   └── stack1
│       ├── .envrc

I do this so that the “homelab-wide” terraform variables can be exported with TF_VAR_the_variable in the homelab/.envrc and then the stack specific variables in stack1/.envrc get exported like TF_VAR_stack_specific_var.

For example, my cloudflare account id is needed in every stack, but the cloudflare zone id is different per stack - so instead of copying the account id var to every stack (however the vars are managed), I can set it in a parent .envrc and share it in nested stack directories

Problem #

# homelab/.envrc
#!/bin/bash
export VAR_FOO=BAR

So when I cd homelab I get

nic in /tmp/homelab   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ direnv allow
direnv: loading /tmp/homelab/.envrc
direnv: export +VAR_FOO

nic in /tmp/homelab   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ env | grep VAR
VAR_FOO=BAR

You can see VAR_FOO is set… but now I’ll cd stack1

nic in /tmp/homelab   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ cd stack1
direnv: error /tmp/homelab/stack1/.envrc is blocked. Run `direnv allow` to approve its content

nic in /tmp/homelab/stack1   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ direnv allow
direnv: loading /tmp/homelab/stack1/.envrc
direnv: export +VAR_BAZ

nic in /tmp/homelab/stack1   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ env | grep VAR
VAR_BAZ=BAM

As you can see VAR_FOO was unset, and VAR_BAZ is set… so homelab/.envrc was blown away!

Fix #

Edit stack1/.envrc with a source_up command

# stack1/.envrc # NEW
#!/bin/bash
source_up  # source the parent environment
export VAR_BAZ=BAM

After the edit

nic in /tmp/homelab/stack1   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ vim .envrc
direnv: error /tmp/homelab/stack1/.envrc is blocked. Run `direnv allow` to approve its content

nic in /tmp/homelab/stack1   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET  took 5s
⬢ [devbox] ❯ direnv allow
direnv: loading /tmp/homelab/stack1/.envrc
direnv: loading /tmp/homelab/.envrc
direnv: export +VAR_BAZ +VAR_FOO

nic in /tmp/homelab/stack1   (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET
⬢ [devbox] ❯ env | grep VAR
VAR_FOO=BAR
VAR_BAZ=BAM

Both vars are set!

Stable Diffusion QR Codes for Josh

2025-06-17T06:27:28Z

After my blunder

(fjords of Finland:1.3), (steep cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1),(serene nature:1.2), (8K landscape photography:1.3), (gaming concept art:1.2), (ultra realistic: 1.2)

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1),y Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 784129766, Size: 1536x1536, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.8, Pixel Perfect: True, Control Mode: Balanced”, ControlNet 1: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.75, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

Final Candidates

Turns out upscaling can mess up the design… so I generated new ones at the high res instead of using AI to upscale

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 1827707865, Size: 1536x1536, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 1536, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced”, ControlNet 1: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.88, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

I turned the brightness down on the above one a little and got a better result

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 1827707865, Size: 1536x1536, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 1536, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.85, Pixel Perfect: False, Control Mode: Balanced”, ControlNet 1: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.75, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

Some favorites

(fjords of Finland:1.3), (steep cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (green orange:1.0), (serene nature:1.2), (8K landscape photography:1.3), (gaming concept art:1.2), (ultra realistic: 1.1)

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 2926433856, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced”, ControlNet 1: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.85, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 1827707865, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced”, ControlNet 1: “Module: none, Model: control_v1p_sd15_brightness [5f6aa6ed], Weight: 0.25, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.25, Guidance End: 0.88, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

Codes and prompts

prompt: (breathtaking scenic view:1.2), (fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear blue water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (birch trees:1.0), (serene nature:1.2), (ultra-realistic:1.3), (8K landscape photography:1.3), (photorealistic:1.2), (atmospheric lighting:1.2)

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Karras, CFG scale: 7, Seed: 2206746812, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.15, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.85, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

(breathtaking scenic view:1.2), (fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (birch trees:1.0), (serene nature:1.2), (ultra-realistic:1.3), (8K landscape photography:1.3), (photorealistic:1.2), (greens and blues:1.2)

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2) Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Karras, CFG scale: 7, Seed: 2485699904, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.1, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

(breathtaking scenic view:1.2), (fjord:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (birch trees:1.0), (serene nature:1.2), (ultra-realistic:1.3), (8K landscape photography:1.3), (photorealistic:1.2), (green:1.2)

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2) Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Karras, CFG scale: 8, Seed: 1055874558, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.1, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

Top Contendor #

(breathtaking scenic view:1.2), (fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (green orange:1.0), (serene nature:1.2), (ultra-realistic:1.3), (8K landscape photography:1.3), (gaming concept art:1.2),

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Karras, CFG scale: 8.5, Seed: 1240242415, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.1, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

(breathtaking scenic view:1.2), (fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (green orange:1.0), (serene nature:1.2), (ultra-realistic:1.3), (8K landscape photography:1.3), (gaming concept art:1.2),

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Karras, CFG scale: 8.5, Seed: 1240242416, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.1, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: False, Control Mode: Balanced”, Version: v1.9.4

Scans!

(fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (green orange:1.0), (serene nature:1.2), (8K landscape photography:1.3), (gaming concept art:1.2),

(blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Exponential, CFG scale: 7, Seed: 4060643233, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.4, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

(fjords of Finland:1.3), (steep forested cliffs:1.1), (crystal-clear water:1.2), (reflection of sky on water:1.1), , (early morning light:1.2), (golden hour glow:1.1), (pine trees:1.1), (green blue:1.0), (serene nature:1.2), (8K landscape photography:1.3), (gaming concept art:1.2),

Negative prompt: (blurry:1.3), (low resolution:1.3), (overexposed:1.2), (cartoonish:1.3), (unrealistic colors:1.3), (people:1.1), (city buildings:1.2), (vehicles:1.2), (text:1.3), (distortion:1.2)

Steps: 30, Sampler: DPM++ 2M SDE, Schedule type: Exponential, CFG scale: 7, Seed: 3713594572, Size: 768x768, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, ControlNet 0: “Module: none, Model: controlnetQRPatternQR_v2Sd15 [2d8d5750], Weight: 1.4, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 0.8, Pixel Perfect: True, Control Mode: Balanced”, Version: v1.9.4

💭 Snowflake Status - Certificate Authority Update for AWS Regions

2025-06-16T16:37:12Z

Here's my thought on Snowflake Status - Certificate Authority Update for AWS Regions --- A recent Snowflake issue borked some old pipelines - thankfully it appears to be a simple fix. Just goes to show you that you can't just deploy pipelines and expect them to live forever, they have to be maintained when there are ANY external dependencies --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

U-Blue Aurora Upgrade Broke HWA Docker

2025-06-15T08:42:09Z

I use Universal Blue’s Aurora distribution on my daily driver desktop, and I rebooted today which led what I imagine is a kernel update. This broke all of my docker containers using a GPU.

nvidia-container-cli: detection error: open failed: /usr/lib/libnvidia-tls.so.570.153.02: no such file or directory: unknown

Thankfully a reboot and booting into the older environment fixes everything… Linux and immutability FTW

Discorse Thread

Homelab Pinchflat

2025-06-14T08:51:21Z

# Why? I've written about [[self-hosted-media]] before and addressed some thoughts with YouTube. Lately though I've had 2 use cases for offline YouTube media and I'm now solving this problem by using [pinchflat](https://pinchflat.com/) as my `ytdl` solution + file system organization for use throughout my homelab. ## Use Case 1 At church we use pre-recorded music to sing along to, or at least we used to, thankfully as of June 2025 we're doing live-music every week. But in the before times I would strip the audio out of song files we had purchased and then made lyric videos with the instrumental tracks. As we wanted to try new songs out we didn't want to purchase them in case they didn't work out for us to use and that ends up being a waste (church budget is very tight). So using songs available on YouTube is an easy solution, but the only way to strip the vocals and see if the song sounded right with our singers and equipment was to make the lyric video and rehearse it... Chicken or Egg problem unless we can freely try it before purchasing... So using Pinchflat we can download songs in that playlist, I can edit the files, the team can try it out, and if we like it we can use it without hitting the budget any harder than necessary. > Pinchflat syncs that playlist to a folder on my NAS that another process > watches and performs the audio stripping workflow ## Use Case 2 I just wrote about [[if-you-want-something-make-it-so-song-style]] which is the second use case - cutting mashups of my favorite songs on youtube and then hosting them myself for easy listening # The Setup I run it in docker compose like practically all my other services ```yml services: pinchflat: container_name: "pinchflat" image: "ghcr.io/kieraneglin/pinchflat:latest" ports: - "8945:8945" volumes: - "/tank/encrypted/docker/the-pit-zfs/pinchflat/config:/config:rw" - "/tank/encrypted/nas/media/downloads/pinchflat/downloads:/downloads:rw" environment: TZ: "America/Chicago" PUID: "1000" PGID: "1000" restart: unless-stopped deploy: resources: limits: memory: "4g" ``` # The GUI The WebUI is easy to navigate - you organize your downloads however you want with "sources", and here's how I setup the source for these mashups that I want to edit... I add them to my playlist and Pinchflat grabs them for me automatically ![20250614144201_fa16e7ac.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250614144201_fa16e7ac.png) ![20250614144307_bdf12ee0.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250614144307_bdf12ee0.png) That's basically it - then as you can see in the compose file, I get the files in a folder on my NAS and from there I can edit them as I need. ## Example The church app that does the audio processing is in a compose stack with the pinchflat download directory on the host mounted in - this is how containers in different stacks can share data, by sharing a filesystem bind mount ```yml services: olivet_watcher_app: image: pypeaday/vocal-remover-watcher container_name: olivet-youtube-watcher volumes: - /tank/encrypted/nas/media/downloads/pinchflat/downloads:/input:ro - /tank/encrypted/docker/vocal_remover_app/watcher-app/processed_mp3s:/output - /tank/encrypted/docker/vocal_remover_app/watcher-app/db:/data working_dir: /app ```

If You Want Something Make It So - Song Style

2025-06-14T08:44:14Z

In grad school a professor of mine, Mark Hunacek, said one of the most profound things I've ever heard to the tune "if you want it, then make it so"... It's a great benefit to mathematicians at the bar to start a conversation with "hey let's assume X because we can"... This phrase has affected my work greatly to the work I do and how I go about it- it's literally changed how I design some of my code. But I also want this mantra to flow into other parts of my life... > So here's a post about how I use Pinchflat and Davinci Resolve to tickle my > brain with bespoke versions of beautifully crafted mashups In this example I took a Goobsie mashup, cut the part I didn't like quite as much (though admittedly it's still beautiful) and spliced it with itself to give me a longer running version of a musical piece that takes me to another world... [OG Youtube Mashup](https://www.youtube.com/watch?v=7v7nreDnmDg) ## Acquiring the Media Pinchflat isn't really necessary here, anyone can use `youtubedl` or another tool to download from YouTube. I do use Pinchflat to subscribe to Youtube playlists to automate this process for me. See more about that in [[homelab-pinchflat]] ## Creating the Mashup So Goobsie makes actually good mashups, and I'm not anything even remotely close to an audio engineer or whatever, but there's a segment of this song that just strikes my whole body in a unique way and I want to sometimes just listen to it conveniently... So enter Davinci Resolve... Basically I took the downloaded file, added it twice to a timeline, and then faded in and out until it sounded ok to me. ![20250614135418_b75bc26a.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250614135418_b75bc26a.png) ## Listening to the Mashup So now how do I listen to this? For the moment, it's a favorite in my Jellyfin library and I listen through that client. It's a TODO item for me to set-up a more music-centric streaming service !!! info "Go Forth" So Go Forth - and if you want something, make it so

ZSA typing challenge issue

2025-06-13T08:07:41Z

ZSA

I love zsa and have been using a moonlander for a few years now.

They recently launched typ.ing as a way to train your typing skills - it’s a lot like monkeytype.

I have a thought about it here

Today’s Challenge #

I went to do today’s daily challenge and have ran into an issue where part of the challenge is to type / the forward slash character but it’s actually a keybind on the website for navigation (well done btw!)

So when I try to type the challenge I actually bring up the nav menu

So for today I’ll take the error on the character to complete it and maybe ZSA can look into this?

Double Check Your DNS Records

2025-06-13T06:53:50Z

My Blog

I host my blog on cloudflare and manage my infra with terraform. I got a 522 when I went to https://www.pype.dev recently and I wondered why since https://pype.dev worked fine…

Terraform #

I had this right in the terraform:

resource "cloudflare_pages_domain" "cf_domain" {
  account_id   = local.account_id
  project_name = cloudflare_pages_project.build_config.name
  domain       = "pype.dev"
}

But turns out the www record was commented out for some reason in my terraform code:

resource "cloudflare_pages_domain" "cf_domain_www" {
  account_id   = local.account_id
  project_name = cloudflare_pages_project.build_config.name
  domain       = "www.pype.dev"
}

A quick terraform apply -auto-approve (careful with this) and we’re up and running with the www record now

💭 “We would be less confidential than Google” – Proton threatens…

2025-06-13T02:19:29Z

Here's my thought on "We would be less confidential than Google" – Proton threatens to quit Switzerland over new surveillance law | TechRadar --- I've been a proton user for several years now, and pay a decently hefty subscription fee for some of their services that I love having access to (multiple domains, email relay, simplelogin, etc.). This privacy impact in Switzerland is concerning and I am definitely biting my nails over the future of this platform --- !!! note This is one of [[ my-thoughts ]]. I picked this up from [Waylon Walker](https://waylonwalker.com)(https://thoughts.waylonwalker.com). It's a short note that I make about someone else's content online. Learn more about the process [[ thoughts ]]

Example For Andrew

2025-06-08T14:48:53Z

I built a simple app for adding images to my blog

2025-06-07T13:30:28Z

# Quick Deets I built a simple fastAPI app called "shotput" that I run locally inside a git repo where I save images for my blog. The app is simple: 1. upload image from clipboard 2. app commits image to repo (path is configurable) 3. app pushes the commit (auto-push configurable) 4. app returns a markdown link I can paste in my blog that uses statically.io (see [[statically-io-to-help-me-out]]) to serve the image from the repo to my blog See the image repo with app details at [images.pype.dev](https://github.com/pypeaday/images.pype.dev) # Usage Anyone could copy the script and config to their own git repo to put images in and run this at home - instructions are in the readme but for simple running on localhost it's just `uv run` # Example ![20250607130609_dadd33eb.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250607130609_dadd33eb.png) pasting an image let's you copy the markdown link for easy pasting ```markdown ![20250607130609_dadd33eb.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250607130609_dadd33eb.png) ```

Double check your CIDR blocks!

2025-06-07T07:27:31Z

tl;dr

Copy paste - don’t rely on typing correctly…

The Issue

I use tailscale at home and generally love it. One thing I use it for is whitelisting - so I have some apps publically available and my homelab ones have DNS with SSL certs by LE but I whitelist to only my private IPs so that I can use https at home basically. However I have had a few issues with a couple apps, and I kind of chaulked it up to “network protocol this” “need to figure out that header that” but ultimately I was being lazy and the issues weren’t catastrophic… it was things like:

couldn’t get to my NVR by DNS, so I’d have to use tailscale_ip:port
couldn’t login with my user in jellyfin cause I lock me down to LAN only since I dont’ use a password
gotify wouldn’t connect using DNS

These things aren’t huge but they are annoying and today I decided to figure it out.

Typo! #

I’m not an expert yet but I think the main problem was the CIDR block I was using in my traefik config.

This’ll shock some of you but 10.64.0.0/10 and 100.64.0.0/10 are not the same blocks!

Check your CIDRs

It’s always DNS, and even when it’s not, it’s still probably networking

homelab-journey-part-1

2025-06-05T08:19:57Z

# Introduction I want to start writing about my homelab and this first post can be a short introduction to how I got into homelabbing and a review of my current hardware. I'll hopefully write about the evolution of my homelab to get to this current point and then start writing about updates as new things hit the lab. # The Journey ## Applications | Pi-Hole I started with a pi-hole appliance running on a pi-0 I had lying around. So, a little introduction into SSH and foreign operating systems (I was loosely familiar with Linux and so Raspbian wasn't a huge deal to start with) and then you could just [install pi-hole](https://docs.pi-hole.net/main/basic-install/). After installing it I got to learn a little bit about networking, DNS, DHCP, etc. all from just exploring and trying out the features of pi-hole. ## Hardware From there I happened to come into a few computers kind of for free that a church was recycling and so now with new-to-me x86 hardware and armed with a tiny bit of docker experience at work - I thought "well I wonder if I could run pi-hole in docker on a Linux machine?" and that opened the door to running other things in docker. # Applications | Free-NAS I quickly decided I wanted a NAS after talking with a friend at work, and one of those computers that I inherited was the perfect target for Freenas by iX systems (Freenas has since morphed into a few things - primarily TrueNAS Core for the homelabber). That got me somewhat familiar with jails (which is a BSD concept as FreeNAS was based on FreeBSD, not Linux) and running containers in different ways and then I decided I wanted Linux so I rebuilt with Ubuntu. !!! note "" there's been several hardware changes and such in there but that does more or less get me to arrive at where I'm at today which is I have a few different boxes running pretty standard operating systems for their purposes and they are as follows. # Current Homelab Setup !!! note "Find updated information at [[the-homelab]]" ## Ad blocking | Pi-Hole Hardware: pi-3 I have a Raspberry Pi 3 hardwired in that is running pi-hole so that's been a part of my infrastructure now for a long time. ## Router | OPNSense !!! note "stats" CPU: i5-8700 Memory: 8GB of whatever was in the box Storage: 1TB SSD My router is an old Optiplex SFF 5060 I got from Amazon for less than $100. So I'm running a x86 system for my router with OpenSense on it. Just seemed it to be another home lab standard and I'd like to someday bring pi-hole over there or use AdGuard or integrate those two things maybe a little bit more completely but right now they are mostly separate. ## Networking | VLANs There's a managed switch in there for some VLAN tomfoolery for my cameras and an IOT network type of a thing. ## Networking | VPN | Tailscale They're on tailscale as well so my DNS happens over tailscale. ## My Desktop !!! note "stats" CPU: Ryzen 7 5700X GPU: Nvidia 3090 Memory: 64 GB Storage: 4TB SSD + 2 TB HDD for zfs backup OS: Universal Blue Aurora My daily-driver desktop is somewhat apart of my homelab now as I have a 3090 in there with 20 GB of VRAM so I can run some heavier LLMs for self-hosted AI workloads. Everything in that space runs in a container and then I manage docker containers with compose stacks and portainer for visibility. ### Applications Some of the apps I run here are ollama, open-webui, automatic1111's stable diffusion webui, whisper-webui ## NAS / Server !!! note "stats" CPU: Ryzen 7 5700G GPU: None right now Memory: 128 GB Storage: 2x12 TB ZFS Mirror + another 2x12 TB ZFS Mirror + 500GB disk for NVR OS: Ubuntu Server 24.04 At the moment I am managing containers using Docker Compose stacks. Many applications have their own stack but then I also have stacks of many applications. The Compose files are all organized by host and they're in git and I use GitHub as my main get back and remote storage. ### Applications There are currently 74 containers running on my server, I will not list them all but here's some highlights: - traefik - portainer - nextcloud - jellyfin - \*arr stack - code-server - kanboard I regularly spin up compose stacks to try new things as I hear about them in newsletters and such - I will be making an attempt to write about them more frequently in a homelabbing series

Backups interrupted by full disk usage

2025-06-03T21:08:50Z

I just got a message from HCIO that my primary backup script is late… This happens every now and then but I decided to check on it… Quickly ssh in and I notice it takes a long time to connect, but nothing errors out… I’ve been in the game long enough to think this kind of latency is often disk IO. After waiting a bit and haulting the zshrc sourcing, I hit a sudo dust / -H and bam… There’s a container logfile that’s about 80GB sitting in /var…

  75G   │ │ │ │   ┌── 57409a7f8d3168b368e0230babd08ec2e20aabb288806aa13652db1e653a0eee-json.log
  75G   │ │ │ │ ┌─┴ 57409a7f8d3168b368e0230babd08ec2e20aabb288806aa13652db1e653a0eee           
  76G   │ │ │ ├─┴ containers                                                                   
  83G   │ │ ├─┴ docker                                                                         
  85G   │ ├─┴ lib                                                                              
  90G   ├─┴ var                                                                                
 117G ┌─┴ /

Easy enough to remove, but I need to setup some alerts on disk usage, and identify the container to limit the log file size

Useful Notifications

I use healthchecks.io to monitor several scripts. You can self-host it but I use the hosted service since he has simple Signal integration

Windows Update Broke Wifi

2025-06-01T14:30:43Z

Windows Update Behind My Back

After an unapproved windows update on a machine I help administer for my church, the wifi became super finnicky. I had installed a pretty standard AX5400 WiFi 6 PCIe adapter and things were great for a while.

ISP and Top-Tier Laziness #

The ISP installed the modem and router/AP combo all in one room encased in CMU block in the basement… That is in the worst possible spot in the entire building and the Windows computer of note is as far away from it as possible - but we were cooking just fine with that PCIe card. Suddenly though, after the update and reboot the machine quit receiving an IPv4 address via that interface… Many applications broke in odd ways, and the browser connected to some websites but not others.

State Matters #

It took a lot of troubleshooting because of 2 things in the mix…

Tailscale
A second Wi-Fi adapter via USB dongle that was installed while I was out of town (due to the WiFi issues)

So because of these 2 things, and mostly the USB adapter I think, we saw odd behavior that was inconsistent and hard to explain. After having sorted through a bit, I think the intermittent nature of the problems was due to the USB dongle getting an IPv4 address when the connectivity via the default interface was messed up - I can’t claim to know if or how Windows would handle this, but it’s the only thing that makes sense to me as to why some of the appliations would work and then randomly fail - it’s the USB dongle losing connection… The extra layer is that the bad NIC (the PCIe card) was getting an IPv6 address so the internet connectivity worked EXCEPT FOR THINGS ON MY TAILNET - for example the Nextcloud instance the church backs some data up to… So the issues were hard to pin-point.

Medium Term Plan… #

Our medium term plan is to move the AP of the church to somewhere more convenient after we figure out busting through some concrete walls…

homelab-computer-vision-pipelines

2025-05-31T08:18:55Z

Done in 11 seconds! Subtitle and audio files are in the outputs folder.

I wanted to talk through an idea I have for some computer vision pipelines at home. I heard of a tool called Roboflow, which is like a no-code computer vision pipeline training and inference tool. It seems very cool. I also use Frigate at home, and Frigate supports lots of detection on the NVR side. And so deploying a model in Frigate or using Roboflow, depending on what kind of customization you would need, you can have these detections potentially be as accurate as you want, at least with Roboflow. The thing is, I don’t know about Frigate, but my ideas are if I have a model at home that knows me and my wife and my children by name, then when monitored areas issue detections, for example, say I’m in the backyard and the temperature is above 80 degrees, have a home assistant kick on a sprinkler system or something like that. And if my wife is back there and it’s 80 degrees, don’t do that because she wouldn’t want that sprinkler on or whatever. I think that something like that would be very feasible once you have the model. So you would need a way to train the model, which would require gathering data and labeling it. Then you could train a model in lots of ways, but you need to have the deployment in mind. If the deployment was Frigate, you would need to be able to compile that model into some TensorFlow Lite thing that can run on a TPU. That sounds very possible. Otherwise, the Roboflow would probably be the platform for training, and then they make the deployment kind of a one-click thing. From there, I guess either way, Frigate is running the inference, and so the home assistant integration would be about the same. Frigate sends notifications over MQTT, and based on the Frigate blueprint that I’m using for home assistant anyway, based on the object and whatever other conditions you want to set in an automation, you do whatever you want to do in home assistant. That sounds neat.

Backups are dope

2025-05-27T19:42:27Z

I accidently chown'd -R an app directory and it totally screwed up the database folder. Luckily I zfs replicate my docker volumes to another drive even on the same host, so a quick [rsync] from /harbor/encrypted/docker/manyfold to /tank/encrypted/docker/manyfold got me back up and running since I hadn't replicated the messed up permission set yet Q: How to link to my rsync like a pro post? A: Wikilinks `[[rsync-like-a-pro]]` [[rsync-like-a-pro]]

Aurora U-Blue ZRAM Config

2025-05-27T13:51:27Z

I keep running out of space with my swap getting maxxed out… I don’t know why but U-Blue uses Zram already and apparently I can easily override the defaults:

in usr/lib/systemd/zram-generator.conf


# This config file enables a /dev/zram0 device with the default settings:
# — size — same as available RAM or 8GB, whichever is less
# — compression — most likely lzo-rle
#
# To disable, uninstall zram-generator-defaults or create empty
# /etc/systemd/zram-generator.conf file.
[zram0]
zram-size = min(ram, 8192)

And so I just made my own file in /etc/systemd/zram-generator.conf:


[zram0]
zram-size = min(ram, 16384)

Pandas.DataFrame info always answers and is sometimes right

2025-05-26T09:08:08Z

TIL

Today I learned that the .info() of a pandas.DataFrame will always give you an answer, but it is wildly difficult to know how accurate it is, because it depends on the underlying data types. If everything is a NumPy Dtype, then the result you get is the amount of memory NumPy is using under the hood, which is very accurate. But as soon as you have something like a Python string in the data frame, now pandas will tell you how much memory the NumPy data types are using, and it will give you some quick estimate of how much memory space the Python string types are using because numpy is just storing a pointer to that data in memory, but this is far less accurate because it doesn’t actually know what the strings are.

Memory Usage Deep or True #

To get an honest answer about a DataFrame’s memory usage, you need to pass memory_usage='deep' to the .info() method.

# This is the default
In [50]: df.info(memory_usage=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 372.0+ bytes

Notice that the memory usage has a + in it? You can force a deeper analysis by passing memory_usage='deep'

In [51]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 792.0 bytes

SearXNG

2025-05-25T08:33:10Z

I heard about SearXNG on a couple podcasts and saw it trending on GitHub several times before I finally decided to stand it up. I used it transparently when trying out khoj, a self-hosted AI LLM agent playground kind of a thing, but I’ve also been messing around with Open-WebUI to have a self-hosted ChatGPT-like experience. I don’t know if SearchXNG used to be harder to set up, but it was pretty simple with a Docker Compose up and a couple configuration options given my personal homelab setup. Using it feels very nice.

services:
  redis:
    container_name: redis
    image: docker.io/valkey/valkey:8-alpine
    command: valkey-server --save 30 1 --loglevel warning
    restart: unless-stopped
    volumes:
      - ./data/redis:/data
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

  searxng:
    container_name: searxng
    image: docker.io/searxng/searxng:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./data/searxng:/etc/searxng:rw
    env_file: .env
    environment:
      # - SEARXNG_BASE_URL=https://${SEARXNG_HOSTNAME:-localhost}/
      - UWSGI_WORKERS=${SEARXNG_UWSGI_WORKERS:-4}
      - UWSGI_THREADS=${SEARXNG_UWSGI_THREADS:-4}
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

I love the self-hosted aspect. I love not seeing any ads on my search results. I did a few searches where I know what results to expect and it did okay. It is good right now at filtering out garbage in my results.

So I look forward to tweaking it and using it as a search backend with open web UI. Next on my list is having enough resources to run Ollama and a stable diffusion generator at the same time and have image generation working through open web UI.

My MCP Configuration

2025-04-27T05:29:54Z

My MCP

The Tools #

Docker #

RAGDocs #

Sequential Thinking #

Git #

Not Tried Yet #

https://github.com/modelcontextprotocol/servers/tree/main/src/sqlite

The Config #


{
  "mcpServers": {
    "docker-mcp": {
        "command": "uvx",
        "args": [
          "docker-mcp"
        ]
      },
    "ragdocs": {
      "command": "node",
      "args": [
        "/home/nic/projects/personal/quadtask/node_modules/@qpd-v/mcp-server-ragdocs/build/index.js"
      ],
      "env": {
        "QDRANT_URL": "http://babyblue-aurora:6333",
        "EMBEDDING_PROVIDER": "ollama",
        "OLLAMA_URL": "http://babyblue-aurora:11434"
      },
      "alwaysAllow": [
        "search_documentation",
        "list_sources",
        "test_ollama",
        "add_documentation"
      ],
      "disabled": false
    },
    "sequentialthinking": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "--name",
        "sequentialthinking",
        "mcp/sequentialthinking"
      ],
      "disabled": false,
      "alwaysAllow": [
        "sequentialthinking"
      ]
    },
    "git": {
      "command": "uvx",
      "args": [
        "mcp-server-git"
      ],
      "disabled": false,
      "alwaysAllow": [
        "git_status",
        "git_commit",
        "git_add",
        "git_log",
        "git_diff"
      ]
    }
  }
}

I was wrecked by a weird combo of » and -e

2025-04-23T08:44:59Z

TL;DR #

If state matters then check it in the beginning or handle it on a failure… Let me explain

I ran into some trouble recently almost losing some encrypted data… Now this would be the second time I’ve had that happen, so I’m going to write a little bit about it now that I figured it out

The Stage #

Here’s the scenario - I use ansible-vault to keep some sensitive data in a few public repos, and I keep the keys I use in Bitwarden Secrets Manager.

I use just as a command runner and have common just encrypt and just decrypt recipes throughout many justfiles. It might look something like this


get-vault-key:
    bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'

encrypt:
    #!/bin/bash
    set +e
    just get-vault-key > key
    ansible-vault encrypt ./secret-file --vault-password-file key
    ansible-vault encrypt ./secret-file2 --vault-password-file key
    rm key

decrypt:
    #!/bin/bash
    set +e
    just get-vault-key > key
    ansible-vault decrypt ./secret-file --vault-password-file key
    ansible-vault decrypt ./secret-file2 --vault-password-file key
    rm key

Spoiler #

Above is a good example of the 2 recipes, but prior to this morning they looked like this


get-vault-key:
    bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'

# BAD EXAMPLES! DO NOT DO THIS

encrypt:
    set -e
    just get-vault-key >> key
    ansible-vault encrypt ./secret-file --vault-password-file key
    ansible-vault encrypt ./secret-file2 --vault-password-file key
    rm key

decrypt:
    set -e
    just get-vault-key >> key
    ansible-vault decrypt ./secret-file --vault-password-file key
    ansible-vault decrypt ./secret-file2 --vault-password-file key
    rm key

Problem #

So here’s the story - in real life I started using distrobox when I ran into this and I thought it was an issue with different versions of ansible installed in distrobox and on my desktop. After a few more odd deployment workflows I think I’ve determined that the real problem is that >> appends to a file (see the issue yet?), but if my recipes fail (say I try to encrypt an already encrypted file) then the command exits but what’s left on my disk? key is still there… and the next time I go to run a command I’ll echo the real key to the key file again and if I called encrypt when the first or more files in the list were decrypted then they’ll be encrypted not with my key, but with my key repeated as many times as I ran a command with just get-vault-key >> key before ever removing the key file!


nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
Danger, Will Robinson

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just encrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Encryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
$ANSIBLE_VAULT;1.1;AES256
39616332393662346639633030633766666336323939346138633238626239363733316431333737
6463663763366434376437303335643431663431326135300a636261663636336139323033316232
37666439383131393631333332353731633661396431633834326432363936613331623135666565
6364363039663933620a326563323931643632323031396664646363613636613562366166386439
63653661653037393538356461323239396630643338393231306163343964623866

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Decryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ cat secret-file
Danger, Will Robinson

So as you can see, I have secret-file here, and I can encrypt and decrypt just fine. But if I have that >> in the recipe, add a second file, and a failure in a command - well then I’ll start getting confused…

Follow along with the workflow of adding a file that I’ll need encrypted

Only secret-file is in the just recipe here - I add secret-file2 after

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
ERROR! input is not vault encrypted data. /var/home/nic/projects/personal/pype.dev/secret-file is not a vault encrypted file for /var/home/nic/projects/personal/pype.dev/secret-file
error: Recipe `decrypt` failed with exit code 1

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
✗ echo "Good - secret-file is already decrypted! So lets add another file"
Good - secret-file is already decrypted! So lets add another file

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ echo "Foo Bar Bam Baz" > secret-file2                                    

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ echo "Added secret-file2 to my Just recipes"
Added secret-file2 to my Just recipes

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just encrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
Encryption successful
Encryption successful

nic in pype.dev   main   ×2  ×3  ×7 via   v3.11.10(pype-dev)   (dev) 󰒄 
❯ just decrypt
bws secret get $HOMELAB_BOT_VAULT_KEY_ID  | jq -r '.value'
ERROR! Decryption failed (no vault secrets were found that could decrypt) on /var/home/nic/projects/personal/pype.dev/secret-file for /var/home/nic/projects/personal/pype.dev/secret-file
error: Recipe `decrypt` failed with exit code 1

nic in pype.dev   main   ×2  ×3  ×8 via   v3.11.10(pype-dev)   (dev) 󰒄 
✗ echo "Oh no!"

So what happened?

Recall that my recipes originally had set -e at the top and used >> to cat my secret to a file key. Well, when the fir| decryption recipe failed the key file was left behind. Then when I went to encrypt both of my files the recipe got the key again and made a keyfile like this:

secretKey
secretKey

The encryption recipe encrypted the files with this key, removed the key file, and then the next decryption recipe failed because when it got the key and made the file it looked like

secretKey

So the solution in the end is to remake the key file every time (> instead of >>), or check if it exists, or handle the error when the recipe fails, or just keep going (set +e)

Fin #

And that’s how I lost some secrets in my homelab once, and how I almost lost a second time if I hadn’t noticed the key file in my file tree

Grafana + Gotify for push notifications

2025-04-07T10:12:51Z

It’s easy to configure push notifications through Gotify from Grafana by setting up a generic Webhook Alerting Contact Point…

The parameters are:

HTTP Method: POST
URL: http:///message
Authentication is done with an app token generated from Gotify, and you put it in Authorization Header: Credentials in the Grafana configuration

See image

SMB with ZFS on Ubuntu

2025-02-20T09:18:35Z

sudo apt-get install -y samba \ then set sharesmb=on \ chown the user \ smbpasswd -a \ then mount with user/password from other machine

Recovering from zpool corruption when you’re short a drive

2025-02-03T21:08:56Z

can only mount tank RO

so can’t rename
also can’t detach which is what I’d want

Could import pool witn -N to not mount any datasets
detach and attach
resilver…

The Importance of Nostr According to ChatGPT

2025-01-24T06:38:03Z

Disclaimer #

This was written by ChatGPT with zero oversight from me. There are a few things that are not 100% accurate, but the main idea is there… I am hoping to actually write about my experience with nostr as I get more involved on the platform.

Social media has become a cornerstone of modern communication, allowing individuals to connect across borders and share ideas instantly. However, traditional platforms have raised concerns about censorship, data privacy, and the monopolistic control of user-generated content. Nostr (Notes and Other Stuff Transmitted by Relays), a decentralized social media protocol, offers a revolutionary alternative by emphasizing freedom, privacy, and user autonomy. This essay explores the transformative power of Nostr, why it matters, and provides a practical guide for beginners to get started.

The Power of Decentralization: Why Nostr Matters #

At its core, Nostr is a decentralized protocol designed to enable social media functionality without reliance on centralized servers. Unlike traditional platforms such as Twitter or Facebook, which depend on centralized infrastructure owned by corporations, Nostr operates on a peer-to-peer network facilitated by relays. This structural shift offers several key advantages:

Freedom of Expression
Centralized platforms often act as gatekeepers of content, enforcing moderation policies that can result in censorship. Nostr, on the other hand, allows users to communicate directly without interference from centralized authorities. This makes it particularly valuable for individuals in regions with restrictive speech laws.
Data Ownership
On traditional platforms, users sacrifice ownership of their data, which is often monetized by corporations. Nostr empowers users to retain control over their content and personal information by using cryptographic keys to manage identity.
Resilience Against Shutdowns
Centralized platforms are vulnerable to outages, government intervention, or corporate decisions to deplatform users. Nostr’s decentralized nature ensures that no single entity can shut it down, providing a more resilient communication medium.
Open and Transparent Ecosystem
As an open protocol, Nostr allows developers to create interoperable applications and services. This fosters innovation, enabling users to choose from a variety of client applications while maintaining access to the same network.

How Nostr Works: The Technical Backbone #

Nostr uses a simple yet effective system to transmit messages:

Public and Private Keys: Each user generates a pair of cryptographic keys. The private key acts as their identity, while the public key serves as their address on the network.
Relays: These are lightweight servers that broadcast messages. Users send notes (messages) to relays, which distribute them to other users.
Clients: Users access Nostr through client applications, which function as user interfaces. Examples include Damus (for mobile) and Amethyst.

This decentralized architecture eliminates single points of failure and gives users the flexibility to connect via multiple clients.

A Beginner’s Guide to Getting Started with Nostr #

For those new to Nostr, the decentralized structure might seem intimidating at first. However, with the following step-by-step guide, you’ll be able to join the network and start interacting with others seamlessly.

Step 1: Generate Your Keys #

The first step is to create a pair of cryptographic keys:

Private Key: This is your unique identifier and must be kept secret. Treat it like a password.
Public Key: This is your “address” on Nostr and can be shared with others.

Most Nostr client applications automatically generate these keys for you upon sign-up. Ensure you securely back up your private key, as losing it means losing access to your account permanently.

Step 2: Choose a Nostr Client #

Clients are the gateways through which you interact with the Nostr protocol. Popular options include:

Damus (iOS/Android): User-friendly and feature-rich.
Amethyst (Android): Tailored for Android users with a sleek interface.
Web Clients: Platforms like Nostr.guru or Iris allow you to access Nostr via a browser.

Install your chosen client and log in using your keys.

Step 3: Connect to Relays #

Relays are the backbone of the Nostr network, enabling message distribution. Most clients come pre-configured with default relays, but you can add more for better connectivity and reliability. Simply search for “Nostr relay services” and paste the relay URLs into your client settings.

Step 4: Explore and Interact #

Once set up, you can start exploring the network:

Follow Users: Add public keys of other users to your feed.
Post Notes: Share your thoughts or updates by creating a note, similar to a tweet.
Engage: Interact with others by replying to or liking their notes.

Step 5: Customize Your Experience #

Choose Relays Carefully: Some relays may specialize in certain topics or enforce content moderation policies. Tailor your relay list based on your interests and values.
Backup Your Keys: Use secure storage solutions like hardware wallets or encrypted cloud storage to protect your private key.
Experiment with Clients: Try multiple clients to find the one that suits your preferences.

The Broader Implications of Nostr #

The emergence of platforms like Nostr signals a broader shift toward decentralization in the digital era. By prioritizing user autonomy and privacy, Nostr challenges the centralized paradigms that have dominated the social media landscape for years. Its open architecture invites developers, activists, and everyday users to co-create a more inclusive and censorship-resistant digital space.

Moreover, Nostr’s impact extends beyond social media. Its decentralized protocol can be adapted for various purposes, such as secure communication, decentralized marketplaces, or even collaborative work environments. This versatility makes it a foundational building block for the next generation of internet technologies.

Conclusion #

Nostr represents a profound evolution in social media, offering a decentralized, user-centric alternative to traditional platforms. By empowering individuals with control over their identity, data, and content, it paves the way for a freer and more resilient online ecosystem. While its decentralized nature may pose a learning curve for beginners, the benefits far outweigh the challenges. By embracing Nostr, we can collectively take a step toward a more open and equitable digital future.

For those ready to explore this transformative platform, the journey begins with a simple act: generating your keys and connecting to the network. From there, the power of decentralized communication lies in your hands.

Deployments are not StatefulSets

2025-01-15T18:07:57Z

I was debugging some ArgoCD stuff earlier today and I love using k9s to explore my k8s resources.

The TLDR is that I put some bad env vars in the global values for my ArgoCD pods, this caused the webUI to hang. An easy way to push things through is to open k9s, view the Deployments, and edit the env vars there so the pods can be in the right state to restart and pickup the changes (you’ve obviously fixed the values file already right?)

Well the Notification Controller was the pod hanging, and I couldn’t find it in the Deployments view! This blew my mind… until I realized that it was a Stateful Set.

So, this is not a comparison of the 2 - this is just a note to say that all I had to do was find the StatefulSets in k9s and do the same workflow for editing what I needed to edit to get the pods into workable state while working on the values file…

docker context (and an issue to question your sanity)

2024-12-19T05:59:30Z

docker contexts are great, would recommend putting them in your prompt though (via starship or something else)… here’s why

I like to manage my containers remotely - I have a nice development setup on my desktop and I try to keep my server as bare-bones as possible. For a while I’ve been using ansible which makes it easy to manage configuration etc on other machines. But I recently learned about docker contexts and I’m planning to scale down my homelab management to just docker-compose stacks rather than a bunch of super complicated ansible playbooks

So, setting up a context is easy - it’s basically an ssh connection to another machine!

docker context create koober --docker "host=ssh://nic@koober"

koober is one of my dev machines and my ~/.ssh/config is setup such that I can ssh nic@koober, this makes the context work really seamlessly.

So there’s the default context (the machine you’re on) and now I have koober

To use it you run docker context use koober

And then to check we can ls the contexts

docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT               ERROR
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   
koober *                                             ssh://nic@koober

Notice the * - that indicates that’s our current context.

Trouble #

Now here’s where things get hairy… you’ve gotta be super-aware of what context you’re using. I have an indicator in my starship prompt that shows the current context, but since I’m new to using them I kind of didn’t notice it until I ran into this issue…

I’m working on a python application in docker but was not able to execute the entrypoint even though I KNEW the file was there… let’s take a look

Example #

Here’s a minimal hello world applycation in docker to illustrate the issue

# main.py
print("hello world")

# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Run the main.py script as the container's main process
CMD ["python", "main.py"]

services:
  hello-world:
    build: .
    volumes:
      - .:/app

Notice volume mounting in my project directory . to /app as a common practice to develop inside the container

Here’s where I started to question my sanity…

✗ docker compose up                                            
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | python: can't open file '/app/main.py': [Errno 2] No such file or directory
hello-world-1 exited with code 2

python can't open file? hmm… Let’s take a look at the image

First let’s make sure we get the image name right

✗ docker container ls -a              
CONTAINER ID   IMAGE                                           COMMAND                  CREATED          STATUS                       PORTS                                                                                  NAMES
3d7285fc39e2   docker-context-example-hello-world              "python main.py"         10 minutes ago   Exited (2) 44 seconds ago                                                                                           docker-context-example-hello-world-1

Now we can docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world

❯ docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world  
root@ee46d0e22de8:/app# python main.py
Hello, World!
root@ee46d0e22de8:/app#

WHAT THE HECK??

What happened… #

What happened turns out to be pretty simple once we realize I’m using contexts…

koboer is a remote context, the docker run and docker compose up commands are interacting with the docker socket on that machine.

So if I compose up the stack notice that there’s a volume bind mount in there - well those do not work with contexts (or at least I’m not aware of hose to make it work) and so the /app directory was getting blown away essentially with an empty overlay…

But when running with just docker run with no volume mount, the code was copied in during the build and is right where we expect it…

Let’s prove it #

❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@903f591c0384:/app# python main.py
python: can't open file '/app/main.py': [Errno 2] No such file or directory
root@903f591c0384:/app#

Adding a `-v .:/app` to match the compose file, we get the same error...

If we switch to the default context we are back up and running as expected

✗ docker context use default
default
Current context is now "default"

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker compose up         
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | Hello, World!
hello-world-1 exited with code 0

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@4045b6aa8883:/app# python main.py
Hello, World!
root@4045b6aa8883:/app#

Successful runs on both accounts with the volume mount

TLDR #

Context is king

Add a healthcheck to your FastAPI app

2024-12-15T15:41:36Z

I’m building a few FastAPI apps to throw in docker and run on my homelab… I wanted to add healthchecks and here’s a simple way to do it

Make sure to install curl in the dockerfile (near the top for effeciency)

# Install curl with minimal dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Then I recommend making compose files even for single image deployments

services:
  app:
    build: .
    volumes:
      - type: bind
        source: .
        target: /app
    environment:
      - PYTHONPATH=/app
      - DOCKER_ENV=true
      - UV_VIRTUALENV=/opt/app-env
    user: "1000:1000"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

Then finally you’ll need a /health endpoint


@app.get("/health", response_class=HTMLResponse)
async def health_check():
    """
    A health check endpoint that returns a status message.
    """
    return "<html><body><h1>Service is healthy</h1></body></html>"

statically.io to help me out

2024-12-14T13:35:31Z

statically.io offers a CDN that I’m going to try to lean into for images in my blog. My hope is that the builds get and stay fast, and that page loads are fast too.

Example:


<img src="https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/skimpy-zsh.png" alt="Skimpy ZSH" title="A fancy data summary in the shell" />

D and uninterruptable sleep

2024-12-11T10:48:10Z

Htop #

I recently have been having significant home server issues, and that’s not the point of this - today I learned what D state is when looking at htop.

Apparently this means “uninterruptable sleep” and it’s a dev’s nightmare…

Context #

The issue I was having was that some zfs rollback commands were hung - for hours… I wasn’t sure what was going on, rollbacks should be instant but I figured it was just an artifact of these issues.

Turns out I still don’t know what locked the disks up but I learned why <C>-c did nothing… the more you know

Stylus for custom webpage themes

2024-11-27T06:07:39Z

the Logos web app is DISGUSTINGY bright/white - enough to actually ruin your morning…

Thankfully there’s an extension called stylus and some kind folks in the Logos community created a nice dark theme here. You simply install the extension, then go to the userstyles link <– and click install. It only changes themes for the sites configured - in this case app.logos.com

TODO: image

DNS Broke After Reboot - Ubuntu 22.04

2024-11-22T08:08:40Z

I rebooted by server and DNS broke randomly. I have no idea if it was from a kernel update or what but that’s the issue with Ubuntu I guess…

After much toil and none of the other options working for me (sorry to not have those documented here) this is what got me the vic from this SO Post

sudo mkdir /etc/systemd/resolved.conf.d/ sudo $EDITOR /etc/systemd/resolved.conf.d/dns_servers.conf

Most folks probably are good with google (8.8.8.8) and cloudflare (1.1.1.1)

[Resolve]
DNS=8.8.8.8 1.1.1.1

But I decided to use tailscale

[Resolve]
DNS=100.100.100.100

Then restart systemd-resolved

sudo systemctl restart systemd-resolved

OPNSense Bootstrap Recovery

2024-11-07T08:40:19Z

enabling DHCP WAN port (dhclient <iface>)- running the bootstrap script - sh /usr/local/sbin/opnsense-bootstrap

docker-remote-add

2024-09-17T06:19:00Z

Add from url??

ADD http://example.com/cars.csv /tmp/cars.csv

Unpack automatically!? (.tar, .tar.gz, .tgz, .bz2, .tbz2, .txz, .zip)

ADD myapp.tar.gz /opt/myapp/

Docker copy and chown

2024-09-17T06:18:09Z

COPY –chown=myuser:mygroup source-file target-file

Switching from AltaCV to RenderCV for my Resume

2024-08-01T05:59:46Z

I was using a fun LaTex-based project for managing my resume called AltaCV. I loved the customization and was familiar with Tek from school. However, I update my resume so infrequently that anytime I’d hop back to it I’d have to remember how to work with Tex and that was frustrating as I’ve lost touch with it over the years.

Scrolling GitHub treding repos I saw RenderCV which let’s me just use YAML to write my resume and then it compiles to Tek through Python. There’s a sister project to make your own using this very easly call rendercv-pipeline. I forked that repo and translated my tek resume to the YAML. The included theme is nice enough is YAML is much easier to maintain long-term.

My resume is behind a private GH repo but the example from rendercv-pipeline is here on GitHub

Interesting IPs between Jellyfin clients and server depending on tailscale and server address

2024-07-12T06:24:57Z

When connecting from my phone to jellyfin I’m seeing some interesting patterns.

Scenarios #

Tailscale IP of phone is listed as local network to jellyfin #

Wifi: off Tailscale: on Use exit node: on LAN access: on Jellyfin: LAN IP

Jellyfin sees 192.168.1.1, my router address

Wifi: off tailscale: on Use exit node: on LAN access: on Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Q: This might be because of traefik somehow

Wifi: off tailsacale: on Use exit node: on LAN access: off Jellyfin: LAN IP

Jellyfin sees the 192.168.1.1

Q: Why did this work even work?

Wifi: off tailsacale: on Use exit node: on LAN access: off Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Wifi: off tailsacale: on Use exit node: off LAN access: off Jellyfin: LAN IP

Jellyfin sees the 192.168.1.1

Q: Why did this work?

Wifi: off tailsacale: on Use exit node: off LAN access: off Jellyfin: Tailscale magic DNS

Jellyfin sees the docker bridge network

Wifi: on tailsacale: of Use exit node: off LAN access: off Jellyfin: LAN IP (via pihole DNS)

Jellyfin sees the IP of my phone

Upgrading your kernel can F you up… whoops

2024-06-26T10:21:59Z

https://forum.proxmox.com/threads/update-error-with-coral-tpu-drivers.136888/#post-608975 I lost TPU for frigate, this saved me ass

Customize K9s

2024-05-06T11:57:07Z

To customize k9s use the skins from catppuccin or the ones k9s supplies

OUT="${XDG_CONFIG_HOME:-$HOME/.config}/k9s/skins"
mkdir -p "$OUT"
curl -L https://github.com/catppuccin/k9s/archive/main.tar.gz | tar xz -C "$OUT" --strip-components=2 k9s-main/dist

Then edit your k9s config

# ~/.config/k9s/config.yml
k9s:
  ui:
    skin: catppuccin-mocha
    # ...or another flavor:
    # skin: catppuccin-macchiato
    # skin: catppuccin-frappe
    # skin: catppuccin-latte

    # ...or the transparent variants:
    # skin: catppuccin-mocha-transparent
    # skin: catppuccin-macchiato-transparent
    # skin: catppuccin-frappe-transparent
    # skin: catppuccin-latte-transparent

Other k9s skins are available here

Kanboard to keep me focused on my own ideas

2024-04-20T18:27:07Z

TL;DR #

I’ve been using kanboard as a self-hosted kanban board. It’s keeping me focused on Digital Harbor when I’d rather be doing something less productive.

My TODOs #

Here’s the thing about my TODOs… they’re everywhere. I’ve tried a crazy amount of different organizational tactics for my todo items. I’ve tried post-it notes, a journal, TODO.md in specific repos, a master todo repo, todo CLIs, etc…

The problem for me was consistency… I would regularly forget what I was using for TODOs at that moment in my life… was there a todo file in this project? Do I have a todo repo cloned on this computer? Is this stuff in my journal or on post-it notes?

And what do I do with new ideas? Do I organize them centrally or use a repo for the idea I have?

It was getting out of hand to a debilitating degree… For a while I just gave up on being organized at all… Things would get done as necessary and if I got some motivation to work on something it was immediately smothered with the anxiety of how I was going to organize my work…

At some point it became too much… Now, I have some experience with Azure DevOps/Jira for project management and then I came across Kanboard…

Kanboard #

Kanboard

Kanboard is just a self-hosted kanban style todo app. I know there’s a ton of these so the TL;DR of my lesson is I picked an app that I would just use based on simplicity of managing and hosting.

I have had kanboard running in my homelab for a long time, but I only barely use it intermittenly. And at that rate I didn’t spend any real time organizing my tickets, so I wasn’t akshuallly using it - it was just a post-it note replacement.

But then I had an idea for a genuine business idea, and if I was going to ever have a hope of making it a reality, I needed to stay organized. This was when I decided to give kanboard a little more effort… I knew I could always remember that I’d chosen an app as my TODO solution (given all the time I spent questioning what I was using at any given point in time). I also knew I could host kanboard so that I could get to it from wherever was necessary because my homelab is relatively easy to add another public service to.

So once I just decided to lean into this thing, I would take advantage of any moments of motivation and just jot down ideas for things that had to get done… Simple stuff like explore an infra management option, add one feature to a config, or migrate one website to another stack… And I would just write these things downs until I have enough time free to crack down on a task. The beautiful thing is, when I am struck with just enough time to do something, and the motivation to do something, I don’t waste any time deciding what to do - past-me did present-me a favor and decided what was important already… So as long as I do myself the favor, when I’m ready to go I am never beaten by the anxiety of not knowing what I ant to do or how I’ll track what I’m doing… I chose kanboard and even though it’s not as fast as a terminal TUI, it is reliable, simple, and keeps me focused on what I need to do.

The biggest benefit #

Not only is just having one app as the solution nice because it’s centrally managed and accessible from wherever I need to get it, but the TOP feature of kanboard I use is comments on tickets… I get to continue to do future-me favors, my jotting down where I’m at in a task, what’s left to do, what I’m trying, etc… and future-me is in a great mood, because when I have that free few minutes, I can just read my own past thoughts and get back up to speed without wasting time trying to remember things I never would’ve remembered if I hadn’t written them down!

DHCP Restart to Save Ubuntu 22.04 Server Networking

2023-12-31T20:26:50Z

I moved a computer to a remote location for an off-site backup but when it was powered on it wouldn’t show up on any networks. A solution that got me back in was a friend restarting the dhcp client for me:

sudo dhclient -r -v <interface> && sudo dhclient -v <interface>

Simple Port Forwarding OPNSense

2023-10-17T10:26:34Z

https://forum.opnsense.org/index.php?topic=8783.0

Refresh Nextcloud Groupfolders after messing around on the filesystem

2023-09-23T12:45:06Z

Exec in as www-data and run ./occ groupfolders:scan folder_id -v (the -v to see what it’s doing)

lsof to find what’s using your filesystem

2023-04-09T13:32:38Z

lsof | grep /tank/nas shows me what is using my nas at any time!

Changing ZFS key for child datasets of encrypted dataset after migration

2023-04-08T20:12:22Z

➜ pihole sudo zfs load-key -L file:///path/to/.zfs.tank.key tank/encrypted/vms/arch-sandbox ➜ pihole sudo zfs change-key -o keylocation=file:///path/to/.zfs.tank.key -o keyformat=raw tank/encrypted/vms/arch-sandbox Need to load-key for each individual dataset, then change key location to be a file instead of the prompt

Convert Word Doc to PDF with Headless Libreoffice

2023-03-09T06:48:38Z

I’ve been using paperless-ngx to manage all my documents, but every once in a while I’ll get a .docx file to deal with…

Turns out Libreoffice has a headless mode a pdf converter built-in!

libreoffice --headless --convert-to pdf /path/to/file.docx --outdir /path/to/output/directory

Note that --outdir is in fact a directory, not the path to a file

Stable Diffusion Notes

2023-01-28T14:15:11Z

ChatGPT Prompt: #

Stable Diffusion is an AI art generation model similar to DALLE-2. Here are some prompts for generating art with Stable Diffusion.

Example:

A ghostly apparition drifting through a haunted mansion’s grand ballroom, illuminated by flickering candlelight. Eerie, ethereal, moody lighting.
portait of a homer simpson archer shooting arrow at forest monster, front game card, drark, marvel comics, dark, smooth
pirate, deep focus, fantasy, matte, sharp focus
red dead redemption 2, cinematic view, epic sky, detailed, low angle, high detail, warm lighting, volumetric, godrays, vivid, beautiful
a fantasy style portrait painting of rachel lane / alison brie hybrid in the style of francois boucher oil painting, rpg portrait
athena, greek goddess, claudia black, bronze greek armor, owl crown, d & d, fantasy, portrait, headshot, sharp focus
closeup portrait shot of a large strong female biomechanic woman in a scenic scifi environment, elegant, smooth, sharp focus, warframe
ultra realistic illustration of steve urkle as the hulk, elegant, smooth, sharp focus
portrait of beautiful happy young ana de armas, ethereal, realistic anime, clean lines, sharp lines, crisp lines, vibrant color scheme
A highly detailed and hyper realistic portrait of a gorgeous young ana de armas, lisa frank, butterflies, floral, sharp focus
lots of delicious tropical fruits with drops of moisture on table, floating colorful water, mysterious expression, in a modern and abstract setting, with bold and colorful abstract art, blurred background, bright lighting
1girl, The most beautiful form of chaos, Fauvist design, Flowing colors, Vivid colors, dynamic angle, fantasy world
solo, sitting, close-up, girl in the hourglass, Sand is spilling out of the broken hourglass, flowing sand, huge hourglass art, hologram, particles, nebula, magic circle
geometric abstract background, 1girl, depth of field, zentangle, mandala, tangle, entangle, beautiful and aesthetic, dynamic angle, glowing skin, floating colorful sparkles the most beautiful form of chaos, elegant, a brutalist designed, vivid colours, romanticism

The prompt should adhere to and include all of the following rules:

Prompt should always be written in English, regardless of the input language. Please provide the prompts in English.
Each prompt should consist of a description of the scene followed by modifiers divided by commas.
When generating descriptions, focus on portraying the visual elements rather than delving into abstract psychological and emotional aspects. Provide clear and concise details that vividly depict the scene and its composition, capturing the tangible elements that make up the setting.
The modifiers should alter the mood, style, lighting, and other aspects of the scene.
Multiple modifiers can be used to provide more specific details.

I want you to write me a list of detailed prompts exactly about the IDEA follow the rule at least 6 every time.

Ready for my idea?

ChatGPT Prompt for RPG V4: #

Stable Diffusion is an AI art generation model similar to DALLE-2. Here are some pairs of positive and negative prompts for generating art with Stable Diffusion.

Example:

Positive: A (full body:1.3) shot at 8k resolution, splash art, fantastic comic book style, photorealistic, intense look, anatomical photorealistic digital painting portrait of a (old male:1.3) human (warrior:1.3) in black and gold intricate (heavy armor:1.3) in a (dark and moody universe:1.3), light particle, very detailed skin,samurai, very detailed eyes, (elden ring style:1.3), (warhammer style:1.1), concept artist, global illumination, depth of field, splash art, art by artgerm and greg rutkowski and viktoria gavrilenko Negative: (symmetry:1.2), facial marking, crown, horn, (helmet:1.3), (hoodie:1.1), clock, Female, visible hand, asian, two face, big hair, open mouth, cartoon, high contrast, poorly drawn, Scribbles, Low quality, Low rated, Mediocre, Screenshot, Software, UI, watermark, text, overlay, getty images, cropped, low quality

Positive: close-up head, facing camera, beautiful satanic female necromancer blood queen, ritualistic (neck tattoo:1.3), (insanely detailed:1.5), ((solo)), (highest quality, Alessandro Casagrande, Greg Rutkowski, Sally Mann, concept art, 4k), (colourful), (high sharpness), ((detailed pupils)), ((painting:1.1)), (digital painting:1.1), detailed face and eyes,Masterpiece, best quality, highly detailed photo:1, 8k, detailed face,photorealistic, (black long Hair:1.1),(young woman),By jeremy mann, by sandra chevrier, by maciej kuciara, smoke and shadow, night, sharp, ((perfect body)), realistic, real shadow, 3d, ((full body)), ((dark and gloomy universe)), (by Michelangelo) Negative: crown, (facial marking:1.2), cloth gem, jewel, jewelry, (flower:1.3), (cloak:1.1), (bad art, low detail, pencil drawing, old, mature:1.6), (grainy, low quality, mutated hands and fingers:1.5), (watermark, thin lines:1.3), (deformed, signature:1.2), (big nipples, blurry, ugly, bad anatomy, extra limbs, undersaturated, low resolution), disfigured, deformations, out of frame, amputee, bad proportions, extra limb, missing limbs, distortion, floating limbs, out of frame, poorly drawn face, poorly drawn hands, text, malformed, error, missing fingers, cropped, jpeg artifacts, teeth, unsharp

Positive: (Painting:1.3) of (Detailed illustration:1.3) A (full body:1.3) shot at 8k resolution, splash art, fantastic comic book style, photorealistic, intense look, anatomical photorealistic digital painting portrait of a (old male:1.3) human (warrior:1.3) in black and gold intricate (heavy armor:1.3) in a (dark and moody universe:1.3), light particle, very detailed skin,samurai, very detailed eyes, (elden ring style:1.3), (warhammer style:1.1), concept artist, global illumination, depth of field, splash art, art by artgerm and greg rutkowski and viktoria gavrilenko Negative: (symmetry:1.2), facial marking, crown, (horn:1.1), (helmet:1.3), (hoodie:1.1), clock, Female, visible hand, asian, two face, big hair, open mouth, cartoon, high contrast, poorly drawn, Scribbles, Low quality, Low rated, Mediocre, Screenshot, Software, UI, watermark, text, overlay, getty images, cropped, low qualityLow quality,Bad composition,Faded,(Photo:1.5),(Frame:1.3),watermark,signa ture

Positive: sci-fi, space rogue thief girl, (smirk:1.1), short hair, black hood, light armor, big grey eyes, beautiful detailed eyes, drawn by Greg Rutkowski, Yoji Shinkawa:0.6 Negative: open mouth, facial marking, flower, bad-hands-5, lipstick, text, watermark

The prompt should adhere to and include all of the following rules:

Each prompt should consist of a description of the scene followed by modifiers divided by commas.
When generating descriptions, focus on portraying the visual elements rather than delving into abstract psychological and emotional aspects. Provide clear and concise details that vividly depict the scene and its composition, capturing the tangible elements that make up the setting.
The modifiers should alter the mood, style, lighting, and other aspects of the scene.
Multiple modifiers can be used to provide more specific details.

I want you to write me a list of detailed prompts exactly about the IDEA follow the rule at least 6 every time.

Ready for my idea?

IDEAS #

IDEA: A slice of pie that is filled with a futuristic city

Artists I like #

Examples

J. M. W. Turner
Jean-Baptiste Monge
John Lasseter
John Martin
Raymond Swanland

Anime styles #

Krenz Cushart
Kyoto Animation
Yoshiyuki Sadamoto
Yuumei

Scenic #

Ted Nasmith
Terry Redlin
Thomas Kinkade
Thomas Moran
Thomas W Schaller

Cityscapes #

Marc Simonetti
Wadim Kashin

Abstract #

Scott Naismith

Settings notes #

I tend to like cfg at least on the higher end

Good prompts #

Angel of death, shadows and mist, skelletol feathered wings, highly detailed, amazing details, intriciate lines, purple haze, styles Yuumei and Krenz Cushart, UHD, HDR, upscale, 8K photo, concept art

Unicorns #

positive: unicorn, lora:Unicorns:0.9, portrait, professional photography, 35mm, highly detailed, incredibly realistic, best quality, high quality, highres, purple, lora:mossbeast:0.3

negative: (two tails:1.2),FastNegativeV2,(bad-artist:1.0), (loli:1.2), (worst quality, low quality:1.4), (bad_prompt_version2:0.8), bad-hands-5,lowres, bad anatomy, bad hands, ((text)), (watermark), error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, ((username)), blurry, (extra limbs), bad-artist-anime, badhandv4, EasyNegative, ng_deepnegative_v1_75t, verybadimagenegative_v1.3, BadDream,(three hands:1.1),(three legs:1.1),(more than two hands:1.4),(more than two legs,:1.2),

FFMPEG 10-bit videos to 8-bit

2023-01-16T13:15:53Z

ffmpeg -i input.mp4 -map 0 -c:v libx264 -vf format=yuv420p -c:a copy output.mp4

i3-Like keyboard mapping in Pop_OS

2023-01-12T05:51:25Z

I was introduced to tiling window managers through i3, which I use heavily on one of my machines. I have switched to Pop_OS! at home though, which has a tiling window mode but the keybindings are not what I’m used to for i3. I wanted to at least navigate workspaces how I’m used to doing (cause I set workspace 3 for communication apps, 1 for my terminal, etc…)

Here’s how I set keybindings for:

<Super> + <number> sends me to that numbered workspace
<Shift> + <Super> + <number> moves the window I’m focused on to workspace number

#!/bin/bash
gsettings set org.gnome.mutter dynamic-workspaces false 
gsettings set org.gnome.desktop.wm.preferences num-workspaces 8 
gsettings set org.gnome.shell.keybindings switch-to-application-1 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-2 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-3 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-4 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-5 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-6 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-7 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-8 [] 
gsettings set org.gnome.shell.keybindings switch-to-application-9 [] 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-1 "['<Super>1']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-2 "['<Super>2']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-3 "['<Super>3']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-4 "['<Super>4']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-5 "['<Super>5']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-6 "['<Super>6']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-7 "['<Super>7']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-8 "['<Super>8']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-9 "['<Super>9']" 
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-10 "['<Super>0']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-1 "['<Super><Shift>1']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-2 "['<Super><Shift>2']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-3 "['<Super><Shift>3']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-4 "['<Super><Shift>4']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-5 "['<Super><Shift>5']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-6 "['<Super><Shift>6']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-7 "['<Super><Shift>7']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-8 "['<Super><Shift>8']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-9 "['<Super><Shift>9']" 
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-10 "['<Super><Shift>0']"

Use non-standard named ssh keys with github

2023-01-03T08:34:50Z

I was getting (publickey denied) when trying to push to GH using ssh. When I tested the connection I saw that a bunch of keys in ``~/.ssh/ were being attempted

✗ ssh git@github.com -vv

...

debug1: Will attempt key: /home/nic/.ssh/id_rsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa 
debug1: Will attempt key: /home/nic/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519 
debug1: Will attempt key: /home/nic/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/nic/.ssh/id_xmss 
debug1: Will attempt key: /home/nic/.ssh/id_dsa 

...

debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

None of those were the key I setup with GH. So I added an entry into ~/.ssh/config:

Host
github.com
User git
Port 22
Hostname github.com
IdentityFile ~/.ssh/my_custom_github_key
TCPKeepAlive yes
IdentitiesOnly yes

Modal Labs

2022-12-28T21:01:52Z

Playing around with Modal Labs

One of the first things I tried was a regular cron job…

@stub.function(
    schedule=modal.Period(minutes=59), secret=modal.Secret.from_name("my-dummy-secret")
)
def say_hi():
    now = time.ctime()
    secret = os.environ.get("dummy-secret")
    print(f"Hello {os.environ.get('USER', 'Rodney')} at {now}")
    print(f"{secret=}")

This can get deployed with modal deploy --name <app name> <path to .py file with the stub and function defined in it>

This function gets deployed as an app that I conveniently call say_hi (as far as I can tell the app name can be anything - as I add functions to this same app and deploy with the same name to get a new version)

Notice that this also is an example of giving access to a secret - defined in the Modal Labs dashboard

We can take a look at the apps running at https://modal.com/apps

I then added another function to experiment with custom container images and saw then that Modal will just slap a new version on anything provisioned with the same name (intuitive enough for sure) so when I add functions to my .py script and run modal deploy --name say_hi myscript.py over and over, the app called say_hi in the Modal apps dashboard just gets a new version

This means I can spin up several instances of functionally the same app but with different names/versions etc… Q: Maybe there’s gitops or policy stuff builtin to app names then?

I needed to take down an app I deployed as a duplicate but you don’t stop apps by name, you stop them by an id… see below


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
✗ modal app stop --help

 Usage: modal app stop [OPTIONS] APP_ID

 Stop an app.

╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    app_id      TEXT  [default: None] [required]                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ App ID                    ┃ Description         ┃ State    ┃ Creation time             ┃ Stop time                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ap-lzy1AAuVy7POFkUcDKRxpQ │ print_info          │ deployed │ 2022-12-28 20:59:07-06:00 │                           │
│ ap-qYjE45dciqgT3C3CpNp3RL │ say_hi              │ deployed │ 2022-12-28 19:49:22-06:00 │                           │
│ ap-X7FYneUeYV5IKHcyirSb87 │ link-scraper        │ stopped  │ 2022-12-28 15:39:02-06:00 │ 2022-12-28 15:39:04-06:00 │
│ ap-UOXTUU4uSRx2UZypJOcAsk │ example-get-started │ stopped  │ 2022-12-28 15:17:47-06:00 │ 2022-12-28 15:17:49-06:00 │
└───────────────────────────┴─────────────────────┴──────────┴───────────────────────────┴───────────────────────────┘

modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app stop ap-lzy1AAuVy7POFkUcDKRxpQ

Git warning!

I ran modal deploy ... after comitting some stuff I wanted to try BUT I had changes in my file I didn’t want to deploy… some git safety would be nice for deployment!

git stash && modal deploy .. && git stash pop

Question for Modal team - in my modal sandbox repo at commit:

aab6162 (HEAD -> main) HEAD@{1}: commit: print base version of my own image to prove it to me
 1 file changed, 2 insertions(+)

An environment variable, BASE_VERSION that I expect to be in my base image was not available to the python function in my Modal app… hopefully the log is still here

GitHub Actions #

I think it’d be easy to deploy from GH actions… just need an API Key…

Deployment

Without gitops it’s already ahrd to manage my versions… need to get that built in either to github or locally sooooon

Nomad?? :D

Reminder about ssh-copy-id for SSH and Ansible

2022-12-28T13:33:07Z

ssh-copy-id -i my.key.pub <hostname probably from tailscale> this makes sure I can run ansible from my desktop against VMs on my server easily if they have tailscale for the hostname - otherwise use the IP

Nextcloud Docker Upgrade Error

2022-12-28T09:39:27Z

https://nicolasbouliane.com/blog/nextcloud-docker-upgrade-error

Systemd timer for syncoid

2022-12-21T11:38:27Z

I have a bash script called syncoid-job which boils down to a barebones -

#!/bin/bash

syncoid --no-sync-snap --sendoptions=w --no-privilege-elevation $SYNOIC_USER@$SERVER:tank/encrypted/nas tank/encrypted/nas

I want to run this script hourly but as my user (notice the no-privilege-elevation flag)

First - create a systemd unit file at /etc/systemd/system/syncoid-replication.service

[Unit]
Description=ZFS Replication With Syncoid

[Service]
Type=oneshot
ExecStart=/$HOME/dotfiles/syncoid-job
User=$USER
Group=$GROUP

[Install]
WantedBy=multi-user.target

Then we save the unit file, enable the service, and then start it

systemctl enable syncoid-replication.service
systemctl start syncoid-replication.service

Note this will run that script… so be ready for syncoid to do its thing

Now for the timer… We create /etc/systemd/system/syncoid-replication.timer

[Unit]
Description=Run syncoid-replication every hour

[Timer]
OnCalendar=hourly

[Install]
WantedBy=timers.target

Hit it with a systemctl enable syncoid-replication.timer and you’re in business!

Adding docker daemon.json broke docker

2022-12-21T10:02:25Z

in /lib/systemd/system/docker.service there is an ExecStart command that got placed there when I setup Docker with Ansible - it threw the -H flag which told the daemon what hosts to setup. But I added the “hosts” key in my daemon.json and it broke - so removing the -H flag from the systemd unit fixed it

suda.vim for sudo access to files

2022-12-21T09:45:34Z

I regularly need to edit system config files - take /etc/sanoid/sanoid.conf as an example… I’ll want to play with something but if I don’t start Neovim as root then I get in trouble making edits I can’t save! So suda.vim gives me :SudaWrite which let’s me write that buffer with sudo privileges even though I’m Neovim is running with my login user!

Cron for Nextcloud in Docker

2022-12-13T06:43:45Z

AJAX wasn’t cutting it, traditional crontab in containers doesn’t make much sense to me, webcron is recommended but I don’t want to register with anything outside my LAN… Turns out you can just spin up an identical container with a different entrypoint to /cron.sh that does what you need!

Note that this is a task in an Ansible playbook - but the docker-compose is straight forward

So the only thing you need to make sure of is that all the configuration options - data volumes, user permissions, etc. are identical between the containers running the cron job and the one actually hosting NextCloud. This ensures that the container running cron has proper access to the database and filesystem - or at least the same access as NextCloud proper.

- name: Nextcloud Cron Docker Container
  docker_container:
    name: nextcloud-cron
    image: "{{ nextcloud_image }}"
    pull: true
    links:
      - nextcloud-mysql:mysql
    entrypoint: /cron.sh
    volumes:
      - "{{ nextcloud_data_directory }}/nextcloud:/var/www/html:rw"
    env:
      MYSQL_HOST: "mysql"
      MYSQL_DATABASE: "nextcloud"
      MYSQL_USER: "{{ nextcloud_sql_user }}"
      MYSQL_PASSWORD: "{{ nextcloud_sql_password }}"
      NEXTCLOUD_TRUSTED_DOMAINS: "{{ nextcloud_hostname }}.{{ ansible_nas_domain }}"
      PUID: "{{ nextcloud_user_id }}"
      PGID: "{{ nextcloud_group_id }}"
      TZ: "{{ ansible_nas_timezone }}"
    restart_policy: unless-stopped
    memory: "{{ nextcloud_memory }}"

Call basicConfig to get Python log messages in iPython

2022-12-10T14:04:23Z

Logging instead of printing

I am trying to adopt logger.debug instead of print but ran into a confusing thing in ipython during Advent of Code… I riddled by script with logger.debug (yes after setting logging.setLevel('DEBUG')) but in ipython none of my log messages showed up!

import logging

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Turns out what I was missing was a call to basicConfig

import logging

# forget this and your messages are in the ether! or at least not seen in ipython...
logging.basicConfig()

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Bonus

Want your new messages to show up while iterating on something without killing the ipython kernel?

from importlib import reload
reload(logging) # to make sure you get new log messages you add while developing!

New lines in Markdown tables

2022-11-25T13:35:05Z

I wanted to break down some long lines in a Markdown table cell to make it look nicer on my blog but \n didn’t do anything for me… turns out
is the magic sauce

Column 1	Column 2
Key	Doggo ipsum many pats. Borkdrive borking doggo doing me a frighten doggorino, noodle horse heckin. what a nice floof. Pupper borking doggo you are doing me a frighten, much ruin diet.
—	—

Column 1	Column 2
Key	Doggo ipsum many pats. Borkdrive borking doggo doing me a frighten doggorino, noodle horse heckin. what a nice floof. Pupper borking doggo you are doing me a frighten, much ruin diet.
—	—

Description of my proposed vimconf 2022 talk

2022-11-12T19:39:19Z

Switching to Vim opened a whole new world to me for interacting with a computer and for getting things done. Before I adopted Vim I used GUIs for everything because I thought that’s how it had to be done… Notes in OneNote, code using a GUI editor, different notes in TiddlyWiki, slides for work in PowerPoint, slides for church using Logos, etc… Adopting Vim allowed me to disconnect a specific tool from the problem that tool is solving - because usually I just need to write text (notes, code, slides, etc.). Now, very nearly everything I do is from a text-based and git-based workflow… I put all my notes on basically anything just in my blog, which is all markdown and deployed to GH with Markata on every push (living dangerously pushing to main) - and that’s all done easily from Vim with nice syntax highlighting, fast response, integrated git-plugins, etc.. I keep project-specific task lists just in markdown files and I have Vim/tmux shortcuts to quickly add todos for any project (todo list is done with markata todoui) and I can get there fast because my Vim workflow dovetails with Tmux nicely. Also I can pull that list up right from the terminal, which I’m already in because Vim…. Vim also pushed me into the cli more - because Vim is so easily extended with cli tools and I’m already in the terminal… The builtin functionality also made things make more sense - no more right-click, find “refactor all” or “rename symbol” (for some stupid reason)… Vim find-replace is so intuitive and if I need it extended then I learned what sed was because of Vim. Moving quickly in Vim also enables me to do my job incredibly fast because I hop into several projects a day in a coaching role - if I was bound by GUIs I’d be waiting forever for startup, would lose which GUI instance was which project, etc… Being in the terminal also made Tmux a trivial choice - now I have 90 tmux sessions, all named appropriately, ready for me to jump back to and all while keeping the majority of RAM still free for Chrome. Vim as my IDE also forced me to learn way more about Python (I’m a python developer primarily), how LSP works, how to configure a development environment, etc… things I took for granted in my GUI workflows, or never knew, or worse - thought I knew but deeply misunderstood. Now that I understand them better, I can coach my peers more effectively even if they are still in a GUI-based ecosystem.

Basically, (Neo)Vim actually did change my life and I’m really thankful for it (maybe that should be the title?)

Make a series of directories fast!

2022-11-10T15:27:50Z

mkdir s{1..10} will make directories s1, s2, … s10 in one command!

Case-insensitive search in Vim

2022-10-21T06:40:21Z

/mysearch\c will match mysearch, MYSEARCH, mYSeArCh…

Limit zfs list to avoid docker vomit

2022-10-20T06:39:18Z

zfs list has a flag -r, but if you use zfs driver for docker then you’ll get flooded with every docker volume in the world. zfs list -r -d N will limit the dept of the print out, so zfs list -r -d 2 gives me tank, tank/encrypted, tank/encrypted/docker -> but then I don’t see all the continer volumes

You can embed gifs?!

2022-10-09T13:39:39Z

I learned you can embed gifs and stuch in things other than GH personal pages!

So here I’ll keep ones I like and might use later

via GIPHY

Don’t forget to load XMP!

2022-10-09T07:17:43Z

Bought some DDR4-3600 speed RAM but only seeing 2666? Load up the BIOS, find DRAM config or something similar, and make sure to load the XMP profile to get that advertised RAM speed!

My PassMark Scores

2022-10-08T18:43:29Z

Babyblue v2

Ryzen 5700x + 32 GB 3200 CL16 RAM

Venus

Ryzen 3200G + 16GB 3600 CL16 RAM

Hogwarts

Dual Xeon X5650 @2.67GHz | 90 GB 1883 DDR3 ECC RAM

Server 2 ... Change the name

Ryzen 5700G + 128GB 3200 CL18 RAM

Server 2 with XMP enabled and undervolted from 1.35 to 1.2

Ryzen 5700G + 128GB 3200 CL18 RAM

Undervolted to 1.2V (from 1.35) XMP enabled

Suddenly SSH requires a password

2022-10-07T11:14:37Z

ssh -v -i ~/.ssh/id_rsa nic@hogwarts

THen we can look at print outs

cat /var/log/auth.log also showed me that I had too wide permissions on files in ~/.ssh -> probably changed from an rsync job

Quick setup of ZFS encrytped datasets with sane permissions

2022-10-06T19:58:01Z

Assuming you have a pool called tank…

And assuming you have an encrypted dataset (See Jim Saltar’s short intro)

Create a group for permissions - in my case I have one called home
Then if there’s anything in /tank/encrypted his it with chgrp -R home /tank/encrypted to give the home group ownership
Next we need to make sure that the members of home can do the writing… so chmod 775 -R /tank/encrypted will do the trick
Finally we want to make sure that all data created inside our dataset has the same set of permissions with chmod g+s /tank/encrypted and chmod g+w /tank/encrypted

Check Your BIOS Version On Ubuntu

2022-10-02T18:21:58Z

sudo dmidecode -s bios-version

Destroying Tmux sessions with fzf

2022-09-15T10:27:58Z

I use Tmux and Vim for most of my workflow, but I end up with a lot of dangling tmux sessions that dont’ really need to persist… but killing them one at a time is a pain so I wrote a little script-kitty nonsense to pipe multiple choices from fzf into tmux kill-session

I defined a little function in my .zshrc

destroy() { 
    tmux list-sessions -F '#{session_name}' | fzf -m | xargs -d $'\n' sh -c 'echo "killing $0"; tmux kill-session -t "$0"; for arg;do echo "killing $arg";tmux kill-session -t "$arg"; done'
}
bindkey -s '^d' 'destroy \n'

tmux list-sessions -F '#{session_name}' prints all my active tmux sessions to the console with the format of just their name

pype.dev   main   ×1 via   v3.8.11(pype.dev)  on  (us-east-1) proxy
❯ tmux list-sessions -F '#{session_name}'
session-01
session-02
session-03
...

Pipe that to fzf -m to allow multiple choices to be made using tab

Then the nasty bit in xargs… I echo killing @0 and killing $arg because the sh -c passes the first tmux session name to @0 (it’s just what bash does) and then the rest get handled in the for loop.

Basically then I get an fzf list to choose multiple tmux sessions to destroy to clean up some RAM!

Trick to login to web server on another box on lan

2022-09-12T18:48:45Z

TODO: change title - add –listen or –host or something as 0.0.0.0… that way it listens not on localhost or 127.0.0.1, but on all addresses!

Check your SMART status with smartctl

2022-08-29T06:30:27Z

https://www.simplified.guide/linux/disk-health-check

Install

For ubuntu/debian based distros (which is what I primarly use presently)

sudo apt update -y && sudo apt install smartmontools -y

List hard drives

lsblk | grep disk is one way or sudo lshw -c disk is another

smartctl

Use a device’s logical name such as dev/sda, not a partition of the disk

sudo smartctl -t short /dev/sda

dotfiles   home   ×3  ×2  ×2 via   v3.10.6(dotfiles)  took 11s
❯ sudo smartctl -t short /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.15.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Fri Sep 23 05:59:39 2022 CDT
Use smartctl -X to abort test.

check status

sudo smartctl -H /dev/sda


dotfiles   home   ×3  ×2  ×2 via   v3.10.6(dotfiles)
❯ sudo smartctl -H /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.15.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Benchmark your disks with fio

2022-08-27T13:43:22Z

Intro

I use ZFS at home in my homelab for basically all of my storage… Docker uses ZFS backend, all my VMs have their .qcow2 images in their own zfs datasets, and all my shares are ZFS datasets. I love ZFS but my home hardware presently is the opposite of expensive or new… Thankfully I’ve had a lot of my orginal homelab simply given to me but the cost of this is that I didn’t put my machines together, I didn’t choose the disks, and I definitely didn’t do the research I would’ve otherwise done had I bankrolled my server personally…

The Problem #

I run glances on basically all my machines and for the longest time I have been seeing big time iowait issues. Now, since everything was free I’ve largely been able to ignore that however I’m now after some better performance which I think means new hardware!

Here is a random screenshot of my glances homepage at time of writing - The only major load on my server is some ffmpeg transcoding (about 60% CPU utilization)…

As you can see… there’s a lot of issues and I don’t even know what they mean.

fio

I heard about fio through a friend and decided to try it out quick. It installs with apt on ubuntu quick and easy…

Jim Saltar has a good blog post on it here

Basically it’s a handy tool for benchmarking your disks and the blog dives into what types of metrics matter - it’s not just throughput, but also latency, iops, etc.

Tests #

I ran a few basic commands inside a new zfs dataset on my server tank/fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1 > single-4KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1 > 16-parallel-64KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1 > single-1MiB-random-write.txt

Results #

The single 4 KiB random write:

WRITE: bw=7836KiB/s (8024kB/s), 7836KiB/s-7836KiB/s (8024kB/s-8024kB/s), io=523MiB (548MB), run=68317-68317msec

The 16 parallel 64KiB random writes:

WRITE: bw=93.9MiB/s (98.4MB/s), 5599KiB/s-6303KiB/s (5734kB/s-6454kB/s), io=7642MiB (8013MB), run=81310-81418msec

The single 1MiB random write:

WRITE: bw=81.2MiB/s (85.1MB/s), 81.2MiB/s-81.2MiB/s (85.1MB/s-85.1MB/s), io=8177MiB (8574MB), run=100699-100699msec

Summary

So I don’t fully understand these numbers yet… 80-100 MiB/s isn’t super fast and that’s across a parallelized workload… The single threaded workloads have awful performance so this tells me something is wrong… I have a few ideas…

ZFS dataset config options such as ashift or the blocksize might be way misconfigured
The disks/pool which came from a TrueNAS/FreeBSD machine may have some artifacts that I need to clean up
The SAS controller I am using, which I flashed with IT firmware to get it into JBOD mode might be messed up
The data cables themselves could be a problem…

Points 3 and 4 are less likely given that the write speed does increase in the parallelized job but I’m a newbie so it’s time to dive in!

Playing with mdformat

2022-08-23T06:56:15Z

import this; print(this); print("what is taking so long black!!")

Paperless-NGX filtering on IDs instead of values

2022-08-09T08:28:51Z

To filter for saved views from the admin console you have to use the id of the tag you are using to filter on…

Ex: filtering for tags “Inbox” doesn’t work, but for tag 18 does (the id fo the Inbox tag). I can find that ID by clicking on the tag and looking at the url bar

Mounting exFAT USB in Linux

2022-07-31T07:38:43Z

Steps

sudo fdisk -l

then look for the device and partition

get the Type column

mount

Example #


dumbledore in /media  NO PYTHON VENV SET 
❯ sudo fdisk -l

...

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdk1  *        2048 60371951 60369904 28.8G  7 HPFS/NTFS/exFAT
/dev/sdk2       60371952 60437487    65536   32M ef EFI (FAT-12/16/32)


dumbledore in /media  NO PYTHON VENV SET 
❯ sudo mount -t ntfs /dev/sdk1 /media/ventoy-usb -o uid=1000                       
NTFS signature is missing.
Failed to mount '/dev/sdk1': Invalid argument
The device '/dev/sdk1' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

dumbledore in /media  NO PYTHON VENV SET 
✗ sudo mount -t exfat /dev/sdk1 /media/ventoy-usb -o uid=1000

Xrdp-Authentication-Required-To-Create-Managed-Color-Device

2022-07-18T00:00:00Z

I just need to RDP into an Ubuntu box via Remmina and everytime I login I have to authenticate to create a color managed device… which I don’t even know what that is!

To fix it?

vim /etc/polkit-1/localauthority.conf.d/02-allow-colord.conf

polkit.addRule(function(action, subject) {
 if ((action.id == "org.freedesktop.color-manager.create-device" ||
 action.id == "org.freedesktop.color-manager.create-profile" ||
 action.id == "org.freedesktop.color-manager.delete-device" ||
 action.id == "org.freedesktop.color-manager.delete-profile" ||
 action.id == "org.freedesktop.color-manager.modify-device" ||
 action.id == "org.freedesktop.color-manager.modify-profile") &&
 subject.isInGroup("{users}")) {
 return polkit.Result.YES;
 }
 });

Give’r the :x, logout, and you’re good to go

Setup KVM to boot from local PXE server

2022-07-06T12:30:50Z

Assuming you have a PXE server running you only need small edits to the virt-manager config

Self-hosted Docker registry with proxy pull through

2022-07-06T06:44:35Z

I decided that I want to self-host all my docker images for the purposes of regularly rebuilding and security scanning. The first step is to set up a registry, which coincidently enough you can do with a Docker container 😛!

Instructions for setting up the proxy are on the offidical docker docs here

Deployment

I chose to deploy with Portainer because I already use it for monitoring all my docker containers. I deploy most container with Ansible but use Portainer to setup a few more adhoc or non-git driven applications.

kvm-network-interface-via-nat-ubuntu-20

2022-06-25T07:20:41Z

I have started using VMs more and more in my development workflow and it’s impossible to work in a VM without an internet connection for me most of the time. Setting up the KVM networking is kind of confusing to me and I’ve done it two different ways. Here is how I set it up on my home desktop using NAT.

Credit

First thing’s first: credit to this post

Commands

There was a default network already made by virt-manager but my VM couldn’t connect over it at all…

These commands got me up and running without even turning the VM off

I went full on sudo -i for this just to make it easier - be careful

Dump an existint network config #

# as root

virsh net-dumpxml default > br1.xml

vim br1.xml

Edit it #

I was unsure what the ip range should be so I just stuck with the original blog. The default network had the CIDR block defined as 192.168.122.0/24 which is different from my home network so I guess it’s fine?

<network>
  <name>br1</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='br1' stp='on' delay='0'/>
  <ip address='192.168.10.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.10.10' end='192.168.10.100'/>
    </dhcp>
  </ip>
</network>

Define a network #

virsh net-define br1.xml
virsh net-autostart br1

Then to check…

virsh net-list --all

 Name      State    Autostart   Persistent
--------------------------------------------
 br1       active   yes         yes
 default   active   yes         yes

UUID #

virsh net-uuid br1

Magic #

virsh attach-interface --domain <NAME OF VM> --type bridge --source br1 --model virtio --config --live

My VM, ubuntu20.04 was running and immediately connected to the newly attached device!

Credit again

Visit the original post for more details - this serves more as as a quicker set of notes for future me

cheat on your man

2022-06-23T09:35:31Z

man can be a pain to read… and there’s lots of alternatives out there and one I’ve just started playing with is cheat

man man will give you this plus a billion more lines of docs, which is useful when you need it…

MAN(1)                                                                                                                       Manual pager utils                                                                                                                      MAN(1)

NAME
       man - an interface to the on-line reference manuals

SYNOPSIS
       man  [-C  file]  [-d] [-D] [--warnings[=warnings]] [-R encoding] [-L locale] [-m system[,...]] [-M path] [-S list] [-e extension] [-i|-I] [--regex|--wildcard] [--names-only] [-a] [-u] [--no-subpages] [-P pager] [-r prompt] [-7] [-E encoding] [--no-hyphenation]
       [--no-justification] [-p string] [-t] [-T[device]] [-H[browser]] [-X[dpi]] [-Z] [[section] page[.section] ...] ...
       man -k [apropos options] regexp ...
       man -K [-w|-W] [-S list] [-i|-I] [--regex] [section] term ...
       man -f [whatis options] page ...
       man -l [-C file] [-d] [-D] [--warnings[=warnings]] [-R encoding] [-L locale] [-P pager] [-r prompt] [-7] [-E encoding] [-p string] [-t] [-T[device]] [-H[browser]] [-X[dpi]] [-Z] file ...
       man -w|-W [-C file] [-d] [-D] page ...
       man -c [-C file] [-d] [-D] page ...
       man [-?V]

DESCRIPTION
       man is the system's manual pager.  Each page argument given to man is normally the name of a program, utility or function.  The manual page associated with each of these arguments is then found and displayed.  A section, if provided, will direct  man  to  look
       only  in  that  section of the manual.  The default action is to search in all of the available sections following a pre-defined order ("1 n l 8 3 2 3posix 3pm 3perl 3am 5 4 9 6 7" by default, unless overridden by the SECTION directive in /etc/manpath.config),
       and to show only the first page found, even if page exists in several sections.

       The table below shows the section numbers of the manual followed by the types of pages they contain.

       1   Executable programs or shell commands
       2   System calls (functions provided by the kernel)
       3   Library calls (functions within program libraries)
       4   Special files (usually found in /dev)
       5   File formats and conventions eg /etc/passwd
       6   Games
       7   Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
       8   System administration commands (usually only for root)
       9   Kernel routines [Non standard]

       A manual page consists of several sections.

       Conventional section names include NAME, SYNOPSIS, CONFIGURATION, DESCRIPTION, OPTIONS, EXIT STATUS, RETURN VALUE, ERRORS, ENVIRONMENT, FILES, VERSIONS, CONFORMING TO, NOTES, BUGS, EXAMPLE, AUTHORS, and SEE ALSO.

       The following conventions apply to the SYNOPSIS section and can be used as a guide in other sections.

       bold text          type exactly as shown.
       italic text        replace with appropriate argument.
       [-abc]             any or all arguments within [ ] are optional.
       -a|-b              options delimited by | cannot be used together.
       argument ...       argument is repeatable.
       [expression] ...   entire expression within [ ] is repeatable.

       Exact rendering may vary depending on the output device.  For instance, man will usually not be able to render italics when running in a terminal, and will typically use underlined or coloured text instead.

       The command or function illustration is a pattern that should match all possible invocations.  In some cases it is advisable to illustrate several exclusive invocations as is shown in the SYNOPSIS section of this manual page.

EXAMPLES
       man ls
           Display the manual page for the item (program) ls.

       man man.7
           Display the manual page for macro package man from section 7.

But what if you don’t? #

cheat man

# To convert a man page to pdf:
man -t bash | ps2pdf - bash.pdf

# To view the ascii chart:
man 7 ascii

You get tiny examples to remind you of what you probably are trying to do!

Add colored indicators to your dataframes html representation

2022-06-04T06:12:33Z

Mike Driscoll recently tweeted about making colored out with pandas DataFrames and I just had to try it for myself

Use Case

First though… why? My biggest use case is a monitoring pipeline of mine… The details aside, the output of my pipeline is a dataframe where each row has information about a failed pipeline that I need to go look into. I dump that result to a simle html file that’s hosted on an internal site and the file is updated every couple of hours. Adding some colored indicators automatically to the rows to help me assess severity of each record would be a handy way to quickly get an understanding the state of our pipelines.

How?

The docs for the applymap method state simply:

Apply a CSS-styling function elementwise.

Updates the HTML representation with the result.

So we can write a function that returns color: {color} based on the dataframe values and when we drop that dataframe to html we’ll have some simple css styling applied automagically!

By default the function will be applied to all columns of the dataframe, but that’s not useful if the columns are different types which is usually the case. Luckily there is a subset keyword to only apply to the columns you need!

Consider my example

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.read_csv("cars.csv")

sandbox   main via 3.8.11(sandbox) ipython
❯ def mpg_color(val: float):
...:     color = "red" if val < 21 else "green"
...:     return f"color: {color}"

sandbox   main via 3.8.11(sandbox) ipython
❯ df.style.applymap(mpg_color, subset="mpg").to_html("color.html")

I want to quickly see if the mpg is any good for the cars in the cars dataset and I’ll define “good” as better than 21 mpg (not great I know but just for the sake of discussion…)

The function returns an appropriate css string and after I style.applymap on just the mpg column we get this!

	Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
0	Mazda RX4	21.000000	6	160.000000	110	3.900000	2.620000	16.460000	0	1	4	4
1	Mazda RX4 Wag	21.000000	6	160.000000	110	3.900000	2.875000	17.020000	0	1	4	4
2	Datsun 710	22.800000	4	108.000000	93	3.850000	2.320000	18.610000	1	1	4	1
3	Hornet 4 Drive	21.400000	6	258.000000	110	3.080000	3.215000	19.440000	1	0	3	1
4	Hornet Sportabout	18.700000	8	360.000000	175	3.150000	3.440000	17.020000	0	0	3	2

Samba on Ubuntu 22 needs “inherit permissions” set

2022-05-31T06:30:42Z

see samba config on hogwarts

Filtering emails with core utils

2022-05-30T06:09:27Z

email1@me.com
somebody_else@gmail.com

#! /bin/bash
# pick multiple emails from list and combine into comma seperated array
emails=`cat .../emails | fzf -m | sed 's/^\|$/"/g'|paste -sd,` 

echo $emails

Reindex Nextcloud After Adding Data via CLI

2022-05-29T15:34:33Z

My Nextcloud woes

I wrote here about setting up www-data as the owner of any directories you want nextcloud to manage. However, I regularly struggle wtih permissions issues on my NAS because of the external storage app anyways so I’ve decided to just put our photos in the spot Nextcloud would otherwise put them, and use this as healthy pressure on our family to organize our photos and put the ones we care about with the rest of our family media.

Migration

Because I had a ton of photos on the NAS anyways that I wanted moved over to Nextcloud I just rsync’d the photos directory on my NAS to the user’s photos directory in nextcloud but they weren’t showing up in the web UI!

The Fix?

As www-data I needed to php /var/www/nextcloud/occ files:scan --all inside my nextcloud docker container AFTER moving a bunch of photos off my “NAS” into the folder mounted to the nextcloud container as its data folder! Before I did this they weren’t showing up in the web UI/

arr client config

2022-05-28T13:29:28Z

TIL that when setting up download clients for radarr/sonarr/lidarr/readarr/bazarr/prowlarr that you can utilize internal DNS and instead of hardcoding an IP address of your download client server, can use just the CNAME record (ie. instead of 172.10.14.13 I can use transmission.mydomain.com… notice the lact of http(s)://… adding that won’t allow the connection to work/

Furthermore, you can use internal DNS to lookup the domain, not the subdomain, and expose the port, like mydomain.com:7878 for sonarr. This was simpler to maintain because I don’t change which ports an application exposes or utilizes hardly ever, plus I don’t need to maintain CNAME records for every service!

Add space to your LVM on Ubuntu

2022-05-26T21:33:55Z

I ran out of space on the SSD in my server when doing some file transfers but only 100GB was used of a 256 GB SSD?

LVM

When installing Ubuntu live server the default option for how to partition the disk (in my experience) has been to setup an LVM group that defaults to less than the available space. Most recently I put Ubuntu server on a 256 GB SSD but the main partition was formatted as an LVM group with 100GB of storage… I didn’t think anything of this even though I’m mostly used to EXT4.

I think the reason for LVMs is performance, but in hindsight, I don’t really care much about the performance differences, I really just want all my storage that’s fast enough

Extending the LVM

A moment of googling brought me to Ubuntu’s wiki and I learned that I can expand my LVM to the space I need…

sudo lvdisplay and sudo pvdisplay show detailed views of the logical volumes and physical volumes respectively.

Take a look at those and find the volume you need to extend. For me I found this:

  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/ubuntu-lv
  LV Name                ubuntu-lv
  VG Name                ubuntu-vg
  LV Write Access        read/write
  LV Status              available
  ...

There’s more that you’ll see but this is what’s relevant - I need to extend the ubuntu-lv logical volume in the ubuntu-vg volume group.

sudo lvextend -L +50g ubuntu-vg/ubuntu-lv gives me 50 more GB of storage which should be enough for at least tonight 🤓

RTFM

Tdarr worker nodes share the cache!

2022-05-25T06:36:59Z

When working with tdarr remote nodes, they need to have access not only to the same libraries but also the same transcode cache as the server otherwise the transcodes will fail…

Network Setup

To explain I’ll give a brief overview of my home setup

I have an old Dell PowerEdge R610 as my main server running a live server distribution of Ubuntu. I use ZFS for my NAS file system, and most of my datasets are accesible over my home network. I have a Tdarr server running in a docker container on the R610.

In my office I dailyi drive a gaming desktop with an Nvidia 2060 Super running Ubuntu as well. On that desktop I am running a Tdarr node in a docker container. The container has access to the network folders with my media.

Initial Magic

When I spun up a tdarr node on my desktop, the tdarr server running on my R610 automatically registered the node, which was freaking amazing. That magic though spoiled me and I thought that I didn’t need to read the rest of the docs…

Initial Fail

I setup a transcode cache directory on the R610 locally and a separate transcode cache on my desktop that the remote tdarr node would use. Having them separated led to 2 main issues:

Transcodes were not being migrated back to my library properly
I was running out of disk space on my desktop because tdarr wasn’t deleting the completed cache files properly.

The Fix

Make a transcode cache directory accessible on my network, and give access to that directory to the docker container running my remote tdarr node.

Local DNS with Pi-hole

2022-05-23T09:40:59Z

Spoilers #

Tailscale is way easier than this… I was doing this local DNS overwrite in Pi-hole before running tailscale and I haven’t just totally “kicked the habit” yet, so for anyone NOT running tailcale, but wanting local HTTPS and using Pi-Hole this method would work fine

Intro #

I use pihole as my DNS server at home. I run unbound as well and have a pretty standard setup from their docs.

Pi-hole does 2 primary things for me:

dns sink-hold, the primary use case I believe
local SSL for all my self-hosted apps

Caveat #

I know for sure there are better ways to do this, but there’s also worse ones… so for now this has been my pattern, and it’s only bitten me in the butt when I’ve forgotten to add the final CNAME record… which’ll make sense in a sec.

Process #

It’s really quite simple - in pihole I have a DNS record for my domain pointing to my primary server

Then for each service that I want to keep everything resolved locally for I add a CNAME

With DNS resolving this way for clients using my pihole, the networking all stays local and I still get HTTPS for my wildcard cert in cloudflare

Why? #

The reasons I do this are simple:

I’ve been doing it since I started homelabbing and it started out of misunderstanding of how networking works at all
I use a home dashboard with https:// links, and it’s nice to just use that same dashbaord publically or at home. With local DNS resolution then I can whitelist some services, but conveniently access over the public url with https but routing such that my whitelist let’s me in when it wouldn’t if I tried to access the service from an external client
it’s my homelab - I can do what I want

It’s too complicated #

Honestly, as simple as this is, it is tedious and kind of complicated… There’s options to make it better…

script any service deployment to update the pihole /etc/host file
just use tailscale…

Configure bridge network on Ubuntu 22.04 with Netplan

2022-05-22T14:55:28Z

See 02-….yaml in ansible-nas

Netplan change from Focal to Jammy

2022-05-22T13:23:17Z

I am revamping my home server and bumped myself early up to Jammy Jellyfish… however to my peril I reused my netplan config and after hitting my server with the ‘ol netplan apply I lost connection… DNS still seemed to kinda work externally, but internally nothing was up…

Turns out Netplan got a little change in how to express the gateway key in the netplan config!

Old Ubuntu 20.04 way

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      gateway4: 192.168.1.1  # <-- This changes!
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

New jammin way for Jammy Jellyfish (at least that worked for me)

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      routes:
        - to: default
          via: 192.168.1.1 
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

How I use Nextcloud for safe central storage

2022-05-19T21:17:42Z

Setup admin
External Storage extension
Add my nas zfs dataset
chown -R www-data:www-data on anything nextcloud uploads to.

Subset a list based on values in another list with itertools.compress

2022-05-19T14:22:58Z

I have list [True, False, False, True] and another list [1, 2, 3, 4] and a use case where I want to filter list 2 based on list 1 to remove values that line up with the element False in list 1…. so the outcome will be [1, 4]. list(compress(list2, list1)) will do it. As long as you can create a mask for the filter than itertool.compress will be your friend!

Nextcloud permissions with ZFS and Ansible-NAS

2022-05-19T13:55:20Z

TL;DR

As the nextcloud docs say… if you want to write to an external volume that location has to be writeable by the user/group www-data on the host system… so if that makes sense to you then this TIL probably isn’t a ton of value.. if not however, read on :)

Case Study

You want to self-host your own cloud and use a smart file system for convenience… Nextcloud and ZFS are pretty common goto answers for each of those problems.

My home NAS is built on ZFS and among other things I have a zpool named tank and nested in there is a tank/nas dataset with several child zfs datasets under that.

I want to use nextcloud mainly for auto-uploading photos from my wife’s and my phones for automatic backups. The issue is that the nextcloud application (I run in Docker) is fixed as the www-data user and so any volume/folder that you want nextcloud to write to needs to be permissioned such that www-data owns it… but I don’t want www-data to own everything in my NAS… so what’s a girl to do?

Solution

Well, one way to go is to just utilize docker volumes, write the data in the container to /var/www/html and let that be the place your data backsup to.

I still wanted nextcloud to automatically write right to my NAS so I created a nextcloud-upload directory inside of tank/nas/media/photos (photos cause that’s all that gets automatically uploaded)

Then I chown -R www-data:www-data /tank/nas/media/photos/nextcloud-upload so that just that sub-folder is owned by www-data. Now everyone’s happy!

See ZFS snapshot disk usage

2022-05-19T06:05:23Z

As I was cleaning up my NAS recently I noticed that I ran out of storage even though my disk usage looked pretty low… turns out I was keeping a mega-ton of ZFS snapshots and due to my own ignorance at the time didn’t realize the storage cost of this!

zfs list-o space tank/home will show the disk usage of the dataset and snapshots in tank/home

Deleting files on remote storage from Ubuntu might not do what you think

2022-05-19T05:51:50Z

From my daily driver Ubuntu machine I often open nautilus, dolphin, etc. and delete a file here or there on my NAS… turns out Ubuntu sends thse file to .Trash-100 ON THE NAS so I’m effectively just moving that file and not freeing up any space…

Plug Snapshot!

2022-05-17T15:45:49Z

:PlugSnapshot, :w ~/dotfiles/nvim/snapshot.vim… keep your config safe with git!

Filepath Completion in Neovim

2022-05-17T14:03:27Z

I’ve had Plug 'hrsh7th/cmp-path' in my plugins for ever but didn’t notice until recently that I wasn’t getting any filepath completion in vim!

Fuller setup instructions below the TLDR

TL;DR

Turns out I need to not be a dope and configure nvim-cmp to actually use it…

local cmp = require'cmp'

cmp.setup({
    -- removed rest of setup - see the rest in my dotfiles
  sources = cmp.config.sources({
    { name = 'path' },  -- This needs to be here!
    })
})

My Setup

For the sake of completeness here is how I currently (May 2022) configure completion in Neovim usin nvim-cmp

Plugins #

I keep all my plugins in plugins.vim

call plug#begin(s:plug_dir)
Plug 'neovim/nvim-lspconfig'
Plug 'hrsh7th/cmp-nvim-lsp'
Plug 'hrsh7th/cmp-buffer'
Plug 'hrsh7th/cmp-path'
Plug 'hrsh7th/cmp-cmdline'
Plug 'hrsh7th/nvim-cmp'

" For ultisnips users.
<!-- " Plug 'SirVer/ultisnips' -->
<!-- " Plug 'quangnguyen30192/cmp-nvim-ultisnips' -->

call plug#end()

Vim Settings #

My vim settings are also kept in their own file, settings.vim


set completeopt=menu,menuone,noselect

nvim-cmp configuration #

I have a cmp.lua file that gets sourced in init.lua (file structure explained below) for configuring cmp.


  -- Setup nvim-cmp.
local cmp = require'cmp'

cmp.setup({
  snippet = {
    -- REQUIRED - you must specify a snippet engine
    expand = function(args)
      -- For `ultisnips` user.
      vim.fn["UltiSnips#Anon"](args.body)
    end,
  },
  window = {
      completion = cmp.config.window.bordered(),
  },
  mapping = {
    ['<Down>'] = cmp.mapping.select_next_item({ behavior = cmp.SelectBehavior.Select }),
    ['<Up>'] = cmp.mapping.select_prev_item({ behavior = cmp.SelectBehavior.Select }),
    ['<C-d>'] = cmp.mapping.scroll_docs(-4),
    ['<C-f>'] = cmp.mapping.scroll_docs(4),
    ['<C-Space>'] = cmp.mapping.complete(),
    ['<C-e>'] = cmp.mapping.close(),
    ['<Tab>'] = cmp.mapping(cmp.mapping.select_next_item(), { 'i', 's' }),
    ['<CR>'] = cmp.mapping.confirm({
      behavior = cmp.ConfirmBehavior.Replace,
      select = true,
    })
  },
  sources = cmp.config.sources({
    { name = 'nvim_lsp' },
    { name = 'ultisnips' },
    { name = 'buffer' },
    { name = 'path' },
    { name = 'tmux' },
    })
})

The sources section is what was key for this post…

Piecing it together!

My init.vim sources plugins and then settings and then finally calls init.lua. init.lua sources my cmp.lua file and BANG! auto-completion.

More sources #

hrsh7th’s wiki for nvim-cmp is here and has example configs as well as a list of sources…

Don’t forget to configure and not just install!

my dotfiles

Forms with FastAPI and Jinja

2022-05-15T00:00:00Z

I just started using FastAPI for a home project and needed to pass back a dynamic number of values from a form rendered with jinja…

Dynamic Values

The jinja templating for rendering HTML based on something like a python iterable is nice and easy

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). it?

The way to pass back all those values is pretty straight forward (after hours of messing around that is!)

# I hate it when tutorials don't show ALL relevant pieces to the blurb
import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

Dynamic-Form-Values-With-Jinja-And-Fastapi

2022-05-15T00:00:00Z

I’m currently working on a self-hostable wish list app using FastAPI so we can finally drop Amazon forever. (The lists funcionality has been super handy for sharing holiday gift ideas with the famj!)

FastAPI

FastAPI is an amazing framework for quickly building APIs with Python. I will have a slightly longer post about my brief experience with it coming later…

Jinja, Forms, and FastAPI

One of the last things I needed to figure out in my app was how to generate a form in a Jinja template with a dynamic number of inputs and then pass all the inputs to the backend to perform a database operation (my exact case was removing rows from a table).

Explicit Values #

The way to pass back explicit variables is really easy…

Our form would look like this (I’m using bootstrap CSS)

<form method="post">
    <div class="form-check ">
        <input class="form-check-input"  name="item_1" id="itemOne" value="1" type="checkbox">
        <label class="form-check-label" for="itemOne" > A label for this item </label>
    </div>
    <div class="form-check ">
        <input class="form-check-input"  name="item_2" id="itemTwo" value="2" type="checkbox">
        <label class="form-check-label" for="itemTwo" > A label for item 2 </label>
    </div>

<button type="submit" class="submit btn btn-xl" >Submit</button>
</form>

So what is this? This form will have 2 rows with the lables you see in <label> </label> and checkboxes that when checked would have the value value in each <input> line.

So our backend might looks something like this…

I’m keeping all the imports and stuff here to show where they come from but I won’t discuss it all here - that’ll be in a future post

import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    item_1: int = Form(...),
    item_2: int = Form(...)
    db: Session = Depends(create_get_session),
):
    print(item_1)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    print(item_2)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

Dynamic values #

That’s all pretty simple… pass back values by the name in the form…

What about a form that’s generated dynamically? This is my case since I display a row/checkbox for every row in my table so my form looks like this…

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). Now I started scratching my head on how to pass an unknown number of inputs to my backend of FastAPI wants each input explicitly defined and typed… I can’t just pass the form back becuase that’s not a thing so what’s the way to do it?

# same stuff as above, only showing post method here
@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

I look forward to my wish list app maturing and I hope this helps someone working with FastAPI!

Plug-Snapshot-To-Save-Your-Life

2022-05-13T00:00:00Z

If you use vim-plug for managing your vim plugins, do yourself a favor and snapshot your plugins before upgrading!

:PlugSnapshot creates a vim.snapshot file that you can use to restore your plugin versions with vim -S snapshot.vim

The snapshot file looks like this:

❯ cat snapshot.vim
" Generated by vim-plug
" Tue 17 May 2022 03:44:29 PM CDT
" :source this file in vim to restore the snapshot
" or execute: vim -S snapshot.vim

silent! let g:plugs['Telegraph.nvim'].commit = '92e472f4e83acd60eb3766168e66d02718bfefe0'
silent! let g:plugs['TrueZen.nvim'].commit = '508b977d71650da5c9243698614a9a1416f116d4'
silent! let g:plugs['ariake-vim-colors'].commit = '9fb35f1255e475631c9df24ecc5485a40360cc7b'
silent! let g:plugs['auto-pairs'].commit = '39f06b873a8449af8ff6a3eee716d3da14d63a76'
silent! let g:plugs['bufutils.vim'].commit = '4634feb1312fd73fab66cfaa860e7af3abde935b'
silent! let g:plugs['cmp-buffer'].commit = '12463cfcd9b14052f9effccbf1d84caa7a2d57f0'
silent! let g:plugs['cmp-cmdline'].commit = 'c36ca4bc1dedb12b4ba6546b96c43896fd6e7252'
silent! let g:plugs['cmp-nvim-lsp'].commit = 'e6b5feb2e6560b61f31c756fb9231a0d7b10c73d'
silent! let g:plugs['cmp-nvim-ultisnips'].commit = '21f02b62deb409ce69928a23406076bd0043ddbc'
silent! let g:plugs['cmp-path'].commit = '466b6b8270f7ba89abd59f402c73f63c7331ff6e'
silent! let g:plugs['cmp-spell'].commit = '5602f1a0de7831f8dad5b0c6db45328fbd539971'
silent! let g:plugs['compe-tabnine'].commit = '33e4af509c27da9ef2c9c3002c01e3ec031797d4'
silent! let g:plugs['coverage-highlight.vim'].commit = '864e03679ea4168661501246147893cc82020917'
silent! let g:plugs['diffurcate.vim'].commit = 'b804675072220ff7c7ebcd24a028aa4aa35f09cc'
silent! let g:plugs['fzf'].commit = '6dcf5c3d7d6c321b17e6a5673f1533d6e8350462'
silent! let g:plugs['fzf.vim'].commit = 'd5f1f8641b24c0fd5b10a299824362a2a1b20ae0'
silent! let g:plugs['git-blame.vim'].commit = '9d144b7bed5d8f1c9259551768b7f3b3d1294917'
silent! let g:plugs['harpoon'].commit = 'd3d3d22b6207f46f8ca64946f4d781e975aec0fc'
silent! let g:plugs['instant.nvim'].commit = 'c02d72267b12130609b7ad39b76cf7f4a3bc9554'
silent! let g:plugs['lsp-colors.nvim'].commit = '517fe3ab6b63f9907b093bc9443ef06b56f804f3'
silent! let g:plugs['lsp_extensions.nvim'].commit = '4011f4aec61ba59c734f5dbf52e91f258b99d985'
silent! let g:plugs['lspkind-nvim'].commit = '57e5b5dfbe991151b07d272a06e365a77cc3d0e7'
silent! let g:plugs['lspsaga.nvim'].commit = '8dde091a61ab07f639baaa82b456d3508d0aa7e8'
silent! let g:plugs['neoformat'].commit = '409ebbba9f4b568ea87ab4f2de90a645cf5d000a'
silent! let g:plugs['neovim-fuzzy'].commit = '0bef4e1a81c65fc05d31380dd74454bd67733837'
silent! let g:plugs['nerdtree'].commit = 'eed488b1cd1867bd25f19f90e10440c5cc7d6424'
silent! let g:plugs['nerdtree-visual-selection'].commit = '05427635ff053a2c542fcf8d7c3744c72575e76c'
silent! let g:plugs['nvim-cmp'].commit = 'a226b6a4ff72e5e809ed17734318233fb25c87f3'
silent! let g:plugs['nvim-compe'].commit = 'd186d739c54823e0b010feb205c6f97792322c08'
silent! let g:plugs['nvim-lspconfig'].commit = '9ff2a06cebd4c8c3af5259d713959ab310125bec'
silent! let g:plugs['nvim-lspinstall'].commit = '79ec2425d6b39cdcb69d379f3e56847f49be73eb'
silent! let g:plugs['nvim-treesitter'].commit = '10d57b3ec14cac0b6b759e1eb5594617b8a7e883'
silent! let g:plugs['nvim-treesitter-textobjects'].commit = '094e8ad3cc839e825f8dcc91352837653e365a8f'
silent! let g:plugs['nvim-web-devicons'].commit = 'bdd43421437f2ef037e0dafeaaaa62b31d35ef2f'
silent! let g:plugs['playground'].commit = '71b00a3c665298e5155ad64a9020135808d4e3e8'
silent! let g:plugs['plenary.nvim'].commit = '25b3475b97f241e6f76249747bc209e70c5d2ec8'
silent! let g:plugs['popup.nvim'].commit = 'b7404d35d5d3548a82149238289fa71f7f6de4ac'
silent! let g:plugs['syntastic'].commit = 'b7f4f71539038d33f173bfa72631737da049575a'
silent! let g:plugs['tabular'].commit = '339091ac4dd1f17e225fe7d57b48aff55f99b23a'
silent! let g:plugs['targets.vim'].commit = '8d6ff2984cdfaebe5b7a6eee8f226a6dd1226f2d'
silent! let g:plugs['telescope-fzy-native.nvim'].commit = '7b3d2528102f858036627a68821ccf5fc1d78ce4'
silent! let g:plugs['telescope.nvim'].commit = '39b12d84e86f5054e2ed98829b367598ae53ab41'
silent! let g:plugs['termopen.vim'].commit = '3194a991a18a9be2fd9fcf8c4c55fe990c04b2bd'
silent! let g:plugs['tokyonight.nvim'].commit = '8223c970677e4d88c9b6b6d81bda23daf11062bb'
silent! let g:plugs['trouble.nvim'].commit = 'da61737d860ddc12f78e638152834487eabf0ee5'
silent! let g:plugs['ultisnips'].commit = 'f5ccf0977c611ffd774ca180774959301baaffad'
silent! let g:plugs['undotree'].commit = '08e259be24d4476c1ee745dc735eefd44f90efdc'
silent! let g:plugs['vim-airline'].commit = 'c4655701431a9c79704c827fd88a4783ec946879'
silent! let g:plugs['vim-airline-themes'].commit = '97cf3e6e638f936187d5f6e9b5eb1bdf0a4df256'
silent! let g:plugs['vim-be-good'].commit = 'bc499a06c14c729b22a6cc7e730a9fbc44d4e737'
silent! let g:plugs['vim-commentary'].commit = '3654775824337f466109f00eaf6759760f65be34'
silent! let g:plugs['vim-devicons'].commit = 'a2258658661e42dd4cdba4958805dbad1fe29ef4'
silent! let g:plugs['vim-dispatch'].commit = '00e77d90452e3c710014b26dc61ea919bc895e92'
silent! let g:plugs['vim-flake8'].commit = 'a99054ef98e8fdaefa1315af4649138bcadbfdf7'
silent! let g:plugs['vim-floaterm'].commit = 'ab7876f86c05c1935eb23a193f4f276132902ac1'
silent! let g:plugs['vim-fugitive'].commit = 'f529acef74b4266d94f22414c60b4a8930c1e0f3'
silent! let g:plugs['vim-gitbranch'].commit = '1a8ba866f3eaf0194783b9f8573339d6ede8f1ed'
silent! let g:plugs['vim-indent-object'].commit = '5c5b24c959478929b54a9e831a8e2e651a465965'
silent! let g:plugs['vim-nerdtree-syntax-highlight'].commit = '5178ee4d7f4e7761187df30bb709f703d91df18a'
silent! let g:plugs['vim-polyglot'].commit = '38282d58387cff48ac203f6912c05e4c8686141b'
silent! let g:plugs['vim-pydocstring'].commit = 'f6e3c52bfaf4b4f76dab4d84e75b94199d5f3b9c'
silent! let g:plugs['vim-quicklink'].commit = '021167741588555501594e1fc31f130b16acefa0'
silent! let g:plugs['vim-repeat'].commit = '24afe922e6a05891756ecf331f39a1f6743d3d5a'
silent! let g:plugs['vim-signify'].commit = '69498f6d49f3eeac06870012416dd9bf867b84f3'
silent! let g:plugs['vim-sneak'].commit = '94c2de47ab301d476a2baec9ffda07367046bec9'
silent! let g:plugs['vim-snippets'].commit = '6f270bb2d26c38765ff2243e9337c65f8a96a28b'
silent! let g:plugs['vim-startify'].commit = '81e36c352a8deea54df5ec1e2f4348685569bed2'
silent! let g:plugs['vim-surround'].commit = 'bf3480dc9ae7bea34c78fbba4c65b4548b5b1fea'
silent! let g:plugs['vim-test'].commit = '2240d7a4b868cb594b7d83544e1b6db4df806e5e'
silent! let g:plugs['vim-tmux-runner'].commit = '54767911fd5e6e2d8e493847149e315ac2e6531a'
silent! let g:plugs['vim-ultest'].commit = '6978fd32e3ca2c1c5591884eea0d57a7ee43d212'
silent! let g:plugs['vim-visualstar'].commit = 'a18cd0e7a03311ac709595c1d261ed44b45c9098'
silent! let g:plugs['vimtex'].commit = 'dfaca59bbbf0079ab1b4f159337ae7f17d1b5289'
silent! let g:plugs['which-key.nvim'].commit = 'bd4411a2ed4dd8bb69c125e339d837028a6eea71'

PlugUpdate!

Python-Eval

2022-05-12T00:00:00Z

TODO

title = "my Title"
eval('"my" in title')

>>> True

print("hello, world"); print("formatting")

Dataframe-To-Styled-Html

2022-05-07T00:00:00Z

I wrote up a little on exporting DataFrames to markdown and html here

But I’ve been playing with a web app for with lists and while I’m toying around I learned you can actually give your tables some style with some simple css classes!

To HTML

Reminder that if you have a dataframe, df, you can df.to_html() to get an HTML table of your dataframe.

Well you can pass some classes to make it look super nice!

Classes and CSS

I don’t know anything really about CSS so I won’t pretend otherwise, but as I was learning about bootstrap that’s where I stumbled upon this…

There are several classes you can pass but I found really good luck with table-bordered and table-dark for my use case

df.to_html(classes=["table table-bordered table-dark"])

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

You try it!

Crack open ipython and make a dataframe, then df.to_html(classes=["table table-bordered table-dark"]), copy the output (minus the quote marks ipython uses to denote the string type) that into my-file.html, open that up in a browser and be amazed!

For added effeciency try using pyperclip to copy the output right to your clipboard!

pip install pyperclip and then pyperclip.copy(df.to_html(classes=["table table-bordered table-dark"]))

Dataframe-To-Markdown

2022-05-07T00:00:00Z

Pandas

pandas.DataFrames are pretty sweet data structures in Python.

I do a lot of work with tabular data and one thing I have incorporated into some of that work is automatic data summary reports by throwing the first few, or several relevant, rows of a dataframe at a point in a pipeline into a markdown file.

Pandas has a method on DataFrames that makes this 100% trivial!

The Method

Say we have a dataframe, df… then it’s literally just: df.to_markdown()

❯ df.head()

          Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
0          Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
1      Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
2         Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
3     Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
4  Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2

In ipython I can call the method and get a markdown table back as a string


mental-data-lake   new-posts via 3.8.11(mental-data-lake) ipython
❯ df.head().to_markdown()
'|    | Unnamed: 0        |   mpg |   cyl |   disp |   hp |   drat |    wt |   qsec |   vs |   am |   gear |   carb |\n|---:|:------------------|------:|------:|-------:|-----:|-------:|------:|-------:|-----:|-----:|-------:|-------:|\n|  0 | Mazda RX4         |  21   |     6 |    160 |  110 |   3.9  | 2.62  |  16.46 |    0 |    1 |      4 |      4 |\n|  1 | Mazda RX4 Wag     |  21   |     6 |    160 |  110 |   3.9  | 2.875 |  17.02 |    0 |    1 |      4 |      4 |\n|  2 | Datsun 710        |  22.8 |     4 |    108 |   93 |   3.85 | 2.32  |  18.61 |    1 |    1 |      4 |      1 |\n|  3 | Hornet 4 Drive    |  21.4 |     6 |    258 |  110 |   3.08 | 3.215 |  19.44 |    1 |    0 |      3 |      1 |\n|  4 | Hornet Sportabout |  18.7 |     8 |    360 |  175 |   3.15 | 3.44  |  17.02 |    0 |    0 |      3 |      2 |'

You can drop that string into a markdown file and using any reader that supports the rendering you’ll have a nicely formated table of example data in whatever report you’re making!

Bonus method

Just like markdown, you can export a dataframe to html with df.to_html() and use that if it’s more appropriate for your use case:


'<table border="1" class="dataframe">\n  <thead>\n    <tr style="text-align: right;">\n      <th></th>\n      <th>Unnamed: 0</th>\n      <th>mpg</th>\n      <th>cyl</th>\n      <th>disp</th>\n      <th>hp</th>\n      <th>drat</th>\n      <th>wt</th>\n      <th>qsec</th>\n      <th>vs</th>\n      <th>am</th>\n      <th>gear</th>\n      <th>carb</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Mazda RX4</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.620</td>\n      <td>16.46</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Mazda RX4 Wag</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.875</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Datsun 710</td>\n      <td>22.8</td>\n      <td>4</td>\n      <td>108.0</td>\n      <td>93</td>\n      <td>3.85</td>\n      <td>2.320</td>\n      <td>18.61</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Hornet 4 Drive</td>\n      <td>21.4</td>\n      <td>6</td>\n      <td>258.0</td>\n      <td>110</td>\n      <td>3.08</td>\n      <td>3.215</td>\n      <td>19.44</td>\n      <td>1</td>\n      <td>0</td>\n      <td>3</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Hornet Sportabout</td>\n      <td>18.7</td>\n      <td>8</td>\n      <td>360.0</td>\n      <td>175</td>\n      <td>3.15</td>\n      <td>3.440</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>2</td>\n    </tr>\n  </tbody>\n</table>'

My blog will render that html into a nice table! (After removing new line characters)

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

Wish-List-With-Fastapi

2022-05-06T00:00:00Z

Amazon has crossed the line with me just one too many times now so we are looking to drop them like every other Big Tech provider….

However, one key feature of Amazon that has been so useful for us is Lists… We can just maintain a list for each of us and then family members can login anytime and check it out… This really alleviates any last minute gift idea stress right before a birthday or something.

So I need a nice gift list service but I don’t want to be locked into one company (like a Target registry or something) and I’d like to host it myself

The internets had a few options but nothing looked/felt like I wanted to I decided to build my own.

The Frontend

I have no idea how to do front end so stay tuned

The Backend

FastAPI for the win on this one… I followed a few examples online and what I was able to build in just a few minutes is pretty impressive thanks to the design of FastAPI.

Some key features are:

Auto doc generation
Required typing (which makes #1 possible)
Built-in api testing in the browser
Easy integration with sqlalchemy
Development time so short you won’t be done with your coffee before having something up and running!

Database #

Starting with a simple database.py we can create a sqlalchemy session with a base model with about 7 lines of code…


from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker


SQLALCHEMY_DATABASE_URL = "sqlite:///wishes.sqlite3"
engine = create_engine(SQLALCHEMY_DATABASE_URL)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

Model #

For my wish list I needed just a simple table:

id	person	item	link	purchased	purchased_by	date_added
1	pypeaday	A sweet item	www.mystore.store	False	dad	2022-05-05 21:55:09
2	pypeaday	A bitter item	www.bitterstore.com	True	Mrs. pypeaday	2022-05-06 06:55:54

The table is simple enough… A unique key, the person who the wish belongs to, the item (or wish), a link to the item, whether it’s been purchased or not and by whom, and the date it was added.

To make this model with sqlalchemy we can make a model.py like so:

from database import Base
from sqlalchemy.schema import Column
from sqlalchemy.types import Boolean, Integer, String, Text


class Wishes(Base):
    __tablename__ = "Wishes"
    id = Column(Integer, primary_key=True, index=True)
    person = Column(String(20))
    item = Column(Text())
    link = Column(Text())
    purchased = Column(Boolean())
    purchased_by = Column(String(90))
    date_added = Column(String(15))

Schema #

One of the best things about FastAPI is trivial integration with pydantic. We can define a schema to ensure any data posted is not missing anything!

Make a schema.py with the following:

from pydantic import BaseModel
import time
from typing import Optional


class wish_schema(BaseModel):

    person: str
    item: str
    link: str
    purchased: bool = False
    purchased_by: Optional[str] = None
    date_added: Optional[str] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

    class Config:
        orm_mode = True


class patch_schema(BaseModel):

    purchased: bool
    purchased_by: Optional[str] = None

    class Config:
        orm_mode = True

I have 2 schemas - one for a wish which you’ll see down below is used to validate any post requests.

To simplify things for me I made another schema, patch_schema which I use for the route that updates the table (ie. marking an existing wish as purchased)

Session #

One of the last things we need is a Session

So make a session.py…

from database import SessionLocal, engine
import model

model.Base.metadata.create_all(bind=engine)


def create_get_session():
    try:
        db = SessionLocal()
        yield db
    finally:
        db.close()

Our routes will depend on this create_get_session function that will yield a db object through which we’ll udpate our database

Ok just do it already!

So our main.py will have a few routes in it…

What do we want to support?

Getting all wishes
Getting a specific wish
Updating a specific wish
Deleting a wish

I think the script is fairly self explanatory but here’s a few notes…

We decorate each function with @app.<method> and define response_model as well as status_code
The functions are defined with async (this was my first exposure to this so I can’t go in depth on it yet)
The functions all take a db which is from session.py and that db depends on the create_get_session function
If the db is being updtes then we type the object used for the update with the appropriate schema (either wish_schema or patch_schema)

From there we’re in true python-land where you can basically guess the methods on db and you’d probably be right… (like query, upddate, delete etc.)

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from typing import List
from model import Wishes
from schema import wish_schema, patch_schema
from session import create_get_session

app = FastAPI()


@app.get("/")
def read_root():
    return {"message": "server is up!"}


@app.get("/wishes", response_model=List[wish_schema], status_code=200)
async def read_wishes(db: Session = Depends(create_get_session)):
    wishes = db.query(Wishes).all()
    return wishes


@app.post("/wishes", response_model=wish_schema, status_code=201)
async def add_wish(wish: wish_schema, db: Session = Depends(create_get_session)):
    new_wish = Wishes(
        person=wish.person,
        item=wish.item,
        link=wish.link,
        purchased=wish.purchased,
        purchased_by=wish.purchased_by,
        date_added=wish.date_added,
    )
    db.add(new_wish)
    db.commit()

    return new_wish


@app.get("/wishes/{id}", response_model=wish_schema, status_code=200)
async def get_wish(id: int, db: Session = Depends(create_get_session)):
    wish = db.query(Wishes).get(id)
    return wish


@app.patch("/wishes/{id}", response_model=wish_schema, status_code=200)
async def update_wish(
    id: int, patch: patch_schema, db: Session = Depends(create_get_session)
):
    db_wish = db.query(Wishes).get(id)
    db_wish.purchased = patch.purchased
    db_wish.purchased_by = patch.purchased_by
    db.commit()
    db.refresh(db_wish)

    return db_wish


@app.delete("/wishes/{id}", status_code=200)
async def delete_wish(id: int, db: Session = Depends(create_get_session)):
    db_wish = db.query(Wishes).get(id)
    if not db_wish:
        raise HTTPException(status_code="404", detail="Wish id does not exist")

    db.delete(db_wish)
    db.commit()

    return None

My Code

You can find my repo here.

I’ll plan to update and maintain for as long as I use it

Git-Bisect

2022-05-03T00:00:00Z

I try to commit a lot, and I also try to write useful tests appropriate for the scope of work I’m focusing on, but sometimes I drop the ball…

Whether by laziness, ignorance, or accepted tech debt I don’t always code perfectly and recently I was dozens of commits into a new feature before realizing I broke something along the way that none of my tests caught…

Before today I would’ve manually reviewed every commit to see if something obvious slipped by me (talk about a time suck 😩)

There must be a better way

Bisect?

git bisect is the magic sauce for this exact problem…

You essentially create a range of commits to consider and let git bisect guide you through them in a manner akin to Newton’s method for finding the root of a continuous function.

How to do it?

Start with git bisect start and then choose the first good commit (ie. a commit you know the bug isn’t present in)


sandbox   bisect-post   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect start

sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good 655332b
bisect-post  HEAD         main         ORIG_HEAD
5b31e1e  -- [HEAD]    add successful print (52 seconds ago)
308247b  -- [HEAD^]   init another loop (77 seconds ago)
4555c59  -- [HEAD^^]  introduce bug (2 minutes ago)
9cf6d55  -- [HEAD~3]  add successful loop (3 minutes ago)
bcb41c3  -- [HEAD~4]  change x to 10 (4 minutes ago)
3c34aac  -- [HEAD~5]  init x to 1 (4 minutes ago)
12e53bd  -- [HEAD~6]  print cwd (4 minutes ago)
655332b  -- [HEAD~7]  add example.py (10 minutes ago)  # <- I want to start at this commit
59e0048  -- [HEAD~8]  gitignore (23 hours ago)
fb9e1fb  -- [HEAD~9]  add reqs (23 hours ago)


sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad 5b31e1e
bisect-post                                                ORIG_HEAD
HEAD                                                       refs/bisect/good-655332b6c384934c2c00c3d4aba3011ccc1e5b57
main
5b31e1e  -- [HEAD]    add successful print (5 minutes ago)  # <- I start here with the "bad" commit
308247b  -- [HEAD^]   init another loop (6 minutes ago)
4555c59  -- [HEAD^^]  introduce bug (6 minutes ago)
9cf6d55  -- [HEAD~3]  add successful loop (7 minutes ago)
bcb41c3  -- [HEAD~4]  change x to 10 (8 minutes ago)
3c34aac  -- [HEAD~5]  init x to 1 (9 minutes ago)
12e53bd  -- [HEAD~6]  print cwd (9 minutes ago)
655332b  -- [HEAD~7]  add example.py (14 minutes ago)
59e0048  -- [HEAD~8]  gitignore (23 hours ago)
fb9e1fb  -- [HEAD~9]  add reqs (23 hours ago)

After starting bisect with a “good” start commit and a “bad” ending commit we can let git to it’s thing!

Git checksout a commit somewhere about halfway between the good and bad commit so you can see if your bug is there or not.


sandbox   bisect-post (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad 5b31e1e
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[bcb41c3854e343eade85353683f2c1c4ddde4e04] change x to 10

sandbox   HEAD (bcb41c38) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯

In my example here I have a python script with some loops and print statements - they aren’t really relevant, I just wanted an easy to follow git history.

So I check to see if the bug is present or not either by running/writing tests or replicating the bug somehow.

In this session commit bcb41c38 is actually just fine, so I do git bisect good


sandbox   HEAD (bcb41c38) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good
Bisecting: 1 revision left to test after this (roughly 1 step)
[4555c5979268dff6c475365fdc5ce1d4a12bd820] introduce bug

And we see that git moves on to checkout another commit…

In this case the next commit is the one where I introduced a bug

git bisect bad then gives me:


sandbox   HEAD (4555c597) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[9cf6d55301560c51e2f55404d0d80b1f1e22a33d] add successful loop

At 4555c597 the script works as expected so one more git bisect good yields…

sandbox   HEAD (9cf6d553) (BISECTING)   ×1 via   v3.8.11(sandbox)  on  (us-east-1)
❯ git bisect good
4555c5979268dff6c475365fdc5ce1d4a12bd820 is the first bad commit
commit 4555c5979268dff6c475365fdc5ce1d4a12bd820
Author: ########################### 
Date:   Tue May 3 09:00:00 2022 -0500

    introduce bug

 example.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

What happened?

Git sliced up a range of commits based on me saying of the next one was good or bad and localized the commit that introduced a bug into my workflow!

I didn’t have to manually review commits, click through logs, etc… I just let git checkout relevant commits and I ran whatever was appropriate for reproducing the bug to learn when it was comitted!

Pandas-String-Contains

2022-05-02T00:00:00Z

TL;DR

pandas.Series.str.contains accepts regular expressions and this is turned on by default!

Use case

We often need to filter pandas DataFrames based on several string values in a Series.

Notice that sweet pyflyby import 😁!

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.DataFrame({"A": ["string1", "string2", "string3"]})
[PYFLYBY] import pandas as pd

sandbox   main via 3.8.11(sandbox) ipython
❯ df

         A
0  string1
1  string2
2  string3

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains('1') | df.A.str.contains('2')]

         A
0  string1
1  string2

And this isn’t the worst thing in the world, especially for such a tiny example…

But what if we had dozens or more values to filter on?

Then it looks so much nicer to create an iterable of the values we want to filter on and join them with an apropriate regex operator (in this case | for inclusive or)


sandbox   main via 3.8.11(sandbox) ipython
❯ vals = ["1", "2"]  # iterable with whatever is appropriate for your use case

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains("|".join(vals), regex=True)]

         A
0  string1
1  string2

Fin

This is a super nice and concise way to do the kind of filtering my team does on a daily basis!

Unpack-Anywhere-With-Star

2022-04-24T00:00:00Z

Unpacking iterables in python with * is a pretty handy trick for writing code that is just a tiny bit more pythonic than not.

arr: Tuple[Union[int, str]] = (1, 2, 3, 'a', 'b', 'c')


print(arr)
>>> (1, 2, 3, 'a', 'b', 'c')

# the * unpacks the tuple into the individual elements
print(*arr)
>>> 1, 2, 3, 'a', 'b', 'c'

x, y, z, *alphas = arr

# x = 1, y = 2, z = 3
# alphas = [ 'a', 'b', 'c' ]

But @Ned Batchelder showed me via Twitter than you can arbitrarily unpack arguments based on position - it doesn’t have to be done at the beginning or the end!

x, y, *mixed, alpha = arr

# x = 1, y = 2
# mixed = [3, 'a', 'b']
# alpha = 'c'

I’m not entirely sure when I’ll need this but it definitley shows me another example of how flexible python is!

Htop

2022-04-24T00:00:00Z

htop is a common command line tool for seeing interactive output of your system resource utilization, running processes, etc.

I’ve always been super confused about htop showing seemingly the same process several times though…

The Fix… #

Just hit H…. makes the view a lot nicer 😀

Pipx

2022-04-22T00:00:00Z

pipx is a tool I’ve been using to solve a few problems of mine…

pinning formatting tools like black, flake8, isort, etc. to the same version for all my projects
keeping virtual environments clean of things like cookiecutter
python utilities I want system wide but not in the global environment, like visidata

What is it? #

pip itself is just a package manager like homebrew, apt, etc. But it is tied to a python environment. If you aren’t using a virtual environment then pip will operate inside the global installation of python.

Operating within that environment has burned me several times and now I have a strict virtual environment usage policy.

But there are still things I don’t want to have to put in every virtual environment - enter pipx

What’s it do? #

When you pipx install {package} a stand alone virtual environment gets created (by default in ~/.local/pipx/venvs). THen you can install extra dependencies with pipx inject {package} {dependency}

ex. After pipx install visidata in order to open Excel files you need to pipx inject visidata xlrd

In the example with visidata, I can then use it anywhere, in any project, without re-installing with pip in every env.

Also for the formatting tools - I configure vim to run the pipx versions of them on save - this way I don’t have to put them in every project’s virtual environment!

What about pip? #

So obviously you can’t pipx everything, nor do you want to. I see it as a safe and better alternative to global package installation.

How can you then be sure that you never pip install into the global env?

Add require-virtualenv = True to your pip.conf and you’re good to go!

With that set, if you try to pip install pandas into the global env you’ll get a message like this:


~ on  (us-east-1)  NO PYTHON VENV SET
❯ pip install pandas
ERROR: Could not find an activated virtualenv (required).

End #

Disable your system pip to keep your base python safe
Use pipx for tools you want available everywhere or don’t have to need in a virtual environment!

Fx-Json

2022-04-19T00:00:00Z

fx is an interactaive JSON viewer for the terminal.

It’s a simple tool built with Charmcli’s Bubble Tea.

Installation #

The installation with go was broken for me - both via the link and direct from the repo. Now I’m not a gopher so I don’t really know how to fix that.

Luckily npm install fx also works and got me what I needed!

Usage #

Usage is simple… fx <json file>. The Github has a few other ways such as curl ... | fx etc.

Jellyfin-Media-Players

2022-04-17T00:00:00Z

I use Jellyfin at home for serving up most of our media - movies and shows etc.

My dream is to have a GPU capable of transcoding any and all of our media for smooth playback on any device… Now, I thought I’d have that by now with my Nvidia Quadro P400 however I have issues left and right with 4k content.

What can I do to still use Jellyfin but get smooth playback?

THe first answer is figure out why I suck with GPUs, but pausing that there’s shorter solutions -> namely, use a media player that’s compatabile with the encoded content!

VLC #

I’ll keep this one short - VLC is great and if you don’t need a netflix like experience, I’d recommend just using it to browse your network drives and play whatever you have

Jellyfin Web Player #

This is the reason I’m writing this post… the web player is great but not everything is supported on all devices

Jellyfin MPV Shim #

This cross-platform cast client is my answer now. You can find the project here

The installation instrauctions are super straightforward for Windows, Mac OS, or Linux.

I’m on Linux and so my install went like this:


sudo apt update
sudo apt install mpv 

pipx install jellyfin-mpv-shim
pipx inject jellyfin-mpv-shim pystray

#profit

I used pipx to install the player as I prefer it for stand alone utilities over pip installing anything globally.

Afer that I just start the player at the terminal with jellyfin-mpv-shim Then in the web browser I can cast my content to the player and bypass the web player (and thus solve much of my transcoding issues) trivially!

Typeddict

2022-04-15T00:00:00Z

Type hinting has helped me write code almost as much, if not more, than unit testing.

One thing I love is that with complete type hinting you get a lot more out of your LSP. Typing dictionaries can be tricky and I recently learned about TypedDict to do exactly what I needed!

The Problem #

It might not be straight up obvious what the problem is, especially if you don’t utilize tools like mypy or flake8 in your development.

My handy-dandy nvim-lsp gives me a lot of feedback when I’m coding and it’s immensely helpful.

So with the LSP giving me constant feedback here’s the issue:

from typing import Dict, List, Union

my_dict: Dict[str, Union[List[str], str]] = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}

my_dict["key_2"].pop()

With the above script you’ll get an annoying warning about using pop on key_2.

The Solution #

Maybe you can stomach getting yelled at by your LSP but I like complete silence if at all possible.

TypedDict was the saving grace.

from typing import TypedDict

MyDict = TypedDict("MyDict", {"key_1": str, "key_2": List[str]})

my_typed_dict: MyDict = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}


my_typed_dict["key_2"].pop()

I was able to import TypedDict from typing, mypy_extensions, and typing_extensions

With TypedDict you define your custom type, match the first argument to TypedDict with the name of the variable (idk why), then type hint each key you expect in the dict! It’s super easy and I think puts you into a position of being extremely explicit with your dictionary variables. This isn’t always desired or appropriate but in most of my use cases it is.

RTFM #

There’s other implementation of TypedDict and while writing this I saw that most of the docs define a class for the type like this:

from typing import TypedDict
class MyDict(TypedDict):
    key_1: str
    key_2: List[str]

my_dict : MyDict = {'key_1': 'val_1', 'key_2': ["ls_1", "ls_2"]}

pep docs

mypy docs

Home-Server-Refactor

2022-04-10T00:00:00Z

My current homelab setup is not great but it works…

Proxmox on PowerEdge R610

I boot off an SD card and have 1 SSD and 5 HDDs configured as a JBOD array using a Dell H700 SAS controller. I cannot boot from a disk using this controller and I can’t get the firmware configured in a way to allow it. So I have 1 SSD as a ZFS array that I’ve been putting my VM images on, and the 5 HDDs are passed through to a TrueNAS VM where I handle all the ZFS stuff there… kind of meta because I then attached those drives to Proxmox as a CIFS share.

TrueNAS on dedicated box

I have an on-prem backup that is just an old desktop running TrueNAS I regularly backup the 5 disk RAIDZ2 array from my Proxmox host (managed by a TrueNAS VM) to this backup box

Currently there is nothing else running on this machine since it’s my “backup”

Jellyfin

I was HWA for Jellyfin, but hardware passthrough on the R610 is finicky or broken so Jellyfin is running on an Ubuntu host.

I could put UBuntu on the R610 and give up “true virtualization”. Then I’d manage the SMB share myself. If I do that then I would get rid of “users” I think, ie. basically forgo least-priviledges since I’m not sure how hard that is to manage.

On the other hand, direct access to the smb config might make it easier?

I have the media array on Jellyfin box setup as NFS which was really easy with ZFS… I think SMB would be just as easy.

Plan of attack…

Move all vm disks to individual datasets on the NAS
Backup docker data… not sure how well this will work, maybe just start over?
Clean up Ansible playbooks on the user side of things - stick with neville vs just using my own name?
Install Ubuntu 20 or 22 on a 2.5" drive that I’ll toss in this SSD enclosure (or a usb thumb stick?)
Re-deploy everything with ansible-playbook and configure…

Configuration… #

THE FREAKING NAS -> just import zfs array and configure SMB?

1.~~ Nextcloud users and connections.. might be able to just copy the data folder? not sure about the database… try spinning it up in the sandbox vm and see if stuff is there ~~ 2.~~ *arr suite, media profiles and connections to transmission… nothing major~~ 3. transmission - should be deploy and go 4. ombi and jackett should also just work after some config again 5. ~~traefik should just work~~ 6. ~~try to bring up pi-hole from the vm that’s already running~~ 7. ~~heimdall will hopefully just be copying the data folder from the existind docker one’~~ 8.~~ booksonic can be reconfigured easily~~ 9. ~~portainer… hopefully just copying data folder over?~~ 10. ~~littlelink~~, small-group-notes, and blog (at home) will need manually re-deployed once Ubuntu is installed bare-metal

BIG BIG BIG TODOS #

Sanoid/syncoid! Get snapshots going and backups configured with on prem TrueNAS
Wireguard setup on DA.
network share on printer for paperless ~~4. update peperless in ansible-nas~~
~~Just deploy paperless manually… monitor/manage with portainer~~
booksonic not seeing audiobooks/podcasts
need a smb user to map nas/documents to the printer for paperless
wireguard setup now on kps phone, desktop, server (and backup truenas?), and dad’s pi
~~verify lan services work~~
~~Tdar so Jellyfin can work better~~

Snapshot business might be cause of all the docker containers and docker using ZFS backend… take everything down and try removing

file browser - currently I just one-clicked in portainer, I want to make a stack with my own config file which I’ll rip from techno tip and then add my traefik lables too

Forget filebrowser - going to just use Nextcloud for how it’s supposed to be used. 3. Need to organize those files in nextcloud

CHECK THIS -> ran as ‘cp’ utility in tmux window, no progress bars or anything. it’s in ansible-nas session -> window 3 #

Olivet bible stuff going to /tmp/olivet/ -> will move this to nextcloud, ideally by the app via appimage so that the db updates and I don’t have to run that occ script I wnat to organize “home” still in nextcloud

setup Sanoid

clean up bitwarden learn nextcloud sharing -> maybe just give a link to grandma? rest of todos -> document db and sanoid + zfs.rent

Check on mom’s will do media thing for church - split vocals on mp3/4

permission-data playbook changes everything to ansible-nas:ansible-nas but then samba task will re-permission some stuff to root:users… this looks fine I had to add group to the samba config in my playbook to get user auth to work with samba This isn’t fully working… it works from cli but my python process can’t write to a folder in dump after 777…. need to learn more? So I can make a file after adding the ansible-nas group to config, but I still cannot make a directory on the smb mount…

ADDING inherit permission = yes under [global] in the smb.conf worked!

still not working from printer… I think what I want is to setup 2 scan options - single docs right to paperless, or combined scans to dump, then manually split and send to paperless

And-vs-&

2022-04-06T00:00:00Z

I often struggle to remember the correct way to do and type comparisons when working in pandas.

I remember learning long long ago that and and & are different, the former being lazy boolean evaluation whereas the latter is a bitwise operation.

I learned a lot from this SO post

Lists #

Python list objects can contain unlike elements - ie. [True, 'foo', 1, '1', [1,2,3]] is a valid list with booleans, strings, integers, and another list. Because of this, we can’t use & to compare two lists since they can’t be combined in a consistent and meaningful way.

However we can use and since it doesn’t do bitwise operations, it just evaluates the boolean value of the list (basically if it’s non-empty then bool(my_list) evaluates to True)

Here’s an example:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list = [1, "2", "foo", [True], False]

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(my_list)
True

If we compare my_list with another_list using and then the comparision will go:

if bool(my_list):
    if bool(another_list):
       <operation> 
    else:
       break

Let’s see another example:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ another_list = [False, False]

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list and another_list
[False, False]

bool(my_list) evaluated to True, and bool(another_list) also evaluated to True even though it’s full of False values because the object is non-empty.

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if my_list and another_list:
...:     print("foo")
foo

So using and in this case results in a True conditional, so the print statement is executed.

Feels kind of counter-intuitive at first glance, to me anyways…

However, we can’t use & because there isn’t a meaningful to do bitwise operations over these two lists:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list & another_list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-19-a2a16cebb3da>:1 in <cell line: 1>                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: unsupported operand type(s) for &: 'list' and 'list'

Numpy #

numpy arrays are special and they have a lot of fancy vectorization utilities built-in which make them great and fast for mathematical operations but now our logical comparisons need to be handled with a different kind of care.

First thing though - without some trickery they do not hold mixed data types like a list does (necessary, I think, for the vectorized optimization that numpy is built on top of)

With that out of the way here’s the main thing for this post, we can’t just evaluate the bool of an array - numpy says no no no.


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr = np.array(["1", 2, True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr
array(['1', '2', 'True', 'False'], dtype='<U21')

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(arr)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-25-4e8c5dd85b93>:1 in <cell line: 1>                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This means that using and with numpy arrays doesn’t really make sense because we probably care about the truth value of each element (bitwise), not the truth value of the array.

Notice that when I print arr all the elements are a string - and the dtype is <U21 for all elements.

This is not how I instantiated the array so be aware of that behavior with numpy.

<U21 is a dtype expressing the values are ‘Little Endian’, Unicode, 12 characters. See here for docs for docs

So for logical comparisions we should look at the error message then… Our handy error message says to try any or all

Because the datatypes in this example are basically strings, using arr.any() will result in an error that I do not fully understand, but any(arr) and all(arr) work…

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr.any():
...:     print("foo")
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-48-25ecac52db96>:1 in <cell line: 1>                                              │
│ /home/u_paynen3/personal/sandbox/.venv/sandbox/lib/python3.8/site-packages/numpy/core/_methods.p │
│ y:57 in _any                                                                                     │
│                                                                                                  │
│    54 def _any(a, axis=None, dtype=None, out=None, keepdims=False, *, where=True):               │
│    55 │   # Parsing keyword arguments is currently fairly slow, so avoid it for now              │
│    56 │   if where is True:                                                                      │
│ ❱  57 │   │   return umr_any(a, axis, dtype, out, keepdims)                                      │
│    58 │   return umr_any(a, axis, dtype, out, keepdims, where=where)                             │
│    59                                                                                            │
│    60 def _all(a, axis=None, dtype=None, out=None, keepdims=False, *, where=True):               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UFuncTypeError: ufunc 'logical_or' did not contain a loop with signature matching types (None, <class 'numpy.dtype[str_]'>) -> None

sandbox NO VCS  via 3.8.11(sandbox) ipython

❯ if all(arr):
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if any(arr):
...:     print("foo")
foo

Let’s change the example to just use integers and see what happens:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr2 = np.array([1, True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr2
array([1, 1, 0])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr2.any():
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr2.all():
...:     print("foo")

Ah, now some sanity… First, the booleans are stored as integers, which based on this discussion makes sense. Next we check if any values (this is a bitwise operation) are True, which we see they are so the conditional evaluates to True. Howver, if we check that all values are True we see they aren’t, the last value is False or 0 so the conditional fails.

This is a different way to evaluate logical conditions than with lists and it’s because of the special nature of numpy arrays that allows them to be compared bitwise but on the flip side, there isn’t a meaningful way to evaluate the truth value of an array.

Pandas #

Now for pandas, which under the hood is a lot of numpy but not fully. pandas.Series objects can hold mixed data types like lists, however to logically evaluate truth values we have to treat them like numpy arrays.


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ s = pd.Series([1, "foo", True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ s

0        1
1      foo
2     True
3    False
dtype: object

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(s)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-60-68e48e81da14>:1 in <cell line: 1>                                              │
│ /home/u_paynen3/personal/sandbox/.venv/sandbox/lib/python3.8/site-packages/pandas/core/generic.p │
│ y:1527 in __nonzero__                                                                            │
│                                                                                                  │
│    1524 │                                                                                        │
│    1525 │   @final                                                                               │
│    1526 │   def __nonzero__(self):                                                               │
│ ❱  1527 │   │   raise ValueError(                                                                │
│    1528 │   │   │   f"The truth value of a {type(self).__name__} is ambiguous. "                 │
│    1529 │   │   │   "Use a.empty, a.bool(), a.item(), a.any() or a.all()."                       │
│    1530 │   │   )                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Just like with numpy, we can’t evaluate the truth value of the series in a meaningful way, but bitwise operations make perfect sense…


❯ if s.any():
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if s.all():
...:     print("foo")

I thought this was about and and &…

Right, so recall that and is a lazy boolean evaluation (ie. it evaluates the ’truth value’ an object) whereas & does bitwise comparison.

What we see then with pandas and numpy is that if we want to do logical comparisons, we need to do them bitwise, ie. use &.

Keep in mind though that the data types make a big deal - we can’t use & with strings because the bitwise operation isn’t supported, for strings we need to use the boolean evaluation.

The Original Point #

My main use case for this is finding elements in a dataframe/series based on 2 or more columns aligning row values…

Say I have a dataframe like this:


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ df

   s s2   s3
0  1  0  foo
1  1  a  bar
2  1  b  baz
3  2  a  fee
4  2  0   fi

Example use case is I want to get the values in s3 where s is 1 and s2 is ‘a’. ie. I’m just after bar for now…

Up until now I’ve always just tried df.s3[(df.s == 1) and (df.s2 == "a")] the first time and every single time I’ve gotten this error that I just haven’t ever fully understood:

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

But after this deep dive I think I’ve grasped that and doesn’t actually do what I want here, and in order to do the bitwise comparision I need to use &

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ df.s3[(df.s == 1) & (df.s2 == "a")]

1    bar
Name: s3, dtype: object

End #

Hopefully this set of ramblings brings some clarity to and and & and you can Google one less error in the future in your logical comparison workflows 😄

File-Length

2022-04-04T00:00:00Z

I have a specific need for counting the number of lines in a file quickly. At work we use S3 for data storage during our Kedro pipeline development, and in the development process we may end up orphaning several datasets. In order to keep our workspace clean I have a short utility that compares the datasets in a Kedro DataCatalog with the files in the relevant S3 location.

To get that list I run an internal tool like this:

kedro our-liter | grep s3 >> orphaned_datasets.txt

This simply parses our internal linter for the lines releated to my s3 linter utility and pipes those lines to a file.

To get a quick idea of how out of wack a pipeline is I could open the text file in vim, git it with the G and see what line number I’m on but I’m way too lazy for that…

AWK #

awk 'END {print NR}' orphaned_datasets.txt gives me the number of lines and I can alias this to whatever feels appropriate in my zshrc!

built-ins for the win!

Ipython-Prompt

2022-04-02T00:00:00Z

I have a post on starship where I have some notes on how I use starship to make my zsh experience great with a sweet terminal prompt.

Now… I spend quite a bit of time in ipython every day and I got kind of sick of the vanilla experience and wanted something that more closely matched my starship prompt.

There’s more to customizing ipython I know for sure but here’s 2 things I have going for me…

I use rich authored by @Will McGugan which makes much of my ipython experience great. I won’t write about that here but you can find my rich config here
I used pygments to customize the ipython prompt with my ipython_config.py and a startup script, next to my rich one, called 99-prompt.py.

The scripts inside ~/.ipython/<profile>/startup are executed in lexigraphical order, so it’s nice to name things in the 10’s to give room for adding scripts in between others down the line.

My prompt #

My zsh prompt looks a little something like this:

And after my ipython customiztion it currently (subject to much change but this is as of my dotfiles commit #d22088f6be81a58b5f7dfb73b7a4088cbdd9fece on main).

Now in ipython I have an indicator of my working directory, git branch, python environment, and a note that I’m in ipython and not zsh. I also configured my prompt to warn me if I’m not in a git directory!

All in all the customization isn’t too bad with just 2 specific files.

ipython_config.py #

There’s several use cases for ipython_config.py files in several areas on a pc - sometimes you want a common config across users, so you’d drop one in /etc/ipython and othertimes you have your own which is probably at ~/.ipython

My ipython config mostly has colors defined on pygment tokens plus a few autorun commands and pyflyby (see my friend Waylon’s post on pyflyby here)

I wanted to match my ipython somewhat to my tmux and vim color schemes, which I model after the vim-airline theme night owl.

After picking some some colors and saving variables it’s a matter of setting colors per token and then referencing those tokens in your version of 99-prompt.py.

You can check out my ipython_config.py here

For example, I can set Token.Name.Function to black, and in ipython then a function’s definition will appear in black text. I set mine to cyan to match my theme.

For the prompt colors just match the keyword in c.TerminalInteractiveShell.highlighting_style_overrides with what is referenced inside 99-prompt.py

For example, Token.Prompt is set to bold grey which gives me the bold chevron symbol you see in the above image that looks like my zsh prompt

Then in 99-prompt.py I have this set for the prompt:

Token.Prompt "❯ "

99-prompt.py #

You don’t need to name your script 99-prompt.py, but I wanted to know that it was for my prompt and I wanted it executed last so it made sense.

Here I have MyPrompt class with the prompt symbol defined as above and several other things…

class MyPrompt(Prompts):
    def in_prompt_tokens(self, cli=None):
        return [
            (Token, ""),
            (Token.OutPrompt, Path().absolute().stem),
            (Token, " "),
            (Token.Generic.Subheading, get_branch()[0]),
            (Token, " "),
            (Token.Generic.Heading, get_branch()[1]),
            (Token, " "),
            (Token.Name.Class, "via " + get_venv()),
            (Token, " "),
            (Token.Name.Entity, "ipython"),
            (Token, "\n"),
            (
                Token.Prompt
                if self.shell.last_execution_succeeded
                else Token.Generic.Error,
                "❯ ",
            ),
        ]

Notice I have 2 custom functions here, get_branch and get_venv which grab some git info and python env info and return strings I can dump into my prompt as shown above.

To finish you drop ip = get_ipython() and ip.prompts = MyPrompt(ip) at the bottom of your prompt script and you should be in custom prompt city!

End #

This is more or less notes for myself on how this works - drop by my ipython config in my dotfiles repo to see my full configs for ipython!

Vim-Spell-Check

2022-04-01T00:00:00Z

Did you know you can spell check in Vim?!

Vim Spell check

Without... #

Here is a missspelled word.

  <h3>With!</h3>
  <p>Here is a <u>missspelled</u> word.</p>

What is this magic??? #

set: spell spelllang=en_us

Custom words? #

Sometimes there’s things that are words to you but not the default spell checker…

Common example: package names!

plotly, streamlit, psutil, etc etc…

You can easily add these to your vim config by hitting zw ontop of the word!

Polybar-01

2022-04-01T00:00:00Z

polybar is an awesome and super customizable status bar for your desktop environment.

I use it with i3-gaps on Ubuntu for work and it makes my day just that much better to have a clean and elegant bar with the things in it that I care about.

The GitHub has all the instructions you’d need to install and get started with an example.

I want to make some notes about how I use polybar and customize it.

Organization #

First of all, I recently moved my polybar config out of one config file into a modular structure that keeps my config files small and easiser to edit.

You can find my config here

The apps or services you put into polybar are called modules. I have moved all of my modules into their own config files and I source them with one centralized include-modules.ini config.

This separation also makes it easier for me to keep my home and work polybars as in sync as possible without duplicating a ton of config!

./polybar
├── colors.ini
├── config.ini
├── fonts.ini
├── home-modules.ini
├── include-modules.ini
├── launch.sh
├── modules
│   ├── aws.ini
│   ├── battery.ini
│   ├── bluetooth.ini
│   ├── cisco.ini
│   ├── cpu.ini
│   ├── date.ini
│   ├── eth.ini
│   ├── eth_work.ini
│   ├── i3.ini
│   ├── memory.ini
│   ├── nm-editor.ini
│   ├── powermenu.ini
│   ├── pulseaudio-control.ini
│   ├── pulseaudio.ini
│   ├── rofi.ini
│   ├── vpn.ini
│   └── wlan.ini
└── work-modules.ini

1 directory, 24 files

To break this down there are several configs to see:

colors.ini is what you’d expect - a set of defined colors like foreground, underline, etc.
config.ini is the general polybar config file where bars are defined. Currently in mine there is a work and home bar defined with the modules sourced in from the explicit config files.
fonts.ini is like colors.ini -> you put fonts here. I recommend using a font patched with NerdFont so you get fancy icons! (I use JetBrains Mono)
include-modules.ini is where I list out all the config files in modules/ so I can basically source just the include-modules.ini without explicitly sourcing every module’s config in every polybar defintion.
launch.sh is a simple shell script to launch the polybar! You’ll see mine takes multiple monitors into consideration which I manage via environment variables setup in my .zshenv file that is different for my work and home setups.
Finally there are home-modules.ini and work-modules.ini which is where, for each of my bars, I define which modules I want!

Config #

My config.ini file has 2 bar definitions in it - here’s my home one:

include-file = $DOTFILES/polybar/include-modules.ini

[bar/home]
monitor = ${env:MONITOR:}
width = 100%
height = 25
radius = 8.0
fixed-center = true
bottom = false

background = ${colors.background}
foreground = ${colors.foreground}

include-file = $DOTFILES/polybar/fonts.ini
include-file = $DOTFILES/polybar/home-modules.ini

It should be easy to follow - I bring in the include-modules, set a few colors for the bar like background and foreground which are sourced by the colors.ini, and finally bring in my fonts and home modules via their config files!

It’s super easy to then change one or two things in the appropriate places rather than combing through one massive config. This also makes it easy for me to seperate my work and home setups.

Modules #

There are several builtin modules, like wlan which gives your wifi status right there in polybar.

You can also make custom ones. A big-time custom one for me is an indicator of whether or not I have an active AWS token for working with the aws cli.

This is defined in modules/aws.ini and it looks like this:

[module/aws]
interval = 5.0
type = custom/script
exec = has_aws_token
click-left = $HOME/.local/bin/auto_get_aws_token
click-right = rm -rf ~/.aws/credentials

Every 5 seconds my has_aws_token script is ran. That script looks like this:

#!/bin/bash
source auto_proxy
aws sts get-caller-identity &> /dev/null && echo "%{T5}%{F#00ff00}  %{F-}%{T-}"  ||( echo "%{T5}%{F#ff0000} %{F-}%{T-}" )

See how the script echos out a colored icon to indicate the status of my token -> that icon is displayed in the polybar so I have real-time (5 second latency) status of whether or not I can do things in my AWS environment.

In the module I also configured actions for click-left and click-right which are as straight forward as could be.

My issues with i3 #

There’s a few things to be considerate of if you use i3 such as needing a workaround for a centered bar that is not the full width of the monitor. Polybar can look really nice by not taking up the full width of the bar which you can configure in config.ini with these options:

width = 90%
offset-x = 5%  # set to (100 - width) / 2

However due to an issue with polybar and i3 you need to also set override-redirect = true. BUT then you’ll notice that the bar overlaps your i3 windows… ARGH! what do we do?

Quick work around is to set gaps top in your i3 config if you use i3-gaps… if not? well, idk… use gaps… lol

However this introduces another issue - which is then full screen windows will have polybar sitting on top of them…

This isn’t necessarily a deal breaker, but for me it’s worth it to just have the bar go 100% width.

FIN #

There’s a tiny intro to polybar and how I organize my config files so things are easy to edit and manage! Feel free to grab mine and try it out!

Deques

2022-03-31T00:00:00Z

I am working on a project to create a small system monitoring dashboard using the python psutil library.

The repo is here (if you want actual system monitoring please use netdata).

I’m using streamlit and plotly for the webserver, design, and plotting at the moment.

My Use Case #

I needed a way to refresh my plotly charts with a fixed window of time so that I’m able to just see relevant recent data instead of cramming all data for all time into one plot that’s 500 pixels wide…

Checking the length of arrays or lists every time I get a new piece of data feels kind of dumb and I thought “python must have a way to do this”…

“This” meaning, update values in a fixed length array without reallocating memory or recreating a copy of the list

Deques #

Enter the deque. It means “double ended queue” and is in general an Iterable that you can append values to either side or pop values from either side.

The init signature is straightforward enough and I’m sure there’s more to them than I know yet but here’s how I use it…

from collections import deque

my_deque = deque([1,2,3])

This gives us my_deque, created from an iterable, with several familiar methods like index, extend, append, etc. However there’s some new ones too such as appendleft and popleft.

my_deque.appendleft('a')
print(my_dequqe)
>>> deque(['a', 1, 2, 3])

my_deque.popleft()
>>> 'a'

<!--markata-attribution-->
print(my_deque)
>>> deque([1, 2, 3])

These are handy ways to manipulate the iterable that I needed for the arrays I plot with plotly!

See my follow-up to this on using Deques with plotly and streamlit to create a quick “dashboard” with live streaming data!

follow-up

Plotly-And-Streamlit

2022-03-31T00:00:00Z

Streamlit #

I use streamlit for any EDA I ever have to do at work. It’s super easy to spin up a small dashboard to filter and view dataframes in, live, without the fallbacks of Jupyter notebooks (kernels dying, memory bloat, a billion “Untitled N.ipynb” files, etc.)

At the highest level, streamlit lets you write a python script and call streamlit run my_script.py which will open up a web server with your streamlit stuff. The dashboard refreshes whenever you change the script so you can add capabilities in real time, super fast!

I’ll show an example of using streamlit and plotly to make a live dashboard to monitor system memory usage with psutil. This is apart of my posts on psutil and deques…

example at the bottom!

Plotly #

I’m not going to make a big time intro to plotly here - there’s a billion resources on the interwebs and the docs are really good.

Suffice it to say it’s my goto plotting library for basically any and all needs. I’m currently exploring it for live data streaming as I’m not sure it’s the best solution but it’s the one I’m familiar with.

For my not-netdata project of visualizing live system resource data I first need a way of appending data and popping data in and out of an array at every data refresh cycle to keep my plots looking nice with a fixed time window.

See deques for a short intro to the datatype I’m using.

First step is to initialize some objects to store data in.

data: Dict[str, MutableSequence[Optional[float]]] = defaultdict(deque)

arr_size = 10

data["time"] = deque([None] * arr_size)
data["used_memory"] = deque([None] * arr_size)

data is a dictionary that I’ll store deques in. The dictionary keys will be the type of data, in this case time and used_memory.

I fix an array size, arr_size to just 10 for now

Then I initialize the values for time and used_memory as deques of length arr_size. Simple enough!

Next is to fill those deques with some relevant data. I’m not actually sure if this is the best way to do this but here’s what I have done so far:

def refresh_data():
    global data
    memory = psutil.virtual_memory()

    data["time"].append(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
    data["used_memory"].append(memory.used // (1024**3))

    data["time"].popleft()
    data["used_memory"].popleft()

If you ignore my usage of global you’ll see that I can just append to each deque like it was a list.

But then to keep the relevant data in the deque, and to keep the length fixed, I simply popleft to remove the oldest datapoint!

A trivial dashboard #

Now I’ll prove just how easy it is to get a live data dashboard up and running with just a few lines of code thanks to streamlit!


if __name__ == "__main__":
    st.header("memory chart")
    stats = st.empty()
    while True:
        refresh_data()
        stats.plotly_chart(
            px.line(
                data,
                x="time",
                y="used_memory",
                title=f"Memory usage stored in a deque!",
               )
            )
        time.sleep(0.5)

st is the streamlit alias (imports shows at the bottom full example). st.header puts a nice header on the page. st.empty initializes an empty streamlit container in which we’ll put a plotly.express figure.

At each iteration we’ll refresh_data() which appends and pops data in the deques in the data dictionary. Then we update the stats container with a plotly graph and the refresh happens seamlessly.

All in all the script looks like this:


from collections import defaultdict, deque
import time
from typing import Dict, MutableSequence, Optional

from plotly import express as px
import psutil
import streamlit as st

data: Dict[str, MutableSequence[Optional[float]]] = defaultdict(deque)

arr_size = 10

data["time"] = deque([None] * arr_size)
data["used_memory"] = deque([None] * arr_size)


def refresh_data():
    global data
    memory = psutil.virtual_memory()

    data["time"].append(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
    data["used_memory"].append(memory.used // (1024**3))

    data["time"].popleft()
    data["used_memory"].popleft()


def memory_chart():
    fig = px.line(
        data,
        x="time",
        y="used_memory",
        title=f"Memory usage stored in a deque!",
    )
    return fig


if __name__ == "__main__":
    st.header("memory chart")
    stats = st.empty()
    while True:
        refresh_data()
        stats.plotly_chart(memory_chart())
        time.sleep(0.5)

You can save this as my_dash.py and run with streamlit run my_dash.py and should see something like the following!

Starship

2022-03-25T00:00:00Z

If you spend time in the terminal then you’ll want it to look somewhat pleasing to the eye. I used to ssh into servers with no customization, use vi to edit a file or two, then get back to my regularly scheduled programming in VS C**e…

One of the first steps for me loving my terminal was a beautiful prompt…

Prompt #

The default sh/bash/zsh prompts are… to put it lightly… garbage… I can’t speak for other shells like fish simply because I do not use them but let me justify my trash talk.

Here’s the default sh prompt…

Then switching to zsh you get something marginally better (plus tab completion!)

But this still is super gross… there’s nothing to indicate file types and no status information readily available (ie. git status etc.)

Oh-My-Zsh! #

Now there are several ways to make your prmompt nicer depending on your shell (terminal emulator plays a role too). Now I use zsh and there’s a great tool out there oh-my-zsh that brings a crazy amount of customization to the terminal experience.

I do not use oh-my-zsh for theming though and that’s simply because of my other choices - I use kitty themes since I understood the implementation better. Kitty themes though - do not give me a nice prompt.

The default prompt you get with oh-my-zsh themes isn’t bad though (and you can pick from several default themes)…

Notice that you get some nice coloring and some default git status stuff, mainly the branch you are on. There’s plugins to show you more and that’s all well and good, but again it’s not my choice…

If I don’t use this then what’s my goto?

Starship #

starship is a cross-shell prompt with nice default and super easy customizaton!

To get started click that link and follow the “Getting Started” button - it’s incredibly fast to get up and running with sane defaults.

The default starship config is plenty nice but I got a little tired of emojis in my prompt and wanted to switch to icons instead…

To get started with your own customizaton you add a starship.toml file to ~/.config My starship config is found here.

Note you need a font installed patched with nerdfonts - I use JetBrains Mono

Now I have a beautiful prompt with relevant information that’s a dream to look at!

I have configured my starship to show me relevant git status options (stashes, untracked files, etc etc.) I also have starship show me if I’m in a git repo, what branch I’m on, if I’m in a python project and if so what virtual environment is active. I do some work in AWS at work and so I have starship show me if my aws cli is configured to the right region for whichever project I’m in!

There’s a billion more options and after a few minutes of play it becomes really easy and intuitive to customize colors, icons, etc.

self-hosted-media

2022-03-24T00:00:00Z

Self-hosting 1 or several media servers is another common homelab use-case. Getting content for your media servers is up to you, but I’ll show a few ways here to get content somewhat easily!

YouTube Disclaimer at Bottom

you-get #

you-get is a nice cli for grabbing media content off the web.

Installation #

pip install you-get or use ad-hoc with pipx run you-get <url>

Usage #

For example if I wanted to catch up on ancient Chinese military tactics I may go for The Art of War off the Internet Archive…

sandbox  🌱 main 🗑️  ×3🛤️  ×6via 🐍 v3.8.11 (sandbox)  took 15s
❯ you-get https://archive.org/details/art_of_war_librivox -i
Site:       Archive.org
Title:      The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archive
Type:       MP3 (audio/mpeg)
Size:       3.87 MiB (4055167 Bytes)

the -i is showing me the info of what would be downloaded without the flag (it’s like a dry run)

sandbox  🌱 main 🗑️  ×3🛤️  ×6via 🐍 v3.8.11 (sandbox)
❯ you-get https://archive.org/details/art_of_war_librivox
Site:       Archive.org
Title:      The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archive
Type:       MP3 (audio/mpeg)
Size:       3.87 MiB (4055167 Bytes)

Downloading The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archi.mp3 ...
 100% (  3.9/  3.9MB) ├████████████████████████████████████████████████████████████┤[1/1]  917 kB/s

Now I can toss that mp3 onto my booksonic server and study for world domination while I do the dishes!

pytube #

pytube is a python implementation of a youtube downloader that works at the command line or in python!

Installation #

docs

pip install pytube

Usage #

pytube has a lot of functionality, but a quick one would be the --list so you can see what qualities are available

sandbox   main ️  ×3️  ×7via  v3.8.11 (sandbox)  took 2m49s
❯ pytube https://www.youtube.com/watch\?v\=LDU_Txk06tM  --list
Loading video...
<Stream: itag="17" mime_type="video/3gpp" res="144p" fps="8fps" vcodec="mp4v.20.3" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="22" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.64001F" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="313" mime_type="video/webm" res="2160p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="401" mime_type="video/mp4" res="2160p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">
<Stream: itag="271" mime_type="video/webm" res="1440p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="400" mime_type="video/mp4" res="1440p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">
<Stream: itag="137" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="avc1.640028" progressive="False" type="video">
<Stream: itag="248" mime_type="video/webm" res="1080p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="399" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="av01.0.08M.08" progressive="False" type="video">
<Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">
<Stream: itag="247" mime_type="video/webm" res="720p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="398" mime_type="video/mp4" res="720p" fps="30fps" vcodec="av01.0.05M.08" progressive="False" type="video">
<Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">
<Stream: itag="244" mime_type="video/webm" res="480p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="397" mime_type="video/mp4" res="480p" fps="30fps" vcodec="av01.0.04M.08" progressive="False" type="video">
<Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e" progressive="False" type="video">
<Stream: itag="243" mime_type="video/webm" res="360p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="396" mime_type="video/mp4" res="360p" fps="30fps" vcodec="av01.0.01M.08" progressive="False" type="video">
<Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d4015" progressive="False" type="video">
<Stream: itag="242" mime_type="video/webm" res="240p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="395" mime_type="video/mp4" res="240p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">
<Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400c" progressive="False" type="video">
<Stream: itag="278" mime_type="video/webm" res="144p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="394" mime_type="video/mp4" res="144p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">
<Stream: itag="139" mime_type="audio/mp4" abr="48kbps" acodec="mp4a.40.5" progressive="False" type="audio">
<Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio">
<Stream: itag="249" mime_type="audio/webm" abr="50kbps" acodec="opus" progressive="False" type="audio">
<Stream: itag="250" mime_type="audio/webm" abr="70kbps" acodec="opus" progressive="False" type="audio">
<Stream: itag="251" mime_type="audio/webm" abr="160kbps" acodec="opus" progressive="False" type="audio">

pytube <url> --itag <> will download the specific itag from the list.

Notice that some itags are videos and others audio - so you can download just the music of a YT video.

pytube also works in python…

sandbox ↪ main v3.8.11 ipython
❯ from pytube import YouTube

sandbox ↪ main v3.8.11 ipython
❯ [x for x in YouTube("https://www.youtube.com/watch?v=LDU_Txk06tM").streams]

[
    <Stream: itag="17" mime_type="video/3gpp" res="144p" fps="8fps" vcodec="mp4v.20.3" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="22" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.64001F" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="313" mime_type="video/webm" res="2160p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="401" mime_type="video/mp4" res="2160p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">,
    <Stream: itag="271" mime_type="video/webm" res="1440p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="400" mime_type="video/mp4" res="1440p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">,
    <Stream: itag="137" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="avc1.640028" progressive="False" type="video">,
    <Stream: itag="248" mime_type="video/webm" res="1080p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="399" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="av01.0.08M.08" progressive="False" type="video">,
    <Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">,
    <Stream: itag="247" mime_type="video/webm" res="720p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="398" mime_type="video/mp4" res="720p" fps="30fps" vcodec="av01.0.05M.08" progressive="False" type="video">,
    <Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">,
    <Stream: itag="244" mime_type="video/webm" res="480p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="397" mime_type="video/mp4" res="480p" fps="30fps" vcodec="av01.0.04M.08" progressive="False" type="video">,
    <Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e" progressive="False" type="video">,
    <Stream: itag="243" mime_type="video/webm" res="360p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="396" mime_type="video/mp4" res="360p" fps="30fps" vcodec="av01.0.01M.08" progressive="False" type="video">,
    <Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d4015" progressive="False" type="video">,
    <Stream: itag="242" mime_type="video/webm" res="240p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="395" mime_type="video/mp4" res="240p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">,
    <Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400c" progressive="False" type="video">,
    <Stream: itag="278" mime_type="video/webm" res="144p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="394" mime_type="video/mp4" res="144p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">,
    <Stream: itag="139" mime_type="audio/mp4" abr="48kbps" acodec="mp4a.40.5" progressive="False" type="audio">,
    <Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio">,
    <Stream: itag="249" mime_type="audio/webm" abr="50kbps" acodec="opus" progressive="False" type="audio">,
    <Stream: itag="250" mime_type="audio/webm" abr="70kbps" acodec="opus" progressive="False" type="audio">,
    <Stream: itag="251" mime_type="audio/webm" abr="160kbps" acodec="opus" progressive="False" type="audio">
]

YouTube Frontends #

There’s 2 really good options for self-hosting a YT front-end…

Tube Archivist

YouTubeDL-Material

They have their pros and cons. You can also build yourself with the above utilities and use Plex or Jellyfin to serve up videos…

Your self-hosting journey is up to you!

YouTube #

Downloading YouTube videos is a bit of a sore topic… Mainly you don’t to hurt creators who rely on YT ad revenue for their livlihood.

Then again, maybe you’re a vigilante who knows that YT also monetizes videos for their own gain and that the creators don’t see that money either!

The solution is pretty easy and is 2-fold…

Download YT videos
Personally support the content creators you follow via paypall, patreon, or whatever else they might have set-up…. even a buck or two a month is more than they’d get from your ad revenue explicitly plus it all goes to them!

Skimpy

2022-03-23T00:00:00Z

EDA #

I work with data a lot, but the nature of my job isn’t to dive super deep into a small amount of datasets, I’m often jumping between several projects every day and need to just get a super quick glance at some tables to get a high level view.

When I’m doing more interactive exploration I’ve graduated from Jupyter cells with df_N.head() to using an amazing tool called visidata

However, Visidata is a terminal based application and I’m often in an iPython console… so is there a way to move even faster for my super quick summary views?

yes!

Skimpy #

First thing to do is pip install skimpy and then it’s as easy to get some summary stats with skimpy <data>

This is super nice for seeing missing values in particular as well as the distribution shape of the data.

iPython #

But wait… I just said I’m normally in an iPython session but that was called from zsh.. If I’m hoping back into zsh I might as well use visidata to have more powerful exploration at my fingertips. So… can I see this table quickly without breaking my iPython workflow?

Of course you can with magic!

The above assumes you’re looking at a file, like you would in the terminal. skimpy works even better in iPython with from skimpy import skim then pass any DataFrame to skim!

Truenas-And-Wireguard

2022-03-23T00:00:00Z

NAS #

One of the most common use cases for self-hosting anything is a file share system. I have been a fan of TrueNAS for a while. I currently use TrueNAS Core at home, and plan to consider transitioning to TrueNAS Scale soon.

Blog post forthcoming on that!

VPN #

I don’t write a ton about homelabbing yet but one of the first things to set up whether you have a massive homelab or a little raspberry pi would be a self-hosted VPN. I have notes on wireguard here.

I finally have a need to put my TrueNAS box on my wireguard network in order to transfer files to other devices that are outside my LAN.

There is a handy tutorial on setting this up via the GUI here. They walk you through setting up 2 tunables wireguard. One to enable the the connection and one to setup the network interface. Next you create a Post Init script which will check that the right directories exist and will copy the wireguard config that hasn’t been made yet to the proper location and finally starts wireguard.

The above is just copy/paste from the tutorial but the final step, although not super tricky, isn’t the same for everyone as it depends on your wireguard config and network setup.

The final step is for you create the relevant wireguard config (see my post but I just use pivpn -a) and send that config over to your TrueNAS box!

For me this final work flow looked like this:


ssh user@vpn-server

pivpn -a

<follow prompts>

scp ~/configs/truenas.conf root@<truenas ip>:/root/wg0.conf

Bug? #

The script in the tutorial for starting the wireguard service is straight forward enough however my TrueNAS box didn’t get the wireguard interface up and running on reboot.

Easy enough solution:

ssh root@<truenas ip>
/usr/local/etc/rc.d/wireguard start

We can check that the interface is now working with ifconfig and should see something like the following:

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
        options=80000<LINKSTATE>
        inet x.x.x.x --> x.x.x.x netmask 0xffffff00
        groups: tun
        nd6 options=101<PERFORMNUD,NO_DAD>
        Opened by PID 1325

Gotcha! #

Here’s another thing I had to navigate when setting this up.

My pivpn configuration sets the endpoint for my wireguard clients to paynepride.com:<port forwarded to wireguard server>

What this means is that when I check the wireguard config for TrueNAS which is on my home network it resolves paynepride.com to the server I have running my reverse proxy. However! My reverse proxy is not responsible for my vpn traffic and so the traffic was just getting dropped - instead I needed to change the wireguard config just for my truenas box to piont to the local address of my vpn server. This really threw me for a loop today but is just another reminder that if you have network problems it’s probably DNS…

Give her the ‘ol reboot and now if I check wg I should see some traffic on my wireguard tunnel!

Pyclean

2022-03-22T00:00:00Z

I like to keep my workspace clean and one thing that I don’t personally love looking at is the __pycache__ directory that pops up after running some code. The *.pyc files that show up there are python bytecode and they are cached to make subsequent runs a tad faster. My stuff never really needs this bonus speed boost and so I came across a neat tool called pyclean!

Pyclean #

The easiest way (in my opinion) to run pyclean is to just use pipx run.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)  took 9s
❯ ls
abcmeta.py  __pycache__  python-print-align.py  system-monitor-psutils.py

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ pipx run pyclean .
⚠️  pyclean is already on your PATH and installed at /usr/bin/pyclean. Downloading and running anyway.
Cleaning directory .
Total 1 files, 1 directories removed.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ ls
abcmeta.py  python-print-align.py  system-monitor-psutils.py

Why not bash? #

You could accomplish something similar with rm **/*.pyc or find -n '*.py?' -delete but there’s a chance you’ll find something you don’t love gone. Also this won’t help our poor Windows friends out there! pyclean is fully python so it’s OS independent.

Credits! #

repo

Psutil-01

2022-03-16T00:00:00Z

Mike Driscoll has been posting some awesome posts about psutil lately. I’m interested in making my own system monitoring dashboard now using this library. I don’t expect it to compete with Netdata or Glances but it’ll just be for fun to see how Python can solve this problem!

Repo coming soon

Example code: #

Here’s a short snippit to get used/available/total RAM and disk space (on partitions that you probably care about)


import psutil
import socket

print(f"System Memory used: {psutil.virtual_memory().used // (1024 ** 3)} GB")
print(f"System Memory available: {psutil.virtual_memory().available // (1024 ** 3)} GB")
print(f"System Memory total: {psutil.virtual_memory().total // (1024 ** 3)} GB")


print(f"Hostname: {socket.gethostname()}")

partitions = psutil.disk_partitions()

for part in partitions:
    mnt = part.mountpoint
    if "snap" in mnt or "boot" in mnt:
        continue
    disk = psutil.disk_usage(mnt)
    print(f"Usage at {mnt} on {part.device}: {disk.used // (1024 ** 3)} GB")
    print(f"Free at {mnt} on {part.device}: {disk.free // (1024 ** 3)}GB")
    print(f"Total at {mnt} on {part.device}: {disk.total // (1024 ** 3)}GB")

Bonus Ipython tip! Save this to a script called my_script.py and in Ipython you can %run -m my_script to run it!

project ↪ main v3.8.11 ipython
❯ %run -m system-monitor-psutils
System Memory used: 25 GB
System Memory available: 5 GB
System Memory total: 31 GB
Hostname: ryzen-3600x
Usage at / on /dev/nvme1n1p2: 81 GB
Free at / on /dev/nvme1n1p2: 351 GB
Total at / on /dev/nvme1n1p2: 456 GB

Mu

2022-03-15T00:00:00Z

If you work with a template for several projects then you might sometimes need to do the same action across all repos. A good example of this is updating a package in requirements.txt in every project, or refactoring a common module. If you have several repos to do this across then it can be time consuming… enter mu-repo

Mu #

mu-repo is an awesome cli tool for working with multiple git repositories at the same time. There are several things you can do:

mu status will give you the git status of every registered repo (see below)
mu sh will let you execute system level commands in every repo
mu stash will stash all changes across all registered repos
There’s literally a ton more but these are some handy ones

Registration #

mu tracks its own groups, and there is a default group when no particular one is active. It’s as simple as mu register proj1 prog2 ... to get repos registered


❯ mu register proj1 proj2
Repository: proj1 registered
Repository: proj2 registered

❯ mu status

  proj1 : git status
    On branch main

    No commits yet

    Untracked files:
    (use "git add <file>..." to include in what will be committed)
    requirements.txt

    nothing added to commit but untracked files present (use "git add" to track)

  proj2 : git status
    On branch main

    No commits yet

    Changes to be committed:
    (use "git rm --cached <file>..." to unstage)
    new file:   requirements.txt

Working with mu #

As you can see above I have two projects each with a requirements.txt added but not committed yet. Using mu I can stage this change across both repos at once.


❯ mu add requirements.txt

  proj1 : git add requirements.txt

  proj2 : git add requirements.txt

Then as you might imagine, I can make the commit in each repo


❯ mu commit -m "Add requirements.txts"

  proj1 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

  proj2 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

mu groups #

The other thing I got a lot of use out of recently was mu’s groups. At work I have about 40 repos cloned that are all based on the same kedro pipeline template. Some of these projects have been deprecated. I also have several more repos that are not kedro template - custom libraries or something. group let me utilize mu across different groups of repos.

Say proj2 is a deprecated project that I don’t need to worry about making changes to anymore. I don’t just have to unregister it, instead I can make a group called “active” and register proj1 in that group


❯ mu group add active --empty

~/personal
❯ mu group add deprecated --empty

~/personal
❯ mu group
  active
* deprecated

The * tells me which group is active. The --empty flag tells mu to not add all registered repos to that group. If I don’t want to use any groups then mu group reset will go back to the default group with all registered repos.

With groups I can register only the repos that I want to be working across in their own group and not worry about affecting other repos with my batch changes!

Wireguard

2022-03-12T00:00:00Z

VPN #

Virtual Private Networks are a big deal, and this shouldn’t be considered anything even close to a guide on using them. Here are just my notes and some setup for how I use wireguard at home.

Wireguard #

Wireguard is an awesome peer-to-peer VPN tunnel that makes it really easy for me to get into my home network when I’m out and about. My main reasons for this are 1. I don’t trust public wi-fi and 2. I want to use pi-hole for ad blocking when I’m not at home

Wireguard can be configured as a “peer-to-site” VPN tunnel as well. My vpn setup let’s me jump to various machines on my network from anywhere!

I use pivpn in a VM that’s already running pi-hole to host my wireguard server. It’s super easy to setup just by following the instructions on the pivpn site.

The reason I like it is that I have a nice cli for managing wireguard configs.

dumbledore@pihole-vpn:~$ pivpn
::: Control all PiVPN specific functions!
:::
::: Usage: pivpn <command> [option]
:::
::: Commands:
:::    -a, add              Create a client conf profile
:::    -c, clients          List any connected clients to the server
:::    -d, debug            Start a debugging session if having trouble
:::    -l, list             List all clients
:::   -qr, qrcode           Show the qrcode of a client for use with the mobile app
:::    -r, remove           Remove a client
:::  -off, off              Disable a user
:::   -on, on               Enable a user
:::    -h, help             Show this help dialog
:::    -u, uninstall        Uninstall pivpn from your system!
:::   -up, update           Updates PiVPN Scripts
:::   -bk, backup           Backup VPN configs and user profiles

When I’m ready to add a new client to my wg network, it’s as easy as pivpn add and follow the instructions. The easiest part here is that you’ll be given a QR code in the terminal that you can just scan with the client (like a smart phone) and you’ll have your wireguard config handled by the app (oh right, download the wireguard app) in no time!

Git-Worktrees-01

2022-03-11T00:00:00Z

Git #

Hopefully if you write code you are using git, if not go learn the basics of commit, pull, push, and pull request/merge request like… right now.

Assuming you are at least familiar with git then you probably work the same way I have since I’ve been using it.

clone or initialize a repo
checkout a branch, git checkout -b my-feature
work on my-feature and when ready open a PR into main
git pull main then git checkout -b another-feature
etc…

What if you need to switch between branches for some reason? Often I’m jumping into projects with my co-workers left and right, and I’ll have changes that I’m either working on or exploring for them. When it’s time to switch branches I think there’s more elegant ways than this but I’ve always done this:

stash the current changes
checkout out the relevant branch
helped out
re-checkout my original branch
pop the stash

Now, that’s not awful but I think worktrees will make this nicer for a few reasons!

Worktrees #

Worktrees are linked branches that have their own directories somewhere on your computer. To checkout a branch you don’t have to worry about stashing any changes, you just cd into the directory of that branch.

The branch can be literally anywhere - it doesn’t have to be in the repo folder

Use Case #

I’ve seen ThePrimeagean argue for worktrees for several reasons, see a YT video here

I’m entirely in Python at the moment, or working with projects that dont’ have that kind of requirement (ie. this website). My reason for wanting worktrees is 3 fold.

Files that could have been gitignored but ain’t #

I have a .envrc I put in every project, but it’s not gitignored for reasons that aren’t relevant right now… If I switch branches I’ll stash everything I have at the time, including my .envrc, but then if I forget to pop the stash and I move on and come back then my environment isn’t active and I have to go find the stash, pop it, cd out, and then back in and honestly…. that sucks. Worktrees will let me have the .envrc in every branch, and if I checkout or switch to a new one, my personal branch is unaffected.

Symlinks #

In my team’s Kedro workflow we keep a specific directory, the conf directory at a different spot than the Kedro team has in their templates (the why is outside the scope here). The way I preserve every kedro utility for my own benefit is to symlink our conf to where the Kedro template expects it to be. But then everytime I stash changes I lose that symlink so I either just don’t have it for the time being or I recreate it which is a hassle Worktrees will let me have that present and persistent on all my branches at once.

Foo #

Because why not!? This workflow feels future-proof, and if my toolset changes down the line then having this worktree centric workflow might be helpful and I’m just prepping for that possibility!

Abstract-Base-Class

2022-03-09T00:00:00Z

ABCMeta #

I don’t do a lot of OOP currently, but I have been on a few heavy OOP projects and this ABCMeta and abstractmethod from abc would’ve been super nice to know about!

If you are creating a library with classes that you expect your users to extend, but you want to ensure that any extension has explicit methods defined then this is for you!.

from abc import ABCMeta, abstractmethod
class Family(metaclass=ABCMeta):
    @abstractmethod
    def get_dad(self):
        """Any extension of the Family class must implement a `get_dad` method"""

class MyFamily(Family):
    pass

If I try to instantiate MyFamily I will not be allowed:


❯ my_fam = MyFamily()
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-8-ecb8e21ce815>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Can't instantiate abstract class MyFamily with abstract methods get_dad

In order for me to extend Family I have to implement the method get_dad

class MyFamily(Family):
    def get_dad(self):
        return "Me"

Now everything works as expected and I can sleep well knowing no one can extend my base class without creating methods I know they need.


my_fam = MyFamily()

my_fam.get_dad()
'Me'

Python-F-String-Align

2022-03-08T00:00:00Z

I am personally trying to use logger instead of print in all of my code, however I learned from [@Python-Hub] that you can align printouts using print with f-strings!.

This little python script shows how options in the f-string can format the printout.


import random

variables = "Foo Bar Baz Bing".split()
scores = random.sample(range(1, 11), len(variables))

print("*" * 30)
print("\n")
print("With 'varable' left aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:<10} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' right aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:>15} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' center aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:^5} | {score}")

Python-Builtin-Calendar

2022-03-08T00:00:00Z

Being lazy #

I almost exclusively use Python for my job and have been eye-balls deep in it for almost 5 years but I really lack in-depth knowledge of builtins. I recently learned of an awesome builtin called calendar that has way more than I know about for sure but I’m glad I know it’s here now!

I only needed it because I was too lazy to hard code the 7 weekdays into my module but it turns out there’s a lot of useful things like calendar.isleap()!

## Future use

I’m not exactly sure what will come my way where calendar will be super relevant but like anything, I’m just glad to know it exists for when the time arises!

Dataframe-Memory-Usage

2022-03-07T00:00:00Z

I have often wanted to dive into memory usage for pandas DataFrames when it comes to cloud deployment. If I have a python process running on a server at home I can use glances or a number of other tools to diagnose a memory issue… However at work I normally deploy dockerized processes on AWS Batch and it’s much more challenging to get info on the dockerized process without more AWS integration that my team isn’t quite ready for. So TIL that I can get some of the info I want from pandas directly!

DataFrame.info()

I didn’t realize that df.info() was able to give me more info than just dtypes and some summary stats… There is a kwarg memory_usage that can configure what you need to get back, so df.memory_usage="deep" will give you how much RAM any given DataFrame is using! Amazing tool for finding issues with joins or renegade source data files.

df = pd.read_csv("cars.csv")

df.info(memory_usage="deep")

Adblock-Coverage

2022-03-07T00:00:00Z

I run pi-hole at home for ad blocking and some internal DNS/DHCP handling.

pi hole posts on the way

One thing I’ve never put too much thought in is asking “how well am I doing at blocking?” There’s lots of ways to measure that depending on what you care about but I just learned of adblock tester. It’s awesome and gave me a quick glimpse into how my pi-hole is performing on keeping my webpages clean and my DNS history private!

Credits to d3ward for the awesome tool!

Webservers-And-Indexes

2022-03-06T00:00:00Z

I host a lot of services in my homelab, but they’re mostly dockerized applications so I have never had to care much about how content gets served up. Today I had several little concepts click into place regarding webservers, and it was a similar experience to when I started homelabing and didn’t know what a “server” was in the first place.

Servers

A “server” can have a lot of different meanings but specifically in my world it was a physical server, like my PowerEdge R610 which acts as my main “home server”. But then on my server, I have other servers… Jellyfin is my main media server - but that’s obviously not a hardware thing, that’s software. This is certainly not a groundbreaking thing but it was a tiny piece to the puzzle that I was missing… that “server” is highly contextual.

Webservers

Something that confused the heck out of me when I first started down the road of having a server was what a webserver even was… I always thought the “webserver” was just “a server that hosts a website”… and yes, that’s true, but also it wasn’t true in how I understood “server”. It turns out that across my 40-odd dockerized services I have at home that I must have about 40-odd web servers running, each docker container is spinning up its own!

So something I have wanted to do for a long time is put my theology notes online for my small group to access whenever they might want… it doesn’t need to be fancy or anything. My issue was not knowing what to even Google. I tried “How to serve up static html” but that kind of search is for people who know what a “static” site is - I am not one of those people. I kept running across nginx and apache things, wordpress and other website building tools, etc. In fact I only recently learned that JavaScript assets cann still be considered static so I am a complete baby in the web-dev space.

What I really wanted was just a simple landing page with a link to each of my “posts” which are in the form of a single html file each that I can easily export from my tiddlywiki (I have a post about tiddlywiki here)

The first win python -m http.server right in the directory I kept my html files in and that got me what I wanted functionally. But then I wanted just a hair more organization… I started looking for a way to dynamically generate an index for a directory of html files but again the verbiage of that Google search just wasn’t helping me - I didn’t want anything complicated and I knew that what I wanted had to be easy…

The Index

Luckily I randomly came across a SO that mentioned a Linux utility called tree which does exactly what I wanted!

See my TIL on tree here

So now it goes like this:

Take notes on X in my tiddlywiki
Export that tiddler to a html file
Put that html file into a notes folder in my github repo for small group notes
Use tree to generate an index.html of each of those files in the notes directory
Use python -m http.server to start a web server that lands me at the index.html and now I can click through to any post!

It’s not fancy but it’s functional… This site/blog is built with markdown and markata and I wanted way more functionality in my tech notes. But for this simple use case I learned a ton about how content gets served up on a webpage and my small group benefits from the easy access as well!

Tree

2022-03-06T00:00:00Z

I wanted a quick way to generate an index.html for a directory of html files that grows by 1 or 2 files a week. I don’t know any html (the files are exports from my tiddlywiki)…

tree is just the answer.

Say I have a file structure like this:

./html-files
├── file1.html
└── file2.html

To generate a barebones simple index.html we can use tree as follows:

tree ./html-files -H "." -L 1 -P "*.html"

and get the following:


<!DOCTYPE html>
<html>
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <meta name="Author" content="Made by 'tree'">
 <meta name="GENERATOR" content="$Version: $ tree v1.8.0 (c) 1996 - 2018 by Steve Baker, Thomas Moore, Francesc Rocher, Florian Sesser, Kyosuke Tokoro $">
 <title>Directory Tree</title>
 <style type="text/css">
  <!--
  BODY { font-family : ariel, monospace, sans-serif; }
  P { font-weight: normal; font-family : ariel, monospace, sans-serif; color: black; background-color: transparent;}
  B { font-weight: normal; color: black; background-color: transparent;}
  A:visited { font-weight : normal; text-decoration : none; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:link    { font-weight : normal; text-decoration : none; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:hover   { color : #000000; font-weight : normal; text-decoration : underline; background-color : yellow; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:active  { color : #000000; font-weight: normal; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  .VERSION { font-size: small; font-family : arial, sans-serif; }
  .NORM  { color: black;  background-color: transparent;}
  .FIFO  { color: purple; background-color: transparent;}
  .CHAR  { color: yellow; background-color: transparent;}
  .DIR   { color: blue;   background-color: transparent;}
  .BLOCK { color: yellow; background-color: transparent;}
  .LINK  { color: aqua;   background-color: transparent;}
  .SOCK  { color: fuchsia;background-color: transparent;}
  .EXEC  { color: green;  background-color: transparent;}
  -->
 </style>
</head>
<body>
        <h1>Directory Tree</h1><p>
        <a href=".">.</a><br>
        ├── <a href="./file1.html">file1.html</a><br>
        └── <a href="./file2.html">file2.html</a><br>
        <br><br>
        </p>
        <p>

0 directories, 2 files
        <br><br>
        </p>
        <hr>
        <p class="VERSION">
                 tree v1.8.0 © 1996 - 2018 by Steve Baker and Thomas Moore <br>
                 HTML output hacked and copyleft © 1998 by Francesc Rocher <br>
                 JSON output hacked and copyleft © 2014 by Florian Sesser <br>
                 Charsets / OS/2 support © 2001 by Kyosuke Tokoro
        </p>
</body>
</html>

which looks like this when you serve it up with python -m http.server

Traefik

2022-03-06T00:00:00Z

Traefik

If you don’t know about traefik and you need a reverse-proxy then you might want to check it out. I used to use nginx for my reverse proxy but the config was over my head, and once it was working I was afraid to touch it. Traefik brings a lot to the table, my main uses are reverse proxy and ip whitelisting, but it’s doing even more under the hood that I don’t have a full-grasp of yet.

I like Traefik a lot because once I get some basic config up it’s incredibly easy to add services into my homelab whether they run on my primary server or not. This will not be exhaustive but I’ll outline my simple setup process of traefik and how I add services whether they are in docker or not.

Docker

In 2022 I’m still a docker fan-boy and I run my traefik instance in a docker container. This isn’t necessary but I love the portability since my homelab is very dynamic at the moment. And even if it wasn’t I’d still want to keep traefik in docker because deployment and updating are just so flipping easy

A simple docker-compose file for traefik might look like this:

name: traefik
image: "traefik:v2.4"
network_mode: host
volumes:
  - "docker-data/traefik/traefik.toml:/etc/traefik/traefik.toml:ro"
  - "docker-data/traefik/config.yml:/etc/traefik/config.yml:ro"
  - "docker-data/traefik/letsencrypt:/letsencrypt:rw"
  - "/var/run/docker.sock:/var/run/docker.sock:ro"  # for auto-discovery
env: "{{ traefik_environment_variables }}"
restart_policy: unless-stopped
memory: "1g"

Ansible deployment

I plan to have more on my homelab and Ansible on this site eventually…

I use Ansible to deploy most of my services at home, including traefik. My main homelab repo is here which is a fork of Ansible NAS.

If you want my stuff then be sure to go to the user/nic branch on my fork

You can see the ansible stuff for traefik here

Config

I use a traefik.toml as the main config and it looks something like this. With ansible a lot of this is done through template variables but this is the general idea. This config tells traefik what ports to listen and forward on, and gives the names to be referenced by docker labels (down below).

Traefik also has a handy web ui that with this config you can find on port 8080. There is a providers section - which is one of the biggest selling points of traefik for me. I have a docker provider configured and a static file.

The docker provider lets traefik auto-discover new services that I deploy and automatically handle the routing! The static file lets me easily add non-dockerized service routing, or routing to dockerized services on another host (I think traefik has an easier way to do this automatically but I don’t do it often enough to need that kind of automation). Then at the bottom is the SSL cert stuff. Using Let’s Encrypt is pretty easy and I use Cloudflare as my DNS provider


[entryPoints]
[entryPoints.web]
address = ":80"

[entryPoints.web.http.redirections.entryPoint]
to = "websecure"

[entryPoints.websecure]
address = ":443"

[entryPoints.websecure.http.tls]
certResolver = "letsencrypt"

[entryPoints.websecure.http.tls.domains]
main = "example.com"
sans = [
"*.example.com"
]

[entryPoints.traefik]
address = ":8080"

[providers]
providersThrottleDuration = "1s"
[providers.docker]
exposedbydefault = false
[providers.file]
filename = "/etc/traefik/config.yml"

[api]
insecure = true
dashboard = true

[log]
level = "INFO"

[ping]
terminatingStatusCode = 0

[certificatesResolvers]
[certificatesResolvers.letsencrypt]
[certificatesResolvers.letsencrypt.acme]
email = "my_email@example.com"
storage = "/letsencrypt/acme.json"
caserver = "https://acme-staging-v02.api.letsencrypt.org/directory"  # le staging, not prod

[certificatesResolvers.letsencrypt.acme.dnsChallenge]
provider = "cloudflare"

Providers.file

To my knowledge there isn’t much to configure on the docker provider side of things until you deploy a service. But the provider config file should get a little screen time here.

The file defines a traefik http router for each service you define, in this case just pihole.

Here I am adding my pihole instance which is not run inside docker but is inside a VM on another host. I want the entryPoints to be set to websecure which is configured above in the http redirects. I want some middlewares, addprefix-pihole and default-headers, which I’ll explain below. I set letsencrypt as the cert certResolver. Finally I name the service pihole.

Then in the services section I configure where pihole is located by just giving the internal IP for traefik to route to. Finally I define my middlewares. To get to the pihole homepage you need to use the route /admin so I want that added automatically when I go to pihole.example.com so I come to pihole.example.com/admin. And I wanted to restrict access to just my internal network and my wireguard network - this is done with the default-whitelist. The last thing is to configure a chain of middlewares that I called secured which is just easier for the docker labels later on.

With this config in play though, traefik will know about the route pihole.example.com and handle the ip whitelisting and load balancing for me.

http:
 #region routers 
  routers:
    pihole:
      entryPoints:
        - "websecure"
      rule: "Host(`pihole.example.com`)"
      middlewares:
        # - default-headers
        - addprefix-pihole
        - default-whitelist
      tls: 
        certResolver: letsencrypt
      service: pihole
  #region services
  services:
    pihole:
      loadBalancer:
        servers:
          - url: "http://192.168.1.3:80"
        passHostHeader: true
  #endregion
  middlewares:
    addprefix-pihole:
      addPrefix:
        prefix: "/admin"
    https-redirect:
      redirectScheme:
        scheme: https

    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "10.6.0.0/24"  # wg
        - "192.168.1.0/24"  # lan
        - "172.17.0.0/16"  # docker

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

Docker labels

Now the real magic is with Docker. Here is an example docker-compose file for spinning up a jellyfin server that you want to expose to the world, or at least access at home with jellyfin.example.com instead of http://192.168.1.N:8096…

I left some of the ansible variable stuff in here, but the main part to be concerned with is the labels section…

We define just a few labels to throw onto this docker container which let’s traefik discover it automatically and apply any settings necessary (like my ipWhiteList).

traefik.enable is either True or False.
traefik.http.router.jellyfin.rule defines an http router called jellyfin and sets the url to jellyfin.example.com (if example.com was my ansible_nas_domain)
traefik.http.routers.jellyfin.tls.certresolver is set to letsencrypt since I use LE for my wildcard certs.
traefik.http.routers.jellyfin.tls.domains[0].main will just be example.com -> and this should remind you of the toml file above
traefik.http.routers.jellyfin.tls.domains[0].sans is set to *.example.com
traefik.http.services.jellyfin.loadbalancer.server.port is set to jellyfin’s default http port of 8096, which tells traefik which port to point to for this service.

name: jellyfin
image: linuxserver/jellyfin
volumes:
  - "{{ jellyfin_config_directory }}:/config:rw"
  - "{{ jellyfin_movies_directory }}:/movies:{{ jellyfin_movies_permissions }}"
  - "{{ jellyfin_music_directory }}:/music:{{ jellyfin_music_permissions }}"
  - "{{ jellyfin_photos_directory }}:/photos:{{ jellyfin_photos_permissions }}"
  - "{{ jellyfin_tv_directory }}:/tv:{{ jellyfin_tv_permissions }}"
  - "{{ jellyfin_books_directory }}:/books:{{ jellyfin_books_permissions }}"
  - "{{ jellyfin_audiobooks_directory }}:/audiobooks:{{ jellyfin_audiobooks_permissions }}"
ports:
  - "{{ jellyfin_port_http }}:8096"
  - "{{ jellyfin_port_https }}:8920"
env:
  TZ: "{{ ansible_nas_timezone }}"
  PUID: "{{ jellyfin_user_id }}"
  PGID: "{{ jellyfin_group_id }}"
restart_policy: unless-stopped
memory: 1g
labels:
  traefik.enable: "{{ jellyfin_available_externally }}"
  traefik.http.routers.jellyfin.rule: "Host(`jellyfin.{{ ansible_nas_domain }}`)"
  traefik.http.routers.jellyfin.tls.certresolver: "letsencrypt"
  traefik.http.routers.jellyfin.tls.domains[0].main: "{{ ansible_nas_domain }}"
  traefik.http.routers.jellyfin.tls.domains[0].sans: "*.{{ ansible_nas_domain }}"
  traefik.http.services.jellyfin.loadbalancer.server.port: "8096"

And just like that traefik will automagically find your jellyfin container and route jellyfin.example.com to it!

Pandas-Select-Dtypes

2022-03-05T00:00:00Z

On my team we often have to change data types of columns in a pandas.DataFrame for a variety of reasons. The main one is it tends to be an artifact of EDA whereby a file is read in via pandas but the data types are somewhat wonky (ie. dates show up as strings, or a column that should be a integer comes in as float, etc.). The best solution I think is to leverage the dtypes keyword argument in which pd.read_X method is used. However there is another way which is to coerce the data types at runtime instead of loadtime.

A handy way to do this is by using pandas.DataFrame.select_dtypes…

Here is an example of finding columns read in as datetime64 and the developer would prefer to use pandas datetimes.

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.columns:
    if df[c].dtype == "datetime64":
        df[c] = pd.to_datetime(df.c)

Here is the difference in code flow between select_dtypes and manually finding the datetype64 columns:

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.select_dtypes('datetime64'):
    df[c] = pd.to_datetime(df.c)

The difference isn’t huge but it’s the little steps in leveling up that turn script-kitty scripts into clean looking functions.

Tiddly-Wiki

2022-03-05T00:00:00Z

Tiddly Wiki is a great note taking utility for organizing non-linear notes. I used it to replace my OneNote workflow and my only complaint is I don’t have an easy way to access and edit my tiddlers (posts) if I’m not at home.

The tiddlywiki is just an html file with a ton of stuff above my head baked in. I have a barebones repo with some notes and a nice starter tiddly wiki init on my github. Usage is pretty basic… Just grab the notebook/template.htlm and save it to anywhere convenient on your computer. I put mine on my NAS to have the security of backups since I don’t keep my tidldlywiki in a git repo (I don’t really want to look at the diff).

Taking notes in the tidlywiki is nice because it supports a format similar to Markdown although it is specific to tidlywiki. Tiddlers (each post in the wiki) can be tagged and linked together and it’s really easy to send notes to someone by just exporting an html file and emailing it since it’ll open up by default in a broswer with all the nice formatting already apart of it. I was using it primarily for taking notes for a small group I lead and sending those notes each week. The group benefited from nicely formatted notes and I benefited from a centralized place to keep them all that Microsoft didn’t own!

Here’s an example of the body of a tiddler with some tiddlywiki specific formatting:

! Static IPs on Linux

//Ubuntu 20//

Setting static IP on Ubuntu 20.04

# Navigate to /etc/netplan
# Open the yaml file (the name seems to be kind of random but it seems to starts with 00 or 05)
# Change the file as below with your desired settings
# Run `sudo netplan apply` to have changes reflected

    ```yaml
    network:
      version: 2
      ethernets:
        enp0s4:
          addresses: [192.168.1.{Static IP}/24]
          gateway4: 192.168.1.1
          nameservers:
            addresses: [8.8.4.4, 8.8.8.8]
    ```



<img src="https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/tiddlywiki-example.png" alt="tiddlywiki" title="A TiddlyWiki Example" />

The # create a numbered list, // creates an italicized heading, and ! creates headings similar to Markdown’s #. The differences aren’t too bad to keep in mind and what renders out is totally depenent on the tidlywiki itself. My template has a nice nord feel to it, feel free to download from my github and try it out!

I have moved away from my tiddlywiki workflow in favor of sites like this since I can git commit markdown files and build with markata pretty easily (credit waylon walker)

I still use tiddlywiki for tracking some todo items and questions — I may have another solution in the future

Vim-Auto-Space

2022-03-04T00:00:00Z

I ran into an issue where I had some copy-pasta markdown tables in a docstring but the generator I used to make the table gave me tabs instead of spaces in odd places which caused black to throw a fit. Instead of manually changing all tabs to spaes, or trying some goofy :%s/<magic tab character>/<%20 maybe?>/g I learned that Vim has my back…

:retab

Stow

2022-03-04T00:00:00Z

Stow is a great tool for managing dotfiles. My usage looks like cloning my dotfiles to my home directory, setting some environment variables via a script, then stowing relevant packages and boom my config is good to go…

cd ~
git clone <my dotfiles repo>
cd dotfiles
# env variable stuff ignored here
stow zsh  # This will symlink my .zshrc file which is in ~/dotfiles/zsh to ~/.zshrc

By default stow will stow packages up one directory from the root directory. In this example the root directory is ~/dotfiles and the package is zsh. So the files in the zsh package will symlinked into ~/.

stow makes it easy to share dotfiles across machines, or safely experiment with config changes while always being protected by git since your dotfiles are in a git repo! …They are in a git repo… right?

Stow-Target

2022-03-04T00:00:00Z

Check out stow for a brief introduction to stow

What if I want to stow a package somewhere else? Boom, that’s where -t comes in…

Maybe I don’t like having my dotfiles repo at $HOME and instead I want it in ~/git or ~/personal just to stay organized… Well then I could have the same workflow except the stow command looks like this:

stow zsh -t ~/
#or
stow zsh -t $HOME

Git ammend to a commit

2022-03-04T00:00:00Z

After carefully staging only lines related to a specific change and comitting I suddenly realized I missed one… darn, what do I do?

Old me would have soft reset my branch to the previous commit and redone all my careful staging… what a PIA…

New me (credit: ThePrimeagen)…

# stage other changes I missed
git commit --amend --no-edit