{ "version": "https://jsonfeed.org/version/1.1", "title": "Thoughts on tech", "home_page_url": "https://pype.dev/tech/", "feed_url": "https://pype.dev/tech/feed.json", "description": "My thoughts and streams of consciousness organized into barely coherent posts about things", "authors": [ { "name": "Nic Payne" } ], "items": [ { "id": "https://pype.dev/harbor-faulted-disk-replacement/", "url": "https://pype.dev/harbor-faulted-disk-replacement/", "title": "The Faulted Disk: harbor Replacement Writeup", "content_html": "\u003cp\u003eThe sequel to \u003ca href=\"/panicking-led-to-losing-my-desktop\"\u003epanicking-led-to-losing-my-desktop\u003c/a\u003e — this time the monitoring actually caught the disk dying, and nothing was lost.\u003c/p\u003e\n\u003ch2 id=\"what-happened\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eWhat happened\u003c/span\u003e \u003ca href=\"#what-happened\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003ccode\u003eharbor\u003c/code\u003e is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from \u003ccode\u003etank\u003c/code\u003e. One side of the mirror, a Seagate Exos \u003ccode\u003eST12000NM0127\u003c/code\u003e (serial \u003ccode\u003eZJV4QFLB\u003c/code\u003e, \u003ccode\u003e/dev/sdb\u003c/code\u003e), went \u003cstrong\u003eFAULTED\u003c/strong\u003e with 14 read + 22 checksum errors.\u003c/p\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emirror-0 DEGRADED 0 0 0\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e ata-ST12000VN0008-2PH103_ZTM0NFDW ONLINE 0 0 0\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e ata-ST12000NM0127_ZJV4QFLB FAULTED 14 0 22 too many errors\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe IronWolf mirror side carried the pool — \u003ccode\u003eNo known data errors\u003c/code\u003e. ZFS redundancy did exactly its job.\u003c/p\u003e\n\u003ch2 id=\"the-difference-from-last-time\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe difference from last time\u003c/span\u003e \u003ca href=\"#the-difference-from-last-time\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLast failure: no monitoring, found out by accident months later, desktop died.\u003c/p\u003e\n\u003cp\u003eThis failure: SigNoz + node-exporter\u0026rsquo;s ZFS collector → \u003ccode\u003enode_zfs_zpool_state{state=\u0026quot;degraded\u0026quot;}\u003c/code\u003e → alert rule → Gotify → my phone. The gotify notification fired \u003cem\u003ebefore\u003c/em\u003e I knew anything was wrong.\u003c/p\u003e\n\u003ch2 id=\"diagnosis\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eDiagnosis\u003c/span\u003e \u003ca href=\"#diagnosis\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBefore \u003ccode\u003ezpool clear\u003c/code\u003e or replace — check SMART. \u003ccode\u003esmartctl-exporter\u003c/code\u003e already scrapes all disks into SigNoz, so I didn\u0026rsquo;t even need sudo:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eReallocated_Sector_Ct\u003c/code\u003e raw = \u003cstrong\u003e3024\u003c/strong\u003e and counting\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eOffline_Uncorrectable\u003c/code\u003e value/worst still 100 but raw errors climbing\u003c/li\u003e\n\u003cli\u003eSMART overall: still PASS (SMART\u0026rsquo;s overall bit is conservative until threshold)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don\u0026rsquo;t clear.\u003c/p\u003e\n\u003ch2 id=\"the-swap-the-annoying-part\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe swap (the annoying part)\u003c/span\u003e \u003ca href=\"#the-swap-the-annoying-part\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHot-plug is never as smooth as it should be:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003ccode\u003ezpool offline harbor \u0026lt;old\u0026gt;\u003c/code\u003e to stop writes to the dead drive — actually skipped; drive fell off the bus on its own\u003c/li\u003e\n\u003cli\u003eNew IronWolf \u003ccode\u003eZTN1CQ07\u003c/code\u003e in hand → plugged into ghost\u0026rsquo;s SATA → \u003cstrong\u003edidn\u0026rsquo;t enumerate\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eForced rescan (\u003ccode\u003eecho \u0026quot;- - -\u0026quot; | sudo tee /sys/class/scsi_host/host*/scan\u003c/code\u003e) → nothing\u003c/li\u003e\n\u003cli\u003eMoved it to the \u003cem\u003eold drive\u0026rsquo;s port\u003c/em\u003e → nothing\u003c/li\u003e\n\u003cli\u003eSabrent USB dock on aurora → dock enumerated as \u003ccode\u003e0B\u003c/code\u003e device, no disk behind it → reseated + replugged → \u003cstrong\u003edrive spun up and appeared\u003c/strong\u003e: \u003ccode\u003e/dev/sdd\u003c/code\u003e, 10.9T, old ZFS partition table on it (used drive — \u003ccode\u003epart1\u003c/code\u003e/\u003ccode\u003epart9\u003c/code\u003e layout)\u003c/li\u003e\n\u003cli\u003eBack to ghost, direct SATA → enumerated as \u003ccode\u003esdb\u003c/code\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eLesson: \u0026ldquo;spins up but doesn\u0026rsquo;t enumerate\u0026rdquo; on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.\u003c/p\u003e\n\u003ch2 id=\"the-replace\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe replace\u003c/span\u003e \u003ca href=\"#the-replace\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esudo zpool replace harbor \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e ata-ST12000NM0127_ZJV4QFLB \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e ata-ST12000VN0008-2PH103_ZTN1CQ07\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eResilver: \u003cstrong\u003e8.80T in 18h38m, 0 errors\u003c/strong\u003e (~154M/s). Pool stayed online and usable the whole time.\u003c/p\u003e\n\u003cp\u003eOne gotcha: after resilver, \u003ccode\u003ezpool status\u003c/code\u003e showed \u003ccode\u003eerrors: 1 data errors\u003c/code\u003e — but \u003ccode\u003ezpool status -v\u003c/code\u003e showed an \u003cstrong\u003eempty error list\u003c/strong\u003e. The corrupted data was already repaired; the counter was just stale. \u003ccode\u003esudo zpool clear harbor\u003c/code\u003e → clean.\u003c/p\u003e\n\u003ch2 id=\"the-monitoring-that-made-this-possible\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe monitoring that made this possible\u003c/span\u003e \u003ca href=\"#the-monitoring-that-made-this-possible\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWired during this session (all now in SigNoz):\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003enode_zfs_zpool_state\u003c/code\u003e — pool health (node-exporter zfs collector)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esmartctl-exporter\u003c/code\u003e — SMART attributes incl. reallocated sectors (the early-death signal)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ezfs-metrics.sh\u003c/code\u003e textfile bridge — sanoid \u003ccode\u003e--monitor-*\u003c/code\u003e exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success\u003c/li\u003e\n\u003cli\u003eAlert rules → Gotify → phone\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.\u003c/p\u003e\n\u003ch2 id=\"remaining-homework\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eRemaining homework\u003c/span\u003e \u003ca href=\"#remaining-homework\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eBuy the replacement spare (the shelf\u0026rsquo;s empty now)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003e10Fold\u003c/code\u003e datasets are garbage + have zero snapshots — destroy\u003c/li\u003e\n\u003cli\u003eEvery scheduled job emits \u003ccode\u003ecron_last_success_epoch\u003c/code\u003e now — if a job dies silently again, I\u0026rsquo;ll know\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSame failure as April, opposite outcome. Redundancy did the protection; monitoring did the \u003cem\u003edetection\u003c/em\u003e. You need both.\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cem\u003eCo-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.\u003c/em\u003e\u003c/p\u003e\n", "content_text": "\nThe sequel to [panicking-led-to-losing-my-desktop](/panicking-led-to-losing-my-desktop) — this time the monitoring actually caught the disk dying, and nothing was lost.\n\n## What happened\n\n`harbor` is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from `tank`. One side of the mirror, a Seagate Exos `ST12000NM0127` (serial `ZJV4QFLB`, `/dev/sdb`), went **FAULTED** with 14 read + 22 checksum errors.\n\n``` text\nmirror-0 DEGRADED 0 0 0\n ata-ST12000VN0008-2PH103_ZTM0NFDW ONLINE 0 0 0\n ata-ST12000NM0127_ZJV4QFLB FAULTED 14 0 22 too many errors\n```\n\nThe IronWolf mirror side carried the pool — `No known data errors`. ZFS redundancy did exactly its job.\n\n## The difference from last time\n\nLast failure: no monitoring, found out by accident months later, desktop died.\n\nThis failure: SigNoz + node-exporter's ZFS collector → `node_zfs_zpool_state{state=\"degraded\"}` → alert rule → Gotify → my phone. The gotify notification fired *before* I knew anything was wrong.\n\n## Diagnosis\n\nBefore `zpool clear` or replace — check SMART. `smartctl-exporter` already scrapes all disks into SigNoz, so I didn't even need sudo:\n\n- `Reallocated_Sector_Ct` raw = **3024** and counting\n- `Offline_Uncorrectable` value/worst still 100 but raw errors climbing\n- SMART overall: still PASS (SMART's overall bit is conservative until threshold)\n\n3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don't clear.\n\n## The swap (the annoying part)\n\nHot-plug is never as smooth as it should be:\n\n1. `zpool offline harbor \u003cold\u003e` to stop writes to the dead drive — actually skipped; drive fell off the bus on its own\n2. New IronWolf `ZTN1CQ07` in hand → plugged into ghost's SATA → **didn't enumerate**\n3. Forced rescan (`echo \"- - -\" | sudo tee /sys/class/scsi_host/host*/scan`) → nothing\n4. Moved it to the *old drive's port* → nothing\n5. Sabrent USB dock on aurora → dock enumerated as `0B` device, no disk behind it → reseated + replugged → **drive spun up and appeared**: `/dev/sdd`, 10.9T, old ZFS partition table on it (used drive — `part1`/`part9` layout)\n6. Back to ghost, direct SATA → enumerated as `sdb`\n\nLesson: \"spins up but doesn't enumerate\" on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.\n\n## The replace\n\n``` bash\nsudo zpool replace harbor \\\n ata-ST12000NM0127_ZJV4QFLB \\\n ata-ST12000VN0008-2PH103_ZTN1CQ07\n```\n\nResilver: **8.80T in 18h38m, 0 errors** (~154M/s). Pool stayed online and usable the whole time.\n\nOne gotcha: after resilver, `zpool status` showed `errors: 1 data errors` — but `zpool status -v` showed an **empty error list**. The corrupted data was already repaired; the counter was just stale. `sudo zpool clear harbor` → clean.\n\n## The monitoring that made this possible\n\nWired during this session (all now in SigNoz):\n\n- `node_zfs_zpool_state` — pool health (node-exporter zfs collector)\n- `smartctl-exporter` — SMART attributes incl. reallocated sectors (the early-death signal)\n- `zfs-metrics.sh` textfile bridge — sanoid `--monitor-*` exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success\n- Alert rules → Gotify → phone\n\nThe replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.\n\n## Remaining homework\n\n- Buy the replacement spare (the shelf's empty now)\n- `10Fold` datasets are garbage + have zero snapshots — destroy\n- Every scheduled job emits `cron_last_success_epoch` now — if a job dies silently again, I'll know\n\nSame failure as April, opposite outcome. Redundancy did the protection; monitoring did the *detection*. You need both.\n\n---\n\n*Co-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.*\n", "summary": "The sequel to panicking-led-to-losing-my-desktop — this time the monitoring actually caught the disk dying, and nothing was lost.", "date_published": "2026-10-06T21:30:00Z", "date_modified": "2026-10-06T21:30:00Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "homelab", "zfs", "tech", "backup" ] }, { "id": "https://pype.dev/it-will-be-hard-to-do-more/", "url": "https://pype.dev/it-will-be-hard-to-do-more/", "title": "It Will Be Hard To Do More", "content_html": "\u003cp\u003eRecently I was given a raise at work after I broached the subject with my\nmanager. I wasn\u0026rsquo;t sure what the outcome would be but I have learned two things\nin nearly a decade in corporate America:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eNo one will fight for you harder than yourself\u003c/li\u003e\n\u003cli\u003eThe worst they can say is no\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"the-raise\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Raise\u003c/span\u003e \u003ca href=\"#the-raise\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSo I talked with my manager about my workload and compensation, and to my\nsurprise he took me seriously. He went up the chain with some accomplishments\nof the year that I wrote up for him (it\u0026rsquo;s his job to advocate for me, but it\u0026rsquo;s\ncertainly my job to give him the ammunition to take into battle). Before much\ntime had passed, a big wig put some time on my calendar. I had a good\nrelationship with my (then) manager, and even with the big wig, so there was\nalways some amount of personal connection amidst the money-talk.\u003c/p\u003e\n\u003ch2 id=\"the-reorg\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Reorg\u003c/span\u003e \u003ca href=\"#the-reorg\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eCue a managerial change and reorganization of the team. The dust settled, and\nmy manager wasn\u0026rsquo;t my manager and the big wig wasn\u0026rsquo;t a big wig over my team\nanymore. The details aren\u0026rsquo;t that relevant, but the outcome was that my \u003cem\u003enow\u003c/em\u003e\nmanager is someone I don\u0026rsquo;t know personally — back to square one of\nprofessional-relationship-building.\u003c/p\u003e\n\u003ch2 id=\"the-conversation\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Conversation\u003c/span\u003e \u003ca href=\"#the-conversation\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDue to the timing of it all, when my raise became official it was this new\nmanager who signed off on it. It was their happy-responsibility to have a quick\nchat to tell me it was official. But that conversation was the most\nawkward of the three.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eDoes this raise of X% close the gap with the other competing opportunities\nyou mentioned to (previous manager)? Because it\u0026rsquo;ll be hard to do more than\nthis.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cdiv class=\"admonition danger\"\u003e\n\u003cp class=\"admonition-title\"\u003e???\u003c/p\u003e\n\u003cp\u003eHard to do more?\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eI was taken aback initially. It felt almost threatening.\u003c/p\u003e\n\u003cp\u003eI really wasn\u0026rsquo;t sure how to respond. I think I was grateful and cordial,\ntalked about how I really do believe in the mission of our work and I\u0026rsquo;m\nthankful that I can receive a raise. This lady is nice but she doesn\u0026rsquo;t know\nwhat my \u0026ldquo;other opportunities\u0026rdquo; are, so there\u0026rsquo;s no honest answer for me to give.\nI wanted to be sarcastic, crack a joke about record profits, make light of her\ndescribing me as a \u0026ldquo;critical player in enterprise deliverables,\u0026rdquo; but I\u0026rsquo;m not\ntrying to make enemies.\u003c/p\u003e\n\u003ch2 id=\"the-reframe\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Reframe\u003c/span\u003e \u003ca href=\"#the-reframe\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSo I called a friend to vent, of course, and he gave me some good advice.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eIt was a threat.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eBut not in the way I took it at first. My friend has been at the same company\nlonger than me, and he expounded on her words: \u0026ldquo;it will be hard to do more.\u0026rdquo;\nI felt like she was annoyed with me for fighting for a raise, but my friend\nsaid she was probably just trying to give me a realistic view of my position\nin the company. \u0026ldquo;It will be hard to do more\u0026rdquo; means there are processes in\nplace at such a large enterprise. Rewarding start-up-like behavior and\ncontribution just isn\u0026rsquo;t well-supported. It\u0026rsquo;s much easier to let someone quit\nwho\u0026rsquo;s been with the company a decade and replace them for 20% more than it is\nto just pay them 20% more. If someone (like me) isn\u0026rsquo;t satisfied, that\u0026rsquo;s not the\nonly thing in the conversation - time at the company, time in position, time\nunder specific manager, etc. all are hurdles to more compensation, and I\u0026rsquo;m\nreferring to salary anyways, equity feels like it\u0026rsquo;s out of my grasp. But if I\u0026rsquo;m\ngone and market conditions dictate that my position and responsibilities are\nworth 20% more than what I was being paid, that\u0026rsquo;s what the position will be\nposted at. My previous manager and the big wig were the types to say\npolicy-be-damned when it would get in the way - it might be how I got this\nraise in the first place. But no shade to my new manager, they\u0026rsquo;re just doing their job.\u003c/p\u003e\n\u003cp\u003eIn the end, they were just being honest, and I should appreciate that.\u003c/p\u003e\n", "content_text": "\nRecently I was given a raise at work after I broached the subject with my\nmanager. I wasn't sure what the outcome would be but I have learned two things\nin nearly a decade in corporate America:\n\n1. No one will fight for you harder than yourself\n2. The worst they can say is no\n\n## The Raise\n\nSo I talked with my manager about my workload and compensation, and to my\nsurprise he took me seriously. He went up the chain with some accomplishments\nof the year that I wrote up for him (it's his job to advocate for me, but it's\ncertainly my job to give him the ammunition to take into battle). Before much\ntime had passed, a big wig put some time on my calendar. I had a good\nrelationship with my (then) manager, and even with the big wig, so there was\nalways some amount of personal connection amidst the money-talk.\n\n## The Reorg\n\nCue a managerial change and reorganization of the team. The dust settled, and\nmy manager wasn't my manager and the big wig wasn't a big wig over my team\nanymore. The details aren't that relevant, but the outcome was that my _now_\nmanager is someone I don't know personally — back to square one of\nprofessional-relationship-building.\n\n## The Conversation\n\nDue to the timing of it all, when my raise became official it was this new\nmanager who signed off on it. It was their happy-responsibility to have a quick\nchat to tell me it was official. But that conversation was the most\nawkward of the three.\n\n\u003e Does this raise of X% close the gap with the other competing opportunities\n\u003e you mentioned to (previous manager)? Because it'll be hard to do more than\n\u003e this.\n\n!!! danger \"???\"\n\n Hard to do more?\n\nI was taken aback initially. It felt almost threatening.\n\nI really wasn't sure how to respond. I think I was grateful and cordial,\ntalked about how I really do believe in the mission of our work and I'm\nthankful that I can receive a raise. This lady is nice but she doesn't know\nwhat my \"other opportunities\" are, so there's no honest answer for me to give.\nI wanted to be sarcastic, crack a joke about record profits, make light of her\ndescribing me as a \"critical player in enterprise deliverables,\" but I'm not\ntrying to make enemies.\n\n## The Reframe\n\nSo I called a friend to vent, of course, and he gave me some good advice.\n\n\u003e It was a threat.\n\nBut not in the way I took it at first. My friend has been at the same company\nlonger than me, and he expounded on her words: \"it will be hard to do more.\"\nI felt like she was annoyed with me for fighting for a raise, but my friend\nsaid she was probably just trying to give me a realistic view of my position\nin the company. \"It will be hard to do more\" means there are processes in\nplace at such a large enterprise. Rewarding start-up-like behavior and\ncontribution just isn't well-supported. It's much easier to let someone quit\nwho's been with the company a decade and replace them for 20% more than it is\nto just pay them 20% more. If someone (like me) isn't satisfied, that's not the\nonly thing in the conversation - time at the company, time in position, time\nunder specific manager, etc. all are hurdles to more compensation, and I'm\nreferring to salary anyways, equity feels like it's out of my grasp. But if I'm\ngone and market conditions dictate that my position and responsibilities are\nworth 20% more than what I was being paid, that's what the position will be\nposted at. My previous manager and the big wig were the types to say\npolicy-be-damned when it would get in the way - it might be how I got this\nraise in the first place. But no shade to my new manager, they're just doing their job.\n\nIn the end, they were just being honest, and I should appreciate that.\n", "summary": "Recently I was given a raise at work after I broached the subject with my manager. I wasn't sure what the outcome would be but I have learned two things in...", "date_published": "2026-05-22T06:23:20Z", "date_modified": "2026-05-22T06:23:20Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "work", "tech" ] }, { "id": "https://pype.dev/panicking-led-to-losing-my-desktop/", "url": "https://pype.dev/panicking-led-to-losing-my-desktop/", "title": "Panicking Led to Losing My Desktop", "content_text": "\n## False Sense of Security\n\nI thought I had backups handled... can you imagine how the rest of this post is\ngoing to go with that intro?\n\nTo be fair, I do have backups figured out on my NAS - simple ZFS +\nsanoid/syncoid + replica pool + off-site backup with simple restore pathways.\nHowever, my desktop has been another story entirely. My desktop OS didn't\nsupport ZFS when I started checking it out, and I spent weeks thinking through\nhow I would backup my HOME directory and projects mostly. I landed on a\nsolution that I did validate once, but it fell off my radar and lo' and behold\nthat was problematic...\n\nSo that backup was based on restic for my home directory, but it was lazy. I\nverified it one time but I had built it with ai, thought I understood the\nrestic repo part, and then promptly moved on with my life never buttoning it\nall up. That home directory backup got too big for where I was going to end up\nrestoring it. My desktop system was installed on a 4 TB NVMe drive and due to\nthe circumstances spawning this blog post I was gonna have to drop to a 500 GB\nboot drive with some extra disks as the storage layer. Overall it looked like:\n\n- A 4 TB SSD that was going bad - old OS\n- A 500 GB SSD, that was going to be my new operating system boot disk\n- A 2 TB SSD that was originally going to be this external storage volume\n anyways but I never set it up because the version of Aurora I was running\n didn't have ZFS, I was married to the idea of using ZFS, so I never ended up\n taking advantage of the space. However it was moot to me because my boot drive\n was 4 TB, high quality drive, so I was \"just sure\" I didn't need it.\n- AND a 4 TB rust disk as well, which was already a ZFS pool, left over from a\n previous desktop configuration, and admittedly I had forgotten it was even in the system.\n\n## The Storm\n\nIf it wasn't clear the problem is that my super-nice high-speed 4TB NVMe drive\nwas going bad, like really bad. Eventually my OS stopped booting, it was even\ndifficult to live-boot from any other ISO due to, I think ultimately, that disk\ncausing such extreme latency in the start-up processes that they just failed.\nSo I quickly found myself with little-to-no access to my primary desktop's\ndata...\n\n## Where It Went Wrong\n\nWhat I did is I live booted into an Ubuntu server environment (which took blood\nsweat and tears to successfully get into), mounted my home directory from the 4 TB SSD, and\ntried to continue my restic backup to my NAS, like an idiot. But at the same\ntime I also tried to prune it by only backing up a few projects because I\nwas getting worried about time. This was the first primary mistake - trying to\nmuck with my backup script under duress.\n\nThen over the course of the whole thing it ended up taking over a week to solve\nthis when it could've been 2-3 days. So say it with me kids - \"Don't make\ndecisions under duress\"\n\n## Climbing Out\n\nI downloaded opencode and had it help me write the right excludes syntax in my\nrestic backup script and got it back up going. That went ok but opencode agents\nhad no historical context for why anything was the way it was, and frankly an\nagent would've been misled thinking the backup solution was much more solid\nthan it was due to how I documented it.\n\nAgents also miss things... in my chat sessions it knew about the other 2\navailable disks on the desktop system, I could have done a fresh backup to the 4 TB\nspinning rust disk no problem: install zfs, mount the pool, change target of\nrestic, run full... that would've been beautifully simple. But instead I\ntrimmed it down and backed not-everything up to the NAS over the network, and\nto a different backup target nonetheless... SMH.\n\nAs I started to consider which OS I was going to go with next I failed to\ninstall Pop_OS! or Ubuntu onto the new disc... Then I tried Omarchy and the\ninstall script just looped. So, I reinstalled Aurora onto the new 500 GB disk\nand then quickly realized I don't have Firefox tabs, my SSH keys are in that\nrestic backup, my ssh config, api keys in hidden files.... Everything is in\nthat restic backup... The backup that's too big to restore to my new boot drive.\n\nBut you know what I have? That 2 terabyte disk mounted just fine as a\nZFS dataset. And I could mount the 4 TB rust disk with zfs as well because this\nversion of Aurora has zfs working flawlessly!\n\n## Hindsight\n\nWhat I should've done is so simple... While in that ubuntu live environment I\nshould've just either updated restic to be a local backup to the 4 TB rust\ndisk, or rsync'd my home directory to it plain and simple... I got all in my\nhead about not backing up python venvs, node_modules, etc. that I didn't think\nto just basically carbon copy it all to a healthy disk and then prune it later.\nThen I could've synced everything back over that I needed to the new Desktop's\n$HOME and then scheduled the rsync or restic again to that locally mounted disk.\n\n## The Detail I Left Out\n\nThe keen reader might stop to think... why not just mount the old 4TB disk and\ncopy what you need to your new desktop? And that's a prudent question...\nHowever, in order to get anything installed I had to physically remove the 4TB\nSSD from the motherboard, which was basically a full PC tear-down. From there I\nwas able to at least boot in and out of iso's like you'd otherwise expect, and\nI have a USB/NVMe adapter so I planned to mount the old drive and copy things over from\nthere... But sadly... it won't mount. it's dead-dead and it appears that\nanything I didn't save in my days-long-panicked-state is just. gone.\n\nI feel pretty stupid to have not taken advantage of the 2 available disks local\nto the machine, to have naively copied stuff over and dealt with the\norganization later once my OS was back up. I tried to be smart and efficient\nand ended up wasting so much time and losing quite a lot of \"stuff\"... ideas,\nblog posts that I never committed, etc.\n\n## Current Status\n\nSo a few lessons...\n\n1. untested backups are not backups\n2. false backups might be worse than none, although I did at least save a few things so maybe the jury is out here\n3. making decisions while stressed out will lead to missing obviously better pathways... slow down, talk it out\n\nAs for my current status - I'm working on [[desktop-setup-2026]] and recovering what I can from my haphazard'd rsyncs in the live ubuntu env I got into. I'm also setting up a new Linux laptop at work at the same time so maybe I'll hve some workflow changes to write about in the future. For now, it's nice to be forced to accept that not every idea was that important, the good stuff will come back around, and ultimately computers and shit are just things, they're not life.\n", "summary": "I thought I had backups handled... can you imagine how the rest of this post is going to go with that intro?", "date_published": "2026-05-13T08:24:00Z", "date_modified": "2026-05-13T08:24:00Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "backup", "tech" ] }, { "id": "https://pype.dev/the-relief-of-earning/", "url": "https://pype.dev/the-relief-of-earning/", "title": "The Relief of Earning", "content_html": "\u003ch2 id=\"the-setup\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Setup\u003c/span\u003e \u003ca href=\"#the-setup\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWork has been crazy for months, and I haven\u0026rsquo;t written too much about it, a few\nnotes here and there but nothing substantial. The craziness is sourced in a\npretty cliched-scenario where we have lofty goals and important things to build\nbut the foundation on which we are building, like architectural decisions, is\nbasically wet sand. So as I\u0026rsquo;ve been trying to find my footing in the mud I feel\nlike I end up spinning my wheels moreso than making any progress. The chaos led\nme eventually to approach my manager about my compensation - I make a nice\nliving but for the stress I\u0026rsquo;d been feeling I could certainly go make more. My\nmanager successfully fought for me up the chain and I was given a slightly bigger\ncarrot, but that\u0026rsquo;s not the point of this post. I started to feel even more\nanxiety after I was told about the raise because, just like with AI and agentic\ncoding, now I was sure the expectations would be even higher. I didn\u0026rsquo;t think\nanything would actually come from me asking my boss about a raise, but now that\nit did I was afraid the ceiling of expectations was just going up up up.\u003c/p\u003e\n\u003ch2 id=\"the-carrot-and-the-anxiety\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe Carrot and the Anxiety\u003c/span\u003e \u003ca href=\"#the-carrot-and-the-anxiety\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThat\u0026rsquo;s where the relief of earning comes in\u0026hellip; I mentioned this to my project\nmanager, and we have candid conversations regularly, and he told me something\nthat I think some people might have been offended by but I was genuinely\nrelieved to hear it. He said he\u0026rsquo;s glad I got a raise, and that he told our boss\nto \u003cem\u003enot\u003c/em\u003e really pursue it for me until I had actually delivered something. So even\nthough I feel like I\u0026rsquo;m spinning my wheels a lot, my direct leadership sees real\nvalue - and the relief is that I can rest in confirmation that the carrot is a\nresponse to work I already did, not a quiet elevation of expectations.\u003c/p\u003e\n", "content_text": "\n## The Setup\n\nWork has been crazy for months, and I haven't written too much about it, a few\nnotes here and there but nothing substantial. The craziness is sourced in a\npretty cliched-scenario where we have lofty goals and important things to build\nbut the foundation on which we are building, like architectural decisions, is\nbasically wet sand. So as I've been trying to find my footing in the mud I feel\nlike I end up spinning my wheels moreso than making any progress. The chaos led\nme eventually to approach my manager about my compensation - I make a nice\nliving but for the stress I'd been feeling I could certainly go make more. My\nmanager successfully fought for me up the chain and I was given a slightly bigger\ncarrot, but that's not the point of this post. I started to feel even more\nanxiety after I was told about the raise because, just like with AI and agentic\ncoding, now I was sure the expectations would be even higher. I didn't think\nanything would actually come from me asking my boss about a raise, but now that\nit did I was afraid the ceiling of expectations was just going up up up.\n\n## The Carrot and the Anxiety\n\nThat's where the relief of earning comes in... I mentioned this to my project\nmanager, and we have candid conversations regularly, and he told me something\nthat I think some people might have been offended by but I was genuinely\nrelieved to hear it. He said he's glad I got a raise, and that he told our boss\nto _not_ really pursue it for me until I had actually delivered something. So even\nthough I feel like I'm spinning my wheels a lot, my direct leadership sees real\nvalue - and the relief is that I can rest in confirmation that the carrot is a\nresponse to work I already did, not a quiet elevation of expectations.\n", "summary": "Work has been crazy for months, and I haven't written too much about it, a few notes here and there but nothing substantial. The craziness is sourced in a...", "date_published": "2026-05-08T05:43:39Z", "date_modified": "2026-05-08T05:43:39Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "work", "tech" ] }, { "id": "https://pype.dev/fixing-firefox-launcher-on-kubuntu-22/", "url": "https://pype.dev/fixing-firefox-launcher-on-kubuntu-22/", "title": "Fixing Firefox Launcher on Kubuntu 22", "content_html": "\u003cul\u003e\n\u003cli\u003eProblem: Firefox installed but missing from KDE/Plasma app menu\u003c/li\u003e\n\u003cli\u003eRoot cause: No .desktop launcher file — Firefox only worked from terminal\u003c/li\u003e\n\u003cli\u003eSolution: Created desktop entry with standard KDE fields\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre\u003e\u003ccode\u003e\n[Desktop Entry]\nVersion=1.0\nName=Firefox\nGenericName=Web Browser\nComment=Web Browser\nExec=firefox %u\nTerminal=false\nIcon=firefox\nType=Application\nCategories=Network;WebBrowser;\nMimeType=x-scheme-handler/http;x-scheme-handler/https;\nStartupNotify=true\nStartupWMClass=firefox\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cul\u003e\n\u003cli\u003eLocation: ~/.local/share/applications/firefox.desktop (user-local, no sudo needed)\u003c/li\u003e\n\u003cli\u003eDiscovery step: Checked /usr/share/applications/ — empty for Firefox\u003c/li\u003e\n\u003cli\u003eApplied fix: Ran kbuildsycoca5 to rebuild Plasma\u0026rsquo;s menu cache\u003c/li\u003e\n\u003cli\u003eResult: Firefox now appears in app menu (Network category) and Alt+F2 autocomplete\u003c/li\u003e\n\u003c/ul\u003e\n", "content_text": "\n- Problem: Firefox installed but missing from KDE/Plasma app menu\n- Root cause: No .desktop launcher file — Firefox only worked from terminal\n- Solution: Created desktop entry with standard KDE fields\n\n```\n\n[Desktop Entry]\nVersion=1.0\nName=Firefox\nGenericName=Web Browser\nComment=Web Browser\nExec=firefox %u\nTerminal=false\nIcon=firefox\nType=Application\nCategories=Network;WebBrowser;\nMimeType=x-scheme-handler/http;x-scheme-handler/https;\nStartupNotify=true\nStartupWMClass=firefox\n\n```\n\n- Location: ~/.local/share/applications/firefox.desktop (user-local, no sudo needed)\n- Discovery step: Checked /usr/share/applications/ — empty for Firefox\n- Applied fix: Ran kbuildsycoca5 to rebuild Plasma's menu cache\n- Result: Firefox now appears in app menu (Network category) and Alt+F2 autocomplete\n", "summary": "Problem: Firefox installed but missing from KDE/Plasma app menu - Root cause: No .desktop launcher file — Firefox only worked from terminal - Solution:...", "date_published": "2026-04-11T10:32:50Z", "date_modified": "2026-04-11T10:32:50Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "agents", "tech" ] }, { "id": "https://pype.dev/data-loading-is-a-huge-deal/", "url": "https://pype.dev/data-loading-is-a-huge-deal/", "title": "Data Loading is a Huge Deal", "content_text": "\nI've been thinking about the work I am doing and have to do in my role at Cat,\nin Cat Autonomy, building Forge (see [[forge-ahead]]). I feel like I have\nlittle revelations almost every day now, not that it means I'm writing\nsomething amazing and producing it really fast but there's just a whole suite\nof problems that different technologies solve at different levels and the more\nI become aware of the problems that exist, the more the existence of some\nsolutions makes sense.\n\n!!! note \"The Problem Perspective\"\n\n I don't know if this is a real thinking technique or if I'm onto something\n novel(doubt) but I think a lot in terms of problems - \"what problem needs\n solving?\" and that's how I've come to prioritize my work, it's only been very\n recently that I've realized I do this and I think I should highlight it's very\n important to document the problem, otherwise every day you might try to solve a\n different problem but be working on the same code\n\n!!! note \"Problem Space\"\n\n Kedro solves a lot of these problems, so when making rada in Reman the problem\n space was already more contained and narrow, Forge's problem-space is much more\n vast\n\nThe one problem I'm really fixated on right now is data loading, the problem I\nneed to solve is accessing data from a wide-variety of scripts/tools, without a\nframework or standard method/library of accessing data in the first place. I've\nseen lots of projects on GitHub claim to make data loading easier and I didn't\nquite understand what problem they were solving... one example to name is [Data\nLoad Tool](https://dlthub.com/). I've seen similar ones I don't have the name\nfor right now that claim to make it easy/fast to load data from s3, or ways to\nmake s3 and a database both abstract in the user-experience for loading data.\nBut I hadn't really understood why these tools existed. I have a lot of\nexperience with [Kedro](https://dlthub.com/) and their\n[DataCatalog](https://docs.kedro.org/en/latest/catalog-data/introduction/)\nwhich provides a python object over a set of yamlfiles that makes it pretty\nsimple to load and save data in a way that isolates I/O from the business\nlogic. But what I didn't realize at the time how powerful that catalog was, the\npower of standard patterns and shared libraries. Now that I don't have it\navailable to me, I'm quite aware of the absence.\n\nIn my new role something I'm realizing is that for all the developers my team\nnow supports, there isn't a canonical way to access data. When I was in Reman\nand working with kedro, the DataCatalog was the access pattern and so when I\nwas developing a platform I never really had to think about it - it was an\nestablished pattern that I treated as a constraint and then built processes\naround it. I've been battling some mental block for weeks on Forge because of\nthe lack of that canonical pattern, and as I've talked with other engineers it\nseems like the baseline assumption is that data is just available on a\nfilesystem, but everyone's code loads data in different ways. On my small Reman\nteam, with common patterns to build on, it was easy to make things cloud-native\nor shim in some devops to improve people's lives. But when everyone's doing\ntheir own thing, and everyone's \"own thing\" is very much built-on some rigid tribal\npatterns then it's hard to really move fast cause everyone isn't already moving\nin the same direction.\n\nThat made me realize that the first problem Forge needed to solve was in\nproviding a way for engineers to have filesystem-native data access in the\nCloud, where we are S3-first in our storage philosophy. I didn't need to figure\nout a way for everyone to name a dataset, define the dataset in the first\nplace, and give a nice `my_dataset.load` that worked in python, bash, cpp, and\nwho knows what else.... I reframed the problem from \"how do engineers load up\nthe data\" to \"how do engineers have access to the data\". The requirements of\nthe Cat Autonomy group was pretty simple: POSIX-compliant storage.\n\nMy pathway to solving this problem is initially underway, I can't imagine it'll\nbe too difficult to setup for FSx instances for teams and give them an api to\nrun a Batch Job with the FSx mounted. From there, their code can load data from\n`/mnt/fsx/\u003cwhatever\u003e` just like they otherwise could be doing locally. Or maybe\nFSx will let us setup mounts to very flexible mount points and their local\nscripts will \"just work\" :shrugs:. I don't know the exact shape, but after\nrealizing the loading data is a big deal, I'm thankful I have a narrower\nproblem to solve first.\n\n!!! note \"S3 Files\"\n\n Literally yesterday, AWS launched \"S3 Files\" offering an NFS filesystem service over buckets. I'm not sure if NFS is going to be a viable filesystem protocol for all of our use cases, but looks like we're not the only people who need the filesystem access patterns over S3.\n\n!!! warning \"A Future Problem - Canonical Reference\"\n\n Another high-value thing Forge needs to solve is \"what is data\". The data\n formats we have are not super simple, it's not just a set of SQL tables. We\n have files that relate to each other based on hard-filepath patterns, and those\n patterns are full of tribal knowledge and distributed processes. So a simple\n question like \"How do I use forge to access my data\" is hard. In Reman a data\n scientist would ask \"how do I use rada to access my data?\" and the answer is\n \"We use Kedro, and Kedro solved that problem for us via the Catalog\" but\n without Kedro, without 100% being in python (devs are also in embedded systems,\n cpp code, and more), without even consistent practices in the existing \"how do\n I access my data\" workflows, it's really impossible to systemetize and codify\n it. It is my next challenge to tackle though...\n", "summary": "I've been thinking about the work I am doing and have to do in my role at Cat, in Cat Autonomy, building Forge (see forge-ahead). I feel like I have little...", "date_published": "2026-04-08T07:40:14Z", "date_modified": "2026-04-08T07:40:14Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "forge", "rada", "kedro", "tech" ] }, { "id": "https://pype.dev/agents-can-miss-obvious-things-too/", "url": "https://pype.dev/agents-can-miss-obvious-things-too/", "title": "Agents can miss obvious things too", "content_text": "\nI am working with MiniMax M.2 to vibe up some webapp ideas for [[Nexus]] and I\njust spent many minutes not having the app render correctly because the agent\nmounted the wrong directory into the docker compose stack. I noticed it about 5\nseconds after opening the file, but wasted nearly a half hour thinking it was a\ndocker issue in distrobox...\n\n!!! danger \"\"\n\n Clankers are still just clankers man\n", "summary": "I am working with MiniMax M.2 to vibe up some webapp ideas for Nexus and I just spent many minutes not having the app render correctly because the agent...", "date_published": "2026-04-03T14:39:51Z", "date_modified": "2026-04-03T14:39:51Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "agents", "til", "tech" ] }, { "id": "https://pype.dev/the-irony-of-10x/", "url": "https://pype.dev/the-irony-of-10x/", "title": "The Irony of 10x", "content_text": "\n## Opener\n\nAgentic coding has been an exciting change to me over the last couple of months\nspecifically. I've been using AI tools for a few years now but something really\nshifted with Opus 4.5 as well as the tools/harnesses getting better and more\nuseful around the same time. I've been influenced by people like Simon Willison\nand Steve Yegge who have been on the forefront of agentic coding and [vibe\nengineering](https://simonwillison.net/2025/Oct/7/vibe-engineering/) since the\ndawn of ChatGPT and in my small circles of work I'm definitely on the bloodiest\nbleeding edge of the adoption of these practices. Are they the future? I don't\nknow - I tried to maintain a skeptical posture but until the bubble pops it's\nlooking like this is at least a direction the future of my line of work is\ngoing.\n\n!!! danger \"\"\n\n The tough part is mixing the new world of agentic coding with\n developers on wildly different points in the spectrum of adoption and maturity.\n\nHere's a short anecdote about what I mean...\n\nIn a new project that I'm on, I've refactored/reimplemented a lot of legacy\ncode, producing about 140k lines of code, configuration, and\ndocs in about 2 months. It's an insane amount of \"product\" and I've done it\nentirely with agents. But I didn't do it with a handful of vanilla chat\nsessions like \"Hey Copilot, reimplement this API, no mistakes\". I've been building\nout my own process for using LLMs effectively. I have no actual idea besides my\nown experience if what I'm building is useful, but it feels pretty good -\nagents for planning, building, reviewing, testing, etc. My \"harness\" includes\nspecialized agents and opinionated development workflows\nto try to ensure that code is never one-shotted into production.\n\nThe catch though is that I'm working with a handful of developers and they are not\nearly-adopters or aggressive experimenters with these new agentic tools. Most\nof them are still in Steve Yegge's stage 2 or 3 of AI coding as he's outlined\n[in this medium\narticle introducing\nGasTown](https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04). They\nopen a chat session, and say yes or no to Opus. I've been in-between stages\n6 and 7 for a while now - managing multiple agentic sessions that themselves\nrun specialized subagents primarily for context management, although I don't\nfeel quite ready for full blown stage 8 (agents running agents running agents).\nSo this isn't me saying that I'm\n\"better\" than the guys I'm working with, just the project we're on\ntogether is now being touched by people on wildly different ends of the agentic\ncoding spectrum, and the project itself is an experiment to me of living in the \"New World\".\n\nFinally the anecdote - I'm getting PRs from guys in stages 2 and 3,\nthousands of lines of code and config, that I know they are not necessarily\nexperts in, but they're producing that code with fairly vanilla practices. How\ndo I verify it? I know my own prompts, I know what agents generated my code, I know\nthe direction and prodding I gave as they're building, but all\nI know in a PR review from someone else is the diff - not how they tested it\n(unless of course there's tests, and there must be, but tests are only as good\nas the tests are...), not the steering they gave, not the manual UAT validation\nthey did, or even the full intent... And ultimately I own the code they merge\nbecause chances are I outlast their contract with the team. It's kind of a\nscary thing to review and accept... do I trust my\nagentic practices to validate their work? Cause I certainly don't trust myself\nto validate it perfectly.\n\n!!! note \"a note about contractors\"\n\n I'm not implying anything about anyone who works contract, I think the facts is that contractors are typically dis-incentivized to really \"own\" something - an unfortunate consequence of the world.\n\n## This is about me, not everyone\n\nI want to make sure to reiterate the context of this post and my thoughts -\nit's really about me. I'm not saying anything here is true for anyone else, I'm\nnot making predictions about tomorrow, and I'm not a prominent FOSS developer or a\nhigh-profile ex-FAANG engineer. I'm not talking about every project under the sun.\nI work at a big company, not a fast-paced startup. I work on internal tooling,\nnothing that should even be seen outside our network. I'm also a self-taught\ndeveloper, not a trained software engineer.\n\nAs far as AI goes, lately I'm mostly all in on Github Copilot\nCLI with a sprinkling of Opencode. I'm not using Claude Code or Codex which\nseem to have their own communities around plugins and usage.\n\nAnd since we're so focused on me right now, the important thing to keep as the\nbackdrop for this post is my temperament and make-up. I'm definitely made a\ncertain kind of way, not different than every single person necessarily, but\ndifferent enough, from enough people, to not blend in with the 9-5 guy who can\nlog out and not think about work until the morning (I'm not saying that's a\nstrength either, my boundary problems are for another time).\n\nI feel an aggressive burden to solve problems and own those solutions, call it\nwhite-knighting or a savior complex if you want, but I've got enough of a\nreputation at Caterpillar now (and anywhere else I've worked) to become a go-to\nperson for more than I think is necessarily appropriate. And with that burden\nI'll bring results - if a problem hooks into my brain it. will. be. solved. I\nprobably won't do the best most clean-code solution right out of the gate, but\nI'll do whatever I can to find a solution.\n\n## Who I was before agents\n\nThat is who I was before agents made code easy to produce. My first boss at\nCaterpillar used the phrase \"tenacious learner\" to describe me in several\nreviews. I kind of rejected the description because I basically refused to\nbelieve that I was really any different, any harder of a worker, than my peers.\n\nBut with almost a decade of experience in the corporate world, and some adult\nperspective on my life, I think it's accurate... I am a harder worker, to my\ndetriment sometimes, than a fair number of people I've worked with\\*\\*.\n\nI'm not brilliant but I can focus for a long time in the right circumstances.\nThe gift of perseverance (or the curse of not being able to let something go,\ndepending on how you look at it) has led to blessing in my life in both reward\nand skill.\n\n!!! note \"\\*\\*\"\n\n There's nothing wrong with it either, Cat's somewhat noticed that work in my EOY reviews and I'm certainly not against \"just doing your job\".\n\nAnd then AI came along and with another set of the right circumstances\ncatalyzed a new way to work.\n\n## Leaning into \"Agentic Engineering\"\n\nI jumped onto experimenting with AI coding tools as soon as they became\navailable, but mostly I just tried vibe coding rather than using tools for real\nengineering work. I vibed up an API at one of my jobs that went into production\nway too early, with far too little validation, and it was scary to support it\nfrom then on out. I also did the meme, vibe-coded a TODO app, and threw that\npuppy into the internet without locking down my API endpoints... That was\nbefore agents were quite as useful as they can be\nnow, but that experience along with a handful of other stepping stones (like\nlearning some real actual fundamentals about security) began to\ngive me confidence in using the AI as a tool, like my IDE is a tool, for\nproducing **solutions** that take the form of code.\n\n!!! note \"\"\n\n As one-shot apps got better and better, and as I learned about scoping work\n more appropriately for agentic tools my confidence in them grew.\n\nThere's quite a difference between \"Claude make me a todo app, no mistakes\" and\nscoping out a solution in natural language, with some technical guardrails, and\nhaving agents tackle the implementation methodically.\n\n## What actually changed\n\nWhat's actually changed for me is quite a lot... I haven't opened my IDE to\nseriously write code for months now. I've oscillated between Opencode and\nCopilot CLI, leaning moreso into Copilot since it's an approved tool at work\nand as of mid-February is quite good. Mentally I'm approaching problems with a\nlittle more thought on the front-end than before because prior to agents I\nwould think as I implemented. At the scale of work that I do, this was really\nfine - working on CLI utilities to solve simple problems, developing an\niterative testing cycle for each problem that allowed me to move fast, and once\nI found a groove I was cooking. But now I don't even need to find it, I open\nOpencode or Copilot CLI with my Planner agent, describe what I want to happen and have\nOpus or GPT scope out a plan for me. Usually there's some back and forth on\nfeature scoping, then I review a markdown file it produces, and once it looks\ndecent enough to me I say \"go\" and it goes.\n\nThat works a lot better than I even care to admit because at the same time as\nI've been leaning harder into agents, I've been building my own harness of\nsorts - not a replacement for Copilot CLI or a competitor to Opencode, but\nmoreso an opinionated workflow spine that I force agents into to give strict\ngates to the SDLC (software development lifecycle).\n\n!!! warning \"Problem Solving Workflows\"\n\n Plan and implement is fine for a lot of things, and I do think it's only getting better. My harness,\n mentioned a few times around here before, called Nexus, is a set of agents and\n rules that I want the code I'll be responsible for to go through before it\n lands in production. That cycle isn't too complex, and there's only about\n 10,000 similar tools to Nexus on Github trending right now. I've thought about dropping my idea and picking up\n something more popular, like\n [superpowers](https://www.github.com/obra/superpowers) but at the moment I'm\n continuing to develop on and lean into my own idea here.\n\n!!! note \"Mini post on Nexus\"\n\n I keep saying a blog post is coming, but the high level of Nexus is that it's a task\n tracker with a CLI that agents use to advance a ticket through a plan -\u003e build\n -\u003e test -\u003e review -\u003e verify -\u003e merge lifecycle that is almost exactly how I\n would otherwise have solved a problem by hand. I think it needs work, I need to\n be harder on TDD methodlogies with agents, and work on verification gating a\n bit more (shoutout to [showboat](https://github.com/simonw/showboat) by Simon\n Willison) but overall it's a system of thinking that I already participate in\n so I'm doing my best to farm out specific parts of my workflow to agents rather\n than trying to one-shot enterprise problems and solutions.\n\n!!! danger \"Who's doing the thinking?\"\n\n I've noticed that as I've developed Nexus out though, I lean on the agents for\n more and more of my own thinking, and am trusting my problem solving\n **process** moreso than my actual problem solving abilities.\n\n## Hidden costs\n\nThe cost of this increase in speed is a lack of familiarity - and the fallout\nof lack of familiarity is hard to express. There's also many facets to it.\nFor me, the first facet is that Nexus helps me move fast, but as I've leaned\ninto it for more and more of the planning, I'm less and less familiar with the\nstate of the code. I find myself asking my reviewer agents in fresh sessions\noften to explain it to me, and thankfully they're usually consistent, but\nnonetheless I'm still not intimately familiar with the code. And on Nexus it's\nnot a big deal, that's low stakes, it's just me and my workflow.\n\nI'm using Nexus + Copilot at work and that feels like higher\nstakes... I have my agents explain the status of our project and although they're\nalso somewhat consistent the thing that's scary is that other people are\nworking on that repo with me, and that's where another layer of complexity\nmanifests itself. If it's just me and my [[clankers]], let's go all day long,\nrebuild, ask questions, etc... but I have other developers I rub shoulders with\nnow, and if they ask me a question what am I going to say? \"Hold on, let me prompt\nmy agent for you\" - it's LMGTFY on steroids. And the burden becomes if I\nfeel like I can own and support what those other developers push into the repo.\n\n## Murky responsibility boundaries\n\nWhy do I own their work? Well for the third time, this post is pretty\nself-centered and all about me, and my situation is that the other developers I\nwork with presently are all contractors. Their work agreement with Cat could\nend at any second, for practically any reason. The incentive structure isn't\nthere for these guys who technically work for an agency... Their bonuses aren't\nbigger (or even exist) if Cat performs well, there's no extra vacation days in\nit for em (aye, contractors don't get vacation days anyways), and not that it's\na problem, moreso just the nature of the world we're in - but they're basically\nmercenaries out to the highest bidder and I happen to know of **multiple times\nwhere Cat lost a good person to a higher bidder**.\n\nSo this isn't really me trying to be negative about contractors at all, I'm\nhere for a pay-check as well but Cat at least gives me SOME incentive to work\nhard with the goal of compensation regardless of how altruistic I feel in my\nown circumstances.\n\n!!! note \"Incentive\"\n\n Better ratings mean marginally better end-of-year salary increases, and I've received some other awards that certainly give me pause about jumping ship to another long-term place even when things can be crazy at Cat.\n\nIt's more than just the contrator-ownership dilema, I've dug myself quite a\nhole over the last 8-10 years, gaining a reputation that I think many would\nappreciate, but for me only lately increases the stress. I don't need to parrot\nevery accolade I've ever received, that's not the point, but to make the point\nas clear as I can - I have a lot of respect from quite a few people at\nCaterpillar. I'm blessed to have that reputation, and it's not like I haven't\nworked hard for it - but people talk about me in a way so flattering I feel\nlike the main character in a fictional story sometimes.\n\nIn a fictional story I can check out the ending, hit up spark notes, or ask AI\nhow it ends... but there isn't an \"end\" in my real world scenario, there's only\ntomorrow and I feel the pressure of not knowing what tomorrow holds now more\nthan ever.\n\n!!! danger \"\"\n\n Being noticed is starting to feel more costly than rewarding...\n\n## Financial irony\n\nWhat's the cost? It's hard to get specific without writing a novel but here's\nthe TLDR - because I've been pretty good at what I do I've been able to do this\ntype of work outside my normal 9-5 responsibilities and with that extra work\nhas been some pretty great financial benefits. However with Cat changes,\nresponsibility increases, and now owning code that others (and their clankers)\nwrite, the extra time I gained for myself is eaten-up and has been reclaimed by\nthe mega-corp... \"Exceeding expectations\" every year just meant the bar is\nraised, the expectations are higher, the time-commitment requirement is higher,\nand as I've had to meet the requirements of both the new world and the curse of\nbeing noticed, I've lost the time for the extra work... For years I've realized\nthe benefit of my own skills and drive, but the irony of agents (and a handful\nof other things) is that with the dramatic increase in expectations, not only\non me but on those I work with and therefore their output, I don't get to\nrealize the benefits of my own gifts anymore.\n\n## Meaning and fatigue\n\nI feel very torn because the work I've been called into with Cat is good, I\nsaid in [[cat-autonomy-2-0]] that autonomy will save people's lives. I love\ngetting to participate in that mission, it's the primary reason I didn't jump\nship to try to maintain the levels, and type, of work I was doing before... But\nin a few short months the mission is being drowned out by expectations and\nrequirements that are so high I'm losing the grip on my own life.\n\n## Open questions\n\nThat leads me to questions that I can't answer, the question I ask daily now of\n\"What about tomorrow?\". What will agents do for us tomorrow, what problems will\nbe solved, what bugs will I create (by agents of course because I've never\nwritten a bug by hand in my whole life \\s). If I stopped using agents would\npeople still be impressed? Would it even matter?\n\n## Fin\n\nI'm certainly not anti-AI, it's typing all my code. I'm not anti-collaboration,\nalthough I do wish I could work alone with just my clanker-army to worry about.\nI'm not sure what I am anymore though... AI has changed how I work, what I work\non, and who I work with... Everything has changed in such a short period of\ntime and like the ending of this post, it's pretty jarring.\n\n!!! danger \"\"\n\n Death comes to us all - James Acaster.\n\nThanks for reading.\n", "summary": "Agentic coding has been an exciting change to me over the last couple of months specifically. I've been using AI tools for a few years now but something...", "date_published": "2026-03-26T06:00:50Z", "date_modified": "2026-03-26T06:00:50Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "ai", "tech", "agents", "personal" ] }, { "id": "https://pype.dev/remote-terraform-state-requires-working-traefik-duh/", "url": "https://pype.dev/remote-terraform-state-requires-working-traefik-duh/", "title": "Remote Terraform State Requires Working Traefik… DUH!", "content_html": "\u003cp\u003eI\u0026rsquo;m working on some spring cleaning in my homelab and backed myself into a\nhilarious corner yesterday. I use Open Tofu for any of my Terraform needs now,\nand although I don\u0026rsquo;t manage a ton with terraform, I do manage all my cloudflare\nstuff with it. I decided I wanted to use my own minio instance as the s3 remote\nstate backend for my workspaces so I could rely on my typical NAS data\nbackup/retention workflow for the buckets in case anything went wrong, as\nopposed to a local state file that I\u0026rsquo;m not taking a lot of precautions with.\nWell during my Spring Cleaning I was working towards replacing ingress into my\nhome network with Cloudflare tunnels and in the midst of that update I took\ndown traefik, no matter a simple \u0026rsquo;tofu apply\u0026rsquo; should get me right back to\nworking order\u0026hellip;\u003c/p\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"err\"\u003e╷\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"err\"\u003e│\u003c/span\u003e \u003cspan class=\"k\"\u003eError\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"k\"\u003eError\u003c/span\u003e \u003cspan class=\"k\"\u003einspecting\u003c/span\u003e \u003cspan class=\"k\"\u003estates\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"k\"\u003ethe\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;s3\u0026#34;\u003c/span\u003e \u003cspan class=\"k\"\u003ebackend\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"err\"\u003e│\u003c/span\u003e \u003cspan class=\"k\"\u003eoperation\u003c/span\u003e \u003cspan class=\"k\"\u003eerror\u003c/span\u003e \u003cspan class=\"k\"\u003eS3\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"k\"\u003eListObjectsV2\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"k\"\u003ehttps\u003c/span\u003e \u003cspan class=\"k\"\u003eresponse\u003c/span\u003e \u003cspan class=\"k\"\u003eerror\u003c/span\u003e \u003cspan class=\"k\"\u003eStatusCode\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e404\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"k\"\u003eRequestID\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"k\"\u003eHostID\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"k\"\u003eapi\u003c/span\u003e \u003cspan class=\"k\"\u003eerror\u003c/span\u003e \u003cspan class=\"k\"\u003eNotFound\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e \u003cspan class=\"k\"\u003eNot\u003c/span\u003e \u003cspan class=\"k\"\u003eFound\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHilarious problem with thankfully an easy fix\u0026hellip; downloading the state file\nfrom Minio wasn\u0026rsquo;t a big deal since the container was still running without\nissue, and placing the state file in the folder to use as the local state\nsolution for the interim went totally smooth, but this highlights the set of\ninterdependencies I\u0026rsquo;m creating for myself and as I take the next few days/weeks\nto do some spring cleaning I\u0026rsquo;m hoping I can separate out the external ingress\nfrom internal with a bit more clear boundaries so that I never lock myself out\nof a workflow I only execute on my LAN in the first place!\u003c/p\u003e\n", "content_text": "\nI'm working on some spring cleaning in my homelab and backed myself into a\nhilarious corner yesterday. I use Open Tofu for any of my Terraform needs now,\nand although I don't manage a ton with terraform, I do manage all my cloudflare\nstuff with it. I decided I wanted to use my own minio instance as the s3 remote\nstate backend for my workspaces so I could rely on my typical NAS data\nbackup/retention workflow for the buckets in case anything went wrong, as\nopposed to a local state file that I'm not taking a lot of precautions with.\nWell during my Spring Cleaning I was working towards replacing ingress into my\nhome network with Cloudflare tunnels and in the midst of that update I took\ndown traefik, no matter a simple 'tofu apply' should get me right back to\nworking order...\n\n```hcl\n╷\n│ Error: Error inspecting states in the \"s3\" backend:\n│ operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NotFound: Not Found\n```\n\nHilarious problem with thankfully an easy fix... downloading the state file\nfrom Minio wasn't a big deal since the container was still running without\nissue, and placing the state file in the folder to use as the local state\nsolution for the interim went totally smooth, but this highlights the set of\ninterdependencies I'm creating for myself and as I take the next few days/weeks\nto do some spring cleaning I'm hoping I can separate out the external ingress\nfrom internal with a bit more clear boundaries so that I never lock myself out\nof a workflow I only execute on my LAN in the first place!\n", "summary": "I'm working on some spring cleaning in my homelab and backed myself into a hilarious corner yesterday. I use Open Tofu for any of my Terraform needs now, and...", "date_published": "2026-03-23T08:12:25Z", "date_modified": "2026-03-23T08:12:25Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "traefik", "tech" ] }, { "id": "https://pype.dev/to-live-in-a-world-without-ai/", "url": "https://pype.dev/to-live-in-a-world-without-ai/", "title": "To Live In A World Without AI", "content_html": "\u003cp\u003eI\u0026rsquo;m finding lately that I wish we could go back to pre-ChatGPT\u0026hellip; A world\nwithout a code-gen easy button, where \u0026ldquo;easy\u0026rdquo; was LSP autocomplete, where tools\nwere at my fingertips rather than remote inference endpoints, and where I and\neveryone I worked with, was paid and judged based on what we could produce with\nour own 2 hands, even if \u003ccode\u003ectrl + c\u003c/code\u003e and \u003ccode\u003ectrl + v\u003c/code\u003e was sometimes a common set of keys\u0026hellip; it\nsure beat answering \u0026ldquo;yes\u0026rdquo; to a robot every 38 seconds.\u003c/p\u003e\n", "content_text": "\nI'm finding lately that I wish we could go back to pre-ChatGPT... A world\nwithout a code-gen easy button, where \"easy\" was LSP autocomplete, where tools\nwere at my fingertips rather than remote inference endpoints, and where I and\neveryone I worked with, was paid and judged based on what we could produce with\nour own 2 hands, even if `ctrl + c` and `ctrl + v` was sometimes a common set of keys... it\nsure beat answering \"yes\" to a robot every 38 seconds.\n", "summary": "I'm finding lately that I wish we could go back to pre-ChatGPT... A world without a code-gen easy button, where \"easy\" was LSP autocomplete, where tools were...", "date_published": "2026-03-22T14:50:38Z", "date_modified": "2026-03-22T14:50:38Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "ai", "tech" ] }, { "id": "https://pype.dev/paynepride-dot-com-outage-on-vacation/", "url": "https://pype.dev/paynepride-dot-com-outage-on-vacation/", "title": "paynepride dot com outage on vacation", "content_html": "\u003cp\u003eThe day after I leave for vacation I start getting SSL errors on every homelab\nservice I host for myself and others. The culprit was my Cloudflare API token\nexpiring. It was easy to find the 403s in the logs for Traefik (thank goodness\nfor Tailscale getting me into the lab from afar). The solution was to rotate the\nAPI token, replace the value in Traefik\u0026rsquo;s .env file, and hit it with the \u0026ldquo;just\ndeploy\u0026rdquo; button. Now I don\u0026rsquo;t know why this expired - the key looks like it has\nno expiration to me - and I\u0026rsquo;m too tired from the beach to dig in further.\nUntil next time, I expect this error to come back March 16 2027 I suppose.\u003c/p\u003e\n", "content_text": "\nThe day after I leave for vacation I start getting SSL errors on every homelab\nservice I host for myself and others. The culprit was my Cloudflare API token\nexpiring. It was easy to find the 403s in the logs for Traefik (thank goodness\nfor Tailscale getting me into the lab from afar). The solution was to rotate the\nAPI token, replace the value in Traefik's .env file, and hit it with the \"just\ndeploy\" button. Now I don't know why this expired - the key looks like it has\nno expiration to me - and I'm too tired from the beach to dig in further.\nUntil next time, I expect this error to come back March 16 2027 I suppose.\n", "summary": "The day after I leave for vacation I start getting SSL errors on every homelab service I host for myself and others. The culprit was my Cloudflare API token...", "date_published": "2026-03-16T13:21:31Z", "date_modified": "2026-03-16T13:21:31Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "ofc", "tech" ] }, { "id": "https://pype.dev/my-thoughts-on-beads/", "url": "https://pype.dev/my-thoughts-on-beads/", "title": "My Thoughts on Beads", "content_text": "\n[Steve Yegge](https://en.wikipedia.org/wiki/Steve_Yegge) is a pretty well-known individual in the tech field, having been\naround for a long time at some of the larger companies. He's making quite a\nsplash in the agentic coding world as well. I've appreciated Steve's posts and\nprojects lately and wanted to put some thoughts on one called\n[beads](https://github.com/steveyegge/beads).\n\n## Beads\n\nBeads is an issue tracker with links - issues relate to and block each other,\nbut agents use beads to keep track of information and dependencies without\nstoring it in their context 100% of the time. It seems like a very popular and\nuseful tool - but I am not using it, and that's what I wanted to capture... why\nnot?\n\nThe answer for me is about **where** the organization layer is for the\ndeveloper. Beads exists in a single repo - it's a system-wide CLI but you 'bd\ninit' in a git repo, and beads uses the `.git/` folder, worktrees, [dolt](https://docs.dolthub.com/), and some\ngit hooks to operate within that git repo. Outside the repo, it takes another\ntool to tie together all the beads databases you might have.\n\nFor me, I'm hardly \"in\" a git repo anymore. My workflow is that when I have\nsomething to work on, I create a \"workspace\" ([self-defined concept](https://pypeaday.github.io/dotfiles/terminal/workspaces/#installation)) which is\njust a folder on my filesystem where I check-out git worktrees from any of the\nrepos related to the work I'm doing. Sometimes it's 1 worktree from 6 repos,\nsometimes it's 6 worktrees from 1 repo for parallel work...\n\nSo because I like to organize myself in this way, beads is already \"out\" for\nme. That's the main reason - I don't have any real technical issues with beads\nor any criticism, it just is designed for a workflow that is not how I work.\n\nThis is why I'm building [[nexus]], something I hope to be able to put out\nthere \"soon\". It won't be as general-purpose as beads, but my goal with it is\nto be plug-and-play for any agentic harness (copilot cli, claude code,\nopencode, etc.). It's a challenge thinking about it as a personal tool but also\nas a tool to share someday, but agentic coding is making it possible to make\nsome cool shareable stuff and I'm excited for my own workflow-task-manager to\nmature and at least become something useful to me (it already is, but building\nthe plane in the air makes it kind of hard to enjoy the plane).\n\n### Credit\n\n- banner image from ChatGPT\n", "summary": "Steve Yegge is a pretty well-known individual in the tech field, having been around for a long time at some of the larger companies. He's making quite a...", "date_published": "2026-03-03T05:00:47Z", "date_modified": "2026-03-03T05:00:47Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "ai", "tech" ] }, { "id": "https://pype.dev/forge-ahead/", "url": "https://pype.dev/forge-ahead/", "title": "Forge Ahead", "content_text": "\nYesterday's [[reflection-contentment-and-work]] has a second-part this morning.\nAs I was wrapping up a project I didn't realize the closed-off-ness of\nleaving... I handed in some notes, and mid-message to someone the clock struck\nmidnight and I was locked out. It's fitting to be honest, and now in the wake\nof yesterday's contentment post, experiencing some more loss than I expected,\ntoday we forge ahead.\n\nMy main focus for work now is a project I will refer to as `forge`. It is what\nI will bring to Caterpillar Autonomy but I can only build it thanks to the\ngifts and experience God has given me. The thing I want to put on paper is a\nshort list of experiences I think God in his sovereignty, gave me over the last\nfew years and what they provide for me for Forge.\n\n## Cat Reman Platform\n\nOne of the first things that comes to mind is another project I was sad to\nlose: a platform I started for an analytics team in Cat Reman. That began as a\nsimple python cli to automate some developer operations that was otherwise a\ndozen clicks through the AWS console. Using [boto3] and some simple python I\ngave that team the start of some real velocity gains. Eventually that grew into\na larger kubernetes-based service where the data science operations: code quality,\ndeployment, updates, webapps, etc. were all handled by our platform. It wasn't\nperfect, it wasn't self-contained, it was a set of things kind of glued\ntogether with systems and scripts, but it worked, it works today and is under\nfantastic ownership.\n\nWhat I learned just from starting that cli was to be passionate about solving\nproblems. No one asked me to make it, but it needed made, and the team is in\nsuch a better place for me having started it.\n\n## Kedro and OpenShift\n\nAnother short project I was able to participate in a few years ago was leading\na data-syncronization task into a fiery horrific crash that lasted weeks -\nneigh months longer - than was necessary or appropriate... The details aren't\nrelevant - I was the lead dev in a new place tasks with syncing up data between\n2 applications. Ultimately, could've been a python script but I over-designed a\nkedro-based solution because of some requirements I misunderstood. Part of that\nmisunderstanding was not knowing how to get the requirements I needed, but they\nweren't provided in full, I didn't know any better, so for weeks we were a\ncorporate meme trying to use python to update a database that no one on our\nteam understood if we were allowed to write to.... It was very confusing.\n\nBut what I learned was a lot about gathering requirements, questioning\nassumptions, and the importance of understanding your constraints as fully as\npossible as early as possible.\n\n## A Real Platform\n\nThen one of the biggest blessings I see for Forge, is the experience I've\ngotten recently with AWS at scale... Not 1 or 2 services created with\nMedium.com copy pasta tutorials from docs... but experience working in a large\nproject across many accounts, supporting several teams, using a wide variety of\ntechnologies from Terraform to Kubernetes.\n\nOn this project I learned about practical system design and gained a lot of\nconfidence in supporting systems that are complicated... I'm not the smartest\nguy in the world but I'm no dummy, and even while lacking fluency in the system\nI was supporting on this project I learned a lot of troubleshooting skills and\ngained confidence in my ability to troubleshoot complex systems.\n\nI'll need this for Forge - which will break certainly, but I'll be there to fix\nit and I'm sure I can because I've done these other things.\n\n## Fin\n\nThe Lord has been with me through these projects - he has certainly blessed me\nwith a skillset and personality that lends itself to being really useful in the\nTech world I've landed in (which is a whole 'nother story of God's sovereignty\nand provision). I am very grateful for the swath of experience I've gotten over\nthe last 8 years or so, and this season I'm in of change is rocking me a little\nmore than I anticipated but by God's grace I think I see the purpose, or at\nleast **a** purpose, and I pray I am making the choices for work he wants me to\nmake.\n", "summary": "Yesterday's reflection-contentment-and-work has a second-part this morning. As I was wrapping up a project I didn't realize the closed-off-ness of leaving......", "date_published": "2026-02-17T05:14:15Z", "date_modified": "2026-02-17T05:14:15Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "work", "tech" ] }, { "id": "https://pype.dev/kubernetes-external-secrets-operator/", "url": "https://pype.dev/kubernetes-external-secrets-operator/", "title": "Kubernetes External Secrets Operator", "content_text": "\nI wanted to put a short demo together of using External Secrets Operator (ESO)\nto expose secrets from a vault (like Hashicorp Vault, AWS Secrets Manager, etc)\nto services running in kubernetes\n\nDemo code is [here in this github repo](https://github.com/pypeaday/blog-kubernetes-external-secrets-operator-demo)\n\nThis post is a high level overview of the components, see the repo for the full example.\n\n## Setup\n\n- [[docker]] for containerized development\n- [[kind]] for setting up a quick cluster\n- [[kubectl]] for accessing the cluster\n- [[helm]] for installing ArgoCD and ESO\n\n- and then [justfile](https://github.com/casey/just) is there to wrap the commands to easier execution\n\n## Step 0 - Vault\n\n- for the demo we'll setup Hashicorp Vault in docker compose to easily bring it\n up and down\n- and the init-script is in the repo - it uses curl to make some secrets in\n vault that we'll reference later\n\n```yml\nservices:\n vault:\n image: hashicorp/vault:1.18\n container_name: vault\n ports:\n - \"58200:8200\n environment:\n VAULT_DEV_ROOT_TOKEN_ID: root\n VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200\n volumes:\n - vault-data:/vault/file\n cap_add:\n - IPC_LOCK\n command: server -dev -dev-root-token-id=root\n\nvolumes:\n vault-data:\n```\n\n- bringing up the vault instance is a simple `docker compose up` (use the just recipes which some `curl` commands for checking status etc.)\n\n```bash\ncurl -s http://localhost:58200/v1/sys/health | jq . # or just vault-status\n{\n \"initialized\": true,\n \"sealed\": false,\n \"standby\": false,\n \"performance_standby\": false,\n \"replication_performance_mode\": \"disabled\",\n \"replication_dr_mode\": \"disabled\",\n \"server_time_utc\": 1770893657,\n \"version\": \"1.18.5\",\n \"enterprise\": false,\n \"cluster_name\": \"vault-cluster-acfb9930\",\n \"cluster_id\": \"4d9162f4-e501-371b-7f94-bd60052b40a3\",\n \"echo_duration_ms\": 0,\n \"clock_skew_ms\": 0,\n \"replication_primary_canary_age_ms\": 0\n}\n```\n\n## Step 1 - App\n\n- We need an app that requires secrets\n- app code in repo, essentially it's a python webserver to show the vault\n values (obviously this would expose real secrets so it's just a demo)\n- Below is one of the endpoints in the vibe-coded app, just to illustrate that\n we're going to give secrets to the app as environment variables (or as mounted\n files!)\n- In the repo, the app is included and there's a `just build` which builds the docker image\n- There is also `just deploy` which handles loading the image into `kind`'s image cache\n\n```py\n\n# example route from demo-app - see git repo\n@app.get(\"/env\", response_class=HTMLResponse)\ndef show_env():\n # ESO brings Vault secrets into environment variables\n env_vars = dict(os.environ)\n\n # Sort by category, then by key\n sorted_items = sorted(env_vars.items(), key=lambda x: (classify_env(x[0]), x[0]))\n\n cards = \"\".join(create_card(k, v) for k, v in sorted_items)\n\n secret_count = sum(1 for k in env_vars if k in SECRET_KEYS)\n config_count = sum(1 for k in env_vars if k in CONFIG_KEYS)\n system_count = len(env_vars) - secret_count - config_count\n\n html = HTML_TEMPLATE.format(\n cards=cards,\n secret_count=secret_count,\n config_count=config_count,\n system_count=system_count,\n )\n\n return HTMLResponse(content=html)\n\ndef read_mounted_files(directory: str) -\u003e dict:\n \"\"\"Read all files from a mounted directory.\"\"\"\n files_data = {}\n if os.path.exists(directory) and os.path.isdir(directory):\n for filename in os.listdir(directory):\n filepath = os.path.join(directory, filename)\n if os.path.isfile(filepath):\n try:\n with open(filepath, \"r\") as f:\n files_data[filename] = f.read().strip()\n except Exception as e:\n files_data[filename] = f\"\u003cError reading file: {e}\u003e\"\n return files_data\n\n```\n\n## Step 2 - Cluster\n\n- use `kind` to bring up a cluster\n- this will start a few docker containers to act as your control-plane and workers\n\n```yml\n# kind-config.yml\nkind: Cluster\napiVersion: kind.x-k8s.io/v1alpha4\nname: eso-demo\nnodes:\n - role: control-plane\n extraPortMappings:\n - containerPort: 30080\n hostPort: 58080\n protocol: TCP\n - role: worker\n```\n\n```\nkind create cluster --config kind-config.yaml --name eso-demo\n```\n\n## Step 3 - External Secrets Operator\n\n- installed with [[helm]] from the official helm chart\n- NOTE: this is the Operator, not the secrets... this is the thing which goes\n to the secrets backend and creates kubernetes secrets\n\n```\nhelm repo add external-secrets https://charts.external-secrets.io 2\u003e/dev/null || true\nhelm repo update\nhelm install external-secrets external-secrets/external-secrets \\\n --namespace external-secrets \\\n --create-namespace \\\n --wait\n```\n\nIn the repo this is mostly `just eso-install`\n\n## Step 3.5 - Secretstore\n\n- You need a `clustersecretstore` to be the place that ESO puts secrets\n\n```\napiVersion: external-secrets.io/v1\nkind: ClusterSecretStore\nmetadata:\n name: vault-backend\nspec:\n provider:\n vault:\n server: \"http://10.10.0.1:58200\"\n path: \"secret\"\n version: \"v2\"\n auth:\n tokenSecretRef:\n name: vault-token\n key: token\n namespace: external-secrets\n\n```\n\n## Step 4 - Secrets\n\n- Secrets go in the `clustersecretstore`\n - in this example it's called 'vault-backend'\n- In the demo we can just `kubectl apply -f \u003cmanifest\u003e` to deploy the secret to\n the cluster\n- In practice this should be handled by something more mature than raw-doggin\n kubectl commands\n\n```yml\n# manifests/external-secrets.yml\n---\napiVersion: external-secrets.io/v1\nkind: ExternalSecret\nmetadata:\n name: demo-app-secrets\n namespace: default\nspec:\n refreshInterval: \"10s\"\n secretStoreRef:\n kind: ClusterSecretStore\n name: vault-backend\n target:\n name: demo-app-secrets\n creationPolicy: Owner\n data:\n - secretKey: DATABASE_PASSWORD\n remoteRef:\n key: secret/data/demo-app/secrets\n property: database_password\n - secretKey: API_KEY\n remoteRef:\n key: secret/data/demo-app/secrets\n property: api_key\n```\n\n## Step 4.1 - Files\n\n- ESO supports mounting files to containers as well through special `ExternalSecret` resources\n- One of the example seecrets is a TLS certificate\n\n```yml\n# manifests/external-secrets-files.yml\n---\n# File-based ExternalSecret for TLS certificates\n# These will be mounted as files in /etc/secrets/\napiVersion: external-secrets.io/v1\nkind: ExternalSecret\nmetadata:\n name: demo-app-tls-files\n namespace: default\nspec:\n refreshInterval: \"10s\"\n secretStoreRef:\n kind: ClusterSecretStore\n name: vault-backend\n target:\n name: demo-app-tls-files\n creationPolicy: Owner\n # Template to ensure proper file formatting\n template:\n type: Opaque\n data:\n tls.crt: \"{{ .tls_crt }}\"\n tls.key: \"{{ .tls_key }}\"\n data:\n - secretKey: tls_crt\n remoteRef:\n key: secret/data/demo-app/tls-files\n property: tls.crt\n - secretKey: tls_key\n remoteRef:\n key: secret/data/demo-app/tls-files\n property: tls.key\n```\n\n- Notice how there's a `spec.target.template` which templates out the file\n contents from the secret contents\n\n## Step 5 - Helm Chart\n\n- This isn't about setting up a helm chart so I'm not going to explain a lot\n but the working example is simple, not secure, and in the repo\n- The helm chart renders manifests - I've paired one down and added comments to\n the relevant things\n- The thing to just take note of is the reference of the secrets in the `envFrom` section\n\n```yml\n# deployment.yml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n annotations:\n meta.helm.sh/release-name: demo-app\n meta.helm.sh/release-namespace: default\n name: demo-app\n namespace: default\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app.kubernetes.io/instance: demo-app\n app.kubernetes.io/name: demo-app\n spec:\n containers:\n - envFrom:\n - secretRef:\n name: demo-app-secrets # name of example secret from section 4\n - secretRef:\n name: demo-app-config # another example in the repo\n image: demo-app:latest # the image you built and loaded into kind - simple 'just' recipe in the repo\n imagePullPolicy: Never\n name: demo-app\n volumeMounts:\n - mountPath: /etc/secrets\n name: secrets-volume\n readOnly: true\n - mountPath: /etc/config\n name: configs-volume\n readOnly: true\n volumes:\n - name: secrets-volume\n secret:\n defaultMode: 420\n secretName: demo-app-tls-files # example secret file from section 4.1\n - name: configs-volume\n secret:\n defaultMode: 420\n secretName: demo-app-config-files\n```\n\n## Step 5.1 - Deploy\n\n- We can deploy the demo-app from the git repo to the cluster\n- For a local demo a few things happen\n - local image build\n - loading that image into [[kind]] (`kind` doesn't have access to your host's docker image cache, so images need to be loaded into the cluster cache)\n- `just deploy` takes care of this for you, read the recipe in the repo if\n you're interested in more there, the focus of this post and example are to\n briefly show how to use ESO though\n\n## Step 6 - Profit\n\nThe example app just displays things that are mounted in - totally vibe-coded\nto illustrate the secrets mounting, not the appropriate way to leak secrets.\n\n![20260210233804_614254b7.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260210233804_614254b7.png)\n\n![20260210233828_19852225.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260210233828_19852225.png)\n", "summary": "I wanted to put a short demo together of using External Secrets Operator (ESO) to expose secrets from a vault (like Hashicorp Vault, AWS Secrets Manager,...", "date_published": "2026-02-10T07:48:34Z", "date_modified": "2026-02-10T07:48:34Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "devops", "tech" ] }, { "id": "https://pype.dev/cat-autonomy-2-0/", "url": "https://pype.dev/cat-autonomy-2-0/", "title": "Cat Autonomy 2.0", "content_text": "\nI wrote about my new role [[new-job-caterpillar-autonomy]] a bit a couple weeks\nago during an insanely busy time - having just started the role and wrapping up\nwhat ended up being side-work, paired with some sleep-deprived ADHD hyperfocus\non my new responsibilities and [[nexus]], that post is less of an update and\nmore of a mind-dump. This post is meant to be a calmer update about my upcoming\ntime with Caterpillar Autonomy.\n\nIt starts, like all good stories, with Caterpillar's RTO (return to office)\nmandate and how it has affected a lot of people negatively (in my opinion). I\nhaven't seen one positive remark about it fromanyone making less than $250k a\nyear + Bonus + Equity if you catch my drift...\n\nI know of people who really did lose their jobs at the end of 2025 for refusing\nto move. My story with RTO is covered elsewhere, I'd been blessed with remote\nwork and was even secure in a remote position in the face of Caterpillar being\nwilling to fire me [[after-exceeding-expectations-for-4-years]].\n\nBut time passes and life happens, and the sovereign Lord brought me to this current\ncircumstance: where I'm fully committed to Caterpillar Autonomy, and it\nfeels weird to say it. I'm going to skip the RTO details, if you know you know.\nWhere I'm at right now is on the other side (or very nearly on the other side)\nof a choice I didn't feel freedom to make because it came from a conviction.\n\nThe manager of the group I'm in told me a story about working for another\nmining company, where he was managing mines in the Congo. These mines can be in\nthe most remote of remote places on earth - in cultures where human life is\ndefinitely treated differently than in the first-world midwest USA. Operators of\nthese huge mining trucks in these parts of the world can be smoking meth in the\ncabs - which obviously leads to unsafe operation. Safety of the mines\nthemselves can be a second-thought as well, they collapse and then the people\ninside have to be excavated out. Sean told me he had come back to work after a\nweekend just to be told, as if it was no big deal, that \"Motombu died\" and to\nfind another operator. Very crass attitude towards human life...\n\nI'm still pretty frustrated at Caterpillar for how they've handled RTO (and\ncorporitisms like \"Caterpillar family\" and \"we bleed Cat yellow\" are\ndistracting nonsense), and part of me wants to leave still over the personal\noffence, but Caterpillar will make the safest mining equipment, without a\ndoubt. And given that Cat will make the safest equipment then the story I was told and\nthe mission I couldn't help but believe in is that rolling out good autonomy\nsoftware and solutions to these remote mines will save lives.\n\n!!! warning \"\"\n\n I'm obviously not like superman or anything, but something feels better about enabling good autonomy rather than supporting an online sales platform, or working for a bank...\n\nI was invited (coerced?) into that mission - and my role in it is enabling the\nAutonomy group to manage data for their software stack. I get to help these\nengineers be experts in the things they're experts in by offloading some\noperational overhead (data movement, pipelineing, infrastructure and devops,\netc.) so that they can produce the best autonomy software on the planet.\n", "summary": "I wrote about my new role new-job-caterpillar-autonomy a bit a couple weeks ago during an insanely busy time - having just started the role and wrapping up...", "date_published": "2026-02-09T06:07:12Z", "date_modified": "2026-02-09T06:07:12Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "work", "tech" ] }, { "id": "https://pype.dev/learning-how-to-agent/", "url": "https://pype.dev/learning-how-to-agent/", "title": "Learning How To Agent", "content_text": "\n## Introduction \u0026 Background\n\nI've been using AI tools for codegen for a few years now, but not super\nheavily. I either use the in-line copilot stuff, which is like LSP on\nseteroids, or I have gone all the way to the other side of full on vibecoding\nwith Windsurf. That's been fun enough, but not super fulfilling and at the end\nof that I either have a simple webapp that does what I want but I don't\nunderstand, or else a half-broken thing I tried to understand but couldn't\nprompt in the right direction.\n\n!!! note \"One Caveat\"\n\n My one caveat with Windsurf, which is a full IDE that I don't have a lot of comfortable navigation in due to all the Cascade keybindings clobbaring my own keymaps - is that with a HEAVY spec, it's done will with the Claude models at implementing a real idea I have. It's closer to vibe-engineering than vibe-coding, but that's my only instance, the rest of my Windsurf usage is \"make me a cool app - no mistakes\"\n\nThis year it feels like some tools exploded and\nI mentioned this in [[new-job-caterpillar-autonomy]]. I've been using\n[opencode](https://opencode.ai) a lot over the last few weeks and have iterated\nmany times already on a system of work that I'm trying to lean into for\nimproving my efficiency.\n\n## The Problem Space\n\nMy desire for a great Developer Experience is years old now, shout out to\nThePrimeagen for his [FEM\ncourse](https://frontendmasters.com/courses/developer-productivity-v2/) and\n[Waylon Walker](https://waylonwalker.com) for being a constant source of\nencouragement to be the best developer I can be. I sometimes (often) get\ntunnel-visioned on developer-productivity initiatives and lose the forest\nthrough the trees when ironing out a workflow - generally to find out I way\nover complicated the solution OR worse, started solving a problem I don't even\nhave.\n\n## First Attempt: Local Progress Tracker\n\nWell that's where I've been for a few weeks... I'm building Nexus, my\nsecond-brain at work to collaborate with agents on the truckload of stuff I'm\nexpected to get done. For a while now I've had a \"working-notes\" repo, which is\nbasically a blog, built with markata and navigated via markdown-lsp, where I do\nlike what I do here - take daily notes, track projects and status, and it gets\nbuilt into a nice little website I can reference with my boss.\n\nNow that we have copilot in full-swing, I'm trying to integrate agents a bit\nmore. Opencode has made this so nice - so many tools and modes of interaction,\nhighly customizable interface but also an amazing default experience... So I\nwas trying to lean into using agents and subagents for more work I started down\nthe path of building out a progress tracker. I started with a simple \"skill\"\nthat told the agent to put some info into a sqlite file, and even create the\nfile and schema based on the work. This worked fine, but was specific to each\nrepo (and actually each worktree I was in) and it was hard for me to get\nvisibility into all the work my fleet of agents was doing. Now that's actually\nproblem 1 - I don't have a fleet of agents, I had some terminal sessions going\nwith opencode, but I got it in my head that I was going to have an army of\nClaude's on my computer, constantly and autonomously knocking out tickets and I\nneeded to know who was doing what, where, when, and why..\n\n## Nexus V1 \u0026 V2: The Over-Engineering Phase\n\nSo, I dropped the local \"progress tracker\" and jumped into a huge FastAPI\nproject that I called Nexus. It was a python cli + api, with a server for\ncentralized management. It presented a kanban board, had policy gates on\n\"plans\" being approved before work could start on an Epic (and therefore any\nchild tickets). It has worktree tracking and automation, etc. It had a lot...\nit didn't all work, and it was hard to build the autonomous system... I wanted\nagentic feedback loops where `voidshaper` and I made epic plans for Epics (see the pun?) and then once the Plan was approved `star-commander` comes on the scene and makes tickets or checks tickets, depending on what's already ready to go it farmed out the work to `starsmith` (and variants for complexity) to build and then automatically calls in `recon-officer` and `qa-engineer` and at the end of it `gatekeeper` came in and\napproved or denied the changes. If denied - automatically start the loop again,\ntracking the work in Nexus, if approved - rebase and merge the branch, clean up\nthe worktree, update the ticket, close it, and get working on the next thing\nthat opens up. Ticket dependencies were in there, tracking stale agent\nsessions, clever routing of tasks to smaller models where appropriate.... you\ncan see that I went too far too fast too hard.\n\n![20260203124603_d5948740.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20260203124603_d5948740.png)\n\nNexus went through 2 or 3 iterations of this feature set. I was building it with several constraints in mind - specifically at work. 1. rate-limiting on large models (so not just using opus for everything), 2. good stewardship (not burning down cities for docstrings - farm out that work to haiku or a -mini model), 3. Copilot support - even though I like to use opencode, I wanted to supper copilot cli and in vscode as a means to share this with my coworkers who were mostly using vscode, 4. agent sessions were dying so I needed some kind of liveness probe for manual intervention in sessions where maybe the vpn died and the agent lost network... stuff like this was on my mind - but you know what wasn't? actually doing some work... I was solving problems I don't have, but were fun to think about.\n\n## The Reality Check\n\nSo, what are my problems? I boiled them down to a few things...\n\n1. I want to manage my workstreams in workspaces - folders on my computer where I put git worktrees\n2. I want varying levels of ai automation in my workflows... some things could be handled by an agent fully autonmously, but most things I'm at least paired up, reviewing diffs still, doing research while working, etc.\n3. In those varying levels of automation, I wanted the same amount of task tracking to a centralized location\n\nAnd after 2 weeks of ADHD-driven hyper focus, and many iterations on Nexus,\nthen Nexus V2.... what happened is that Nexus V2 died, and we say long live\nNexus V3!\n\n## Nexus V3: The Pragmatic Pivot\n\nNexus V3 is the same idea, but different approach... I'm leaning into opencode\ntooling specifically, dropping my care to support copilot given the lack of\nfeatures. I'm using a few opencode [plugins](https://opencode.ai/docs/plugins/)\n([opencode-notify](https://github.com/kdcokenny/opencode-notify) and\n[opencode-background-agents](https://github.com/kdcokenny/opencode-background-agents)\nand a few [commands](https://opencode.ai/docs/commands/). Instead of building\nmy own tracker, I'm using [kanboard](https://kanboard.org/) because it's\nbasically a feature-complete agile/sprint/kanban board that I use at home, has\na simple plugin ecosystem for light customization, and solves practically every\nstatus-tracking problem I tried to build from the ground up initially - ticket\ndependencies/linkages, actions to change ticket colors for a simple intuitive\nUI based on state, easy columns and tagging configuration, a simple API and\nthere's even an [mcp server](https://github.com/bivex/kanboard-mcp).\n\nSo, I've given up on the full automation for now, although\n[opencode-pilot](https://github.com/athal7/opencode-pilot) looks VERY PROMISING\nfor this in the future. Today though, through some simple commands to give to\nagents for updating kanboard, I manually put them in worktrees, and they get to\nwork - the tracking and human-in-the-loop model is going well for the work I\nneed done.\n\n## Lessons Learned\n\nI learned and relearned plenty of lessons on this over the last 2 weeks... Data\nmodels matter more than almost anything, well-defined workflows are required if\nyou want agents to help you iterate, and not everything has to be a product...\nThat last one's personal, but every time I have an idea I **think** is good,\nI'm sure it'll be something to share, but a good lesson for me is to just build\nthe things I need for me, and **eventually** maybe it can be cleaned up to\nshare, but when I start building something with anyone other than **me** in\nmind, I'm in for a long hard journey\n\n## Current State \u0026 Future\n\nSo what's the summary? I don't think I know how to agent super well yet - but\nI'm trying to get better to stay on the forefront of my co-workers who I see\nusing AI in simple and sometimes scary ways\n\n!!! danger \"Copilot x sudo\"\n\n Do not give copilot `sudo` on your CI server and say \"fix my problem\"... are you retarded???\n\nI am not going hardcore with [ralph\nwiggum](https://awesomeclaude.ai/ralph-wiggum) or\n[gastown](https://github.com/steveyegge/gastown) - although those inspired\nNexus v1 and v2, but I am dialing in my agentic workflow with some simple\nspecialized agents, farming out work to subagents for context management,\nplanning ahead of time to put appropriate context in a ticket, and getting\nclose to having agents check out tickets, make worktrees, and do simple work by\nthemselves (this was working in Nexus V2 but only intermittenly).\n\nSomeday I'll open-source Nexus and share the configuration and workflow, for\nnow it's private as I'm actually using it to build out what I want rather than\ntrying to recreate gastown with a cool space theme.\n", "summary": "I've been using AI tools for codegen for a few years now, but not super heavily. I either use the in-line copilot stuff, which is like LSP on seteroids, or I...", "date_published": "2026-02-03T05:50:33Z", "date_modified": "2026-02-03T05:50:33Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "genai", "tech" ] }, { "id": "https://pype.dev/new-job-caterpillar-autonomy/", "url": "https://pype.dev/new-job-caterpillar-autonomy/", "title": "New Job - Caterpillar Autonomy", "content_text": "\nIn [[im-back-from-the-dead]] I mentioned my new role that started this year -\nit's a return to Caterpillar Autonomy. I built some data pipelines and\njunior-grade infrastructure 5 years ago but left over burn-out at the prospect of\na job with smaller scoped work and more technical guidance. That was a good\nmove, and in God's sovereignty he's brought me back. Now I'm looking at code,\nthankfully not that I wrote, but that the guy after me wrote - it accomplished\na job, like my original work did, but also like what I built back then -\nthere's better ways today. The amount of things in front of me is absolutely\ndaunting, and the people on the team have been seeded with a very high opinion\nof me... I don't know how much is valid and how much is hype, but I'm gonna try\nto live up to it.\n\nOne way I'm striving to do that is to maximize my efficiency with the AI coding\ntools available to me. We have Github Copilot and I've picked up a lot over the\ncourse of the last few years (and especially more recently) in using agents,\nplanning out work, documenting plans for agents to follow, splitting up work\nvia worktrees, etc. I feel pretty good about where I'm going and someday I\nmight even release Nexus. Nexus is serving as my second-brain at work, the hub\nwhere I collaborate with my fleet of agents on the work that must get done.\nDetails to come on this, but the thing that matters is that in ironing out\nworkflow I've moved from prompt + chat to really managing a long-lived stream\nof work. AI has been changing the world, and there's been developer hype for\nyears now. But \"make me a cool app, no mistakes\" isn't going to cut it. Even\n\"make an app but generate a plan first\" isn't going to cut it... Developers\nhave to adopt a higher level role, a systems-oriented and architecture-driven\nworldview must become primary in order to keep up. Code-gen and syntax writing\nare not what developers were **ever** paid to do, although many believed so.\nOur jobs have been to solve problems and deliver code that solves the problem.\nThe code is cheap now, but solving problems still is a human task.\n\nIn the Autonomy group - there's problems all up and down the stack. My focus is\non infrastructure and developer operations - it's become my bread and butter\nover the last several years. I'm excited to help the team grow, and I'm excited\nto grow personally/technically as I lean into the agentic workflow to produce\ncode that I'm actually proud of, that has my name on the commit, and that\nsolves real problems.\n\n## Example - Local Development\n\nOne of the first things I'm tackling is a developer-pain-point of working on\ntheir laptop. I've been in this space for years, mostly with python programmer\nwho are writing data science code. They don't know about virtual environments,\nchecking $PATH, assuming bad state in their terminal session, how to configure\nVS Code, etc. Often they just want to write some scripts and somehow test it.\nThe solution I see most often is for devs to write code in JupyterLab/Notebooks\nin AWS or some environment close to their data - this is fine I guess, but it's\nnot developing good pipelines, and it's tedious as hell. In my last job I\nhelped set developers up with workflows that allowed them to run their IDE of\nchoice locally (getting all the goodies of syntax highlighting, LSP, etc) and a\nCLI that took their code and ran it in the cloud, right next to data, in the\nsame way that prod runs. It was a hit. After that I introduced some tools to\nhelp them manage python environments - we had strict templating requirements in\nour projects, so making tools to automate those things wasn't too hard - it's\nmuch easier than trying to make something flexible for every use case. The\nopinions made the automation and tooling easy to make and distribute.\n\nWell I'm up against a similar task now, but oh so much worse... Larger team,\nlarger environment sprawl, larger infrastructure mismanagement, the whole\ngambit. And I'm here for it... Here's the first problem I'm addressing - local\ndevelopment for Airflow DAGs that run in an Airflow deployment on Kubernetes.\nThe deployment itself is a little odd, Airflow is an orchestrator, all the\npipelines run in external AWS Batch jobs - so a DAG hits the Batch API to run\nthe code. The design there is actually nice, but how are devs testing code?\n\nOh that's easy... they SSH into the prod server, which is a 5 year old desktop\nTHAT I BUILT WITH A CO-WORKER BEFORE I WAS IN AUTONOMY THE FIRST TIME... hold\non, WHAT!? Yes, it's true.. so they SSH into the prod server, run some bash\nscripts in their userspace that setup airflow and a few db utilities in a\ndocker compose stack, authenticate with AWS themselves from that server, and\nthen they run DAGs against real data to test it... I am beyond shook.\n\nHere's what I've put together - a bootstrap process (I like `just` + PEP 723\npython scripts as opposed to bash, but to each their own, and bash of course\nhas its place) that spins up a [kind](https://kind.sigs.k8s.io/) cluster on\ntheir laptop, the process pulls some private images and loads them into the\nkind cluster, it installs airflow from a helm chart (the same helm chart we'll\nuse in dev and prod... no more docker compose over here, kubectl over there),\nand everything just. comes. up. No SSH into ancient server, no touching cloud\ninfra (they get MinIO and a DB container to emulate S3 and RDS in our AWS\naccounts), no sweat on testing DAGs. They stage some data in MinIO (`just open\nminio` handles the port-forward and opens the browser), then `just open\nairflow` (their DAGs are hot-reloaded via hostPath mounting), and they can run\nDAGs locally until they're satisfied with the results.\n\nIt's taken me about a week of split-focused effort (I mentioned Nexus and I've\nbeen co-building and dogfooding that at the same time) but I'm proud of that\nlocal setup now. I am a bit shocked they've dealt with a brittle, hacky, often\nbroken development workflow for the last 4 years or so, but that development I\nsuppose.\n\n## The Point\n\nThis post wasn't meant to be me glazing myself for awesome local development\npractices, I am simply excited about this new chapter. I love solving problems,\nand I love owning those solutions. There's a whole mess of things to address,\nmy mind is buzzing, and I feel like God has blessed me with renewed passion\n(again see [[im-back-from-the-dead]]). I've given up some pay and some freedom\nto take this role, but I think it'll pay dividends.\n\nLife happens to all of us - this role change affected a lot for me, it's been\nhard to digest some of those changes, but the work is good, the development is\nfun, the new world of using agents to fly through things you're fluent in is\nexciting, and I'm here to help a team that desperately needs it to improve\ntheir lives and the work we do for Cat Autonomy.\n\nI'm still pissed at Caterpillar Executives for RTO ruining parts of my life,\nbut in God's sovereignty their idiocy has led to non-trivial blessing, so I can\nsay \"Praise the Lord\"\n", "summary": "In im-back-from-the-dead I mentioned my new role that started this year - it's a return to Caterpillar Autonomy. I built some data pipelines and junior-grade...", "date_published": "2026-01-26T10:24:54Z", "date_modified": "2026-01-26T10:24:54Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "work", "tech" ] }, { "id": "https://pype.dev/increase-inotify-limit-in-your-ci-workers/", "url": "https://pype.dev/increase-inotify-limit-in-your-ci-workers/", "title": "Increase inotify limit in your CI workers", "content_html": "\u003cp\u003eToday I tripped over a CI failure that I had to think about for a while.\u003c/p\u003e\n\u003cp\u003eI build \u003ca href=\"https://github.com/zensical/zensical\"\u003ezensical\u003c/a\u003e static sites in CI on\nmy Forgejo instance. These builds had been working fine, then suddenly started\nfailing with no code changes. Naturally, I assumed something upstream broke —\nmaybe a new uv release, maybe zensical.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eI pinned versions.\u003c/li\u003e\n\u003cli\u003eI tested older versions.\u003c/li\u003e\n\u003cli\u003eSame failure every time.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThat ruled out regressions and pushed me toward the environment.\u003c/p\u003e\n\u003cp\u003eI pulled the runner and worker images locally and built the sites just fine\u0026hellip;\nBut that doesn\u0026rsquo;t perfectly emulate the CI setup - my forgejo runner relies on\ndocker-in-docker and so we aren\u0026rsquo;t \u003cstrong\u003ejust\u003c/strong\u003e running a container on a host, we have\nthis middle layer to consider\u0026hellip; I wasn\u0026rsquo;t sure how to really test this out\nlocally so I succomed to AI and here\u0026rsquo;s where Jipity got me in about 5\nminutes\u0026hellip;\u003c/p\u003e\n\u003ch2 id=\"the-failure\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe failure\u003c/span\u003e \u003ca href=\"#the-failure\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe builds blew up with:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003ethread 'zrx/monitor' panicked at .../zensical-watch/src/agent/monitor.rs:154:49:\ncalled `Result::unwrap()` on an `Err` value:\nError { kind: Io(Os { code: 24, message: \u0026quot;Too many open files\u0026quot; }) }\n\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAt first glance it means nothing to me but Jipity says this screams ulimit.\u003c/p\u003e\n\u003cp\u003eSo following the AI overlords I checked:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eulimit -n\u003c/code\u003e was already very high\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003e/proc/sys/fs/inotify/max_user_watches\u003c/code\u003e was also very high\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003ePID limits\u003c/code\u003e were not constrained\u003c/p\u003e\n\u003cp\u003eEverything looked fine according to Jipity.\u003c/p\u003e\n\u003cp\u003eYet the panic persisted.\u003c/p\u003e\n\u003ch2 id=\"the-real-culprit\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe real culprit\u003c/span\u003e \u003ca href=\"#the-real-culprit\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe actual limit being hit was:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003e/proc/sys/fs/inotify/max_user_instances\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eIn my Forgejo runner container, it was set to 128.\u003c/p\u003e\n\u003cp\u003eThat turns out to be far too low for zensical.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s what ChatGPT said:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eEven during a normal zensical build, the tool spins up its watch subsystem,\nwhich creates many inotify instances. Once it crosses the kernel limit,\ninotify_init() fails with EMFILE, and the process panics because the error is\nunwrapped.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"the-fix\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eThe fix\u003c/span\u003e \u003ca href=\"#the-fix\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRaising the inotify instance limit fixed it immediately:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003eecho 1024 \u0026gt; /proc/sys/fs/inotify/max_user_instances\nRUST_BACKTRACE=full uvx zensical build --clean\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAfter that, the build succeeded consistently.\u003c/p\u003e\n", "content_text": "\nToday I tripped over a CI failure that I had to think about for a while.\n\nI build [zensical](https://github.com/zensical/zensical) static sites in CI on\nmy Forgejo instance. These builds had been working fine, then suddenly started\nfailing with no code changes. Naturally, I assumed something upstream broke —\nmaybe a new uv release, maybe zensical.\n\n- I pinned versions.\n- I tested older versions.\n- Same failure every time.\n\nThat ruled out regressions and pushed me toward the environment.\n\nI pulled the runner and worker images locally and built the sites just fine...\nBut that doesn't perfectly emulate the CI setup - my forgejo runner relies on\ndocker-in-docker and so we aren't **just** running a container on a host, we have\nthis middle layer to consider... I wasn't sure how to really test this out\nlocally so I succomed to AI and here's where Jipity got me in about 5\nminutes...\n\n## The failure\n\nThe builds blew up with:\n\n```\nthread 'zrx/monitor' panicked at .../zensical-watch/src/agent/monitor.rs:154:49:\ncalled `Result::unwrap()` on an `Err` value:\nError { kind: Io(Os { code: 24, message: \"Too many open files\" }) }\n\n```\n\nAt first glance it means nothing to me but Jipity says this screams ulimit.\n\nSo following the AI overlords I checked:\n\n`ulimit -n` was already very high\n\n`/proc/sys/fs/inotify/max_user_watches` was also very high\n\n`PID limits` were not constrained\n\nEverything looked fine according to Jipity.\n\nYet the panic persisted.\n\n## The real culprit\n\nThe actual limit being hit was:\n\n`/proc/sys/fs/inotify/max_user_instances`\n\nIn my Forgejo runner container, it was set to 128.\n\nThat turns out to be far too low for zensical.\n\nHere's what ChatGPT said:\n\n\u003e Even during a normal zensical build, the tool spins up its watch subsystem,\n\u003e which creates many inotify instances. Once it crosses the kernel limit,\n\u003e inotify_init() fails with EMFILE, and the process panics because the error is\n\u003e unwrapped.\n\n## The fix\n\nRaising the inotify instance limit fixed it immediately:\n\n```\necho 1024 \u003e /proc/sys/fs/inotify/max_user_instances\nRUST_BACKTRACE=full uvx zensical build --clean\n```\n\nAfter that, the build succeeded consistently.\n", "summary": "Today I tripped over a CI failure that I had to think about for a while.", "date_published": "2025-12-30T06:11:49Z", "date_modified": "2025-12-30T06:11:49Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "forgejo", "tech" ] }, { "id": "https://pype.dev/prettier-docker-ps-command/", "url": "https://pype.dev/prettier-docker-ps-command/", "title": "Prettier Docker ‘ps’ Command", "content_html": "\u003cp\u003eThe \u003ccode\u003edocker ps\u003c/code\u003e command is very useful, but I hate reading the output. Turns\nout, you can make it prettier:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003edocker ps\u003c/code\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u0026ndash;format \u0026ldquo;table\u0026rdquo; is implied with the command.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cpre\u003e\u003ccode\u003e\n❯ docker ps --format \u0026quot;table\u0026quot; | grep can\n9b716a7d1ab0 postgres:17 \u0026quot;docker-entrypoint.s…\u0026quot; 13 hours ago Up 13 hours (healthy) 0.0.0.0:5432-\u0026gt;5432/tcp, [::]:5432-\u0026gt;5432/tcp cannalyzer-db-1\nf7708ea0c112 adminer \u0026quot;entrypoint.sh docke…\u0026quot; 13 hours ago Up 13 hours 0.0.0.0:8081-\u0026gt;8080/tcp, [::]:8081-\u0026gt;8080/tcp cannalyzer-adminer-1\n93cd67719ed4 frontend:latest \u0026quot;/docker-entrypoint.…\u0026quot; 13 hours ago Up 13 hours 0.0.0.0:5173-\u0026gt;80/tcp, [::]:5173-\u0026gt;80/tcp cannalyzer-frontend-1\nf517625fca98 traefik:3.0 \u0026quot;/entrypoint.sh --pr…\u0026quot; 13 hours ago Up 13 hours 0.0.0.0:80-\u0026gt;80/tcp, [::]:80-\u0026gt;80/tcp, 0.0.0.0:8090-\u0026gt;8080/tcp, [::]:8090-\u0026gt;8080/tcp cannalyzer-proxy-1\nf4daa036216e schickling/mailcatcher \u0026quot;sh -c 'mailcatcher …\u0026quot; 13 hours ago Up 13 hours 0.0.0.0:1025-\u0026gt;1025/tcp, [::]:1025-\u0026gt;1025/tcp, 0.0.0.0:1080-\u0026gt;1080/tcp, [::]:1080-\u0026gt;1080/tcp cannalyzer-mailcatcher-1\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eBut you can pass a template string to the \u003ccode\u003e--format\u003c/code\u003e option, like so:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003edocker ps --format \u0026quot;table {{.Names}}\u0026quot;\u003c/code\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e\n✗ docker ps --format \u0026quot;table {{.Names}}\u0026quot; | grep can\ncannalyzer-db-1\ncannalyzer-adminer-1\ncannalyzer-frontend-1\ncannalyzer-proxy-1\ncannalyzer-mailcatcher-1\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003e\u003ccode\u003edocker ps --format \u0026quot;table {{.Names}}\\t{{.Status}}\\t{{.Ports}}\u0026quot;\u003c/code\u003e\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e\n❯ docker ps --format \u0026quot;table {{.Names}}\\t{{.Ports}}\u0026quot; | grep can\ncannalyzer-db-1 0.0.0.0:5432-\u0026gt;5432/tcp, [::]:5432-\u0026gt;5432/tcp\ncannalyzer-adminer-1 0.0.0.0:8081-\u0026gt;8080/tcp, [::]:8081-\u0026gt;8080/tcp\ncannalyzer-frontend-1 0.0.0.0:5173-\u0026gt;80/tcp, [::]:5173-\u0026gt;80/tcp\ncannalyzer-proxy-1 0.0.0.0:80-\u0026gt;80/tcp, [::]:80-\u0026gt;80/tcp, 0.0.0.0:8090-\u0026gt;8080/tcp, [::]:8090-\u0026gt;8080/tcp\ncannalyzer-mailcatcher-1 0.0.0.0:1025-\u0026gt;1025/tcp, [::]:1025-\u0026gt;1025/tcp, 0.0.0.0:1080-\u0026gt;1080/tcp, [::]:1080-\u0026gt;1080/tcp\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2 id=\"picture\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003ePicture\u003c/span\u003e \u003ca href=\"#picture\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eI noticed my template isn\u0026rsquo;t folding the codeblocks in a way that actually makes this post look like I\u0026rsquo;m lying!\u003c/p\u003e\n\u003cfigure\u003e\n\u003cimg src=\"https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251229122717_67e76a67.png\" alt=\"20251229122717_67e76a67.png\"\u003e\n\u003c/figure\u003e\n", "content_text": "\nThe `docker ps` command is very useful, but I hate reading the output. Turns\nout, you can make it prettier:\n\n`docker ps`\n\n\u003e --format \"table\" is implied with the command.\n\n```\n\n❯ docker ps --format \"table\" | grep can\n9b716a7d1ab0 postgres:17 \"docker-entrypoint.s…\" 13 hours ago Up 13 hours (healthy) 0.0.0.0:5432-\u003e5432/tcp, [::]:5432-\u003e5432/tcp cannalyzer-db-1\nf7708ea0c112 adminer \"entrypoint.sh docke…\" 13 hours ago Up 13 hours 0.0.0.0:8081-\u003e8080/tcp, [::]:8081-\u003e8080/tcp cannalyzer-adminer-1\n93cd67719ed4 frontend:latest \"/docker-entrypoint.…\" 13 hours ago Up 13 hours 0.0.0.0:5173-\u003e80/tcp, [::]:5173-\u003e80/tcp cannalyzer-frontend-1\nf517625fca98 traefik:3.0 \"/entrypoint.sh --pr…\" 13 hours ago Up 13 hours 0.0.0.0:80-\u003e80/tcp, [::]:80-\u003e80/tcp, 0.0.0.0:8090-\u003e8080/tcp, [::]:8090-\u003e8080/tcp cannalyzer-proxy-1\nf4daa036216e schickling/mailcatcher \"sh -c 'mailcatcher …\" 13 hours ago Up 13 hours 0.0.0.0:1025-\u003e1025/tcp, [::]:1025-\u003e1025/tcp, 0.0.0.0:1080-\u003e1080/tcp, [::]:1080-\u003e1080/tcp cannalyzer-mailcatcher-1\n```\n\nBut you can pass a template string to the `--format` option, like so:\n\n`docker ps --format \"table {{.Names}}\"`\n\n```\n\n✗ docker ps --format \"table {{.Names}}\" | grep can\ncannalyzer-db-1\ncannalyzer-adminer-1\ncannalyzer-frontend-1\ncannalyzer-proxy-1\ncannalyzer-mailcatcher-1\n```\n\n`docker ps --format \"table {{.Names}}\\t{{.Status}}\\t{{.Ports}}\"`\n\n```\n\n❯ docker ps --format \"table {{.Names}}\\t{{.Ports}}\" | grep can\ncannalyzer-db-1 0.0.0.0:5432-\u003e5432/tcp, [::]:5432-\u003e5432/tcp\ncannalyzer-adminer-1 0.0.0.0:8081-\u003e8080/tcp, [::]:8081-\u003e8080/tcp\ncannalyzer-frontend-1 0.0.0.0:5173-\u003e80/tcp, [::]:5173-\u003e80/tcp\ncannalyzer-proxy-1 0.0.0.0:80-\u003e80/tcp, [::]:80-\u003e80/tcp, 0.0.0.0:8090-\u003e8080/tcp, [::]:8090-\u003e8080/tcp\ncannalyzer-mailcatcher-1 0.0.0.0:1025-\u003e1025/tcp, [::]:1025-\u003e1025/tcp, 0.0.0.0:1080-\u003e1080/tcp, [::]:1080-\u003e1080/tcp\n```\n\n## Picture\n\nI noticed my template isn't folding the codeblocks in a way that actually makes this post look like I'm lying!\n\n![20251229122717_67e76a67.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251229122717_67e76a67.png)\n", "summary": "The command is very useful, but I hate reading the output. Turns out, you can make it prettier:", "date_published": "2025-12-29T05:33:05Z", "date_modified": "2025-12-29T05:33:05Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "docker", "til", "tech" ] }, { "id": "https://pype.dev/setup-a-cloudflare-tunnel-with-terraform/", "url": "https://pype.dev/setup-a-cloudflare-tunnel-with-terraform/", "title": "Setup A Cloudflare Tunnel With Terraform", "content_html": "\u003cp\u003eI am cooking up some stuff at home and want to put it on the interwebs, but I\ndon\u0026rsquo;t want it on the same infra as my homelab. Now\u0026hellip; I only have a server or\n2, so to some degree it will be, but networking-wise I didn\u0026rsquo;t want to funnel\nextra traffic through my reverse proxy.\u003c/p\u003e\n\u003cp\u003eSo, I\u0026rsquo;d heard about Cloudflare Tunnels - they sound like P2P VPN to me, but I\nknow there\u0026rsquo;s layers of the networking stack I\u0026rsquo;m blatantly ignoring. \u0026ldquo;What the\ntunnel is\u0026rdquo; isn\u0026rsquo;t much the point - I\u0026rsquo;m here to show you how to set one up and\nget yourself a fancy \u003ca href=\"https://app.mydomain.com\"\u003ehttps://app.mydomain.com\u003c/a\u003e for your web app running\nkind of wherever you want\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eExample Repo linked at the bottom\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"requirements\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eRequirements\u003c/span\u003e \u003ca href=\"#requirements\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003col start=\"0\"\u003e\n\u003cli\u003eTerraform or open-tofu. I currently use open-tofu but either would be fine.\n\u003ccode\u003ebrew install open-tofu\u003c/code\u003e is a simple way to get going\u003c/li\u003e\n\u003cli\u003eCloudflare account with a domain\u003c/li\u003e\n\u003cli\u003eAPI token with permissions:\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eAccount:Cloudflare Tunnel:Edit\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eZone:DNS:Edit\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ecloudflared\u003c/code\u003e (the example repo runs cloudflared in a docker compose stack)\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"tunnel\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eTunnel\u003c/span\u003e \u003ca href=\"#tunnel\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe module is simple and has just a few resources:\u003c/p\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e❯ tofu state list\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emodule.tunnel.cloudflare_record.tunnel\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emodule.tunnel.cloudflare_tunnel_config.this\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emodule.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emodule.tunnel.random_id.tunnel_secret\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eWe see there will be the a DNS record that tofu references by the key \u0026ldquo;tunnel\u0026rdquo;.\nThere is a tunnel configuration resource, the tunnel resource itself, and\nfinally the associated secret required for the cloudflared daemon that will run\nalongside your webapp.\u003c/p\u003e\n\u003cp\u003eTo get started you\u0026rsquo;ll need to fill out the example \u003ccode\u003eterraform.tfvars\u003c/code\u003e file with\nyour info:\u003c/p\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Copy to terraform.tfvars and fill in values\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# DO NOT commit terraform.tfvars to git\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ecloudflare_api_token\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;your-api-token-here\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ecloudflare_account_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;your-account-id\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ecloudflare_zone_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;your-zone-id\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edomain\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;example.com\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003esubdomain\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;app\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003etunnel_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;my-tunnel\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eorigin_service\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;http://localhost:8000\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eYou can grab your account id and zone id from Cloudflare\u0026rsquo;s dashboard for your\ndomain. It\u0026rsquo;s near the bottom of the Overview page\u003c/p\u003e\n\u003cfigure\u003e\n\u003cimg src=\"https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251213113332_0e0f09b8.png\" alt=\"20251213113332_0e0f09b8.png\"\u003e\n\u003c/figure\u003e\n\u003cp\u003eThen I presume you have a domain already, but if not hop over to namecheap to\nsnag one and then register it with cloudflare so they can manage your DNS. I\nhave terraform for this as well, a future blog post will combine this with a\nfuller terraform\u0026rsquo;d cloudflare setup for simple domain use cases\u003c/p\u003e\n\u003cp\u003eOnce you fill those out, hit it with the \u003ccode\u003etofu init\u003c/code\u003e and \u003ccode\u003etofu plan\u003c/code\u003e to see what\u0026rsquo;s up\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eNOTE: \u003ccode\u003eterraform.tfvars\u003c/code\u003e is automatically sourced by terraform/tofu, you can name the file differently and then pass \u003ccode\u003e-var-file=myvars.tfvars\u003c/code\u003e to the commands\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThe initial plan should look something like this:\u003c/p\u003e\n\u003cpre class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eOpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + create\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eOpenTofu will perform the following actions:\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"c1\"\u003e# module.tunnel.cloudflare_record.tunnel will be created\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + resource \u003cspan class=\"s2\"\u003e\u0026#34;cloudflare_record\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;tunnel\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eallow_overwrite\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nb\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ecomment\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Managed by Terraform - soonish-tunnel\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003econtent\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ecreated_on\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ehostname\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eid\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003emetadata\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003emodified_on\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ename\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;app\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eproxiable\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eproxied\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nb\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ettl\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nb\"\u003etype\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;CNAME\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003evalue\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ezone_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026lt;REDACTED\u0026gt;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"c1\"\u003e# module.tunnel.cloudflare_tunnel_config.this will be created\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + resource \u003cspan class=\"s2\"\u003e\u0026#34;cloudflare_tunnel_config\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;this\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eaccount_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026lt;REDACTED\u0026gt;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eid\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003etunnel_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + config \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + ingress_rule \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ehostname\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;app.notifiq.net\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eservice\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;http://localhost:8000\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + ingress_rule \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eservice\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;http_status:404\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"c1\"\u003e# module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this will be created\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + resource \u003cspan class=\"s2\"\u003e\u0026#34;cloudflare_zero_trust_tunnel_cloudflared\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;this\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eaccount_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026lt;REDACTED\u0026gt;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ecname\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eid\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ename\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;soonish-tunnel\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003esecret\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003esensitive value\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003etunnel_token\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003esensitive value\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"c1\"\u003e# module.tunnel.random_id.tunnel_secret will be created\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + resource \u003cspan class=\"s2\"\u003e\u0026#34;random_id\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;tunnel_secret\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eb64_std\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eb64_url\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ebyte_length\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e32\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003edec\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003ehex\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e + \u003cspan class=\"nv\"\u003eid\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eknown after apply\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e \u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ePlan: \u003cspan class=\"m\"\u003e4\u003c/span\u003e to add, \u003cspan class=\"m\"\u003e0\u003c/span\u003e to change, \u003cspan class=\"m\"\u003e0\u003c/span\u003e to destroy.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eAs long as that looks good you to, then we \u003ccode\u003etofu apply\u003c/code\u003e next (type \u003ccode\u003eyes\u003c/code\u003e when\nasked or pass \u003ccode\u003e-auto-approve\u003c/code\u003e)\u003c/p\u003e\n\u003cp\u003eAfterwards \u003ccode\u003etofu state list\u003c/code\u003e should show you the 4 resources, and if you go to\nyour cloudflare zone\u0026rsquo;s dashboard you should see the CNAME associated with the\ntunnel address\u003c/p\u003e\n\u003cfigure\u003e\n\u003cimg src=\"https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251213120004_d199e5fd.png\" alt=\"20251213120004_d199e5fd.png\"\u003e\n\u003c/figure\u003e\n\u003ch2 id=\"daemon\"\u003e\u003cspan class=\"heading-wear-glyph\"\u003eDaemon\u003c/span\u003e \u003ca href=\"#daemon\" class=\"heading-anchor\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRun the compose stack or the binary itself. Get the token from terraform state with \u003ccode\u003etofu output -raw tunnel_token\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eTUNNEL_TOKEN=$(tofu output -raw tunnel_token) docker compose up -d\u003c/code\u003e will do you nicely\u003c/p\u003e\n\u003cp\u003eEnjoy your tunnel!\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/pypeaday/example-terraform-cloudflare-tunnel\"\u003eexample repo\u003c/a\u003e\u003c/p\u003e\n", "content_text": "\nI am cooking up some stuff at home and want to put it on the interwebs, but I\ndon't want it on the same infra as my homelab. Now... I only have a server or\n2, so to some degree it will be, but networking-wise I didn't want to funnel\nextra traffic through my reverse proxy.\n\nSo, I'd heard about Cloudflare Tunnels - they sound like P2P VPN to me, but I\nknow there's layers of the networking stack I'm blatantly ignoring. \"What the\ntunnel is\" isn't much the point - I'm here to show you how to set one up and\nget yourself a fancy \u003chttps://app.mydomain.com\u003e for your web app running\nkind of wherever you want\n\n\u003e Example Repo linked at the bottom\n\n## Requirements\n\n0. Terraform or open-tofu. I currently use open-tofu but either would be fine.\n `brew install open-tofu` is a simple way to get going\n1. Cloudflare account with a domain\n2. API token with permissions:\n - `Account:Cloudflare Tunnel:Edit`\n - `Zone:DNS:Edit`\n3. `cloudflared` (the example repo runs cloudflared in a docker compose stack)\n\n## Tunnel\n\nThe module is simple and has just a few resources:\n\n```bash\n❯ tofu state list\nmodule.tunnel.cloudflare_record.tunnel\nmodule.tunnel.cloudflare_tunnel_config.this\nmodule.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this\nmodule.tunnel.random_id.tunnel_secret\n\n```\n\nWe see there will be the a DNS record that tofu references by the key \"tunnel\".\nThere is a tunnel configuration resource, the tunnel resource itself, and\nfinally the associated secret required for the cloudflared daemon that will run\nalongside your webapp.\n\nTo get started you'll need to fill out the example `terraform.tfvars` file with\nyour info:\n\n```hcl\n# Copy to terraform.tfvars and fill in values\n# DO NOT commit terraform.tfvars to git\n\ncloudflare_api_token = \"your-api-token-here\"\ncloudflare_account_id = \"your-account-id\"\ncloudflare_zone_id = \"your-zone-id\"\ndomain = \"example.com\"\nsubdomain = \"app\"\ntunnel_name = \"my-tunnel\"\norigin_service = \"http://localhost:8000\"\n```\n\nYou can grab your account id and zone id from Cloudflare's dashboard for your\ndomain. It's near the bottom of the Overview page\n\n![20251213113332_0e0f09b8.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251213113332_0e0f09b8.png)\n\nThen I presume you have a domain already, but if not hop over to namecheap to\nsnag one and then register it with cloudflare so they can manage your DNS. I\nhave terraform for this as well, a future blog post will combine this with a\nfuller terraform'd cloudflare setup for simple domain use cases\n\nOnce you fill those out, hit it with the `tofu init` and `tofu plan` to see what's up\n\n\u003e NOTE: `terraform.tfvars` is automatically sourced by terraform/tofu, you can name the file differently and then pass `-var-file=myvars.tfvars` to the commands\n\nThe initial plan should look something like this:\n\n```bash\n\nOpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:\n + create\n\nOpenTofu will perform the following actions:\n\n # module.tunnel.cloudflare_record.tunnel will be created\n + resource \"cloudflare_record\" \"tunnel\" {\n + allow_overwrite = false\n + comment = \"Managed by Terraform - soonish-tunnel\"\n + content = (known after apply)\n + created_on = (known after apply)\n + hostname = (known after apply)\n + id = (known after apply)\n + metadata = (known after apply)\n + modified_on = (known after apply)\n + name = \"app\"\n + proxiable = (known after apply)\n + proxied = true\n + ttl = (known after apply)\n + type = \"CNAME\"\n + value = (known after apply)\n + zone_id = \"\u003cREDACTED\u003e\"\n }\n\n # module.tunnel.cloudflare_tunnel_config.this will be created\n + resource \"cloudflare_tunnel_config\" \"this\" {\n + account_id = \"\u003cREDACTED\u003e\"\n + id = (known after apply)\n + tunnel_id = (known after apply)\n\n + config {\n + ingress_rule {\n + hostname = \"app.notifiq.net\"\n + service = \"http://localhost:8000\"\n }\n + ingress_rule {\n + service = \"http_status:404\"\n }\n }\n }\n\n # module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this will be created\n + resource \"cloudflare_zero_trust_tunnel_cloudflared\" \"this\" {\n + account_id = \"\u003cREDACTED\u003e\"\n + cname = (known after apply)\n + id = (known after apply)\n + name = \"soonish-tunnel\"\n + secret = (sensitive value)\n + tunnel_token = (sensitive value)\n }\n\n # module.tunnel.random_id.tunnel_secret will be created\n + resource \"random_id\" \"tunnel_secret\" {\n + b64_std = (known after apply)\n + b64_url = (known after apply)\n + byte_length = 32\n + dec = (known after apply)\n + hex = (known after apply)\n + id = (known after apply)\n }\n\nPlan: 4 to add, 0 to change, 0 to destroy.\n\n```\n\nAs long as that looks good you to, then we `tofu apply` next (type `yes` when\nasked or pass `-auto-approve`)\n\nAfterwards `tofu state list` should show you the 4 resources, and if you go to\nyour cloudflare zone's dashboard you should see the CNAME associated with the\ntunnel address\n\n![20251213120004_d199e5fd.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20251213120004_d199e5fd.png)\n\n## Daemon\n\nRun the compose stack or the binary itself. Get the token from terraform state with `tofu output -raw tunnel_token`.\n\n`TUNNEL_TOKEN=$(tofu output -raw tunnel_token) docker compose up -d` will do you nicely\n\nEnjoy your tunnel!\n\n[example repo](https://github.com/pypeaday/example-terraform-cloudflare-tunnel)\n", "summary": "I am cooking up some stuff at home and want to put it on the interwebs, but I don't want it on the same infra as my homelab. Now... I only have a server or...", "date_published": "2025-12-12T21:06:59Z", "date_modified": "2025-12-12T21:06:59Z", "authors": [ { "name": "Nic Payne", "url": "https://pype.dev" } ], "tags": [ "tofu", "terraform", "tech" ] } ] }