My Nextcloud woes

I wrote here about setting up www-data as the owner of any directories you want nextcloud to manage. However, I regularly struggle wtih permissions issues on my NAS because of the external storage app anyways so I’ve decided to just put our photos in the spot Nextcloud would otherwise put them, and use this as healthy pressure on our family to organize our photos and put the ones we care about with the rest of our family media.

Migration

Because I had a ton of photos on the NAS anyways that I wanted moved over to Nextcloud I just rsync’d the photos directory on my NAS to the user’s photos directory in nextcloud but they weren’t showing up in the web UI!

The Fix?

As www-data I needed to php /var/www/nextcloud/occ files:scan --all inside my nextcloud docker container AFTER moving a bunch of photos off my “NAS” into the folder mounted to the nextcloud container as its data folder! Before I did this they weren’t showing up in the web UI/

I ran out of space on the SSD in my server when doing some file transfers but only 100GB was used of a 256 GB SSD?

LVM

When installing Ubuntu live server the default option for how to partition the disk (in my experience) has been to setup an LVM group that defaults to less than the available space. Most recently I put Ubuntu server on a 256 GB SSD but the main partition was formatted as an LVM group with 100GB of storage… I didn’t think anything of this even though I’m mostly used to EXT4.

I think the reason for LVMs is performance, but in hindsight, I don’t really care much about the performance differences, I really just want all my storage that’s fast enough

Extending the LVM

A moment of googling brought me to Ubuntu’s wiki and I learned that I can expand my LVM to the space I need…

sudo lvdisplay and sudo pvdisplay show detailed views of the logical volumes and physical volumes respectively.

Take a look at those and find the volume you need to extend. For me I found this:

  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/ubuntu-lv
  LV Name                ubuntu-lv
  VG Name                ubuntu-vg
  LV Write Access        read/write
  LV Status              available
  ...

There’s more that you’ll see but this is what’s relevant - I need to extend the ubuntu-lv logical volume in the ubuntu-vg volume group.

sudo lvextend -L +50g ubuntu-vg/ubuntu-lv gives me 50 more GB of storage which should be enough for at least tonight 🤓

RTFM

See 02-….yaml in ansible-nas

I am revamping my home server and bumped myself early up to Jammy Jellyfish… however to my peril I reused my netplan config and after hitting my server with the ‘ol netplan apply I lost connection… DNS still seemed to kinda work externally, but internally nothing was up…

Turns out Netplan got a little change in how to express the gateway key in the netplan config!

Old Ubuntu 20.04 way

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      gateway4: 192.168.1.1  # <-- This changes!
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

New jammin way for Jammy Jellyfish (at least that worked for me)

network:
  version: 2
  ethernets:
    enp0s4:
      addresses: [192.168.1.{Static IP}/24]
      routes:
        - to: default
          via: 192.168.1.1 
      nameservers:
        addresses: [192.168.1.1, 1.1.1.1]

From my daily driver Ubuntu machine I often open nautilus, dolphin, etc. and delete a file here or there on my NAS… turns out Ubuntu sends thse file to .Trash-100 ON THE NAS so I’m effectively just moving that file and not freeing up any space…

I started my homelab journey being super naive about ZFS and how to manage the filesystem… that bit me in the butt when transfering a ton of files out of folders and into datasets because ZFS is copy on write so I was essentially duplicating my storage until I got a hair smarter about removing files after they’re moved (rsync –remove-source-file ftw). But I had a ton of snapshots of child datasets with a ton of data that I just never will need, so I learned zfs list -H -o name -t snapshot tank/dataset1/dataset2 will list just the snapshots for dataset2 and if you pipe that into xargs -n1 zfs destroy then you have a way to clear out some snapshots you don’t need!

I have a specific need for counting the number of lines in a file quickly. At work we use S3 for data storage during our Kedro pipeline development, and in the development process we may end up orphaning several datasets. In order to keep our workspace clean I have a short utility that compares the datasets in a Kedro DataCatalog with the files in the relevant S3 location.

To get that list I run an internal tool like this:

kedro our-liter | grep s3 >> orphaned_datasets.txt

This simply parses our internal linter for the lines releated to my s3 linter utility and pipes those lines to a file.

To get a quick idea of how out of wack a pipeline is I could open the text file in vim, git it with the G and see what line number I’m on but I’m way too lazy for that…

AWK #

awk 'END {print NR}' orphaned_datasets.txt gives me the number of lines and I can alias this to whatever feels appropriate in my zshrc!

built-ins for the win!

I wanted a quick way to generate an index.html for a directory of html files that grows by 1 or 2 files a week. I don’t know any html (the files are exports from my tiddlywiki)…

tree is just the answer.

Say I have a file structure like this:

./html-files
├── file1.html
└── file2.html

To generate a barebones simple index.html we can use tree as follows:

tree ./html-files -H "." -L 1 -P "*.html"

and get the following:

<!DOCTYPE html>
<html>
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <meta name="Author" content="Made by 'tree'">
 <meta name="GENERATOR" content="$Version: $ tree v1.8.0 (c) 1996 - 2018 by Steve Baker, Thomas Moore, Francesc Rocher, Florian Sesser, Kyosuke Tokoro $">
 <title>Directory Tree</title>
 <style type="text/css">
  <!--
  BODY { font-family : ariel, monospace, sans-serif; }
  P { font-weight: normal; font-family : ariel, monospace, sans-serif; color: black; background-color: transparent;}
  B { font-weight: normal; color: black; background-color: transparent;}
  A:visited { font-weight : normal; text-decoration : none; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:link    { font-weight : normal; text-decoration : none; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:hover   { color : #000000; font-weight : normal; text-decoration : underline; background-color : yellow; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  A:active  { color : #000000; font-weight: normal; background-color : transparent; margin : 0px 0px 0px 0px; padding : 0px 0px 0px 0px; display: inline; }
  .VERSION { font-size: small; font-family : arial, sans-serif; }
  .NORM  { color: black;  background-color: transparent;}
  .FIFO  { color: purple; background-color: transparent;}
  .CHAR  { color: yellow; background-color: transparent;}
  .DIR   { color: blue;   background-color: transparent;}
  .BLOCK { color: yellow; background-color: transparent;}
  .LINK  { color: aqua;   background-color: transparent;}
  .SOCK  { color: fuchsia;background-color: transparent;}
  .EXEC  { color: green;  background-color: transparent;}
  -->
 </style>
</head>
<body>
        <h1>Directory Tree</h1><p>
        <a href=".">.</a><br>
        ├── <a href="./file1.html">file1.html</a><br>
        └── <a href="./file2.html">file2.html</a><br>
        <br><br>
        </p>
        <p>

0 directories, 2 files
        <br><br>
        </p>
        <hr>
        <p class="VERSION">
                 tree v1.8.0 © 1996 - 2018 by Steve Baker and Thomas Moore <br>
                 HTML output hacked and copyleft © 1998 by Francesc Rocher <br>
                 JSON output hacked and copyleft © 2014 by Florian Sesser <br>
                 Charsets / OS/2 support © 2001 by Kyosuke Tokoro
        </p>
</body>
</html>

which looks like this when you serve it up with python -m http.server