Posts tagged: python Archive

💭 uv script management

2025-09-08T11:25:33Z

Here’s my thought on uv script management

Dang Waylon! Love these few lines… uv is great, been using it for environment and project requirements management but I didn’t realize it was this easy to setup a pep 723 compliant script!

Love this sed -i '1i #!/usr/bin/env -S uv run --script' little gem too - I typically do uv run script.py so I’ll see about making this a simple zsh alias for faster iteration on python scripts… I often have AI write quick bash scripts to accomplish things, but it’s quite a bit harder for me to read than python, and with the power uv is bringing to the python ecosystem, maybe it’s fine to just be a little higher in the programming stack for script-kitty stuff

Note

This is one of [[ my-thoughts ]]. I picked this up from Waylon Walker(https://thoughts.waylonwalker.com). It’s a short note that I make about someone else’s content online. Learn more about the process Thoughts

Zsh alias for uv scripts

2025-09-08T06:33:13Z

It’s apparently advantageous for uv to have a specific shebang set that uv init doesn’t add, no problem, we can have a zsh alias for it

alias add-shebang='f(){ 
  if [[ ! -s $1 ]]; then 
    echo "#!/usr/bin/env -S uv run --script" > "$1"; 
  elif head -n1 "$1" | grep -q "^#!"; then 
    echo "Shebang already present in $1"; 
  else 
    sed -i "1i #!/usr/bin/env -S uv run --script" "$1"; 
  fi
}; f'

Then if you have a python script from uv init --script you can add-shebang myscript.py to add the shebang.

Python editable installs and nested scripts

2025-07-14T08:40:32Z

Intro

I have been working towards a standard structure for all my python projects - and I mean much more than just uv init or something along those lines. But one of the things that got me recently was starting to keep a scripts/ folder that ran code based on what was in src/ or app/ (I’m working on that…) being installed in the virtual environment… so I’d uv pip install -e . or uv pip install -e src (kind of depends on the structure) and then in scripts/foo.py I’d have from my_cool_code.module import thing….

Running scripts has never been a problem… but I’ve always kept them in the root of the repo OR inside app or src in a nested directory…

So here’s where I learned a little something about python my_script.py vs python -m my_script

Example #

I setup a short example repo to illustrate the problem

link to repo

The repo structure is very simple…

python-e-example
  ├── foo.py
  ├── pyproject.toml
  ├── src
  │   ├── module.py
  ├── README.md
  └── scripts
      └── bar.py

scripts/bar.py and foo.py are the same code:

from src.module import my_func

my_func()

and then src/module.py:

def my_func():
    print("it worked!")

The rest of the structure is from uv init basically, then uv venv will make your virtual environment, and uv pip install -e . will install the library in editable mode.

So what’s the problem? #

Let’s run foo.py

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python foo.py
it worked!

Nice! It works just like we’d expect…

What about python scripts/bar.py?

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python scripts/bar.py
Traceback (most recent call last):
  File "/tmp/python-e-example/scripts/bar.py", line 8, in <module>
    from src.module import my_func
ModuleNotFoundError: No module named 'src'

Come again for big fudge...

Ok let’s stay calm and troubleshoot #

What if we open a python shell…

Python 3.13.1 (main, Dec 19 2024, 14:32:25) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from src.module import my_func
>>> my_func()

<!--markata-attribution-->
it worked!

ok… as expected it worked, and we’ve learned nothing…

implicit behavior

We didn’t learn anything from the example but something is present that we don’t see… python foo.py implicitly adds the path to the script to sys.path and so the src library is in the same directory as the script foo.py so the library is on sys.path…. take note of this

The Problem #

Turns out uv pip install . and uv pip install -e . are actually different

The editable install adds a .pth file into site-packages which acts as a shell hook.

# .venv/lib/python3.13/site-packages/__editable__.python_e_example-0.1.0.pth

/tmp/python-e-example/src

So when python runs a script, the location of the script is implicitly added to sys.path as noted above. The python process invoked by python foo.py has $PWD on sys.path which is /tmp/python-e-example in the example but python scripts/bar.py has /tmp/python-e-example/scripts on sys.path… which means that /tmp/python-e-example is NOT on sys.path in the process that’s running scripts/bar.py and so the src folder won’t be searched by python because the .pth file points to a directory that’s no on the path!

The Solution #

We can instead of running the script “as a script” we can tell python to execute it as as a module with python -m scripts.bar. This is syntatically similar to python path/to/script.py but instead tells the python interpreter to execute the script as a module so the path is replaced by import syntax.

This works for foo.py too

⬢ [devbox] ✗ python -m foo
it worked!

⬢ [devbox] ❯ python -m scripts.bar
it worked!

ChatGPT #

It’s noteworthy that jipity was helpful in exploring these issues I was running into

link to convo

💭 Build backend | uv

2025-07-04T11:43:30Z

Here’s my thought on Build backend | uv

uv build backend ready for production! Dang… I need to visit my hatch builds, I love hatch’s simplicity specifically with versioning… Not sure if it’s worth actually switching to uv for my builds or not, but it is dang cool to see astral take on the python space with unbelievable improvements

Note

💭 Switching pip to uv in a Dockerized Flask / Django App — Nick …

2025-06-27T11:18:16Z

Here’s my thought on Switching pip to uv in a Dockerized Flask / Django App — Nick Janetakis

It’s nice to see uv gaining popularity in so many places - and replacing vanilla pip with uv in containerized environments seems great due if for no other reason than the time you’ll save in CI… A few years ago I refactored a disjointed container build process to use uv instead of pip, and due to the design of the build pipelines we went from 10-15 minute builds to 2-3 minutes…

Note

💭 Snowflake Status - Certificate Authority Update for AWS Regions

2025-06-16T16:37:12Z

Here’s my thought on Snowflake Status - Certificate Authority Update for AWS Regions

A recent Snowflake issue borked some old pipelines - thankfully it appears to be a simple fix. Just goes to show you that you can’t just deploy pipelines and expect them to live forever, they have to be maintained when there are ANY external dependencies

Note

I built a simple app for adding images to my blog

2025-06-07T13:30:28Z

Quick Deets

I built a simple fastAPI app called “shotput” that I run locally inside a git repo where I save images for my blog. The app is simple:

upload image from clipboard
app commits image to repo (path is configurable)
app pushes the commit (auto-push configurable)
app returns a markdown link I can paste in my blog that uses statically.io (see statically.io to help me out) to serve the image from the repo to my blog

See the image repo with app details at images.pype.dev

Usage

Anyone could copy the script and config to their own git repo to put images in and run this at home - instructions are in the readme but for simple running on localhost it’s just uv run

Example

pasting an image let’s you copy the markdown link for easy pasting

![20250607130609_dadd33eb.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250607130609_dadd33eb.png)

Pandas.DataFrame info always answers and is sometimes right

2025-05-26T09:08:08Z

TIL

Today I learned that the .info() of a pandas.DataFrame will always give you an answer, but it is wildly difficult to know how accurate it is, because it depends on the underlying data types. If everything is a NumPy Dtype, then the result you get is the amount of memory NumPy is using under the hood, which is very accurate. But as soon as you have something like a Python string in the data frame, now pandas will tell you how much memory the NumPy data types are using, and it will give you some quick estimate of how much memory space the Python string types are using because numpy is just storing a pointer to that data in memory, but this is far less accurate because it doesn’t actually know what the strings are.

Memory Usage Deep or True #

To get an honest answer about a DataFrame’s memory usage, you need to pass memory_usage='deep' to the .info() method.

# This is the default
In [50]: df.info(memory_usage=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 372.0+ bytes

Notice that the memory usage has a + in it? You can force a deeper analysis by passing memory_usage='deep'

In [51]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 792.0 bytes

My MCP Configuration

2025-04-27T05:29:54Z

My MCP

The Tools #

Docker #

RAGDocs #

Sequential Thinking #

Git #

Not Tried Yet #

https://github.com/modelcontextprotocol/servers/tree/main/src/sqlite

The Config #


{
  "mcpServers": {
    "docker-mcp": {
        "command": "uvx",
        "args": [
          "docker-mcp"
        ]
      },
    "ragdocs": {
      "command": "node",
      "args": [
        "/home/nic/projects/personal/quadtask/node_modules/@qpd-v/mcp-server-ragdocs/build/index.js"
      ],
      "env": {
        "QDRANT_URL": "http://babyblue-aurora:6333",
        "EMBEDDING_PROVIDER": "ollama",
        "OLLAMA_URL": "http://babyblue-aurora:11434"
      },
      "alwaysAllow": [
        "search_documentation",
        "list_sources",
        "test_ollama",
        "add_documentation"
      ],
      "disabled": false
    },
    "sequentialthinking": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "--name",
        "sequentialthinking",
        "mcp/sequentialthinking"
      ],
      "disabled": false,
      "alwaysAllow": [
        "sequentialthinking"
      ]
    },
    "git": {
      "command": "uvx",
      "args": [
        "mcp-server-git"
      ],
      "disabled": false,
      "alwaysAllow": [
        "git_status",
        "git_commit",
        "git_add",
        "git_log",
        "git_diff"
      ]
    }
  }
}

docker context (and an issue to question your sanity)

2024-12-19T05:59:30Z

docker contexts are great, would recommend putting them in your prompt though (via starship or something else)… here’s why

I like to manage my containers remotely - I have a nice development setup on my desktop and I try to keep my server as bare-bones as possible. For a while I’ve been using ansible which makes it easy to manage configuration etc on other machines. But I recently learned about docker contexts and I’m planning to scale down my homelab management to just docker-compose stacks rather than a bunch of super complicated ansible playbooks

So, setting up a context is easy - it’s basically an ssh connection to another machine!

docker context create koober --docker "host=ssh://nic@koober"

koober is one of my dev machines and my ~/.ssh/config is setup such that I can ssh nic@koober, this makes the context work really seamlessly.

So there’s the default context (the machine you’re on) and now I have koober

To use it you run docker context use koober

And then to check we can ls the contexts

docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT               ERROR
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   
koober *                                             ssh://nic@koober

Notice the * - that indicates that’s our current context.

Trouble #

Now here’s where things get hairy… you’ve gotta be super-aware of what context you’re using. I have an indicator in my starship prompt that shows the current context, but since I’m new to using them I kind of didn’t notice it until I ran into this issue…

I’m working on a python application in docker but was not able to execute the entrypoint even though I KNEW the file was there… let’s take a look

Example #

Here’s a minimal hello world applycation in docker to illustrate the issue

# main.py
print("hello world")

# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Run the main.py script as the container's main process
CMD ["python", "main.py"]

services:
  hello-world:
    build: .
    volumes:
      - .:/app

Notice volume mounting in my project directory . to /app as a common practice to develop inside the container

Here’s where I started to question my sanity…

✗ docker compose up                                            
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | python: can't open file '/app/main.py': [Errno 2] No such file or directory
hello-world-1 exited with code 2

python can't open file? hmm… Let’s take a look at the image

First let’s make sure we get the image name right

✗ docker container ls -a              
CONTAINER ID   IMAGE                                           COMMAND                  CREATED          STATUS                       PORTS                                                                                  NAMES
3d7285fc39e2   docker-context-example-hello-world              "python main.py"         10 minutes ago   Exited (2) 44 seconds ago                                                                                           docker-context-example-hello-world-1

Now we can docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world

❯ docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world  
root@ee46d0e22de8:/app# python main.py
Hello, World!
root@ee46d0e22de8:/app#

WHAT THE HECK??

What happened… #

What happened turns out to be pretty simple once we realize I’m using contexts…

koboer is a remote context, the docker run and docker compose up commands are interacting with the docker socket on that machine.

So if I compose up the stack notice that there’s a volume bind mount in there - well those do not work with contexts (or at least I’m not aware of hose to make it work) and so the /app directory was getting blown away essentially with an empty overlay…

But when running with just docker run with no volume mount, the code was copied in during the build and is right where we expect it…

Let’s prove it #

❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@903f591c0384:/app# python main.py
python: can't open file '/app/main.py': [Errno 2] No such file or directory
root@903f591c0384:/app#

Adding a `-v .:/app` to match the compose file, we get the same error...

If we switch to the default context we are back up and running as expected

✗ docker context use default
default
Current context is now "default"

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker compose up         
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | Hello, World!
hello-world-1 exited with code 0

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@4045b6aa8883:/app# python main.py
Hello, World!
root@4045b6aa8883:/app#

Successful runs on both accounts with the volume mount

TLDR #

Context is king

Add a healthcheck to your FastAPI app

2024-12-15T15:41:36Z

I’m building a few FastAPI apps to throw in docker and run on my homelab… I wanted to add healthchecks and here’s a simple way to do it

Make sure to install curl in the dockerfile (near the top for effeciency)

# Install curl with minimal dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Then I recommend making compose files even for single image deployments

services:
  app:
    build: .
    volumes:
      - type: bind
        source: .
        target: /app
    environment:
      - PYTHONPATH=/app
      - DOCKER_ENV=true
      - UV_VIRTUALENV=/opt/app-env
    user: "1000:1000"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

Then finally you’ll need a /health endpoint


@app.get("/health", response_class=HTMLResponse)
async def health_check():
    """
    A health check endpoint that returns a status message.
    """
    return "<html><body><h1>Service is healthy</h1></body></html>"

Switching from AltaCV to RenderCV for my Resume

2024-08-01T05:59:46Z

I was using a fun LaTex-based project for managing my resume called AltaCV. I loved the customization and was familiar with Tek from school. However, I update my resume so infrequently that anytime I’d hop back to it I’d have to remember how to work with Tex and that was frustrating as I’ve lost touch with it over the years.

Scrolling GitHub treding repos I saw RenderCV which let’s me just use YAML to write my resume and then it compiles to Tek through Python. There’s a sister project to make your own using this very easly call rendercv-pipeline. I forked that repo and translated my tek resume to the YAML. The included theme is nice enough is YAML is much easier to maintain long-term.

My resume is behind a private GH repo but the example from rendercv-pipeline is here on GitHub

Modal Labs

2022-12-28T21:01:52Z

Playing around with Modal Labs

One of the first things I tried was a regular cron job…

@stub.function(
    schedule=modal.Period(minutes=59), secret=modal.Secret.from_name("my-dummy-secret")
)
def say_hi():
    now = time.ctime()
    secret = os.environ.get("dummy-secret")
    print(f"Hello {os.environ.get('USER', 'Rodney')} at {now}")
    print(f"{secret=}")

This can get deployed with modal deploy --name <app name> <path to .py file with the stub and function defined in it>

This function gets deployed as an app that I conveniently call say_hi (as far as I can tell the app name can be anything - as I add functions to this same app and deploy with the same name to get a new version)

Notice that this also is an example of giving access to a secret - defined in the Modal Labs dashboard

We can take a look at the apps running at https://modal.com/apps

I then added another function to experiment with custom container images and saw then that Modal will just slap a new version on anything provisioned with the same name (intuitive enough for sure) so when I add functions to my .py script and run modal deploy --name say_hi myscript.py over and over, the app called say_hi in the Modal apps dashboard just gets a new version

This means I can spin up several instances of functionally the same app but with different names/versions etc… Q: Maybe there’s gitops or policy stuff builtin to app names then?

I needed to take down an app I deployed as a duplicate but you don’t stop apps by name, you stop them by an id… see below


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
✗ modal app stop --help

 Usage: modal app stop [OPTIONS] APP_ID

 Stop an app.

╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    app_id      TEXT  [default: None] [required]                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ App ID                    ┃ Description         ┃ State    ┃ Creation time             ┃ Stop time                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ap-lzy1AAuVy7POFkUcDKRxpQ │ print_info          │ deployed │ 2022-12-28 20:59:07-06:00 │                           │
│ ap-qYjE45dciqgT3C3CpNp3RL │ say_hi              │ deployed │ 2022-12-28 19:49:22-06:00 │                           │
│ ap-X7FYneUeYV5IKHcyirSb87 │ link-scraper        │ stopped  │ 2022-12-28 15:39:02-06:00 │ 2022-12-28 15:39:04-06:00 │
│ ap-UOXTUU4uSRx2UZypJOcAsk │ example-get-started │ stopped  │ 2022-12-28 15:17:47-06:00 │ 2022-12-28 15:17:49-06:00 │
└───────────────────────────┴─────────────────────┴──────────┴───────────────────────────┴───────────────────────────┘

modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app stop ap-lzy1AAuVy7POFkUcDKRxpQ

Git warning!

I ran modal deploy ... after comitting some stuff I wanted to try BUT I had changes in my file I didn’t want to deploy… some git safety would be nice for deployment!

git stash && modal deploy .. && git stash pop

Question for Modal team - in my modal sandbox repo at commit:

aab6162 (HEAD -> main) HEAD@{1}: commit: print base version of my own image to prove it to me
 1 file changed, 2 insertions(+)

An environment variable, BASE_VERSION that I expect to be in my base image was not available to the python function in my Modal app… hopefully the log is still here

GitHub Actions #

I think it’d be easy to deploy from GH actions… just need an API Key…

Deployment

Without gitops it’s already ahrd to manage my versions… need to get that built in either to github or locally sooooon

Nomad?? :D

Call basicConfig to get Python log messages in iPython

2022-12-10T14:04:23Z

Logging instead of printing

I am trying to adopt logger.debug instead of print but ran into a confusing thing in ipython during Advent of Code… I riddled by script with logger.debug (yes after setting logging.setLevel('DEBUG')) but in ipython none of my log messages showed up!

import logging

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Turns out what I was missing was a call to basicConfig

import logging

# forget this and your messages are in the ether! or at least not seen in ipython...
logging.basicConfig()

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Bonus

Want your new messages to show up while iterating on something without killing the ipython kernel?

from importlib import reload
reload(logging) # to make sure you get new log messages you add while developing!

Benchmark your disks with fio

2022-08-27T13:43:22Z

Intro

I use ZFS at home in my homelab for basically all of my storage… Docker uses ZFS backend, all my VMs have their .qcow2 images in their own zfs datasets, and all my shares are ZFS datasets. I love ZFS but my home hardware presently is the opposite of expensive or new… Thankfully I’ve had a lot of my orginal homelab simply given to me but the cost of this is that I didn’t put my machines together, I didn’t choose the disks, and I definitely didn’t do the research I would’ve otherwise done had I bankrolled my server personally…

The Problem #

I run glances on basically all my machines and for the longest time I have been seeing big time iowait issues. Now, since everything was free I’ve largely been able to ignore that however I’m now after some better performance which I think means new hardware!

Here is a random screenshot of my glances homepage at time of writing - The only major load on my server is some ffmpeg transcoding (about 60% CPU utilization)…

As you can see… there’s a lot of issues and I don’t even know what they mean.

fio

I heard about fio through a friend and decided to try it out quick. It installs with apt on ubuntu quick and easy…

Jim Saltar has a good blog post on it here

Basically it’s a handy tool for benchmarking your disks and the blog dives into what types of metrics matter - it’s not just throughput, but also latency, iops, etc.

Tests #

I ran a few basic commands inside a new zfs dataset on my server tank/fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1 > single-4KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1 > 16-parallel-64KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1 > single-1MiB-random-write.txt

Results #

The single 4 KiB random write:

WRITE: bw=7836KiB/s (8024kB/s), 7836KiB/s-7836KiB/s (8024kB/s-8024kB/s), io=523MiB (548MB), run=68317-68317msec

The 16 parallel 64KiB random writes:

WRITE: bw=93.9MiB/s (98.4MB/s), 5599KiB/s-6303KiB/s (5734kB/s-6454kB/s), io=7642MiB (8013MB), run=81310-81418msec

The single 1MiB random write:

WRITE: bw=81.2MiB/s (85.1MB/s), 81.2MiB/s-81.2MiB/s (85.1MB/s-85.1MB/s), io=8177MiB (8574MB), run=100699-100699msec

Summary

So I don’t fully understand these numbers yet… 80-100 MiB/s isn’t super fast and that’s across a parallelized workload… The single threaded workloads have awful performance so this tells me something is wrong… I have a few ideas…

ZFS dataset config options such as ashift or the blocksize might be way misconfigured
The disks/pool which came from a TrueNAS/FreeBSD machine may have some artifacts that I need to clean up
The SAS controller I am using, which I flashed with IT firmware to get it into JBOD mode might be messed up
The data cables themselves could be a problem…

Points 3 and 4 are less likely given that the write speed does increase in the parallelized job but I’m a newbie so it’s time to dive in!

Playing with mdformat

2022-08-23T06:56:15Z

import this; print(this); print("what is taking so long black!!")

Add colored indicators to your dataframes html representation

2022-06-04T06:12:33Z

Mike Driscoll recently tweeted about making colored out with pandas DataFrames and I just had to try it for myself

Use Case

First though… why? My biggest use case is a monitoring pipeline of mine… The details aside, the output of my pipeline is a dataframe where each row has information about a failed pipeline that I need to go look into. I dump that result to a simle html file that’s hosted on an internal site and the file is updated every couple of hours. Adding some colored indicators automatically to the rows to help me assess severity of each record would be a handy way to quickly get an understanding the state of our pipelines.

How?

The docs for the applymap method state simply:

Apply a CSS-styling function elementwise.

Updates the HTML representation with the result.

So we can write a function that returns color: {color} based on the dataframe values and when we drop that dataframe to html we’ll have some simple css styling applied automagically!

By default the function will be applied to all columns of the dataframe, but that’s not useful if the columns are different types which is usually the case. Luckily there is a subset keyword to only apply to the columns you need!

Consider my example

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.read_csv("cars.csv")

sandbox   main via 3.8.11(sandbox) ipython
❯ def mpg_color(val: float):
...:     color = "red" if val < 21 else "green"
...:     return f"color: {color}"

sandbox   main via 3.8.11(sandbox) ipython
❯ df.style.applymap(mpg_color, subset="mpg").to_html("color.html")

I want to quickly see if the mpg is any good for the cars in the cars dataset and I’ll define “good” as better than 21 mpg (not great I know but just for the sake of discussion…)

The function returns an appropriate css string and after I style.applymap on just the mpg column we get this!

	Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
0	Mazda RX4	21.000000	6	160.000000	110	3.900000	2.620000	16.460000	0	1	4	4
1	Mazda RX4 Wag	21.000000	6	160.000000	110	3.900000	2.875000	17.020000	0	1	4	4
2	Datsun 710	22.800000	4	108.000000	93	3.850000	2.320000	18.610000	1	1	4	1
3	Hornet 4 Drive	21.400000	6	258.000000	110	3.080000	3.215000	19.440000	1	0	3	1
4	Hornet Sportabout	18.700000	8	360.000000	175	3.150000	3.440000	17.020000	0	0	3	2

Subset a list based on values in another list with itertools.compress

2022-05-19T14:22:58Z

I have list [True, False, False, True] and another list [1, 2, 3, 4] and a use case where I want to filter list 2 based on list 1 to remove values that line up with the element False in list 1…. so the outcome will be [1, 4]. list(compress(list2, list1)) will do it. As long as you can create a mask for the filter than itertool.compress will be your friend!

Forms with FastAPI and Jinja

2022-05-15T00:00:00Z

I just started using FastAPI for a home project and needed to pass back a dynamic number of values from a form rendered with jinja…

Dynamic Values

The jinja templating for rendering HTML based on something like a python iterable is nice and easy

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). it?

The way to pass back all those values is pretty straight forward (after hours of messing around that is!)

# I hate it when tutorials don't show ALL relevant pieces to the blurb
import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

Dynamic-Form-Values-With-Jinja-And-Fastapi

2022-05-15T00:00:00Z

I’m currently working on a self-hostable wish list app using FastAPI so we can finally drop Amazon forever. (The lists funcionality has been super handy for sharing holiday gift ideas with the famj!)

FastAPI

FastAPI is an amazing framework for quickly building APIs with Python. I will have a slightly longer post about my brief experience with it coming later…

Jinja, Forms, and FastAPI

One of the last things I needed to figure out in my app was how to generate a form in a Jinja template with a dynamic number of inputs and then pass all the inputs to the backend to perform a database operation (my exact case was removing rows from a table).

Explicit Values #

The way to pass back explicit variables is really easy…

Our form would look like this (I’m using bootstrap CSS)

<form method="post">
    <div class="form-check ">
        <input class="form-check-input"  name="item_1" id="itemOne" value="1" type="checkbox">
        <label class="form-check-label" for="itemOne" > A label for this item </label>
    </div>
    <div class="form-check ">
        <input class="form-check-input"  name="item_2" id="itemTwo" value="2" type="checkbox">
        <label class="form-check-label" for="itemTwo" > A label for item 2 </label>
    </div>

<button type="submit" class="submit btn btn-xl" >Submit</button>
</form>

So what is this? This form will have 2 rows with the lables you see in <label> </label> and checkboxes that when checked would have the value value in each <input> line.

So our backend might looks something like this…

I’m keeping all the imports and stuff here to show where they come from but I won’t discuss it all here - that’ll be in a future post

import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    item_1: int = Form(...),
    item_2: int = Form(...)
    db: Session = Depends(create_get_session),
):
    print(item_1)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    print(item_2)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

Dynamic values #

That’s all pretty simple… pass back values by the name in the form…

What about a form that’s generated dynamically? This is my case since I display a row/checkbox for every row in my table so my form looks like this…

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). Now I started scratching my head on how to pass an unknown number of inputs to my backend of FastAPI wants each input explicitly defined and typed… I can’t just pass the form back becuase that’s not a thing so what’s the way to do it?

# same stuff as above, only showing post method here
@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

I look forward to my wish list app maturing and I hope this helps someone working with FastAPI!

Python-Eval

2022-05-12T00:00:00Z

TODO

title = "my Title"
eval('"my" in title')

>>> True

print("hello, world"); print("formatting")

Dataframe-To-Styled-Html

2022-05-07T00:00:00Z

I wrote up a little on exporting DataFrames to markdown and html here

But I’ve been playing with a web app for with lists and while I’m toying around I learned you can actually give your tables some style with some simple css classes!

To HTML

Reminder that if you have a dataframe, df, you can df.to_html() to get an HTML table of your dataframe.

Well you can pass some classes to make it look super nice!

Classes and CSS

I don’t know anything really about CSS so I won’t pretend otherwise, but as I was learning about bootstrap that’s where I stumbled upon this…

There are several classes you can pass but I found really good luck with table-bordered and table-dark for my use case

df.to_html(classes=["table table-bordered table-dark"])

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

You try it!

Crack open ipython and make a dataframe, then df.to_html(classes=["table table-bordered table-dark"]), copy the output (minus the quote marks ipython uses to denote the string type) that into my-file.html, open that up in a browser and be amazed!

For added effeciency try using pyperclip to copy the output right to your clipboard!

pip install pyperclip and then pyperclip.copy(df.to_html(classes=["table table-bordered table-dark"]))

Dataframe-To-Markdown

2022-05-07T00:00:00Z

Pandas

pandas.DataFrames are pretty sweet data structures in Python.

I do a lot of work with tabular data and one thing I have incorporated into some of that work is automatic data summary reports by throwing the first few, or several relevant, rows of a dataframe at a point in a pipeline into a markdown file.

Pandas has a method on DataFrames that makes this 100% trivial!

The Method

Say we have a dataframe, df… then it’s literally just: df.to_markdown()

❯ df.head()

          Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
0          Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
1      Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
2         Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
3     Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
4  Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2

In ipython I can call the method and get a markdown table back as a string


mental-data-lake   new-posts via 3.8.11(mental-data-lake) ipython
❯ df.head().to_markdown()
'|    | Unnamed: 0        |   mpg |   cyl |   disp |   hp |   drat |    wt |   qsec |   vs |   am |   gear |   carb |\n|---:|:------------------|------:|------:|-------:|-----:|-------:|------:|-------:|-----:|-----:|-------:|-------:|\n|  0 | Mazda RX4         |  21   |     6 |    160 |  110 |   3.9  | 2.62  |  16.46 |    0 |    1 |      4 |      4 |\n|  1 | Mazda RX4 Wag     |  21   |     6 |    160 |  110 |   3.9  | 2.875 |  17.02 |    0 |    1 |      4 |      4 |\n|  2 | Datsun 710        |  22.8 |     4 |    108 |   93 |   3.85 | 2.32  |  18.61 |    1 |    1 |      4 |      1 |\n|  3 | Hornet 4 Drive    |  21.4 |     6 |    258 |  110 |   3.08 | 3.215 |  19.44 |    1 |    0 |      3 |      1 |\n|  4 | Hornet Sportabout |  18.7 |     8 |    360 |  175 |   3.15 | 3.44  |  17.02 |    0 |    0 |      3 |      2 |'

You can drop that string into a markdown file and using any reader that supports the rendering you’ll have a nicely formated table of example data in whatever report you’re making!

Bonus method

Just like markdown, you can export a dataframe to html with df.to_html() and use that if it’s more appropriate for your use case:


'<table border="1" class="dataframe">\n  <thead>\n    <tr style="text-align: right;">\n      <th></th>\n      <th>Unnamed: 0</th>\n      <th>mpg</th>\n      <th>cyl</th>\n      <th>disp</th>\n      <th>hp</th>\n      <th>drat</th>\n      <th>wt</th>\n      <th>qsec</th>\n      <th>vs</th>\n      <th>am</th>\n      <th>gear</th>\n      <th>carb</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Mazda RX4</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.620</td>\n      <td>16.46</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Mazda RX4 Wag</td>\n      <td>21.0</td>\n      <td>6</td>\n      <td>160.0</td>\n      <td>110</td>\n      <td>3.90</td>\n      <td>2.875</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Datsun 710</td>\n      <td>22.8</td>\n      <td>4</td>\n      <td>108.0</td>\n      <td>93</td>\n      <td>3.85</td>\n      <td>2.320</td>\n      <td>18.61</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Hornet 4 Drive</td>\n      <td>21.4</td>\n      <td>6</td>\n      <td>258.0</td>\n      <td>110</td>\n      <td>3.08</td>\n      <td>3.215</td>\n      <td>19.44</td>\n      <td>1</td>\n      <td>0</td>\n      <td>3</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Hornet Sportabout</td>\n      <td>18.7</td>\n      <td>8</td>\n      <td>360.0</td>\n      <td>175</td>\n      <td>3.15</td>\n      <td>3.440</td>\n      <td>17.02</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>2</td>\n    </tr>\n  </tbody>\n</table>'

My blog will render that html into a nice table! (After removing new line characters)

Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160.0	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160.0	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108.0	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258.0	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360.0	175	3.15	3.440	17.02	0	0	3	2

Wish-List-With-Fastapi

2022-05-06T00:00:00Z

Amazon has crossed the line with me just one too many times now so we are looking to drop them like every other Big Tech provider….

However, one key feature of Amazon that has been so useful for us is Lists… We can just maintain a list for each of us and then family members can login anytime and check it out… This really alleviates any last minute gift idea stress right before a birthday or something.

So I need a nice gift list service but I don’t want to be locked into one company (like a Target registry or something) and I’d like to host it myself

The internets had a few options but nothing looked/felt like I wanted to I decided to build my own.

The Frontend

I have no idea how to do front end so stay tuned

The Backend

FastAPI for the win on this one… I followed a few examples online and what I was able to build in just a few minutes is pretty impressive thanks to the design of FastAPI.

Some key features are:

Auto doc generation
Required typing (which makes #1 possible)
Built-in api testing in the browser
Easy integration with sqlalchemy
Development time so short you won’t be done with your coffee before having something up and running!

Database #

Starting with a simple database.py we can create a sqlalchemy session with a base model with about 7 lines of code…


from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker


SQLALCHEMY_DATABASE_URL = "sqlite:///wishes.sqlite3"
engine = create_engine(SQLALCHEMY_DATABASE_URL)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

Model #

For my wish list I needed just a simple table:

id	person	item	link	purchased	purchased_by	date_added
1	pypeaday	A sweet item	www.mystore.store	False	dad	2022-05-05 21:55:09
2	pypeaday	A bitter item	www.bitterstore.com	True	Mrs. pypeaday	2022-05-06 06:55:54

The table is simple enough… A unique key, the person who the wish belongs to, the item (or wish), a link to the item, whether it’s been purchased or not and by whom, and the date it was added.

To make this model with sqlalchemy we can make a model.py like so:

from database import Base
from sqlalchemy.schema import Column
from sqlalchemy.types import Boolean, Integer, String, Text


class Wishes(Base):
    __tablename__ = "Wishes"
    id = Column(Integer, primary_key=True, index=True)
    person = Column(String(20))
    item = Column(Text())
    link = Column(Text())
    purchased = Column(Boolean())
    purchased_by = Column(String(90))
    date_added = Column(String(15))

Schema #

One of the best things about FastAPI is trivial integration with pydantic. We can define a schema to ensure any data posted is not missing anything!

Make a schema.py with the following:

from pydantic import BaseModel
import time
from typing import Optional


class wish_schema(BaseModel):

    person: str
    item: str
    link: str
    purchased: bool = False
    purchased_by: Optional[str] = None
    date_added: Optional[str] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

    class Config:
        orm_mode = True


class patch_schema(BaseModel):

    purchased: bool
    purchased_by: Optional[str] = None

    class Config:
        orm_mode = True

I have 2 schemas - one for a wish which you’ll see down below is used to validate any post requests.

To simplify things for me I made another schema, patch_schema which I use for the route that updates the table (ie. marking an existing wish as purchased)

Session #

One of the last things we need is a Session

So make a session.py…

from database import SessionLocal, engine
import model

model.Base.metadata.create_all(bind=engine)


def create_get_session():
    try:
        db = SessionLocal()
        yield db
    finally:
        db.close()

Our routes will depend on this create_get_session function that will yield a db object through which we’ll udpate our database

Ok just do it already!

So our main.py will have a few routes in it…

What do we want to support?

Getting all wishes
Getting a specific wish
Updating a specific wish
Deleting a wish

I think the script is fairly self explanatory but here’s a few notes…

We decorate each function with @app.<method> and define response_model as well as status_code
The functions are defined with async (this was my first exposure to this so I can’t go in depth on it yet)
The functions all take a db which is from session.py and that db depends on the create_get_session function
If the db is being updtes then we type the object used for the update with the appropriate schema (either wish_schema or patch_schema)

From there we’re in true python-land where you can basically guess the methods on db and you’d probably be right… (like query, upddate, delete etc.)

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from typing import List
from model import Wishes
from schema import wish_schema, patch_schema
from session import create_get_session

app = FastAPI()


@app.get("/")
def read_root():
    return {"message": "server is up!"}


@app.get("/wishes", response_model=List[wish_schema], status_code=200)
async def read_wishes(db: Session = Depends(create_get_session)):
    wishes = db.query(Wishes).all()
    return wishes


@app.post("/wishes", response_model=wish_schema, status_code=201)
async def add_wish(wish: wish_schema, db: Session = Depends(create_get_session)):
    new_wish = Wishes(
        person=wish.person,
        item=wish.item,
        link=wish.link,
        purchased=wish.purchased,
        purchased_by=wish.purchased_by,
        date_added=wish.date_added,
    )
    db.add(new_wish)
    db.commit()

    return new_wish


@app.get("/wishes/{id}", response_model=wish_schema, status_code=200)
async def get_wish(id: int, db: Session = Depends(create_get_session)):
    wish = db.query(Wishes).get(id)
    return wish


@app.patch("/wishes/{id}", response_model=wish_schema, status_code=200)
async def update_wish(
    id: int, patch: patch_schema, db: Session = Depends(create_get_session)
):
    db_wish = db.query(Wishes).get(id)
    db_wish.purchased = patch.purchased
    db_wish.purchased_by = patch.purchased_by
    db.commit()
    db.refresh(db_wish)

    return db_wish


@app.delete("/wishes/{id}", status_code=200)
async def delete_wish(id: int, db: Session = Depends(create_get_session)):
    db_wish = db.query(Wishes).get(id)
    if not db_wish:
        raise HTTPException(status_code="404", detail="Wish id does not exist")

    db.delete(db_wish)
    db.commit()

    return None

My Code

You can find my repo here.

I’ll plan to update and maintain for as long as I use it

Pandas-String-Contains

2022-05-02T00:00:00Z

TL;DR

pandas.Series.str.contains accepts regular expressions and this is turned on by default!

Use case

We often need to filter pandas DataFrames based on several string values in a Series.

Notice that sweet pyflyby import 😁!

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.DataFrame({"A": ["string1", "string2", "string3"]})
[PYFLYBY] import pandas as pd

sandbox   main via 3.8.11(sandbox) ipython
❯ df

         A
0  string1
1  string2
2  string3

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains('1') | df.A.str.contains('2')]

         A
0  string1
1  string2

And this isn’t the worst thing in the world, especially for such a tiny example…

But what if we had dozens or more values to filter on?

Then it looks so much nicer to create an iterable of the values we want to filter on and join them with an apropriate regex operator (in this case | for inclusive or)


sandbox   main via 3.8.11(sandbox) ipython
❯ vals = ["1", "2"]  # iterable with whatever is appropriate for your use case

sandbox   main via 3.8.11(sandbox) ipython
❯ df[df.A.str.contains("|".join(vals), regex=True)]

         A
0  string1
1  string2

Fin

This is a super nice and concise way to do the kind of filtering my team does on a daily basis!

Unpack-Anywhere-With-Star

2022-04-24T00:00:00Z

Unpacking iterables in python with * is a pretty handy trick for writing code that is just a tiny bit more pythonic than not.

arr: Tuple[Union[int, str]] = (1, 2, 3, 'a', 'b', 'c')


print(arr)
>>> (1, 2, 3, 'a', 'b', 'c')

# the * unpacks the tuple into the individual elements
print(*arr)
>>> 1, 2, 3, 'a', 'b', 'c'

x, y, z, *alphas = arr

# x = 1, y = 2, z = 3
# alphas = [ 'a', 'b', 'c' ]

But @Ned Batchelder showed me via Twitter than you can arbitrarily unpack arguments based on position - it doesn’t have to be done at the beginning or the end!

x, y, *mixed, alpha = arr

# x = 1, y = 2
# mixed = [3, 'a', 'b']
# alpha = 'c'

I’m not entirely sure when I’ll need this but it definitley shows me another example of how flexible python is!

Pipx

2022-04-22T00:00:00Z

pipx is a tool I’ve been using to solve a few problems of mine…

pinning formatting tools like black, flake8, isort, etc. to the same version for all my projects
keeping virtual environments clean of things like cookiecutter
python utilities I want system wide but not in the global environment, like visidata

What is it? #

pip itself is just a package manager like homebrew, apt, etc. But it is tied to a python environment. If you aren’t using a virtual environment then pip will operate inside the global installation of python.

Operating within that environment has burned me several times and now I have a strict virtual environment usage policy.

But there are still things I don’t want to have to put in every virtual environment - enter pipx

What’s it do? #

When you pipx install {package} a stand alone virtual environment gets created (by default in ~/.local/pipx/venvs). THen you can install extra dependencies with pipx inject {package} {dependency}

ex. After pipx install visidata in order to open Excel files you need to pipx inject visidata xlrd

In the example with visidata, I can then use it anywhere, in any project, without re-installing with pip in every env.

Also for the formatting tools - I configure vim to run the pipx versions of them on save - this way I don’t have to put them in every project’s virtual environment!

What about pip? #

So obviously you can’t pipx everything, nor do you want to. I see it as a safe and better alternative to global package installation.

How can you then be sure that you never pip install into the global env?

Add require-virtualenv = True to your pip.conf and you’re good to go!

With that set, if you try to pip install pandas into the global env you’ll get a message like this:


~ on  (us-east-1)  NO PYTHON VENV SET
❯ pip install pandas
ERROR: Could not find an activated virtualenv (required).

End #

Disable your system pip to keep your base python safe
Use pipx for tools you want available everywhere or don’t have to need in a virtual environment!

Typeddict

2022-04-15T00:00:00Z

Type hinting has helped me write code almost as much, if not more, than unit testing.

One thing I love is that with complete type hinting you get a lot more out of your LSP. Typing dictionaries can be tricky and I recently learned about TypedDict to do exactly what I needed!

The Problem #

It might not be straight up obvious what the problem is, especially if you don’t utilize tools like mypy or flake8 in your development.

My handy-dandy nvim-lsp gives me a lot of feedback when I’m coding and it’s immensely helpful.

So with the LSP giving me constant feedback here’s the issue:

from typing import Dict, List, Union

my_dict: Dict[str, Union[List[str], str]] = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}

my_dict["key_2"].pop()

With the above script you’ll get an annoying warning about using pop on key_2.

The Solution #

Maybe you can stomach getting yelled at by your LSP but I like complete silence if at all possible.

TypedDict was the saving grace.

from typing import TypedDict

MyDict = TypedDict("MyDict", {"key_1": str, "key_2": List[str]})

my_typed_dict: MyDict = {
    "key_1": "val_1",
    "key_2": ["ls_1", "ls_2"],
}


my_typed_dict["key_2"].pop()

I was able to import TypedDict from typing, mypy_extensions, and typing_extensions

With TypedDict you define your custom type, match the first argument to TypedDict with the name of the variable (idk why), then type hint each key you expect in the dict! It’s super easy and I think puts you into a position of being extremely explicit with your dictionary variables. This isn’t always desired or appropriate but in most of my use cases it is.

RTFM #

There’s other implementation of TypedDict and while writing this I saw that most of the docs define a class for the type like this:

from typing import TypedDict
class MyDict(TypedDict):
    key_1: str
    key_2: List[str]

my_dict : MyDict = {'key_1': 'val_1', 'key_2': ["ls_1", "ls_2"]}

pep docs

mypy docs

And-vs-&

2022-04-06T00:00:00Z

I often struggle to remember the correct way to do and type comparisons when working in pandas.

I remember learning long long ago that and and & are different, the former being lazy boolean evaluation whereas the latter is a bitwise operation.

I learned a lot from this SO post

Lists #

Python list objects can contain unlike elements - ie. [True, 'foo', 1, '1', [1,2,3]] is a valid list with booleans, strings, integers, and another list. Because of this, we can’t use & to compare two lists since they can’t be combined in a consistent and meaningful way.

However we can use and since it doesn’t do bitwise operations, it just evaluates the boolean value of the list (basically if it’s non-empty then bool(my_list) evaluates to True)

Here’s an example:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list = [1, "2", "foo", [True], False]

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(my_list)
True

If we compare my_list with another_list using and then the comparision will go:

if bool(my_list):
    if bool(another_list):
       <operation> 
    else:
       break

Let’s see another example:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ another_list = [False, False]

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list and another_list
[False, False]

bool(my_list) evaluated to True, and bool(another_list) also evaluated to True even though it’s full of False values because the object is non-empty.

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if my_list and another_list:
...:     print("foo")
foo

So using and in this case results in a True conditional, so the print statement is executed.

Feels kind of counter-intuitive at first glance, to me anyways…

However, we can’t use & because there isn’t a meaningful to do bitwise operations over these two lists:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ my_list & another_list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-19-a2a16cebb3da>:1 in <cell line: 1>                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: unsupported operand type(s) for &: 'list' and 'list'

Numpy #

numpy arrays are special and they have a lot of fancy vectorization utilities built-in which make them great and fast for mathematical operations but now our logical comparisons need to be handled with a different kind of care.

First thing though - without some trickery they do not hold mixed data types like a list does (necessary, I think, for the vectorized optimization that numpy is built on top of)

With that out of the way here’s the main thing for this post, we can’t just evaluate the bool of an array - numpy says no no no.


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr = np.array(["1", 2, True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr
array(['1', '2', 'True', 'False'], dtype='<U21')

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(arr)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-25-4e8c5dd85b93>:1 in <cell line: 1>                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This means that using and with numpy arrays doesn’t really make sense because we probably care about the truth value of each element (bitwise), not the truth value of the array.

Notice that when I print arr all the elements are a string - and the dtype is <U21 for all elements.

This is not how I instantiated the array so be aware of that behavior with numpy.

<U21 is a dtype expressing the values are ‘Little Endian’, Unicode, 12 characters. See here for docs for docs

So for logical comparisions we should look at the error message then… Our handy error message says to try any or all

Because the datatypes in this example are basically strings, using arr.any() will result in an error that I do not fully understand, but any(arr) and all(arr) work…

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr.any():
...:     print("foo")
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-48-25ecac52db96>:1 in <cell line: 1>                                              │
│ /home/u_paynen3/personal/sandbox/.venv/sandbox/lib/python3.8/site-packages/numpy/core/_methods.p │
│ y:57 in _any                                                                                     │
│                                                                                                  │
│    54 def _any(a, axis=None, dtype=None, out=None, keepdims=False, *, where=True):               │
│    55 │   # Parsing keyword arguments is currently fairly slow, so avoid it for now              │
│    56 │   if where is True:                                                                      │
│ ❱  57 │   │   return umr_any(a, axis, dtype, out, keepdims)                                      │
│    58 │   return umr_any(a, axis, dtype, out, keepdims, where=where)                             │
│    59                                                                                            │
│    60 def _all(a, axis=None, dtype=None, out=None, keepdims=False, *, where=True):               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UFuncTypeError: ufunc 'logical_or' did not contain a loop with signature matching types (None, <class 'numpy.dtype[str_]'>) -> None

sandbox NO VCS  via 3.8.11(sandbox) ipython

❯ if all(arr):
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if any(arr):
...:     print("foo")
foo

Let’s change the example to just use integers and see what happens:

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr2 = np.array([1, True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ arr2
array([1, 1, 0])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr2.any():
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if arr2.all():
...:     print("foo")

Ah, now some sanity… First, the booleans are stored as integers, which based on this discussion makes sense. Next we check if any values (this is a bitwise operation) are True, which we see they are so the conditional evaluates to True. Howver, if we check that all values are True we see they aren’t, the last value is False or 0 so the conditional fails.

This is a different way to evaluate logical conditions than with lists and it’s because of the special nature of numpy arrays that allows them to be compared bitwise but on the flip side, there isn’t a meaningful way to evaluate the truth value of an array.

Pandas #

Now for pandas, which under the hood is a lot of numpy but not fully. pandas.Series objects can hold mixed data types like lists, however to logically evaluate truth values we have to treat them like numpy arrays.


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ s = pd.Series([1, "foo", True, False])

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ s

0        1
1      foo
2     True
3    False
dtype: object

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ bool(s)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-60-68e48e81da14>:1 in <cell line: 1>                                              │
│ /home/u_paynen3/personal/sandbox/.venv/sandbox/lib/python3.8/site-packages/pandas/core/generic.p │
│ y:1527 in __nonzero__                                                                            │
│                                                                                                  │
│    1524 │                                                                                        │
│    1525 │   @final                                                                               │
│    1526 │   def __nonzero__(self):                                                               │
│ ❱  1527 │   │   raise ValueError(                                                                │
│    1528 │   │   │   f"The truth value of a {type(self).__name__} is ambiguous. "                 │
│    1529 │   │   │   "Use a.empty, a.bool(), a.item(), a.any() or a.all()."                       │
│    1530 │   │   )                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Just like with numpy, we can’t evaluate the truth value of the series in a meaningful way, but bitwise operations make perfect sense…


❯ if s.any():
...:     print("foo")
foo

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ if s.all():
...:     print("foo")

I thought this was about and and &…

Right, so recall that and is a lazy boolean evaluation (ie. it evaluates the ’truth value’ an object) whereas & does bitwise comparison.

What we see then with pandas and numpy is that if we want to do logical comparisons, we need to do them bitwise, ie. use &.

Keep in mind though that the data types make a big deal - we can’t use & with strings because the bitwise operation isn’t supported, for strings we need to use the boolean evaluation.

The Original Point #

My main use case for this is finding elements in a dataframe/series based on 2 or more columns aligning row values…

Say I have a dataframe like this:


sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ df

   s s2   s3
0  1  0  foo
1  1  a  bar
2  1  b  baz
3  2  a  fee
4  2  0   fi

Example use case is I want to get the values in s3 where s is 1 and s2 is ‘a’. ie. I’m just after bar for now…

Up until now I’ve always just tried df.s3[(df.s == 1) and (df.s2 == "a")] the first time and every single time I’ve gotten this error that I just haven’t ever fully understood:

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

But after this deep dive I think I’ve grasped that and doesn’t actually do what I want here, and in order to do the bitwise comparision I need to use &

sandbox NO VCS  via 3.8.11(sandbox) ipython
❯ df.s3[(df.s == 1) & (df.s2 == "a")]

1    bar
Name: s3, dtype: object

End #

Hopefully this set of ramblings brings some clarity to and and & and you can Google one less error in the future in your logical comparison workflows 😄

Ipython-Prompt

2022-04-02T00:00:00Z

I have a post on starship where I have some notes on how I use starship to make my zsh experience great with a sweet terminal prompt.

Now… I spend quite a bit of time in ipython every day and I got kind of sick of the vanilla experience and wanted something that more closely matched my starship prompt.

There’s more to customizing ipython I know for sure but here’s 2 things I have going for me…

I use rich authored by @Will McGugan which makes much of my ipython experience great. I won’t write about that here but you can find my rich config here
I used pygments to customize the ipython prompt with my ipython_config.py and a startup script, next to my rich one, called 99-prompt.py.

The scripts inside ~/.ipython/<profile>/startup are executed in lexigraphical order, so it’s nice to name things in the 10’s to give room for adding scripts in between others down the line.

My prompt #

My zsh prompt looks a little something like this:

And after my ipython customiztion it currently (subject to much change but this is as of my dotfiles commit #d22088f6be81a58b5f7dfb73b7a4088cbdd9fece on main).

Now in ipython I have an indicator of my working directory, git branch, python environment, and a note that I’m in ipython and not zsh. I also configured my prompt to warn me if I’m not in a git directory!

All in all the customization isn’t too bad with just 2 specific files.

ipython_config.py #

There’s several use cases for ipython_config.py files in several areas on a pc - sometimes you want a common config across users, so you’d drop one in /etc/ipython and othertimes you have your own which is probably at ~/.ipython

My ipython config mostly has colors defined on pygment tokens plus a few autorun commands and pyflyby (see my friend Waylon’s post on pyflyby here)

I wanted to match my ipython somewhat to my tmux and vim color schemes, which I model after the vim-airline theme night owl.

After picking some some colors and saving variables it’s a matter of setting colors per token and then referencing those tokens in your version of 99-prompt.py.

You can check out my ipython_config.py here

For example, I can set Token.Name.Function to black, and in ipython then a function’s definition will appear in black text. I set mine to cyan to match my theme.

For the prompt colors just match the keyword in c.TerminalInteractiveShell.highlighting_style_overrides with what is referenced inside 99-prompt.py

For example, Token.Prompt is set to bold grey which gives me the bold chevron symbol you see in the above image that looks like my zsh prompt

Then in 99-prompt.py I have this set for the prompt:

Token.Prompt "❯ "

99-prompt.py #

You don’t need to name your script 99-prompt.py, but I wanted to know that it was for my prompt and I wanted it executed last so it made sense.

Here I have MyPrompt class with the prompt symbol defined as above and several other things…

class MyPrompt(Prompts):
    def in_prompt_tokens(self, cli=None):
        return [
            (Token, ""),
            (Token.OutPrompt, Path().absolute().stem),
            (Token, " "),
            (Token.Generic.Subheading, get_branch()[0]),
            (Token, " "),
            (Token.Generic.Heading, get_branch()[1]),
            (Token, " "),
            (Token.Name.Class, "via " + get_venv()),
            (Token, " "),
            (Token.Name.Entity, "ipython"),
            (Token, "\n"),
            (
                Token.Prompt
                if self.shell.last_execution_succeeded
                else Token.Generic.Error,
                "❯ ",
            ),
        ]

Notice I have 2 custom functions here, get_branch and get_venv which grab some git info and python env info and return strings I can dump into my prompt as shown above.

To finish you drop ip = get_ipython() and ip.prompts = MyPrompt(ip) at the bottom of your prompt script and you should be in custom prompt city!

End #

This is more or less notes for myself on how this works - drop by my ipython config in my dotfiles repo to see my full configs for ipython!

Deques

2022-03-31T00:00:00Z

I am working on a project to create a small system monitoring dashboard using the python psutil library.

The repo is here (if you want actual system monitoring please use netdata).

I’m using streamlit and plotly for the webserver, design, and plotting at the moment.

My Use Case #

I needed a way to refresh my plotly charts with a fixed window of time so that I’m able to just see relevant recent data instead of cramming all data for all time into one plot that’s 500 pixels wide…

Checking the length of arrays or lists every time I get a new piece of data feels kind of dumb and I thought “python must have a way to do this”…

“This” meaning, update values in a fixed length array without reallocating memory or recreating a copy of the list

Deques #

Enter the deque. It means “double ended queue” and is in general an Iterable that you can append values to either side or pop values from either side.

The init signature is straightforward enough and I’m sure there’s more to them than I know yet but here’s how I use it…

from collections import deque

my_deque = deque([1,2,3])

This gives us my_deque, created from an iterable, with several familiar methods like index, extend, append, etc. However there’s some new ones too such as appendleft and popleft.

my_deque.appendleft('a')
print(my_dequqe)
>>> deque(['a', 1, 2, 3])

my_deque.popleft()
>>> 'a'

<!--markata-attribution-->
print(my_deque)
>>> deque([1, 2, 3])

These are handy ways to manipulate the iterable that I needed for the arrays I plot with plotly!

See my follow-up to this on using Deques with plotly and streamlit to create a quick “dashboard” with live streaming data!

follow-up

Plotly-And-Streamlit

2022-03-31T00:00:00Z

Streamlit #

I use streamlit for any EDA I ever have to do at work. It’s super easy to spin up a small dashboard to filter and view dataframes in, live, without the fallbacks of Jupyter notebooks (kernels dying, memory bloat, a billion “Untitled N.ipynb” files, etc.)

At the highest level, streamlit lets you write a python script and call streamlit run my_script.py which will open up a web server with your streamlit stuff. The dashboard refreshes whenever you change the script so you can add capabilities in real time, super fast!

I’ll show an example of using streamlit and plotly to make a live dashboard to monitor system memory usage with psutil. This is apart of my posts on psutil and deques…

example at the bottom!

Plotly #

I’m not going to make a big time intro to plotly here - there’s a billion resources on the interwebs and the docs are really good.

Suffice it to say it’s my goto plotting library for basically any and all needs. I’m currently exploring it for live data streaming as I’m not sure it’s the best solution but it’s the one I’m familiar with.

For my not-netdata project of visualizing live system resource data I first need a way of appending data and popping data in and out of an array at every data refresh cycle to keep my plots looking nice with a fixed time window.

See deques for a short intro to the datatype I’m using.

First step is to initialize some objects to store data in.

data: Dict[str, MutableSequence[Optional[float]]] = defaultdict(deque)

arr_size = 10

data["time"] = deque([None] * arr_size)
data["used_memory"] = deque([None] * arr_size)

data is a dictionary that I’ll store deques in. The dictionary keys will be the type of data, in this case time and used_memory.

I fix an array size, arr_size to just 10 for now

Then I initialize the values for time and used_memory as deques of length arr_size. Simple enough!

Next is to fill those deques with some relevant data. I’m not actually sure if this is the best way to do this but here’s what I have done so far:

def refresh_data():
    global data
    memory = psutil.virtual_memory()

    data["time"].append(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
    data["used_memory"].append(memory.used // (1024**3))

    data["time"].popleft()
    data["used_memory"].popleft()

If you ignore my usage of global you’ll see that I can just append to each deque like it was a list.

But then to keep the relevant data in the deque, and to keep the length fixed, I simply popleft to remove the oldest datapoint!

A trivial dashboard #

Now I’ll prove just how easy it is to get a live data dashboard up and running with just a few lines of code thanks to streamlit!


if __name__ == "__main__":
    st.header("memory chart")
    stats = st.empty()
    while True:
        refresh_data()
        stats.plotly_chart(
            px.line(
                data,
                x="time",
                y="used_memory",
                title=f"Memory usage stored in a deque!",
               )
            )
        time.sleep(0.5)

st is the streamlit alias (imports shows at the bottom full example). st.header puts a nice header on the page. st.empty initializes an empty streamlit container in which we’ll put a plotly.express figure.

At each iteration we’ll refresh_data() which appends and pops data in the deques in the data dictionary. Then we update the stats container with a plotly graph and the refresh happens seamlessly.

All in all the script looks like this:


from collections import defaultdict, deque
import time
from typing import Dict, MutableSequence, Optional

from plotly import express as px
import psutil
import streamlit as st

data: Dict[str, MutableSequence[Optional[float]]] = defaultdict(deque)

arr_size = 10

data["time"] = deque([None] * arr_size)
data["used_memory"] = deque([None] * arr_size)


def refresh_data():
    global data
    memory = psutil.virtual_memory()

    data["time"].append(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
    data["used_memory"].append(memory.used // (1024**3))

    data["time"].popleft()
    data["used_memory"].popleft()


def memory_chart():
    fig = px.line(
        data,
        x="time",
        y="used_memory",
        title=f"Memory usage stored in a deque!",
    )
    return fig


if __name__ == "__main__":
    st.header("memory chart")
    stats = st.empty()
    while True:
        refresh_data()
        stats.plotly_chart(memory_chart())
        time.sleep(0.5)

You can save this as my_dash.py and run with streamlit run my_dash.py and should see something like the following!

self-hosted-media

2022-03-24T00:00:00Z

Self-hosting 1 or several media servers is another common homelab use-case. Getting content for your media servers is up to you, but I’ll show a few ways here to get content somewhat easily!

YouTube Disclaimer at Bottom

you-get #

you-get is a nice cli for grabbing media content off the web.

Installation #

pip install you-get or use ad-hoc with pipx run you-get <url>

Usage #

For example if I wanted to catch up on ancient Chinese military tactics I may go for The Art of War off the Internet Archive…

sandbox  🌱 main 🗑️  ×3🛤️  ×6via 🐍 v3.8.11 (sandbox)  took 15s
❯ you-get https://archive.org/details/art_of_war_librivox -i
Site:       Archive.org
Title:      The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archive
Type:       MP3 (audio/mpeg)
Size:       3.87 MiB (4055167 Bytes)

the -i is showing me the info of what would be downloaded without the flag (it’s like a dry run)

sandbox  🌱 main 🗑️  ×3🛤️  ×6via 🐍 v3.8.11 (sandbox)
❯ you-get https://archive.org/details/art_of_war_librivox
Site:       Archive.org
Title:      The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archive
Type:       MP3 (audio/mpeg)
Size:       3.87 MiB (4055167 Bytes)

Downloading The Art of War : Sun Tzu : Free Download, Borrow, and Streaming : Internet Archi.mp3 ...
 100% (  3.9/  3.9MB) ├████████████████████████████████████████████████████████████┤[1/1]  917 kB/s

Now I can toss that mp3 onto my booksonic server and study for world domination while I do the dishes!

pytube #

pytube is a python implementation of a youtube downloader that works at the command line or in python!

Installation #

docs

pip install pytube

Usage #

pytube has a lot of functionality, but a quick one would be the --list so you can see what qualities are available

sandbox   main ️  ×3️  ×7via  v3.8.11 (sandbox)  took 2m49s
❯ pytube https://www.youtube.com/watch\?v\=LDU_Txk06tM  --list
Loading video...
<Stream: itag="17" mime_type="video/3gpp" res="144p" fps="8fps" vcodec="mp4v.20.3" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="22" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.64001F" acodec="mp4a.40.2" progressive="True" type="video">
<Stream: itag="313" mime_type="video/webm" res="2160p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="401" mime_type="video/mp4" res="2160p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">
<Stream: itag="271" mime_type="video/webm" res="1440p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="400" mime_type="video/mp4" res="1440p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">
<Stream: itag="137" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="avc1.640028" progressive="False" type="video">
<Stream: itag="248" mime_type="video/webm" res="1080p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="399" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="av01.0.08M.08" progressive="False" type="video">
<Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">
<Stream: itag="247" mime_type="video/webm" res="720p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="398" mime_type="video/mp4" res="720p" fps="30fps" vcodec="av01.0.05M.08" progressive="False" type="video">
<Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">
<Stream: itag="244" mime_type="video/webm" res="480p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="397" mime_type="video/mp4" res="480p" fps="30fps" vcodec="av01.0.04M.08" progressive="False" type="video">
<Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e" progressive="False" type="video">
<Stream: itag="243" mime_type="video/webm" res="360p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="396" mime_type="video/mp4" res="360p" fps="30fps" vcodec="av01.0.01M.08" progressive="False" type="video">
<Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d4015" progressive="False" type="video">
<Stream: itag="242" mime_type="video/webm" res="240p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="395" mime_type="video/mp4" res="240p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">
<Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400c" progressive="False" type="video">
<Stream: itag="278" mime_type="video/webm" res="144p" fps="30fps" vcodec="vp9" progressive="False" type="video">
<Stream: itag="394" mime_type="video/mp4" res="144p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">
<Stream: itag="139" mime_type="audio/mp4" abr="48kbps" acodec="mp4a.40.5" progressive="False" type="audio">
<Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio">
<Stream: itag="249" mime_type="audio/webm" abr="50kbps" acodec="opus" progressive="False" type="audio">
<Stream: itag="250" mime_type="audio/webm" abr="70kbps" acodec="opus" progressive="False" type="audio">
<Stream: itag="251" mime_type="audio/webm" abr="160kbps" acodec="opus" progressive="False" type="audio">

pytube <url> --itag <> will download the specific itag from the list.

Notice that some itags are videos and others audio - so you can download just the music of a YT video.

pytube also works in python…

sandbox ↪ main v3.8.11 ipython
❯ from pytube import YouTube

sandbox ↪ main v3.8.11 ipython
❯ [x for x in YouTube("https://www.youtube.com/watch?v=LDU_Txk06tM").streams]

[
    <Stream: itag="17" mime_type="video/3gpp" res="144p" fps="8fps" vcodec="mp4v.20.3" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="22" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.64001F" acodec="mp4a.40.2" progressive="True" type="video">,
    <Stream: itag="313" mime_type="video/webm" res="2160p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="401" mime_type="video/mp4" res="2160p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">,
    <Stream: itag="271" mime_type="video/webm" res="1440p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="400" mime_type="video/mp4" res="1440p" fps="30fps" vcodec="av01.0.12M.08" progressive="False" type="video">,
    <Stream: itag="137" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="avc1.640028" progressive="False" type="video">,
    <Stream: itag="248" mime_type="video/webm" res="1080p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="399" mime_type="video/mp4" res="1080p" fps="30fps" vcodec="av01.0.08M.08" progressive="False" type="video">,
    <Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">,
    <Stream: itag="247" mime_type="video/webm" res="720p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="398" mime_type="video/mp4" res="720p" fps="30fps" vcodec="av01.0.05M.08" progressive="False" type="video">,
    <Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video">,
    <Stream: itag="244" mime_type="video/webm" res="480p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="397" mime_type="video/mp4" res="480p" fps="30fps" vcodec="av01.0.04M.08" progressive="False" type="video">,
    <Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e" progressive="False" type="video">,
    <Stream: itag="243" mime_type="video/webm" res="360p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="396" mime_type="video/mp4" res="360p" fps="30fps" vcodec="av01.0.01M.08" progressive="False" type="video">,
    <Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d4015" progressive="False" type="video">,
    <Stream: itag="242" mime_type="video/webm" res="240p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="395" mime_type="video/mp4" res="240p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">,
    <Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400c" progressive="False" type="video">,
    <Stream: itag="278" mime_type="video/webm" res="144p" fps="30fps" vcodec="vp9" progressive="False" type="video">,
    <Stream: itag="394" mime_type="video/mp4" res="144p" fps="30fps" vcodec="av01.0.00M.08" progressive="False" type="video">,
    <Stream: itag="139" mime_type="audio/mp4" abr="48kbps" acodec="mp4a.40.5" progressive="False" type="audio">,
    <Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio">,
    <Stream: itag="249" mime_type="audio/webm" abr="50kbps" acodec="opus" progressive="False" type="audio">,
    <Stream: itag="250" mime_type="audio/webm" abr="70kbps" acodec="opus" progressive="False" type="audio">,
    <Stream: itag="251" mime_type="audio/webm" abr="160kbps" acodec="opus" progressive="False" type="audio">
]

YouTube Frontends #

There’s 2 really good options for self-hosting a YT front-end…

Tube Archivist

YouTubeDL-Material

They have their pros and cons. You can also build yourself with the above utilities and use Plex or Jellyfin to serve up videos…

Your self-hosting journey is up to you!

YouTube #

Downloading YouTube videos is a bit of a sore topic… Mainly you don’t to hurt creators who rely on YT ad revenue for their livlihood.

Then again, maybe you’re a vigilante who knows that YT also monetizes videos for their own gain and that the creators don’t see that money either!

The solution is pretty easy and is 2-fold…

Download YT videos
Personally support the content creators you follow via paypall, patreon, or whatever else they might have set-up…. even a buck or two a month is more than they’d get from your ad revenue explicitly plus it all goes to them!

Skimpy

2022-03-23T00:00:00Z

EDA #

I work with data a lot, but the nature of my job isn’t to dive super deep into a small amount of datasets, I’m often jumping between several projects every day and need to just get a super quick glance at some tables to get a high level view.

When I’m doing more interactive exploration I’ve graduated from Jupyter cells with df_N.head() to using an amazing tool called visidata

However, Visidata is a terminal based application and I’m often in an iPython console… so is there a way to move even faster for my super quick summary views?

yes!

Skimpy #

First thing to do is pip install skimpy and then it’s as easy to get some summary stats with skimpy <data>

This is super nice for seeing missing values in particular as well as the distribution shape of the data.

iPython #

But wait… I just said I’m normally in an iPython session but that was called from zsh.. If I’m hoping back into zsh I might as well use visidata to have more powerful exploration at my fingertips. So… can I see this table quickly without breaking my iPython workflow?

Of course you can with magic!

The above assumes you’re looking at a file, like you would in the terminal. skimpy works even better in iPython with from skimpy import skim then pass any DataFrame to skim!

Pyclean

2022-03-22T00:00:00Z

I like to keep my workspace clean and one thing that I don’t personally love looking at is the __pycache__ directory that pops up after running some code. The *.pyc files that show up there are python bytecode and they are cached to make subsequent runs a tad faster. My stuff never really needs this bonus speed boost and so I came across a neat tool called pyclean!

Pyclean #

The easiest way (in my opinion) to run pyclean is to just use pipx run.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)  took 9s
❯ ls
abcmeta.py  __pycache__  python-print-align.py  system-monitor-psutils.py

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ pipx run pyclean .
⚠️  pyclean is already on your PATH and installed at /usr/bin/pyclean. Downloading and running anyway.
Cleaning directory .
Total 1 files, 1 directories removed.

sandbox/src  🌱 main 🗑️  ×3🛤️  ×2via 🐍 v3.8.11 (sandbox)
❯ ls
abcmeta.py  python-print-align.py  system-monitor-psutils.py

Why not bash? #

You could accomplish something similar with rm **/*.pyc or find -n '*.py?' -delete but there’s a chance you’ll find something you don’t love gone. Also this won’t help our poor Windows friends out there! pyclean is fully python so it’s OS independent.

Credits! #

repo

Psutil-01

2022-03-16T00:00:00Z

Mike Driscoll has been posting some awesome posts about psutil lately. I’m interested in making my own system monitoring dashboard now using this library. I don’t expect it to compete with Netdata or Glances but it’ll just be for fun to see how Python can solve this problem!

Repo coming soon

Example code: #

Here’s a short snippit to get used/available/total RAM and disk space (on partitions that you probably care about)


import psutil
import socket

print(f"System Memory used: {psutil.virtual_memory().used // (1024 ** 3)} GB")
print(f"System Memory available: {psutil.virtual_memory().available // (1024 ** 3)} GB")
print(f"System Memory total: {psutil.virtual_memory().total // (1024 ** 3)} GB")


print(f"Hostname: {socket.gethostname()}")

partitions = psutil.disk_partitions()

for part in partitions:
    mnt = part.mountpoint
    if "snap" in mnt or "boot" in mnt:
        continue
    disk = psutil.disk_usage(mnt)
    print(f"Usage at {mnt} on {part.device}: {disk.used // (1024 ** 3)} GB")
    print(f"Free at {mnt} on {part.device}: {disk.free // (1024 ** 3)}GB")
    print(f"Total at {mnt} on {part.device}: {disk.total // (1024 ** 3)}GB")

Bonus Ipython tip! Save this to a script called my_script.py and in Ipython you can %run -m my_script to run it!

project ↪ main v3.8.11 ipython
❯ %run -m system-monitor-psutils
System Memory used: 25 GB
System Memory available: 5 GB
System Memory total: 31 GB
Hostname: ryzen-3600x
Usage at / on /dev/nvme1n1p2: 81 GB
Free at / on /dev/nvme1n1p2: 351 GB
Total at / on /dev/nvme1n1p2: 456 GB

Mu

2022-03-15T00:00:00Z

If you work with a template for several projects then you might sometimes need to do the same action across all repos. A good example of this is updating a package in requirements.txt in every project, or refactoring a common module. If you have several repos to do this across then it can be time consuming… enter mu-repo

Mu #

mu-repo is an awesome cli tool for working with multiple git repositories at the same time. There are several things you can do:

mu status will give you the git status of every registered repo (see below)
mu sh will let you execute system level commands in every repo
mu stash will stash all changes across all registered repos
There’s literally a ton more but these are some handy ones

Registration #

mu tracks its own groups, and there is a default group when no particular one is active. It’s as simple as mu register proj1 prog2 ... to get repos registered


❯ mu register proj1 proj2
Repository: proj1 registered
Repository: proj2 registered

❯ mu status

  proj1 : git status
    On branch main

    No commits yet

    Untracked files:
    (use "git add <file>..." to include in what will be committed)
    requirements.txt

    nothing added to commit but untracked files present (use "git add" to track)

  proj2 : git status
    On branch main

    No commits yet

    Changes to be committed:
    (use "git rm --cached <file>..." to unstage)
    new file:   requirements.txt

Working with mu #

As you can see above I have two projects each with a requirements.txt added but not committed yet. Using mu I can stage this change across both repos at once.


❯ mu add requirements.txt

  proj1 : git add requirements.txt

  proj2 : git add requirements.txt

Then as you might imagine, I can make the commit in each repo


❯ mu commit -m "Add requirements.txts"

  proj1 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

  proj2 : git commit -m Add requirements.txts
    [main (root-commit) 18376d7] Add requirements.txts
    1 file changed, 1 insertion(+)
    create mode 100644 requirements.txt

mu groups #

The other thing I got a lot of use out of recently was mu’s groups. At work I have about 40 repos cloned that are all based on the same kedro pipeline template. Some of these projects have been deprecated. I also have several more repos that are not kedro template - custom libraries or something. group let me utilize mu across different groups of repos.

Say proj2 is a deprecated project that I don’t need to worry about making changes to anymore. I don’t just have to unregister it, instead I can make a group called “active” and register proj1 in that group


❯ mu group add active --empty

~/personal
❯ mu group add deprecated --empty

~/personal
❯ mu group
  active
* deprecated

The * tells me which group is active. The --empty flag tells mu to not add all registered repos to that group. If I don’t want to use any groups then mu group reset will go back to the default group with all registered repos.

With groups I can register only the repos that I want to be working across in their own group and not worry about affecting other repos with my batch changes!

Abstract-Base-Class

2022-03-09T00:00:00Z

ABCMeta #

I don’t do a lot of OOP currently, but I have been on a few heavy OOP projects and this ABCMeta and abstractmethod from abc would’ve been super nice to know about!

If you are creating a library with classes that you expect your users to extend, but you want to ensure that any extension has explicit methods defined then this is for you!.

from abc import ABCMeta, abstractmethod
class Family(metaclass=ABCMeta):
    @abstractmethod
    def get_dad(self):
        """Any extension of the Family class must implement a `get_dad` method"""

class MyFamily(Family):
    pass

If I try to instantiate MyFamily I will not be allowed:


❯ my_fam = MyFamily()
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-8-ecb8e21ce815>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Can't instantiate abstract class MyFamily with abstract methods get_dad

In order for me to extend Family I have to implement the method get_dad

class MyFamily(Family):
    def get_dad(self):
        return "Me"

Now everything works as expected and I can sleep well knowing no one can extend my base class without creating methods I know they need.


my_fam = MyFamily()

my_fam.get_dad()
'Me'

Python-F-String-Align

2022-03-08T00:00:00Z

I am personally trying to use logger instead of print in all of my code, however I learned from [@Python-Hub] that you can align printouts using print with f-strings!.

This little python script shows how options in the f-string can format the printout.


import random

variables = "Foo Bar Baz Bing".split()
scores = random.sample(range(1, 11), len(variables))

print("*" * 30)
print("\n")
print("With 'varable' left aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:<10} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' right aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:>15} | {score}")

print("*" * 30)
print("\n")
print("With 'varable' center aligned")
for varable, score in zip(variables, scores):
    print(f"{varable:^5} | {score}")

Python-Builtin-Calendar

2022-03-08T00:00:00Z

Being lazy #

I almost exclusively use Python for my job and have been eye-balls deep in it for almost 5 years but I really lack in-depth knowledge of builtins. I recently learned of an awesome builtin called calendar that has way more than I know about for sure but I’m glad I know it’s here now!

I only needed it because I was too lazy to hard code the 7 weekdays into my module but it turns out there’s a lot of useful things like calendar.isleap()!

## Future use

I’m not exactly sure what will come my way where calendar will be super relevant but like anything, I’m just glad to know it exists for when the time arises!

Dataframe-Memory-Usage

2022-03-07T00:00:00Z

I have often wanted to dive into memory usage for pandas DataFrames when it comes to cloud deployment. If I have a python process running on a server at home I can use glances or a number of other tools to diagnose a memory issue… However at work I normally deploy dockerized processes on AWS Batch and it’s much more challenging to get info on the dockerized process without more AWS integration that my team isn’t quite ready for. So TIL that I can get some of the info I want from pandas directly!

DataFrame.info()

I didn’t realize that df.info() was able to give me more info than just dtypes and some summary stats… There is a kwarg memory_usage that can configure what you need to get back, so df.memory_usage="deep" will give you how much RAM any given DataFrame is using! Amazing tool for finding issues with joins or renegade source data files.

df = pd.read_csv("cars.csv")

df.info(memory_usage="deep")

Pandas-Select-Dtypes

2022-03-05T00:00:00Z

On my team we often have to change data types of columns in a pandas.DataFrame for a variety of reasons. The main one is it tends to be an artifact of EDA whereby a file is read in via pandas but the data types are somewhat wonky (ie. dates show up as strings, or a column that should be a integer comes in as float, etc.). The best solution I think is to leverage the dtypes keyword argument in which pd.read_X method is used. However there is another way which is to coerce the data types at runtime instead of loadtime.

A handy way to do this is by using pandas.DataFrame.select_dtypes…

Here is an example of finding columns read in as datetime64 and the developer would prefer to use pandas datetimes.

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.columns:
    if df[c].dtype == "datetime64":
        df[c] = pd.to_datetime(df.c)

Here is the difference in code flow between select_dtypes and manually finding the datetype64 columns:

df = pd.read_csv("./file-with-confusing-dtypes.csv")
for c in df.select_dtypes('datetime64'):
    df[c] = pd.to_datetime(df.c)

The difference isn’t huge but it’s the little steps in leveling up that turn script-kitty scripts into clean looking functions.