Posts tagged: python

💭 uv script management

2025-09-08T11:25:33Z

Here’s my thought on uv script management

Dang Waylon! Love these few lines… uv is great, been using it for environment and project requirements management but I didn’t realize it was this easy to setup a pep 723 compliant script!

Love this sed -i '1i #!/usr/bin/env -S uv run --script' little gem too - I typically do uv run script.py so I’ll see about making this a simple zsh alias for faster iteration on python scripts… I often have AI write quick bash scripts to accomplish things, but it’s quite a bit harder for me to read than python, and with the power uv is bringing to the python ecosystem, maybe it’s fine to just be a little higher in the programming stack for script-kitty stuff

Note

This is one of [[ my-thoughts ]]. I picked this up from Waylon Walker(https://thoughts.waylonwalker.com). It’s a short note that I make about someone else’s content online. Learn more about the process Thoughts

Zsh alias for uv scripts

2025-09-08T06:33:13Z

It’s apparently advantageous for uv to have a specific shebang set that uv init doesn’t add, no problem, we can have a zsh alias for it

alias add-shebang='f(){ 
  if [[ ! -s $1 ]]; then 
    echo "#!/usr/bin/env -S uv run --script" > "$1"; 
  elif head -n1 "$1" | grep -q "^#!"; then 
    echo "Shebang already present in $1"; 
  else 
    sed -i "1i #!/usr/bin/env -S uv run --script" "$1"; 
  fi
}; f'

Then if you have a python script from uv init --script you can add-shebang myscript.py to add the shebang.

Python editable installs and nested scripts

2025-07-14T08:40:32Z

Intro

I have been working towards a standard structure for all my python projects - and I mean much more than just uv init or something along those lines. But one of the things that got me recently was starting to keep a scripts/ folder that ran code based on what was in src/ or app/ (I’m working on that…) being installed in the virtual environment… so I’d uv pip install -e . or uv pip install -e src (kind of depends on the structure) and then in scripts/foo.py I’d have from my_cool_code.module import thing….

Running scripts has never been a problem… but I’ve always kept them in the root of the repo OR inside app or src in a nested directory…

So here’s where I learned a little something about python my_script.py vs python -m my_script

Example #

I setup a short example repo to illustrate the problem

link to repo

The repo structure is very simple…

python-e-example
  ├── foo.py
  ├── pyproject.toml
  ├── src
  │   ├── module.py
  ├── README.md
  └── scripts
      └── bar.py

scripts/bar.py and foo.py are the same code:

from src.module import my_func

my_func()

and then src/module.py:

def my_func():
    print("it worked!")

The rest of the structure is from uv init basically, then uv venv will make your virtual environment, and uv pip install -e . will install the library in editable mode.

So what’s the problem? #

Let’s run foo.py

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python foo.py
it worked!

Nice! It works just like we’d expect…

What about python scripts/bar.py?

nic in python-e-example   main   ×7 via   v3.13.1(python-e-example)   (dev) 󰒄
⬢ [devbox] ❯ python scripts/bar.py
Traceback (most recent call last):
  File "/tmp/python-e-example/scripts/bar.py", line 8, in <module>
    from src.module import my_func
ModuleNotFoundError: No module named 'src'

Come again for big fudge...

Ok let’s stay calm and troubleshoot #

What if we open a python shell…

Python 3.13.1 (main, Dec 19 2024, 14:32:25) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from src.module import my_func
>>> my_func()

<!--markata-attribution-->
it worked!

ok… as expected it worked, and we’ve learned nothing…

implicit behavior

We didn’t learn anything from the example but something is present that we don’t see… python foo.py implicitly adds the path to the script to sys.path and so the src library is in the same directory as the script foo.py so the library is on sys.path…. take note of this

The Problem #

Turns out uv pip install . and uv pip install -e . are actually different

The editable install adds a .pth file into site-packages which acts as a shell hook.

# .venv/lib/python3.13/site-packages/__editable__.python_e_example-0.1.0.pth

/tmp/python-e-example/src

So when python runs a script, the location of the script is implicitly added to sys.path as noted above. The python process invoked by python foo.py has $PWD on sys.path which is /tmp/python-e-example in the example but python scripts/bar.py has /tmp/python-e-example/scripts on sys.path… which means that /tmp/python-e-example is NOT on sys.path in the process that’s running scripts/bar.py and so the src folder won’t be searched by python because the .pth file points to a directory that’s no on the path!

The Solution #

We can instead of running the script “as a script” we can tell python to execute it as as a module with python -m scripts.bar. This is syntatically similar to python path/to/script.py but instead tells the python interpreter to execute the script as a module so the path is replaced by import syntax.

This works for foo.py too

⬢ [devbox] ✗ python -m foo
it worked!

⬢ [devbox] ❯ python -m scripts.bar
it worked!

ChatGPT #

It’s noteworthy that jipity was helpful in exploring these issues I was running into

link to convo

💭 Build backend | uv

2025-07-04T11:43:30Z

Here’s my thought on Build backend | uv

uv build backend ready for production! Dang… I need to visit my hatch builds, I love hatch’s simplicity specifically with versioning… Not sure if it’s worth actually switching to uv for my builds or not, but it is dang cool to see astral take on the python space with unbelievable improvements

Note

💭 Switching pip to uv in a Dockerized Flask / Django App — Nick …

2025-06-27T11:18:16Z

Here’s my thought on Switching pip to uv in a Dockerized Flask / Django App — Nick Janetakis

It’s nice to see uv gaining popularity in so many places - and replacing vanilla pip with uv in containerized environments seems great due if for no other reason than the time you’ll save in CI… A few years ago I refactored a disjointed container build process to use uv instead of pip, and due to the design of the build pipelines we went from 10-15 minute builds to 2-3 minutes…

Note

💭 Snowflake Status - Certificate Authority Update for AWS Regions

2025-06-16T16:37:12Z

Here’s my thought on Snowflake Status - Certificate Authority Update for AWS Regions

A recent Snowflake issue borked some old pipelines - thankfully it appears to be a simple fix. Just goes to show you that you can’t just deploy pipelines and expect them to live forever, they have to be maintained when there are ANY external dependencies

Note

I built a simple app for adding images to my blog

2025-06-07T13:30:28Z

Quick Deets

I built a simple fastAPI app called “shotput” that I run locally inside a git repo where I save images for my blog. The app is simple:

upload image from clipboard
app commits image to repo (path is configurable)
app pushes the commit (auto-push configurable)
app returns a markdown link I can paste in my blog that uses statically.io (see statically.io to help me out) to serve the image from the repo to my blog

See the image repo with app details at images.pype.dev

Usage

Anyone could copy the script and config to their own git repo to put images in and run this at home - instructions are in the readme but for simple running on localhost it’s just uv run

Example

pasting an image let’s you copy the markdown link for easy pasting

![20250607130609_dadd33eb.png](https://cdn.statically.io/gh/pypeaday/images.pype.dev/main/blog-media/20250607130609_dadd33eb.png)

Pandas.DataFrame info always answers and is sometimes right

2025-05-26T09:08:08Z

TIL

Today I learned that the .info() of a pandas.DataFrame will always give you an answer, but it is wildly difficult to know how accurate it is, because it depends on the underlying data types. If everything is a NumPy Dtype, then the result you get is the amount of memory NumPy is using under the hood, which is very accurate. But as soon as you have something like a Python string in the data frame, now pandas will tell you how much memory the NumPy data types are using, and it will give you some quick estimate of how much memory space the Python string types are using because numpy is just storing a pointer to that data in memory, but this is far less accurate because it doesn’t actually know what the strings are.

Memory Usage Deep or True #

To get an honest answer about a DataFrame’s memory usage, you need to pass memory_usage='deep' to the .info() method.

# This is the default
In [50]: df.info(memory_usage=True)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 372.0+ bytes

Notice that the memory usage has a + in it? You can force a deeper analysis by passing memory_usage='deep'

In [51]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   s1      10 non-null     int64 
 1   s2      10 non-null     int64 
 2   s3      10 non-null     object
dtypes: int64(2), object(1)
memory usage: 792.0 bytes

My MCP Configuration

2025-04-27T05:29:54Z

My MCP

The Tools #

Docker #

RAGDocs #

Sequential Thinking #

Git #

Not Tried Yet #

https://github.com/modelcontextprotocol/servers/tree/main/src/sqlite

The Config #


{
  "mcpServers": {
    "docker-mcp": {
        "command": "uvx",
        "args": [
          "docker-mcp"
        ]
      },
    "ragdocs": {
      "command": "node",
      "args": [
        "/home/nic/projects/personal/quadtask/node_modules/@qpd-v/mcp-server-ragdocs/build/index.js"
      ],
      "env": {
        "QDRANT_URL": "http://babyblue-aurora:6333",
        "EMBEDDING_PROVIDER": "ollama",
        "OLLAMA_URL": "http://babyblue-aurora:11434"
      },
      "alwaysAllow": [
        "search_documentation",
        "list_sources",
        "test_ollama",
        "add_documentation"
      ],
      "disabled": false
    },
    "sequentialthinking": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "--name",
        "sequentialthinking",
        "mcp/sequentialthinking"
      ],
      "disabled": false,
      "alwaysAllow": [
        "sequentialthinking"
      ]
    },
    "git": {
      "command": "uvx",
      "args": [
        "mcp-server-git"
      ],
      "disabled": false,
      "alwaysAllow": [
        "git_status",
        "git_commit",
        "git_add",
        "git_log",
        "git_diff"
      ]
    }
  }
}

docker context (and an issue to question your sanity)

2024-12-19T05:59:30Z

docker contexts are great, would recommend putting them in your prompt though (via starship or something else)… here’s why

I like to manage my containers remotely - I have a nice development setup on my desktop and I try to keep my server as bare-bones as possible. For a while I’ve been using ansible which makes it easy to manage configuration etc on other machines. But I recently learned about docker contexts and I’m planning to scale down my homelab management to just docker-compose stacks rather than a bunch of super complicated ansible playbooks

So, setting up a context is easy - it’s basically an ssh connection to another machine!

docker context create koober --docker "host=ssh://nic@koober"

koober is one of my dev machines and my ~/.ssh/config is setup such that I can ssh nic@koober, this makes the context work really seamlessly.

So there’s the default context (the machine you’re on) and now I have koober

To use it you run docker context use koober

And then to check we can ls the contexts

docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT               ERROR
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   
koober *                                             ssh://nic@koober

Notice the * - that indicates that’s our current context.

Trouble #

Now here’s where things get hairy… you’ve gotta be super-aware of what context you’re using. I have an indicator in my starship prompt that shows the current context, but since I’m new to using them I kind of didn’t notice it until I ran into this issue…

I’m working on a python application in docker but was not able to execute the entrypoint even though I KNEW the file was there… let’s take a look

Example #

Here’s a minimal hello world applycation in docker to illustrate the issue

# main.py
print("hello world")

# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Run the main.py script as the container's main process
CMD ["python", "main.py"]

services:
  hello-world:
    build: .
    volumes:
      - .:/app

Notice volume mounting in my project directory . to /app as a common practice to develop inside the container

Here’s where I started to question my sanity…

✗ docker compose up                                            
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | python: can't open file '/app/main.py': [Errno 2] No such file or directory
hello-world-1 exited with code 2

python can't open file? hmm… Let’s take a look at the image

First let’s make sure we get the image name right

✗ docker container ls -a              
CONTAINER ID   IMAGE                                           COMMAND                  CREATED          STATUS                       PORTS                                                                                  NAMES
3d7285fc39e2   docker-context-example-hello-world              "python main.py"         10 minutes ago   Exited (2) 44 seconds ago                                                                                           docker-context-example-hello-world-1

Now we can docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world

❯ docker run --rm -it --entrypoint /bin/bash --name debug docker-context-example-hello-world  
root@ee46d0e22de8:/app# python main.py
Hello, World!
root@ee46d0e22de8:/app#

WHAT THE HECK??

What happened… #

What happened turns out to be pretty simple once we realize I’m using contexts…

koboer is a remote context, the docker run and docker compose up commands are interacting with the docker socket on that machine.

So if I compose up the stack notice that there’s a volume bind mount in there - well those do not work with contexts (or at least I’m not aware of hose to make it work) and so the /app directory was getting blown away essentially with an empty overlay…

But when running with just docker run with no volume mount, the code was copied in during the build and is right where we expect it…

Let’s prove it #

❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@903f591c0384:/app# python main.py
python: can't open file '/app/main.py': [Errno 2] No such file or directory
root@903f591c0384:/app#

Adding a `-v .:/app` to match the compose file, we get the same error...

If we switch to the default context we are back up and running as expected

✗ docker context use default
default
Current context is now "default"

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker compose up         
[+] Running 1/0
 ✔ Container docker-context-example-hello-world-1  Created                                                                                                                                                                                                                                                              0.0s 
Attaching to hello-world-1
hello-world-1  | Hello, World!
hello-world-1 exited with code 0

nic in /tmp/docker-context-example  via   v3.13.0  (dev) 󰒄 󱔎 NO PYTHON ENVIORNMENT SET 
❯ docker run --rm -it --entrypoint /bin/bash --name debug -v .:/app docker-context-example-hello-world
root@4045b6aa8883:/app# python main.py
Hello, World!
root@4045b6aa8883:/app#

Successful runs on both accounts with the volume mount

TLDR #

Context is king

Add a healthcheck to your FastAPI app

2024-12-15T15:41:36Z

I’m building a few FastAPI apps to throw in docker and run on my homelab… I wanted to add healthchecks and here’s a simple way to do it

Make sure to install curl in the dockerfile (near the top for effeciency)

# Install curl with minimal dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Then I recommend making compose files even for single image deployments

services:
  app:
    build: .
    volumes:
      - type: bind
        source: .
        target: /app
    environment:
      - PYTHONPATH=/app
      - DOCKER_ENV=true
      - UV_VIRTUALENV=/opt/app-env
    user: "1000:1000"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

Then finally you’ll need a /health endpoint


@app.get("/health", response_class=HTMLResponse)
async def health_check():
    """
    A health check endpoint that returns a status message.
    """
    return "<html><body><h1>Service is healthy</h1></body></html>"

Switching from AltaCV to RenderCV for my Resume

2024-08-01T05:59:46Z

I was using a fun LaTex-based project for managing my resume called AltaCV. I loved the customization and was familiar with Tek from school. However, I update my resume so infrequently that anytime I’d hop back to it I’d have to remember how to work with Tex and that was frustrating as I’ve lost touch with it over the years.

Scrolling GitHub treding repos I saw RenderCV which let’s me just use YAML to write my resume and then it compiles to Tek through Python. There’s a sister project to make your own using this very easly call rendercv-pipeline. I forked that repo and translated my tek resume to the YAML. The included theme is nice enough is YAML is much easier to maintain long-term.

My resume is behind a private GH repo but the example from rendercv-pipeline is here on GitHub

Modal Labs

2022-12-28T21:01:52Z

Playing around with Modal Labs

One of the first things I tried was a regular cron job…

@stub.function(
    schedule=modal.Period(minutes=59), secret=modal.Secret.from_name("my-dummy-secret")
)
def say_hi():
    now = time.ctime()
    secret = os.environ.get("dummy-secret")
    print(f"Hello {os.environ.get('USER', 'Rodney')} at {now}")
    print(f"{secret=}")

This can get deployed with modal deploy --name <app name> <path to .py file with the stub and function defined in it>

This function gets deployed as an app that I conveniently call say_hi (as far as I can tell the app name can be anything - as I add functions to this same app and deploy with the same name to get a new version)

Notice that this also is an example of giving access to a secret - defined in the Modal Labs dashboard

We can take a look at the apps running at https://modal.com/apps

I then added another function to experiment with custom container images and saw then that Modal will just slap a new version on anything provisioned with the same name (intuitive enough for sure) so when I add functions to my .py script and run modal deploy --name say_hi myscript.py over and over, the app called say_hi in the Modal apps dashboard just gets a new version

This means I can spin up several instances of functionally the same app but with different names/versions etc… Q: Maybe there’s gitops or policy stuff builtin to app names then?

I needed to take down an app I deployed as a duplicate but you don’t stop apps by name, you stop them by an id… see below


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
✗ modal app stop --help

 Usage: modal app stop [OPTIONS] APP_ID

 Stop an app.

╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    app_id      TEXT  [default: None] [required]                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ App ID                    ┃ Description         ┃ State    ┃ Creation time             ┃ Stop time                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ap-lzy1AAuVy7POFkUcDKRxpQ │ print_info          │ deployed │ 2022-12-28 20:59:07-06:00 │                           │
│ ap-qYjE45dciqgT3C3CpNp3RL │ say_hi              │ deployed │ 2022-12-28 19:49:22-06:00 │                           │
│ ap-X7FYneUeYV5IKHcyirSb87 │ link-scraper        │ stopped  │ 2022-12-28 15:39:02-06:00 │ 2022-12-28 15:39:04-06:00 │
│ ap-UOXTUU4uSRx2UZypJOcAsk │ example-get-started │ stopped  │ 2022-12-28 15:17:47-06:00 │ 2022-12-28 15:17:49-06:00 │
└───────────────────────────┴─────────────────────┴──────────┴───────────────────────────┴───────────────────────────┘

modal-sandbox/modal_sandbox   main   ×1  ×9 via   v3.10.6(modal-sandbox)
❯ modal app stop ap-lzy1AAuVy7POFkUcDKRxpQ

Git warning!

I ran modal deploy ... after comitting some stuff I wanted to try BUT I had changes in my file I didn’t want to deploy… some git safety would be nice for deployment!

git stash && modal deploy .. && git stash pop

Question for Modal team - in my modal sandbox repo at commit:

aab6162 (HEAD -> main) HEAD@{1}: commit: print base version of my own image to prove it to me
 1 file changed, 2 insertions(+)

An environment variable, BASE_VERSION that I expect to be in my base image was not available to the python function in my Modal app… hopefully the log is still here

GitHub Actions #

I think it’d be easy to deploy from GH actions… just need an API Key…

Deployment

Without gitops it’s already ahrd to manage my versions… need to get that built in either to github or locally sooooon

Nomad?? :D

Call basicConfig to get Python log messages in iPython

2022-12-10T14:04:23Z

Logging instead of printing

I am trying to adopt logger.debug instead of print but ran into a confusing thing in ipython during Advent of Code… I riddled by script with logger.debug (yes after setting logging.setLevel('DEBUG')) but in ipython none of my log messages showed up!

import logging

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Turns out what I was missing was a call to basicConfig

import logging

# forget this and your messages are in the ether! or at least not seen in ipython...
logging.basicConfig()

logger = logging.getLogger(__name__)
logger.setLevel("DEBUG")

Bonus

Want your new messages to show up while iterating on something without killing the ipython kernel?

from importlib import reload
reload(logging) # to make sure you get new log messages you add while developing!

Benchmark your disks with fio

2022-08-27T13:43:22Z

Intro

I use ZFS at home in my homelab for basically all of my storage… Docker uses ZFS backend, all my VMs have their .qcow2 images in their own zfs datasets, and all my shares are ZFS datasets. I love ZFS but my home hardware presently is the opposite of expensive or new… Thankfully I’ve had a lot of my orginal homelab simply given to me but the cost of this is that I didn’t put my machines together, I didn’t choose the disks, and I definitely didn’t do the research I would’ve otherwise done had I bankrolled my server personally…

The Problem #

I run glances on basically all my machines and for the longest time I have been seeing big time iowait issues. Now, since everything was free I’ve largely been able to ignore that however I’m now after some better performance which I think means new hardware!

Here is a random screenshot of my glances homepage at time of writing - The only major load on my server is some ffmpeg transcoding (about 60% CPU utilization)…

As you can see… there’s a lot of issues and I don’t even know what they mean.

fio

I heard about fio through a friend and decided to try it out quick. It installs with apt on ubuntu quick and easy…

Jim Saltar has a good blog post on it here

Basically it’s a handy tool for benchmarking your disks and the blog dives into what types of metrics matter - it’s not just throughput, but also latency, iops, etc.

Tests #

I ran a few basic commands inside a new zfs dataset on my server tank/fio

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1 > single-4KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1 > 16-parallel-64KiB-random-write.txt
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1 > single-1MiB-random-write.txt

Results #

The single 4 KiB random write:

WRITE: bw=7836KiB/s (8024kB/s), 7836KiB/s-7836KiB/s (8024kB/s-8024kB/s), io=523MiB (548MB), run=68317-68317msec

The 16 parallel 64KiB random writes:

WRITE: bw=93.9MiB/s (98.4MB/s), 5599KiB/s-6303KiB/s (5734kB/s-6454kB/s), io=7642MiB (8013MB), run=81310-81418msec

The single 1MiB random write:

WRITE: bw=81.2MiB/s (85.1MB/s), 81.2MiB/s-81.2MiB/s (85.1MB/s-85.1MB/s), io=8177MiB (8574MB), run=100699-100699msec

Summary

So I don’t fully understand these numbers yet… 80-100 MiB/s isn’t super fast and that’s across a parallelized workload… The single threaded workloads have awful performance so this tells me something is wrong… I have a few ideas…

ZFS dataset config options such as ashift or the blocksize might be way misconfigured
The disks/pool which came from a TrueNAS/FreeBSD machine may have some artifacts that I need to clean up
The SAS controller I am using, which I flashed with IT firmware to get it into JBOD mode might be messed up
The data cables themselves could be a problem…

Points 3 and 4 are less likely given that the write speed does increase in the parallelized job but I’m a newbie so it’s time to dive in!

Playing with mdformat

2022-08-23T06:56:15Z

import this; print(this); print("what is taking so long black!!")

Add colored indicators to your dataframes html representation

2022-06-04T06:12:33Z

Mike Driscoll recently tweeted about making colored out with pandas DataFrames and I just had to try it for myself

Use Case

First though… why? My biggest use case is a monitoring pipeline of mine… The details aside, the output of my pipeline is a dataframe where each row has information about a failed pipeline that I need to go look into. I dump that result to a simle html file that’s hosted on an internal site and the file is updated every couple of hours. Adding some colored indicators automatically to the rows to help me assess severity of each record would be a handy way to quickly get an understanding the state of our pipelines.

How?

The docs for the applymap method state simply:

Apply a CSS-styling function elementwise.

Updates the HTML representation with the result.

So we can write a function that returns color: {color} based on the dataframe values and when we drop that dataframe to html we’ll have some simple css styling applied automagically!

By default the function will be applied to all columns of the dataframe, but that’s not useful if the columns are different types which is usually the case. Luckily there is a subset keyword to only apply to the columns you need!

Consider my example

sandbox   main via 3.8.11(sandbox) ipython
❯ df = pd.read_csv("cars.csv")

sandbox   main via 3.8.11(sandbox) ipython
❯ def mpg_color(val: float):
...:     color = "red" if val < 21 else "green"
...:     return f"color: {color}"

sandbox   main via 3.8.11(sandbox) ipython
❯ df.style.applymap(mpg_color, subset="mpg").to_html("color.html")

I want to quickly see if the mpg is any good for the cars in the cars dataset and I’ll define “good” as better than 21 mpg (not great I know but just for the sake of discussion…)

The function returns an appropriate css string and after I style.applymap on just the mpg column we get this!

	Unnamed: 0	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
0	Mazda RX4	21.000000	6	160.000000	110	3.900000	2.620000	16.460000	0	1	4	4
1	Mazda RX4 Wag	21.000000	6	160.000000	110	3.900000	2.875000	17.020000	0	1	4	4
2	Datsun 710	22.800000	4	108.000000	93	3.850000	2.320000	18.610000	1	1	4	1
3	Hornet 4 Drive	21.400000	6	258.000000	110	3.080000	3.215000	19.440000	1	0	3	1
4	Hornet Sportabout	18.700000	8	360.000000	175	3.150000	3.440000	17.020000	0	0	3	2

Subset a list based on values in another list with itertools.compress

2022-05-19T14:22:58Z

I have list [True, False, False, True] and another list [1, 2, 3, 4] and a use case where I want to filter list 2 based on list 1 to remove values that line up with the element False in list 1…. so the outcome will be [1, 4]. list(compress(list2, list1)) will do it. As long as you can create a mask for the filter than itertool.compress will be your friend!

Forms with FastAPI and Jinja

2022-05-15T00:00:00Z

I just started using FastAPI for a home project and needed to pass back a dynamic number of values from a form rendered with jinja…

Dynamic Values

The jinja templating for rendering HTML based on something like a python iterable is nice and easy

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). it?

The way to pass back all those values is pretty straight forward (after hours of messing around that is!)

# I hate it when tutorials don't show ALL relevant pieces to the blurb
import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

Dynamic-Form-Values-With-Jinja-And-Fastapi

2022-05-15T00:00:00Z

I’m currently working on a self-hostable wish list app using FastAPI so we can finally drop Amazon forever. (The lists funcionality has been super handy for sharing holiday gift ideas with the famj!)

FastAPI

FastAPI is an amazing framework for quickly building APIs with Python. I will have a slightly longer post about my brief experience with it coming later…

Jinja, Forms, and FastAPI

One of the last things I needed to figure out in my app was how to generate a form in a Jinja template with a dynamic number of inputs and then pass all the inputs to the backend to perform a database operation (my exact case was removing rows from a table).

Explicit Values #

The way to pass back explicit variables is really easy…

Our form would look like this (I’m using bootstrap CSS)

<form method="post">
    <div class="form-check ">
        <input class="form-check-input"  name="item_1" id="itemOne" value="1" type="checkbox">
        <label class="form-check-label" for="itemOne" > A label for this item </label>
    </div>
    <div class="form-check ">
        <input class="form-check-input"  name="item_2" id="itemTwo" value="2" type="checkbox">
        <label class="form-check-label" for="itemTwo" > A label for item 2 </label>
    </div>

<button type="submit" class="submit btn btn-xl" >Submit</button>
</form>

So what is this? This form will have 2 rows with the lables you see in <label> </label> and checkboxes that when checked would have the value value in each <input> line.

So our backend might looks something like this…

I’m keeping all the imports and stuff here to show where they come from but I won’t discuss it all here - that’ll be in a future post

import starlette.status as status
from fastapi import APIRouter, Depends, Form, Request
from fastapi.encoders import jsonable_encoder
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session

from app.session.session import create_get_session

router = APIRouter()
templates = Jinja2Templates(directory="templates/")

@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    item_1: int = Form(...),
    item_2: int = Form(...)
    db: Session = Depends(create_get_session),
):
    print(item_1)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    print(item_2)  # will just print 1 to the console where fastapi is running if the checkbox was checked
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

Dynamic values #

That’s all pretty simple… pass back values by the name in the form…

What about a form that’s generated dynamically? This is my case since I display a row/checkbox for every row in my table so my form looks like this…

data is the result of a database query, and item is each row, so the dot notation is the value of each column basically in that row

<form method="post">
  {% for item in data %}
    <div class="form-check ">
        <input class="form-check-input"  name="item_{{ item.id }}" id="{{ item.name }}" value="{{ item.id }}" type="checkbox">
        <label class="form-check-label" for="{{ item.id }}" > Label for: {{ item.name }} </label>
    </div>
  {% endfor %}

<button type="submit" class="submit btn btn-xl btn-outline-danger" >Remove</button>
</form>

This form generates a row with a checkbox for every item in data (in my case each item is an existing row in my table). Now I started scratching my head on how to pass an unknown number of inputs to my backend of FastAPI wants each input explicitly defined and typed… I can’t just pass the form back becuase that’s not a thing so what’s the way to do it?

# same stuff as above, only showing post method here
@router.post("/my_route/do_something_with_form", response_class=HTMLResponse)
async def delete_rows(
    request: Request,
    db: Session = Depends(create_get_session),
):
    form_data = await request.get_form()
    data = jsonable_encoder(form_data)
    # data = {"item_1": 1, "item_2": 2, ... "item_N": N}
    return RedirectResponse("/", status_code=status.HTTP_302_FOUND)

We await request.get_form() and after encoding the data we get a dictionary with key/value pairs of the name/value from the form!

This took me quite a long time to figure out in part because most of the Google-able resources are still on Flask…

I look forward to my wish list app maturing and I hope this helps someone working with FastAPI!