Posts tagged: litestream

Using Litestream to Restore My Database for Easy Development

2025-08-07T15:14:45Z

Litestream

see Using Litestream to Backup QuadTask's SQLite DB for how I setup litestream replication for QuadTask

I have the entrypoint to my app container check if the sqlite database exists or not - in production or with local volume mounts, it will and then we start litestream replication and then the app like normal. But what’s awesome about this little trick I got from this litestream example is I can docker compose up locally without the data directory volume mounted in, and get a fresh copy of my production data right there locally to work with!

My entrypoint script now looks like:

#!/bin/bash

# Exit on error, undefined vars, and fail on pipe errors
set -euo pipefail
# Restore the database if it does not already exist.
if [ -f /app/data/quadtask.db ]; then
 echo "Database already exists, skipping restore"
else
 echo "No database found, restoring from replica if exists"
 litestream restore -v /app/data/quadtask.db
fi
# Run litestream with your app as the subprocess.
exec litestream replicate -exec "uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug"

Appropriate and Safe Replication #

The way to get prod data in automatically is to restore from the production backup but I don’t want to replicate to the production backup with my local work, so it’s not the best but I have a manual step that makes it pretty easy

I first just sync-db

# Sync production database locally
sync-db:
    fly ssh console -a quadtask -C 'cat /app/data/quadtask.db' > ./litestream/source/quadtask-prod.db

and this brings my database down to a folder I maintain locally.

I have litestream installed on my desktop obviously, and here’s the relevant litestream configuration I use:

dbs:
  - path: /home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db
    replicas:
      - name: quadtask-prod
        type: s3
        bucket: litestream
        path: quadtask-local
        endpoint: s3.example.com
        skip-verify: false
        region: us-east-1

So on my desktop if I litestream replicate then basically I’m taking the output of just synd-db and replicating to my litestream bucket to the quadtask-local path.

Then I start up my app, it restores from that point and I’m working with fresh prod data!

How To Handle Environments? #

I have this working for different environments too, with a little bit of process but basically thanks to litestream expanding env vars in the config we can do something like this:

dbs:
  - path: /app/data/quadtask.db
    replicas:
      - name: quadtask
        type: s3
        bucket: litestream
        path: quadtask-${ENVIRONMENT}
        endpoint: s3.example.com
        skip-verify: false
        region: us-east-1

And so wherever I spin up quadtask: local, dev, or prod - the replication will happen to and from a different path in my litestream bucket!

Update

I updated the docker-entrypoint.sh of my quadtask container to look like this now:


# Restore the database if it does not already exist.
if [ -f /app/data/quadtask.db ]; then
 echo "Database already exists, skipping restore"
else
 echo "No database found, restoring from replica"
 ENVIRONMENT="prod" litestream restore -v /app/data/quadtask.db
fi
# Run litestream with your app as the subprocess.
exec litestream replicate -exec "uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug"

The if block checks if there’s a db at the filepath it should be in in the continer - if it’s not there is the little bit of magic to me right now…

My litestream config has a parameterized ENVIRONMENT variable in the db path, so by overwritting it for just the restore command, I can very simply always pull a fresh copy of the prod database into an ephemeral container to test things against real data in a very convenient way

Using Litestream to Backup QuadTask’s SQLite DB

2025-08-05T19:34:19Z

Litestream

This post will be a walkthrough of installing Litestream in a docker container that runs my FastAPI app in order to replicate the sqlite db in the data volume currently hosted in Fly’s volume manager to my MinIO instance. From there, on my development machine I can use litestream to restore the prod database to my dev environment to practice whatever I may need to practice with real production data!

Dockerfile #

Installing litestream in the dockerfile is 2-3 lines of code - no sweat. Configuration file is further down the post.

FROM python:3.11-slim
WORKDIR /app

# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

# Install litestream
ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.8/litestream-v0.3.8-linux-amd64-static.tar.gz /tmp/litestream.tar.gz
RUN tar -C /usr/local/bin -xzf /tmp/litestream.tar.gz
COPY etc/litestream.yml /etc/litestream.yml

# Install dependencies
COPY requirements.txt .
RUN uv pip install --no-cache -r requirements.txt

# Copy the rest of the application
COPY . .

# Install the app package in development mode
RUN uv pip install -e .

# Run the application
CMD ["docker-entrypoint.sh"]

Litestream Configuration #

Taking care to set ENVIRONMENT appropriately in each environment, the litestream configuration gets that env var templated out to the path for the litestream target. See docs

#/etc/litestream.yml
dbs:
- path: /app/data/quadtask.db
  replicas:
  - name: quadtask
    type: s3
    bucket: litestream
    path: quadtask-${ENVIRONMENT}
    endpoint: s3.example.com
    skip-verify: false
    region: us-east-1

Authentication #

In your docker container you’ll need to set LITESTREAM_ACCESS_KEY_ID and LITESTREAM_SECRET_ACCESS_KEY which you setup in Minio - see Setup Remote MinIO S3 Backend Target for Litestream. I have these set as secrets in my fly configuration, and locally in a .env file that gets sourced with my compose stack

Example Log from Dev #

My fly dev machine has this log message showing the ENVIRONMENT var dev gets expanded into the config so the s3 path includes quadtask-dev

replicating to: name="quadtask" type="s3" bucket="litestream" path="quadtask-dev" region="us-east-1" endpoint="s3.<redacted>.com" sync-interval=1s

Example Usage for Local Development #

Using Litestream to Restore My Database for Easy Development

Setup Remote MinIO S3 Backend Target for Litestream

2025-08-05T08:39:16Z

Intro #

I am starting to think through some patterns for replicating sqlite databases, exploring them, standardizing on schemas/models I use across projects, etc… Thanks to Waylon I know about Litestream and finally I started playing with it. Their quickstart is good, but it assumes you’ll run MinIO on the same machine you run litestream on.

I, however, have MinIO setup in my homelab with the api endpoint accessible via DNS - s3.mydomain.com.

Before starting - I realize I left the credential aspect out - Litestream explains how to setup API keys in MinIO for their application here

Basically after generating the keys I set LITESTREAM_ACCESS_KEY_ID and LITESTREAM_SECRET_ACCESS_KEY as env vars

So let’s see what happens and then get to [the fix]

⬢ [devbox] ❯ litestream replicate quadtask-prod.db s3://s3.example.com/litestream/qt.db    
2025/08/05 08:50:09 INFO litestream version=v0.3.13
2025/08/05 08:50:09 INFO initialized db path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db
2025/08/05 08:50:09 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=s3.example.com path=litestream/qt.db region="" endpoint=""
2025/08/05 08:50:10 INFO sync: new generation db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db generation=209692854ccd85d5 reason="no generation exists"
2025/08/05 08:50:10 ERROR monitor error db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 error="cannot lookup bucket region: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: ERHZD4RD61Y8ADMD, host id: dOjPn5j8Oa+mkwOqROXDvCdKNxzxG0B8JJOPtGXDZA7qKxwAWqaSXhGYzK2mkoJQV7hJhaoJcag="

So that 403 is pretty simple… my login is wrong… but wait…

2025/08/05 08:50:09 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=s3.example.com path=litestream/qt.db region="" endpoint=""

Notice this line - endpoint=""… so s3://s3.example.com was not the right replacement for s3://mybkt.localhost:9000/target.db. Also the inferred bucket is the entire s3.example.com…

Oh! Well I didn’t specify a bucket in the subdomain - I did it with a route on the URL…

ok let’s try litestream replicate quadtask-prod.db s3://litestream.example.com/qt.db

We got the same kind of error…

2025/08/05 08:53:45 INFO replicating to name=s3 type=s3 sync-interval=1s bucket=litestream.example.com path=qt.db region="" endpoint=""

This time the inferred bucket was litestream.example.com… ok that’s consistent with above as well - so the endpoint is still wrong/null and the bucket inference is wrong…

Turns out for Litestream, and probably many other tools that use s3-compatible storage, that there’s an inference problem if the configuration leans on the shorthand s3:// - see this GitHub issue and this one about localhost. There’s a few options for using your own s3.mydomain.example.

The Fix #

If you’re familiar with the awscli then maybe you’ve heard of a config option AWS_S3_ENDPOINT_URL which allows you to use the boto sdk with s3-compatibile resources while still using the s3:// shorthand. Litestream has a way to set this configuration right on config:

dbs:
  - path: /home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db
    replicas:
      - type: s3
        bucket: litestream
        endpoint: s3.example.com  # <---- right here!
        skip-verify: true  # set to false if you have TLS and want more security
        # note you acn set these sensitive values as env vars per the Litestream Quickstart
        # access-key-id: <ACCESS-KEY-ID>
        # secret-access-key: <SECRET-ACCESS-KEY>

UPDATE

I added a path key to give the backups a folder basically on s3, so under “bucket: listream” I have “path: ”

So with that config in place we can actually JUST litestream replicate from anywhere…

⬢ [devbox] ❯ litestream replicate                                                      
time=2025-08-05T08:56:11.353-05:00 level=INFO msg=litestream version=v0.3.13
time=2025-08-05T08:56:11.353-05:00 level=INFO msg="initialized db" path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db
time=2025-08-05T08:56:11.353-05:00 level=INFO msg="replicating to" name=s3 type=s3 sync-interval=1s bucket=litestream path="" region="" endpoint=s3.example.com
time=2025-08-05T08:56:12.399-05:00 level=INFO msg="write snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:4152
time=2025-08-05T08:56:12.489-05:00 level=INFO msg="snapshot written" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:4152 elapsed=89.994626ms sz=909437
time=2025-08-05T08:56:12.492-05:00 level=INFO msg="write wal segment" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:0
time=2025-08-05T08:56:12.510-05:00 level=INFO msg="wal segment written" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=s3 position=209692854ccd85d5/00000000:0 elapsed=17.572784ms sz=4152

It works!

Notice that the endpoint is properly s3.example.com, and the bucket is litestream (the name of my taret bucket in MinIO)

You can use a local config with -c config.yaml or something if you want to avoid the global /etc/litestream.yml

What About Restore? #

Should work just fine… Let’s move the database on my host and see if I can use litestream to restore it

nic in quadtask/litestream   database-clean-up   ×21  ×2  ×9 via   v3.11.10(quadtask)   (dev) 󰒄 
⬢ [devbox] ❯ mv ./source/quadtask-prod.db ./source/quadtask-prod.db.backup                 

nic in quadtask/litestream   database-clean-up   ×21  ×2  ×9 via   v3.11.10(quadtask)   (dev) 󰒄 
⬢ [devbox] ❯ litestream restore ./source/quadtask-prod.db              
time=2025-08-05T09:06:25.826-05:00 level=INFO msg="restoring snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 path=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db.tmp
time=2025-08-05T09:06:25.843-05:00 level=INFO msg="restoring wal files" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index_min=0 index_max=0
time=2025-08-05T09:06:25.845-05:00 level=INFO msg="downloaded wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=2.103514ms
time=2025-08-05T09:06:25.856-05:00 level=INFO msg="applied wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=10.871034ms
time=2025-08-05T09:06:25.856-05:00 level=INFO msg="renaming database from temporary location" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod

The command litestream restore <db path> was confusing at first because I was trying to replicate to a new place, but couldn’t get the config to line up. The general understanding here is DR I believe - so the configuration is relatively static and when you restore you are restoring a specific database, not just standing up a copy from a backup…

But would we setup a copy from a backup?

ie. restore to a different file?

Yes, but it’s confusing - at least it was to me… it looks to me like litestream uses the path of a db as the identifier in the config, and you reference the config not by an object’s name (if it has one) but by the path…

So the command is litestream restore -o <output path> <db path>. This makes sense in light of the fact that the configuration is relative to this host - so <db path> being the path of the source db whose backup you are restoring from to <output path> makes some amount of sense… if I wanted to replicate to another machine, then THAT machine would need /etc/litestream.yaml and if the filesystem hierarchy was different then an appropriate path for the “source” of the db to replicate back to.

⬢ [devbox] ❯ litestream restore -o /tmp/db ./source/quadtask-prod.db          
time=2025-08-05T09:09:04.784-05:00 level=INFO msg="restoring snapshot" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 path=/tmp/db.tmp
time=2025-08-05T09:09:04.797-05:00 level=INFO msg="restoring wal files" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index_min=0 index_max=0
time=2025-08-05T09:09:04.799-05:00 level=INFO msg="downloaded wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=2.128594ms
time=2025-08-05T09:09:04.800-05:00 level=INFO msg="applied wal" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod generation=209692854ccd85d5 index=0 elapsed=610.75µs
time=2025-08-05T09:09:04.800-05:00 level=INFO msg="renaming database from temporary location" db=/home/nic/projects/personal/quadtask/litestream/source/quadtask-prod.db replica=quadtask-prod