Skip to content

Deployment Guide

How a project gets onto rethink-net — our Ubuntu 26.x servers. Get your container to follow a few consistent rules and deploying is mostly handing us a compose.yaml.

What can run here

APIs, websites, applets, bots, monitors.

Preparing for Deploy

You build and test locally — leaning on our libraries, AI, and this handbook — then push to git. From there the ops tooling takes over: staging (if you run gitflow), then it builds the Docker image and runs it on the fleet. You don't run any deploy commands — your only job is the three pieces in the tabs below.

flowchart LR subgraph build ["you build"] direction TB local["local testing"] libs["libs<br/>(rethink-public, pip)"] ai["AI-assisted<br/>(Claude Code)"] docs["reading the<br/>handbook"] local --- libs libs --- ai ai --- docs docs --- local end build -->|push| git["git<br/>(Gitea)"] git -.->|develop, if gitflow| stage["staging"] git --> deploy["deployment<br/>(ops tooling)"] stage --> deploy deploy --> docker["docker<br/>(built + run<br/>on the fleet)"] classDef ship fill:#061541,stroke:#569bcc,color:#eef1f6; classDef work fill:#0e1530,stroke:#294274,color:#eef1f6; class git,stage,deploy,docker ship; class local,libs,ai,docs work;

Your compose names nothing repo-specific

No container_name, no hardcoded names/paths. The deploy layer injects identity and host paths so your compose can't collide with any other service on the fleet:

  • the service key is always svc — it never changes, in any repo
  • named volumes use bare names (cache, not myapp_cache)
  • host paths come from ${LOGS_DIR} / ${CONFIG_DIR}

The compose.yaml your repo ships. Copy it verbatim — it names nothing repo-specific, so the deploy layer can inject identity and host paths without collisions.

services:
  svc:                                     # generic service key — ALWAYS svc, no container_name
    build: .
    user: "1337:1337"                      # the shared services account (required)
    restart: unless-stopped                # required — host unit is oneshot; this recovers crashes
    environment:
      HOME: /tmp
    volumes:
      - ${LOGS_DIR:-./logs}:/app/logs        # output   — host log dir, injected at deploy
      - ${CONFIG_DIR:-./config}:/app/config  # input     — host config dir, injected at deploy
      # - ${MOUNTS_DIR:-./mounts}:/app/data  # optional  — arbitrary host data (opt-in)
      - cache:/app/cache                     # your data — ephemeral named volume

volumes:
  cache:                                   # bare name — deploy auto-prefixes it per service

The container path is where your code reads and writes

WORKDIR is /app, so the right-hand side of each volume line is the path your code targets. Write to ./cache (i.e. /app/cache) → the cache volume; logs go to /app/logs, config is read from /app/config.

Storage, three tiers:

  • logs + config — we inject and manage these at deploy time.
  • mounts (optional) — host dir for arbitrary data. We inject MOUNTS_DIR and create the dir; you uncomment the line and pick the container path.
  • Named volumes (cache, …) — yours; Docker owns them, no host paths to manage.

Two kinds of logs — and crashes go to the other one

${LOGS_DIR} holds only what your app writes to disk through its logger (e.g. log_setup writing a file) — your own structured logging. It does not capture the process's stdout/stderr, and that's where startup crashes and uncaught exceptions land — a traceback from a failed import or a missing file never reaches your logger. So if a service dies on startup, or you don't see the error in your log files, it's in the process output, not ${LOGS_DIR}. Make fatal errors visible — and to route uncaught exceptions into your log file too, install a top-level hook:

import sys, logging
sys.excepthook = lambda *exc: logging.getLogger().critical("uncaught", exc_info=exc)

Same file, both places

Locally, docker compose up needs nothing set — the ${VAR:-./default} fallbacks use ./logs / ./config. When deployed, the ops tooling sets the real values. One compose.yaml works everywhere.

Subprocess and browser workloads (bots that spawn Chrome, Xvfb, ffmpeg) need three extra knobs:

services:
  svc:
    build: .
    user: "1337:1337"
    restart: unless-stopped
    init: true              # tini as PID 1 — reaps zombie subprocesses, forwards signals
    shm_size: "2gb"         # Chrome/headless browsers crash on Docker's default 64 MB /dev/shm
    mem_limit: "4g"         # bound memory — raise as worker/browser count grows

The PID-1 gotcha with shell-wrapper CMDs

If your CMD is a shell-script wrapper (e.g. xvfb-run ...), it must not be PID 1, or the real process dies on startup. init: true fixes this — tini takes PID 1, your wrapper runs as a normal child.

Every service runs containerized as the shared services account: uid/gid 1337, fixed fleet-wide. Build the image to be uid-agnostic so it runs cleanly as that account.

FROM python:3.12-slim
ENV HOME=/tmp                                  # services account has no home dir; $HOME must be writable

WORKDIR /app
RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*             # include git ONLY if the container itself needs it

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # deps before code — layer caching

COPY . .
RUN chmod -R a+rwX /app                         # writable by any uid — this is "uid-agnostic"
CMD ["python", "-m", "yourapp"]

Layer caching: copy the deps file and install before COPY . . — Docker caches layers in order, so deps only reinstall when the deps file changes, not on every code edit.

Getting files in: COPY . . grabs the whole repo; be explicit about anything that needs its own place. In COPY <src> <dest>, <src> is relative to the build context (your repo), <dest> is a path in the image.

COPY ./config.toml /app/config.toml   # a single file into a specific path
COPY ./assets /app/assets             # a whole directory (tree mirrors ./assets/)

Faster builds with uv (optional)

uv is a drop-in for pip that reads the same pyproject.toml — no lockfile needed in the image. Add UV_COMPILE_BYTECODE so containers don't pay the first-import .pyc compile cost:

FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/   # pull the uv binary from its image
ENV HOME=/tmp
ENV UV_COMPILE_BYTECODE=1                                # compile bytecode at build, not cold start

WORKDIR /app
COPY pyproject.toml .
RUN uv pip install --system .                            # into the image's Python, no venv/lock
COPY . .
RUN chmod -R a+rwX /app
CMD ["python", "-m", "yourapp"]

Secrets never go in the image

We do not commit secrets (usually, lol). They stay gitignored and live on the host in your config dir, reaching your container read-only via ${CONFIG_DIR}. Add them to .dockerignore so a COPY . . can't sweep them into a layer.

  • Secrets live on the host, in your config dir — never in git, never in the image.
  • They reach the container read-only via the injected ${CONFIG_DIR} mount.
  • Keep them out of the build context: list them in .dockerignore so a blanket COPY . . can't pull them into a layer.

Rotating a secret is a host-side edit — update the file and the service picks it up on restart. No rebuild, and nothing you run: flag it and we handle the restart.

Supporting services live in your compose, not the fleet

Need Redis? Declare it in your own compose.yaml, as a sidecar. There is no shared Redis — nothing fleet-wide, nothing per-workspace, nothing ops provisions for you. A repo that needs Redis brings its own; a repo that doesn't adds nothing.

That's the whole point of the deploy model. Every service already runs in its own compose project on its own network so that one service falling over can't touch another. A shared Redis puts that coupling straight back: one process everything depends on, whose OOM, stray FLUSHALL, single-threaded stall, or restart becomes everyone's outage. A sidecar shares its owning repo's fate and nobody else's — and the isolation is free, because it rides the per-project network you already get. No ACLs, no key-prefix discipline, no shared credentials to manage.

Your app talks to it with the redis lib from the suite (async, config-free, kv/hash/ttl/pubsub), pointed at redis://redis:6379 — the compose service name, not a host port. The sidecar comes up auto-namespaced on your project's network like every other container, exactly as the naming convention above describes.

Ephemeral or persistent — pick deliberately

A sidecar Redis is ephemeral by default: restart it and the data is gone. That's correct for some workloads and quietly destructive for others, so make the call on purpose. Ask one question — if this data vanished on a restart, would anything be lost?

  • No → ephemeral. A scratch cache, a dedupe set, rate-limit counters, transient data you can just re-fetch. Nothing to back up, nothing to grow.
  • Yes → persistent. An outbound webhook or notification queue, a job queue, anything that could be mid-flight when the process dies. Losing it drops real work.

Fine to lose on restart — no volume, no persistence, by design.

redis:
  image: redis:7-alpine
  restart: unless-stopped
  command: redis-server --save "" --maxmemory 256mb --maxmemory-policy allkeys-lru
  # no volume: throwaway by design

Survives restart, rebuild, and reboot — the append-only file lives on the host mounts dir injected at deploy, the same ${MOUNTS_DIR} mechanism described above. Redis just uses it as its backing store.

redis:
  image: redis:7-alpine
  restart: unless-stopped
  command: redis-server --appendonly yes --appendfsync everysec
  volumes:
    - ${MOUNTS_DIR:-./mounts}/redis:/data   # AOF persists on the host mounts dir

The app connects the same way in both modes — REDIS_URL: redis://redis:6379. Only the durability changes.

Two things to know before you rely on a persistent sidecar

--appendfsync everysec can lose ~1 second of the newest entries on a hard crash. For a webhook queue that's an acceptable trade — just know it's there. Use --appendfsync always if you genuinely cannot drop a single entry (safer, slower).

Data under the mounts dir is not backed up. Mounts are excluded from the backup pipeline, and that's right for a queue: a lost queue means some notifications didn't fire, not that business data is gone. That's the dividing line. If losing this data would actually hurt, it isn't queue or cache state — it's a system of record, and it belongs in Postgres (which is backed up), not a local mount.

The rules

Don't do these

  • Don't map Redis to a host port. No ports: - "6379:6379". Two repos both grabbing host 6379 on the same box collide. Keep it internal to the compose network — nothing exposed, nothing to collide.
  • Don't stand up a shared or fleet-wide Redis. Per-repo means per-need. One Redis per project, shared by that project's containers if a repo runs several — never one per fleet.
  • Don't treat persistent Redis as a database. Queues and caches, yes. A durable system of record, no — that's Postgres, and unlike a mount it's backed up.

You can't reach another repo's Redis — different project, different network. That's not a restriction you have to work around; it's the isolation working for you. Nobody else's service can touch your cache or drain your queue either, and you never have to think about whose keys are whose.

What about ACLs?

A shared Redis can be secured — Redis 6+ ACLs scope users by command, key pattern, and channel. But ACLs don't solve resource contention or the noisy-neighbour problem, and they add real management burden, so the fleet uses per-repo sidecars instead. Reserve ACLs for the rare case of a deliberately shared, durable, backed-up Redis run as actual infrastructure.