Deploy to your own box

Renaming the repository or the GitHub owner breaks the deploy until you re-authorize it. The CI runner authenticates to Tailscale with a GitHub OIDC token, and that token's sub claim embeds the repository's name (repo:<owner>/<name>:...). Renaming the repo, transferring it, or moving it to another owner changes the claim, the existing federated identity no longer matches, and every deploy fails with: `` failed to exchange JWT for access token: 403 Unauthorized Visit https://login.tailscale.com/admin/settings/trust-credentials/view/** ` The failure looks like a broken secret but nothing on the GitHub side changed. Fix it by editing the federated identity's subject in the Tailscale admin console to match the new name, or by creating a new one and updating TS_OAUTH_CLIENT_ID / TS_AUDIENCE. Run make rename` before* wiring the deploy, not after.

Renaming also leaves the server in a state the deploy cannot fix by itself. Two artefacts on the box carry the old name, and both must be cleared by hand once, after the rename: 1. The container. The old container (<old-name>) keeps holding the published port, so the new one (<new-name>) fails to start with Bind for 127.0.0.1:8080 failed: port is already allocated. The deploy script reports that only as a healthcheck timeout, and the run still exits green — so it looks like a success while the previous container keeps serving. Remove it: docker rm -f <old-name>. 2. The secrets file. The compose env_file and the CI render target must agree on ~/.secrets/<app>.env. If they disagree you get env file /home/deploy/.secrets/<app>.env not found, again masked as a timeout. make rename renames the repository; it cannot rename a file on a machine it has never seen. Both failures share a nasty property: the deploy exits 0 and the site stays up on the old container. Verify a deploy actually landed by checking the running container's uptime, not the workflow's conclusion: ``bash ssh $DEPLOY_HOST 'docker inspect <app> --format "{{.State.StartedAt}} {{.State.Health.Status}}"' ` A StartedAt` older than the workflow run means the deploy did not apply.

The default workflow is copy the template + make rename + make dev for local work. For a permanent deployment, the project ships a production deploy workflow that publishes to a server of your choosing — recommended: a small Linux box + Tailscale + a Cloudflare-tunneled domain. No registry, no cold starts, full control.

Server layout (multi-project standard)

Every project that adopts the pattern lives under /home/deploy/services/ and shares the same shape — siblings differ only by name:

/home/deploy/services/
└── gogogo/                ← this project
    ├── bin/
    │   ├── gogogo          ← current binary (chmod 755)
    │   └── gogogo.previous ← prior binary, kept for fast rollback
    ├── compose/
    │   └── docker-compose.prod.yml
    ├── env/
    │   └── .env                     ← non-secret env (DATABASE_URL, APP_URL, ...)
    ├── secrets/
    │   └── gogogo.env   ← mode 600, regenerated every deploy from GH Secrets
    ├── data/                        ← persistent volume, survives restarts
    │   (bind-mounted into the container at /var/lib/<app>/data —
    │   app.db, queue.db, nats/, dagnats/; see deploy/docker-compose.prod.yml)
    ├── repo/                        ← git clone of this repo (source for env/.env + deploy-prod.sh)
    └── scripts/
        └── deploy-prod.sh           ← the on-server deploy runner (mode 700)

/home/deploy/services/<other-project>/   ← siblings follow the same shape

Not /opt/. The deploy user is non-root, and /opt would need root to create. The workflow creates the tree with mkdir -p as deploy (.github/workflows/deploy.yml, "Ensure layout + run deploy"), and scripts/deploy-prod.sh reads APP_DIR="/home/deploy/services/${PROJECT}".

First-time setup on the server

Install Docker and create a deploy user with SSH key access.
Add the box to your Tailscale tailnet.

New to Tailscale? It is a private WireGuard mesh between your machines (install, then tailscale up and approve the machine in the admin console). The deploy needs it so the CI runner reaches the server over the tailnet — no public SSH port, no static deploy credential (auth is a short-lived OIDC JWT per run). Nothing in local development needs it; it becomes required only at this step.

Configure a Cloudflare Tunnel routing your domain (e.g. fullstack.example.com) to the Tailscale hostname on port 8080.
Add the GitHub Actions secrets — see the header comment of .github/workflows/deploy.yml for the full list (SSH host, user, key, Tailscale OIDC client id + audience, project name, and the app's own secrets). You do not create the directory tree by hand; the workflow does it on first deploy.

Secrets stay server-side by design: the workflow renders ~/.secrets/<app>.env (600) from GitHub Secrets on every deploy and nothing secret is ever committed. The template also ships age tooling (bin/init-secrets + internal/secrets) for projects that need encrypted config — deliberately not used on a single box, where the identity would live next to the ciphertext and the gain over 600 files + no-repo + monitoring is ~zero. Revisit if secrets ever need to travel through git (multi-operator GitOps).

Two gotchas that bite on a non-root deploy user: - Grant the container write access with setfacl/chmod, never chown — the deploy user is not root. - Never scp into the server's repo clone — git pull --ff-only aborts on a dirty tree. Use the deploy workflow, which ships the binary and compose file to bin/ and compose/ instead.

Password brute-force throttling lives here, not in the app — the app deliberately tracks no per-email login attempts (that would let anyone lock anyone else out; see internal/auth/ratelimit.go). Rate-limit /login and /api/auth/* at this ingress (Caddy rate_limit or the Cloudflare WAF) before exposing the box.

After setup, every push to `master` deploys

.github/workflows/deploy.yml runs on every push to master and:

Builds the project (Go install, npm deps, CSS, then the binary).
Cross-compiles the production binary in the runner — no Docker image is built there. The step is CGO_ENABLED=1 GOOS=linux GOARCH=arm64 CC=aarch64-linux-gnu-gcc go build -trimpath -ldflags="-s -w -extldflags=-static -X main.Version=…", producing bin/<app>.new for the target architecture (arm64 for the reference box; adjust GOARCH/CC for your host).
Uploads that binary as an artifact, then brings up Tailscale in the runner (OIDC workload identity, scoped to the run).
SCPs the binary to the server as gogogo.new, plus the compose file to compose/docker-compose.prod.yml.
Writes the secrets file (/home/deploy/services/gogogo/secrets/gogogo.env) with mode 600, rendered from GitHub Actions secrets.
SSHes in, ensures the directory tree exists, and runs scripts/deploy-prod.sh, which:
atomically renames gogogo.new → gogogo and keeps the old binary as .previous,
restarts the container with docker compose -f deploy/docker-compose.prod.yml up -d --wait — --wait blocks until the compose healthcheck passes, rather than polling a fixed number of seconds,
the healthcheck itself is CMD ["/app","health"] (interval 30s, timeout 5s, retries 3, start_period 10s): the binary's own health subcommand does an internal GET to /health and exits 0 on 200.
Prints the new container status and the last log lines for confirmation.

The Docker image is built on the server from the build: stanza in deploy/docker-compose.prod.yml — the runner only ships a static binary.

Restarting a stack requires cd <project-dir> && docker compose up -d, never a blanket docker restart <container> — compose recreates networks, volume mounts, and dependencies in order.

Secrets are never stored long-term on the server. Every deploy re-renders the secrets file from GitHub Actions secrets. The file is chmod 600, owned by the deploy user, and overwritten on every run — there is no history of secrets on disk.

Build pipeline and version badge metadata

The compile steps outside go build are the CSS bundles (one per skin) and the Go ldflags that bake the version badge into the binary. All assets are embedded into the binary via //go:embed — there is no runtime CSS build step, no JS runtime, and no CDN.

src/css/input.css          →  tailwindcss v4 CLI  →  web/resources/static/app.min.css        (DaisyUI)
src/css/basecoat-input.css →  tailwindcss v4 CLI  →  web/resources/static/basecoat.min.css  (Basecoat)
                                                              │
                                                        //go:embed in the Go binary

The navbar version badge shows BuildLabel (the git tag) and BuildCommit (the short SHA), both baked in via Go ldflags:

# Local build — the Makefile sets these automatically:
make build
#   VERSION   = git describe --tags --abbrev=0 | sed 's/^v//'   (e.g. 0.29.1; "dev" if no tags)
#   COMMIT    = git rev-parse --short HEAD                    (e.g. b54916e)
#   BUILDTIME = date -u +"%Y-%m-%dT%H:%M:%SZ"
# LDFLAGS = -ldflags="-w -X main.Version=$VERSION -X main.CommitHash=$COMMIT -X main.BuildTime=$BUILDTIME"

The CI deploy cross-compiles with the same ldflags (deploy.yml, "Build prod binary"), so the badge on the running box matches the tag that produced it.

Why the tag push is supposed to deploy again (but doesn't). The release flow is push → CI green → tag, so the deploy triggered by the push runs before the tag exists and git describe --tags --abbrev=0 resolves to the previous tag — the box would run the new code while reporting the old version. (Observed for real: the v0.36.1 push deployed a binary labelled 0.36.0, and again at v0.48.0: header 0.47.0-5095026 serving the tagged tree.) deploy.yml keeps a tags: ["v*"] arm for that re-deploy, but tags created by release-please use GITHUB_TOKEN, which starts no workflow runs — so the arm never fires. The schedule reconciler (every 30 min) closes the gap in both dimensions: running build SHA vs master HEAD (code drift), and badge version vs latest tag (label drift — redeploys the same code so the badge reports what is running). Tagging before CI passes would avoid the second restart, but a red CI would then already be on the tag — not a trade worth making.

The Dockerfile declares ARG VERSION COMMIT BUILDTIME at the stage top, not inline inside a RUN chain — an inline ARG breaks the Buildkit parse.

Confirming a deploy. The cheapest proof a fix is live is a byte-diff of an embedded asset:

diff <(curl -s https://<host>/static/app.min.css) \
     <(git show HEAD:web/resources/static/app.min.css)

There is no /api/version endpoint, so do not use one as a check — it 404s. For the badge, compare the navbar's rendered BuildLabel/BuildCommit against git describe --tags --abbrev=0 and git rev-parse --short HEAD.

Docker image

make docker-image   # Build and push the multi-arch image to ghcr.io (LOCAL target)

This is an optional, local convenience for publishing an image. The deploy workflow does not use it: the runner cross-compiles a static binary and the image is built on the server from deploy/docker-compose.prod.yml. Run make docker-image only if you want a registry-hosted image (e.g. to run the same build on another host).

The scratch image has no shell, so its healthcheck is CMD ["/app","health"] — never wget/curl/CMD-SHELL, none of which exist in a scratch base.

Configuration — the env vars the deploy writes.
Local CI — run the same gate before pushing.

Backup and restore

./scripts/backup.sh [DATA_DIR] [DEST_DIR]   # .backup + integrity_check per .db, JetStream when reachable
./scripts/restore.sh --dry-run <backup>     # monthly drill: verifies without touching anything
./scripts/restore.sh --force <backup> [DATA_DIR]  # stop the app first; moves -wal/-shm aside

What gets backed up: data.db, queue.db (both integrity-checked), JetStream streams/KV (only when the nats CLI reaches the server — ephemeral fan-out streams are losable by design, see async-layers). Browser state is never backed up. Restore swaps files with the app stopped; every engine recovers on boot, so there is no cross-store coordination step.

← UI sounds Desktop & Mobile →