Production Deployment — pkg.example.org

Target domain: example.org | Primary hostname: pkg.example.org | Last updated: 2026-04-13

DNS Records
VM Requirements
Secrets & Environment
Generating Signing Keys
Creating the deploy User
Pre-Deployment Checklist
Deployment Steps
Post-Deployment Validation
First Subscriber Onboarding
Run the Verification Suite
Monitoring

1. DNS Records

All records are on the example.org zone. Apply these before starting the deployment — Traefik's ACME TLS-ALPN-01 challenge requires the hostnames to resolve to the VM before it can issue the TLS certificate.

Type	Name	Value	TTL	Notes
A	`pkg.example.org`	`<VM_IPV4>`	300	Primary package serving endpoint
AAAA	`pkg.example.org`	`<VM_IPV6>`	300	Only if VM has a public IPv6 address
A	`admin.pkg.example.org`	`<VM_IPV4>`	300	Admin UI + admin API (OAuth-gated)
AAAA	`admin.pkg.example.org`	`<VM_IPV6>`	300	Only if VM has a public IPv6 address
CAA	`pkg.example.org`	`0 issue "letsencrypt.org"`	3600	Restricts TLS cert issuance to Let's Encrypt
CAA	`pkg.example.org`	`0 iodef "mailto:ops@example.org"`	3600	CAA violation notification address

Admin host: admin.pkg.example.org is gated by OAuth + operator allowlist (see Operator onboarding). The historical loopback admin entrypoint (127.0.0.1:8088) has been retired — the admin API is now exposed over public TLS but requires a valid operator session.

DNS propagation check

dig +short pkg.example.org A
dig +short admin.pkg.example.org A
dig +short pkg.example.org CAA

2. VM Requirements

Resource	Minimum	Notes
OS	Ubuntu 24.04 LTS	Or any Docker-compatible Linux
CPU	2 vCPU	Auth service + nginx are lightweight
RAM	4 GB	Aptly snapshot creation peaks at ~1.5 GB
Disk	100 GB	RPM/DEB/OCI artifact storage; size to expected package volume
Ports open	TCP 22, TCP 443	22 for SSH/operator access; 443 for package serving
Docker	26+	With Compose plugin v2

Firewall rules:

Source	Protocol/Port	Purpose
`0.0.0.0/0`	TCP 443	Package subscribers + TLS-ALPN-01 cert issuance
operator CIDR	TCP 22	SSH access to the VM (host-level ops; admin API is HTTPS-only)

Traefik uses the TLS-ALPN-01 challenge for Let's Encrypt — port 443 is the only port required. Port 80 does not need to be open.

3. Secrets & Environment

`.env` file (on VM, never committed)

# TLS + hostnames
ACME_EMAIL=ops@example.org
PKG_DOMAIN=pkg.example.org
ADMIN_DOMAIN=admin.pkg.example.org

# Admin bootstrap — the first operator inserted when the operators table is
# empty. Idempotent: ignored once any operator exists. See the operator
# onboarding guide for the full flow.
PACKYARD_BOOTSTRAP_OPERATOR_EMAIL=ops@example.org

# OAuth providers — configure at least one; configure both for redundancy.
# Every variable in a provider's group must be set together; partial config
# fails the auth service at startup.

# GitHub
PACKYARD_GITHUB_CLIENT_ID=<from GitHub OAuth App>
PACKYARD_GITHUB_CLIENT_SECRET=<from GitHub OAuth App>
PACKYARD_GITHUB_REDIRECT_URI=https://admin.pkg.example.org/api/v1/auth/callback/github
PACKYARD_GITHUB_ORG=<github-org-login>

# Microsoft Entra (Azure AD)
PACKYARD_MICROSOFT_CLIENT_ID=<from Entra app registration>
PACKYARD_MICROSOFT_CLIENT_SECRET=<from Entra app registration>
PACKYARD_MICROSOFT_REDIRECT_URI=https://admin.pkg.example.org/api/v1/auth/callback/microsoft
PACKYARD_MICROSOFT_TENANT_ID=<from Entra directory>

# Forward-auth public-component cache TTL (optional; unset = 30s, 0 disables,
# max 1h). See docs/reference/configuration.md.
#PACKYARD_PUBLIC_COMPONENT_CACHE_TTL=30s

# RustFS staging storage (generate with: openssl rand -hex 20)
RUSTFS_ACCESS_KEY=<generate>
RUSTFS_SECRET_KEY=<generate>

See Operator onboarding §1 for how to register the GitHub and Microsoft applications and obtain the client id / secret / tenant id values.

GitHub Actions secrets

Secret name	Value source
`HOST`	`pkg.example.org`
`SSH_PRIVATE_KEY`	Private key for the `deploy` user on VM
`SSH_KNOWN_HOST`	`ssh-keyscan pkg.example.org` output
`RUSTFS_ACCESS_KEY`	Same as `.env`
`RUSTFS_SECRET_KEY`	Same as `.env`
`GPG_PRIVATE_KEY`	ASCII-armored LTS GPG signing key
`GPG_KEY_ID`	Key fingerprint (40 hex chars, no spaces)
`GPG_PASSPHRASE`	GPG key passphrase

OCI images are signed keylessly, so no cosign secret is needed. The promote-oci workflow authenticates to Sigstore with its GitHub OIDC token.

4. Generating Signing Keys

Keys are generated once, kept offline in a secrets manager (e.g. 1Password, Vault), and loaded into GitHub Actions secrets. Never commit private keys to the repository.

4.1 GPG signing key

Used to sign RPM and DEB packages at promotion time. Use a dedicated key for LTS — do not reuse an operator's personal key.

# 1. Generate the key (batch mode, no TTY required)
cat > /tmp/lts-gpg-params <<'EOF'
%echo Generating LTS signing key
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Name-Real: LTS
Name-Comment: Package Signing Key
Name-Email: lts-signing@example.org
Expire-Date: 0
Passphrase: <CHOOSE_A_STRONG_PASSPHRASE>
%commit
%echo Done
EOF

gpg --batch --gen-key /tmp/lts-gpg-params
rm /tmp/lts-gpg-params

# 2. Find the 40-character fingerprint — this is GPG_KEY_ID
gpg --list-keys --fingerprint lts-signing@example.org
GPG_KEY_ID="<40-char fingerprint, no spaces>"

# 3. Export ASCII-armored private key — this is GPG_PRIVATE_KEY
gpg --armor --export-secret-keys "$GPG_KEY_ID"

# 4. Export public key and commit it to the repo
gpg --armor --export "$GPG_KEY_ID" > static/content/gpg/lts.asc
# git add static/content/gpg/lts.asc && git commit

Secrets to set:

Secret	Value
`GPG_PRIVATE_KEY`	Output of `gpg --armor --export-secret-keys "$GPG_KEY_ID"`
`GPG_KEY_ID`	40-character fingerprint (no spaces)
`GPG_PASSPHRASE`	Passphrase chosen during key generation

4.2 cosign (keyless, no key to generate)

OCI images are signed by promote-oci.yml with cosign keyless signing. The workflow requests a short-lived certificate from Sigstore's Fulcio CA for its GitHub Actions identity and records each signature in the Rekor transparency log. There is no private key to generate, store or rotate, and no public key to publish.

Requirements:

The workflow job has id-token: write (already set).
The runner can reach fulcio.sigstore.dev and rekor.sigstore.dev. GitHub-hosted runners can.
Subscribers verifying images can reach rekor.sigstore.dev and tuf-repo-cdn.sigstore.dev. Air-gapped subscribers cannot verify keyless signatures. If that becomes a requirement, revert to key-based signing with a published cosign.pub.

4.3 Key storage checklist

GPG private key exported and stored in secrets manager
GPG key ID (fingerprint) noted
GPG passphrase stored in secrets manager
static/content/gpg/lts.asc committed to repository
All 8 secrets set in GitHub Actions repository settings

5. Creating the `deploy` User

Run on the VM as root or via sudo.

# Create user with login shell (required for git clone and docker compose)
useradd --create-home --shell /bin/bash deploy
usermod -aG docker deploy

Generate an SSH key pair on your local machine:

ssh-keygen -t ed25519 -C "packyard-deploy" -f ~/.ssh/packyard_deploy
# ~/.ssh/packyard_deploy      — private key (keep secret)
# ~/.ssh/packyard_deploy.pub  — public key  (goes on the VM)

Authorize the public key on the VM:

# On the VM as root
mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
echo "<paste ~/.ssh/packyard_deploy.pub>" >> /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

Verify access, then capture secrets:

# Test login
ssh -i ~/.ssh/packyard_deploy deploy@pkg.example.org

# SSH_PRIVATE_KEY secret value
cat ~/.ssh/packyard_deploy

# SSH_KNOWN_HOST secret value (run after DNS propagates)
ssh-keyscan pkg.example.org

What the promotion workflows need on the host

The promotion workflows (promote-rpm.yml, promote-deb.yml, promote-oci.yml, promote-release.yml) log in as deploy and run docker compose -f $DEPLOY_DIR/compose.yml exec ... against the running services. Three things make that work:

DEPLOY_DIR is a GitHub repository variable naming the checkout on the host. It defaults to /opt/packyard when unset. Set it under Settings, Secrets and variables, Actions, Variables if the checkout lives elsewhere, for example /etc/docker/pkgs.
COMPOSE_PROJECT_NAME in .env pins the Compose project name. Without it Compose derives the name from the checkout's directory, so renaming or moving the checkout, or reaching it through a symlink, changes the project name and docker compose ... ps -q aptly finds nothing. Pinning it makes the name independent of the path.
.env must be readable by deploy. Compose interpolates ${ADMIN_DOMAIN:?} and friends from .env on every invocation, including exec. Make it root:deploy with mode 0640:

chgrp deploy /path/to/checkout/.env
chmod 640 /path/to/checkout/.env

deploy is in the docker group. On a Docker host that is equivalent to root, so readable .env grants nothing the user does not already have. Treat the SSH_PRIVATE_KEY secret accordingly: it is a root credential for the package host.

The RPM tree lives in the rpm-data volume and is written from inside the rpm container, which ships createrepo_c. The host needs no RPM tooling.

Three processes write that tree and none of them has capabilities: nginx as uid 101, the auth service as root (it creates component directories from the component record), and the promotion exec as root. The rpm image's start script therefore makes the tree group 101, setgid and group-writable, compose.yml gives the auth service group_add: ["101"], and the promotion exec runs as 0:101. A volume created by an image older than 0.5.1 is normalised on the next start of the rpm container.

6. Pre-Deployment Checklist

DNS A record for pkg.example.org propagated (dig confirms VM IP)
DNS CAA record for pkg.example.org present
VM firewall: tcp/443 open to internet
Docker + Compose plugin v2 installed on VM
deploy user created, added to docker group, SSH key authorized (§5)
GPG LTS signing key generated (§4.1); lts.asc committed to static/content/gpg/
.env file written on VM with production values (§3), including COMPOSE_PROJECT_NAME, owned root:deploy mode 0640 (§5)
Repository variable DEPLOY_DIR set if the checkout is not /opt/packyard (§5)
All 8 GitHub Actions secrets set in repository settings (§3)

7. Deployment Steps

# On the VM as the deploy user
git clone <packyard-repo> ~/packyard
cd ~/packyard

# Write .env (see §3)
# Ensure static/content/gpg/lts.asc is present

docker compose pull
docker compose up -d

# Watch for Traefik to obtain the Let's Encrypt certificate (up to 2 min)
docker compose logs traefik -f

Expected cert issuance log line:

traefik  | msg="Certificate obtained successfully" domain=pkg.example.org

8. Post-Deployment Validation

Run these from an external host, not the VM itself.

Replace core and minion in the examples below with component names provisioned via POST /api/v1/components.

# 1. GPG key endpoint — tests TLS + routing (unauthenticated)
curl -sI https://pkg.example.org/gpg/lts.asc
# Expect: HTTP/2 200, Content-Type: text/plain

# 2. Package endpoint rejects unauthenticated requests
curl -sI https://pkg.example.org/rpm/core/2025/el9-x86_64/repodata/repomd.xml
# Expect: HTTP/2 401

# 3. Valid key is accepted
curl -sI -u subscriber:<KEY> https://pkg.example.org/rpm/core/2025/el9-x86_64/repodata/repomd.xml
# Expect: HTTP/2 200 (after first promotion) or 404 if no artifacts yet

# 4. Wrong-component key is rejected
curl -sI -u subscriber:<CORE_KEY> https://pkg.example.org/rpm/minion/2025/el9-x86_64/repodata/repomd.xml
# Expect: HTTP/2 401

# 5. Admin host reachable, but unauthenticated /api/v1 returns 401
curl -sI https://admin.pkg.example.org/api/v1/operators
# Expect: HTTP/2 401 with `code: "UNAUTHORIZED"` body

# 6. Admin login page renders
curl -sI https://admin.pkg.example.org/admin/login
# Expect: HTTP/2 200 with text/html

9. First Subscriber Onboarding

The bootstrap operator (see .env → PACKYARD_BOOTSTRAP_OPERATOR_EMAIL) signs in at https://admin.pkg.example.org/admin/login using their IdP, then uses the SPA to create a subscriber account and issue its first key.

For headless / scripted use, the same flow goes through the API. Capture the session cookie from a browser login (DevTools → Application → Cookies → packyard_session) and pass it via curl --cookie:

COOKIE="packyard_session=…"

# 1. Create a subscriber account
ACCOUNT=$(curl -s -X POST https://admin.pkg.example.org/api/v1/accounts \
  -H 'Content-Type: application/json' \
  --cookie "$COOKIE" \
  -d '{"email":"ops@acme.test","org_name":"Acme Corp"}' | jq -r .id)

# 2. Issue a subscription key for the account
curl -s -X POST "https://admin.pkg.example.org/api/v1/accounts/${ACCOUNT}/keys" \
  -H 'Content-Type: application/json' \
  --cookie "$COOKIE" \
  -d '{"component":"core","label":"Acme — Core"}' | jq .
# The `id` field IS the subscription key — there is no separate `secret`
# field. Share `id` with the subscriber verbatim; rotation requires
# revoking this key and issuing a fresh one. See docs/reference/api.md
# § Subscription keys for the full schema.

See Operator onboarding for how to allowlist additional admins or auditor-style readonly operators.

Example subscriber yum.repos.d entry:

[onms-lts-core]
name=LTS Core
baseurl=https://subscriber:<KEY>@pkg.example.org/rpm/core/2025/el9-x86_64/
enabled=1
gpgcheck=1
gpgkey=https://pkg.example.org/gpg/lts.asc

10. Run the Verification Suite

Two scripts verify the stack end-to-end. Run them in order.

10.1 Container health check

Run on the VM as the deploy user. Checks container states, GPG endpoint, auth service reachability, admin API isolation, RPM routing, network isolation, and RustFS health:

cd ~/packyard
PKG_DOMAIN=pkg.example.org bash scripts/health-check.sh

Expected: all lines start with OK: and the script exits All services healthy.

10.2 Remote smoke test

Run from any machine with network access to the deployment. Requires the subscriber key created in §9:

# Clone the repo locally if needed
git clone <packyard-repo> packyard && cd packyard

bash verify.sh \
  --base-url https://pkg.example.org \
  --test-key "$KEY" \
  --test-component core

Expected output ends with:

=== Results: N passed, 0 failed ===

The remote mode covers: public GPG endpoints, forwardAuth allow/deny, scope enforcement, and OCI scope — without touching the admin API or the Docker socket.

11. Monitoring

Check	Method	SLA
Endpoint availability	HTTP GET `https://pkg.example.org/gpg/lts.asc` from external monitor	99.9% monthly
TLS cert expiry	Alert at ≤ 30 days remaining	—
Auth service health	Traefik health check (auto; returns 503 on failure)	Fail-closed

Prometheus metrics are available at http://auth:9090/metrics (internal Docker network only). Traefik's own metrics are at http://traefik:8082/metrics on the internal metrics entrypoint. Expose either to an internal monitoring stack by adding a Prometheus container on the proxy Docker network, or scrape over SSH if a stack is hosted elsewhere.

Table of Contents​

1. DNS Records​

DNS propagation check​

2. VM Requirements​

3. Secrets & Environment​

.env file (on VM, never committed)​

GitHub Actions secrets​

4. Generating Signing Keys​

4.1 GPG signing key​

4.2 cosign (keyless, no key to generate)​

4.3 Key storage checklist​

5. Creating the deploy User​

What the promotion workflows need on the host​

6. Pre-Deployment Checklist​

7. Deployment Steps​

8. Post-Deployment Validation​

9. First Subscriber Onboarding​

10. Run the Verification Suite​

10.1 Container health check​

10.2 Remote smoke test​

11. Monitoring​

Table of Contents