Skip to main content

Production Install Runbook (Fresh Server) — This runbook is the practical fresh-server process to bring up a brand-new node from scratch. ⬆ índice

Production Install Runbook (Fresh Server)

This runbook is the practical fresh-server process to bring up a brand-new node from scratch.

If the server is already bootstrapped and reachable as externaladmin on the hardened SSH port, do not use this runbook. Use the deploy/update flow referenced from Deployment.

Assumptions​

  • You have a Debian/Ubuntu VPS with a public IP.
  • DNS is under your control.
  • You start with provider access: SSH root@<host> on TCP port 22 using a temporary password.
  • Final access will be: SSH externaladmin@<host> on TCP port 36987.

Inputs you must have​

  • IP_OR_DOMAIN: the server IP or DNS name (example: sd1.desarrolloselectronicos.com)
  • Provider temporary root password (used only for bootstrap): you will be prompted during remote bootstrap (or export REMOTE_ROOT_PASSWORD)
  • Desired admin password (used for sudo during deploy): you will be prompted during remote bootstrap/remote deploy (or export REMOTE_EXTERNAL_ADMIN_PASSWORD)
  • Your SSH public key (recommended): REMOTE_SSH_PUBLIC_KEY_PATH (or export SSH_PUBLIC_KEY)

1) What you must do first​

Before you run any install commands, do this checklist in order.

DNS​

If the domain is delegated to Cloudflare, generate/sync records from the repo:

./infra remote dns cloudflare <host> --zone <domain> --preflight
./infra remote dns cloudflare <host> --zone <domain>
./infra remote dns cloudflare <host> --zone <domain> --apply --wait

For initial account and nameserver setup, see Alta de zona. For the complete workflow and CLI reference, see Cloudflare.

For any other DNS provider, print the hostnames you need to create from your current config:

./infra remote domains

Create A/AAAA records for each hostname (pointing to the server public IP).

Wait until they resolve:

host <any-endpoint>.<domain>

Provider firewall / security group​

Allow inbound:

  • TCP 22 (temporary; bootstrap only)
  • TCP 36987 (final SSH port)
  • TCP 80, 443 (HTTP/HTTPS)
  • any other public ports you intentionally expose (optional)

If this checklist is not green, stop here and fix it first. Otherwise you'll hit noisy, misleading failures later.

2) How to run the process​

Configure local env + secrets​

These are the canonical sources of truth used by scripts and Ansible:

  • Topology/ports (non-secret): deployment/topology.env (generated from template)
  • Public endpoints/domains (non-secret): deployment/remote.env
  • Service runtime secrets for servers: Ansible Vault (recommended) via core2_env_secrets (rendered to {{ core2_repo_root }}/.env.secrets during deploy)
  • Optional local secrets for development: .env.secrets (never commit; not synced to servers)

Choose ONE path: Option A OR Option B (do not do both).

Option A (recommended): guided wizard (standard flow). This walks you through the same steps as the manual path (env -> server registration -> bootstrap -> deploy) and then runs basic verification checks.

./infra init remote

If you choose Option A, you can skip the manual steps below.

Option B: run the manual steps below:

Generate and validate locally:

./infra env generate
./infra env check

Important note:

  • These env files are used from your local machine to drive Ansible and generate configs.
  • remote bootstrap does not "generate env on the server".
  • remote deploy syncs the repo and seeds missing per-project .env files from .env.example/.env.template when needed.

Register the server (inventory)​

./infra server add <IP_OR_DOMAIN> --user root --port 22
./infra server list

Bootstrap (one-time hardening)​

Bootstrap uses the initial provider access (root@22 + temporary password) to harden the server and enable key-based access.

./infra remote bootstrap --host <IP_OR_DOMAIN>

After bootstrap, you should be able to connect on the hardened port:

ssh -p 36987 externaladmin@<IP_OR_DOMAIN>

Deploy (repeatable)​

This repository now uses a staged remote deploy strategy:

  • core: Traefik + private registry
  • system: PostgreSQL + Redis + Zitadel bootstrap
  • apps: application services

Application deploys build locally, push to the private registry, and pull remotely. Normal deploys do not build on the server. Read:

  • docs/internal/operations/remote-minimal-deploy.md
./infra remote deploy <IP_OR_DOMAIN> core
./infra remote deploy <IP_OR_DOMAIN> system
./infra remote deploy <IP_OR_DOMAIN> apps
./infra remote deploy <IP_OR_DOMAIN> <project>
./infra release <IP_OR_DOMAIN> <project>

Use:

  • remote deploy core|system for routing, registry, database, Redis, Zitadel, and other non-image refreshes
  • release <project> for application updates when the project image changed and must be rebuilt/published before deploy
  • after deploys on 8GB hosts, run the memory validation block in:
    • docs/internal/operations/remote-minimal-deploy.md

3) How to verify it worked (and lock it down)​

Prefer SSH keys (no password) for server access​

Before you consider the server "production-ready", ensure you can SSH with keys (no password prompts).

If you provided REMOTE_SSH_PUBLIC_KEY_PATH (or SSH_PUBLIC_KEY), bootstrap should have installed your key. Verify:

ssh -p 36987 -o BatchMode=yes externaladmin@<IP_OR_DOMAIN> "echo ok"

Operational checks​

On the server:

ssh -p 36987 externaladmin@<IP_OR_DOMAIN>
sudo ufw status verbose
sudo systemctl is-active docker fail2ban
sudo fail2ban-client status sshd
docker ps
curl -fsS http://127.0.0.1:5000/v2/

From your local machine:

./infra remote deploy <IP_OR_DOMAIN> <project>
./infra remote logs <IP_OR_DOMAIN> <project>

If public endpoints exist, validate:

  • HTTPS certs (issuer + expiry)
  • app pages return expected content
  • only expected ports are open

Registry (expected to be loopback-only on the server):

./infra remote tunnel <IP_OR_DOMAIN> registry
curl -fsS http://127.0.0.1:5000/v2/

Lock down bootstrap access​

After everything is stable:

  • close inbound TCP 22 at the provider firewall (keep only 36987)
  • keep a break-glass procedure using provider console access

Final post-install checks​

After the first successful deploy, review Backups and confirm the scheduler/snapshot process.

If the host is public, confirm public HTTPS is served by Traefik and Let's Encrypt certificates were issued. For local/Vagrant/private-IP targets, the fallback mode uses local HTTP or self-signed routing depending on deployment/remote.env.

Common install failures (quick fixes)​

  1. Bootstrap fails to connect as root

    • provider firewall blocks port 22, or wrong password in .env.secrets
    • verify: nc -zv <host> 22

  2. Deploy fails on public TLS

    • DNS not pointing to the server yet, or port 80/443 blocked
    • temporary workaround for private/local targets: REMOTE_TLS_MODE=http ./infra remote deploy <host> core

  3. Deploy sync is slow or fails

    • check disk space and network on the server
    • verify: df -h and journalctl -u ssh -n 200 --no-pager