sd1 → sd2 Cutover Runbook — the order-dependent steps for retiring desarrolloselectronicos.com on sd1 and promoting aitha.es on sd2. ↓ Production Install · ⬆ índice

sd1 → sd2 Cutover

sd2 runs the same stack as sd1 on ARM (aarch64) under a new domain. Its data is a snapshot, not a replica: nothing syncs continuously, so everything below has to happen in one window, in this order.

Verify the drift before you start — the gap grows every day:

for H in sd1 sd2; do
  ssh $H 'docker exec postgres-central sh -c "
    psql -U postgres -d chatwoot   -tAc \"select '\''messages'\''||chr(9)||count(*) from messages\";
    psql -U postgres -d thingsboard -tAc \"select '\''ts_kv'\''||chr(9)||count(*) from ts_kv\""'
done

1. Reconcile the secrets

Both hosts' .env.secrets already agree on 46 of 50 keys, database passwords included, so no ALTER ROLE is needed. Merge the rest with:

./deployment/scripts/setup/pull_secrets_from.sh sd1            # dry run
./deployment/scripts/setup/pull_secrets_from.sh sd1 --apply

It merges by key rather than copying the file, which would delete the 15 keys for projects sd1 never had, and prints md5 prefixes instead of values.

Four OIDC client secrets (chatwoot, chat_server, portal, thingsboard) are held back on purpose. They live inside the Zitadel database, so applying them before the restore leaves each app presenting credentials the local instance never issued and every login fails. Apply them with the restore in step 3:

./deployment/scripts/setup/pull_secrets_from.sh sd1 --apply --include-zitadel-apps

The n8n encryption key is not in either secrets file

sd1's running n8n uses a key that exists only in its project .env, not in its .env.secrets:

                 .env.secrets   .env project   running container
sd1              54b1dc6c9c     692f4adbae     692f4adbae
sd2              54b1dc6c9c     54b1dc6c9c     54b1dc6c9c

Two consequences. Merging the secrets files does not carry it, so read it from /opt/core2infra/infrastructure/n8n/.env on sd1 and set it on sd2 before restoring the n8n database, or re-enter the two stored credentials by hand afterwards.

And it is a live hazard on sd1 today, independent of this migration: regenerating that project's .env and redeploying would hand n8n the key from .env.secrets and make sd1's own credentials unreadable. Do not run ./infra env generate infrastructure/n8n against sd1 until this is reconciled.

The same "encrypts data at rest" property applies to Airflow's Fernet key and Chatwoot's SECRET_KEY_BASE; both already match across the hosts.

2. Take a backup on both hosts

ssh sd2 'cd /opt/core2infra && sudo ./infra backup-scheduler run'

The scheduler writes to /var/backups/core2 on the same disk as the data, so it survives a bad restore but not a lost host. Copy the archive off-box before cutting over.

3. Restore sd1's data onto sd2

Snapshot the retired host last, so the window is as short as possible. Restoring the zitadel database also replaces the instance domains — which is why step 4 comes after, not before.

3b. Copy the data that does not live in PostgreSQL

Only PostgreSQL was ever migrated. Everything held on disk is still sd1-only, and a database restore will not bring it:

Path	sd1	sd2
data-layer/qdrant/data	8 collections, 146 MB	empty
platform/chatwoot/storage	52 files, 20 MB	empty
infrastructure/chat-server/uploads	30 files	2 files
platform/thingsboard/data	107 MB	19 MB

Qdrant holds every assistant's vector store: agenciabene, bene-1, bersa-general-or-pw, bersa-servicio-tecnico, bersa-shop, bersashop-ex, desarrolloselectronicos-web, echevarri-sports.

Chatwoot is the urgent one: sd2's database already has 39 active_storage_blobs rows pointing at files that are not there, so those attachments are broken in the UI today.

Stop the consuming service, rsync the directory, start it again — Qdrant and ThingsBoard will not tolerate a copy taken from under them.

Set BACKUP_INCLUDE_QDRANT=true in /etc/core2-backup.conf once sd2 holds the collections. It defaults to false, so today sd1's 146 MB of embeddings are in no backup at all.

4. Re-point the Zitadel instance domain

sd1 has iam.desarrolloselectronicos.com flagged primary; sd2 has no primary domain at all. After the restore sd2 inherits sd1's, still pointing at the retired name.

ssh sd2 'docker exec postgres-central psql -U postgres -d zitadel -tAc \
  "select domain, is_primary, is_generated from projections.instance_domains order by domain"'

iam.aitha.es must come back is_primary = t. Setting it needs the System API: AddCustomDomain rejects the management PAT with 403, and no SetPrimaryDomain method exists on InstanceService v2. Populate ZITADEL_SYSTEMAPIUSERS (security/zitadel/docker-compose.yml already wires it, and the value must be single-quoted in .env or Zitadel crashloops) and call it with a system JWT.

Drop the stale generated domain zitadel-6v4lpk.iam.desarrolloselectronicos.com in the same pass.

5. Provision the field device broker accounts

sd1's broker carries an mqttlow account that sd2 does not; it was added by hand and was never in any config. Devices authenticating with it fail against sd2 with not authorised.

Set GLOBAL_MOSQUITTO_EXTRA_USERS to mqttlow:<sd1 password> and GLOBAL_MOSQUITTO_PASSWORD to sd1's mqttuser password, then deploy mosquitto: provisioning upserts every configured account on each start.

ssh sd2 'docker exec mosquitto cut -d: -f1 /mosquitto/config/passwords'

Both accounts must be listed. Getting this wrong is silent until hardware in the field stops reporting.

6. DNS and certificates

www.desarrolloselectronicos.com stays served from sd2 — it is the one name that outlives the retirement. Everything else moves to aitha.es.