048679065e
Closes failure mode FM5 from docs/bulletproof-containers.md: the v1.7.38 + v1.7.39 rollouts left every affected node on an unreachable UI (nginx 500) with no recovery path short of SSH. This release adds a self-check guardrail to the update flow. What changed: - apply_update() writes a pending-verify marker with old+new version and a 150s deadline immediately before scheduling the service restart. - verify_pending_update() runs from main.rs startup. If the marker is present and within its freshness window, the new binary waits 15s for nginx + backend to settle, then probes https://127.0.0.1/ every 5s for up to 90s (self-signed certs accepted). - On any probe success within the window, the marker is cleared and nothing else happens. - On window-exhaust, the new binary: 1. Moves the broken /opt/archipelago/web-ui to web-ui.failed.<ts> (quarantined, not deleted, so we can post-mortem). 2. Restores web-ui.bak on top of web-ui. 3. Calls rollback_update() to restore the previous binary. 4. Updates state.current_version to reflect the rollback. 5. systemctl --no-block restart archipelago so the OLD binary boots. - Markers older than 10 minutes are treated as stale and cleared without probing, so a crashed-during-startup marker from weeks ago cannot spontaneously roll back a healthy node on a later reboot. - rollback_update() binary copy now goes through host_sudo instead of tokio::fs::copy, so it escapes the service's ProtectSystem=strict mount namespace. Without this, the rollback silently failed with EROFS on /usr/local/bin and orphaned the rollback - the exact opposite of what auto-rollback is for. Tests: 4 new unit tests in update::tests covering marker round-trip, absent-marker noop, no-panic on verify_pending_update with nothing to verify, and an invariant assert that the 90s probe window stays below the 600s stale threshold. All passing. Side fix: scripts/create-release-manifest.sh was dying with exit 141 (SIGPIPE from tar tvzf pipe head pipe awk) under set -euo pipefail. Replaced with a single awk NR==1 that doesn't short-circuit the upstream pipe, so the release-build flow is idempotent again.
27 lines
1.7 KiB
JSON
27 lines
1.7 KiB
JSON
{
|
|
"version": "1.7.41-alpha",
|
|
"release_date": "2026-04-22",
|
|
"changelog": [
|
|
"Updates now self-check. When a new version lands, the node probes its own web UI through nginx within the first 90 seconds after the service restarts. If the frontend isn't answering cleanly, the node automatically rolls back to the previous working version and reboots the service. A bad release can no longer leave the fleet stranded on an unreachable UI — the kind of failure that required SSH recovery on every affected node during the v1.7.38 and v1.7.39 rollouts is now self-healing.",
|
|
"Rollback is hardened against the service's own mount namespace. Restoring the previous binary goes through the same privileged helper as every other write into /opt/archipelago, so it no longer silently fails with EROFS when ProtectSystem is strict. Both the binary and the previous web UI tarball are restored together; the broken web UI is quarantined rather than deleted so you can inspect it after the fact."
|
|
],
|
|
"components": [
|
|
{
|
|
"name": "archipelago",
|
|
"current_version": "1.7.40-alpha",
|
|
"new_version": "1.7.41-alpha",
|
|
"download_url": "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/v1.7.41-alpha/archipelago",
|
|
"sha256": "eb6eeb9720720c566db614861c1a878f48630e6f6c90276cbc8c032bfd910afc",
|
|
"size_bytes": 41215800
|
|
},
|
|
{
|
|
"name": "archipelago-frontend-1.7.41-alpha.tar.gz",
|
|
"current_version": "1.7.40-alpha",
|
|
"new_version": "1.7.41-alpha",
|
|
"download_url": "https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/v1.7.41-alpha/archipelago-frontend-1.7.41-alpha.tar.gz",
|
|
"sha256": "b1ac88b8bb056033aff8818f5a143b69388b66e388aa1e096b064dfbe892130c",
|
|
"size_bytes": 162084894
|
|
}
|
|
]
|
|
}
|