archy/.claude/skills/polish-deploy/SKILL.md

Skill: Polish Deployment Pipeline

Harden deploy-to-target.sh with rollback capability, pre-deploy checks, post-deploy health verification, and deployment locking.

1. Pre-Deploy Checks

Add to the beginning of deploy-to-target.sh:

pre_deploy_checks() {
  echo "Running pre-deploy checks..."

  # SSH key exists
  if [ ! -f "$SSH_KEY" ]; then
    echo "ERROR: SSH key not found at $SSH_KEY"
    exit 1
  fi

  # Target reachable
  ssh $SSH_OPTS "$TARGET_HOST" "echo ok" >/dev/null 2>&1 || {
    echo "ERROR: Cannot reach $TARGET_HOST"
    exit 1
  }

  # Disk space (need 2GB free)
  local free_kb=$(ssh $SSH_OPTS "$TARGET_HOST" "df /home | tail -1 | awk '{print \$4}'")
  if [ "$free_kb" -lt 2097152 ]; then
    echo "ERROR: Need 2GB free disk space, have $(( free_kb / 1024 ))MB"
    exit 1
  fi

  echo "Pre-deploy checks passed"
}

2. Backup Before Deploy

Before overwriting binary or frontend:

backup_current() {
  echo "Backing up current deployment..."
  ssh $SSH_OPTS "$TARGET_HOST" "
    # Backup binary
    if [ -f /usr/local/bin/archipelago ]; then
      sudo cp /usr/local/bin/archipelago /usr/local/bin/archipelago.backup
    fi
    # Backup frontend
    if [ -d /opt/archipelago/web-ui ]; then
      sudo cp -a /opt/archipelago/web-ui /opt/archipelago/web-ui.backup
    fi
    # Backup nginx config
    if [ -f /etc/nginx/sites-available/archipelago ]; then
      sudo cp /etc/nginx/sites-available/archipelago /etc/nginx/sites-available/archipelago.backup
    fi
  "
  echo "Backup complete"
}

3. Post-Deploy Health Check

After restarting services:

health_check() {
  echo "Running post-deploy health check..."
  local max_attempts=15
  local attempt=0

  while [ $attempt -lt $max_attempts ]; do
    attempt=$((attempt + 1))
    local status=$(ssh $SSH_OPTS "$TARGET_HOST" "curl -s -o /dev/null -w '%{http_code}' http://localhost:5678/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
      echo "Health check passed (attempt $attempt)"
      return 0
    fi
    echo "Health check attempt $attempt/$max_attempts (status: $status)"
    sleep 2
  done

  echo "ERROR: Health check failed after $max_attempts attempts"
  return 1
}

4. Rollback on Failure

If health check fails:

rollback() {
  echo "ROLLING BACK deployment..."
  ssh $SSH_OPTS "$TARGET_HOST" "
    # Restore binary
    if [ -f /usr/local/bin/archipelago.backup ]; then
      sudo cp /usr/local/bin/archipelago.backup /usr/local/bin/archipelago
    fi
    # Restore frontend
    if [ -d /opt/archipelago/web-ui.backup ]; then
      sudo rm -rf /opt/archipelago/web-ui
      sudo mv /opt/archipelago/web-ui.backup /opt/archipelago/web-ui
    fi
    # Restore nginx
    if [ -f /etc/nginx/sites-available/archipelago.backup ]; then
      sudo cp /etc/nginx/sites-available/archipelago.backup /etc/nginx/sites-available/archipelago
      sudo nginx -t && sudo systemctl reload nginx
    fi
    # Restart with old binary
    sudo systemctl restart archipelago
  "
  echo "Rollback complete. Previous version restored."
}

5. Deployment Lock

Prevent concurrent deploys:

LOCK_FILE="/tmp/archipelago-deploy.lock"

acquire_lock() {
  exec 9>"$LOCK_FILE"
  flock -n 9 || {
    echo "ERROR: Another deployment is in progress"
    exit 1
  }
  trap "flock -u 9; rm -f $LOCK_FILE" EXIT
}

6. Nginx Config Validation

Before reloading nginx:

validate_nginx() {
  ssh $SSH_OPTS "$TARGET_HOST" "sudo nginx -t" 2>&1 || {
    echo "ERROR: Nginx config invalid. Restoring backup..."
    ssh $SSH_OPTS "$TARGET_HOST" "
      sudo cp /etc/nginx/sites-available/archipelago.backup /etc/nginx/sites-available/archipelago
      sudo nginx -t && sudo systemctl reload nginx
    "
    return 1
  }
}

Integration

The deploy flow becomes:

1. acquire_lock
2. pre_deploy_checks
3. backup_current
4. Build + deploy (existing logic)
5. validate_nginx
6. Restart services
7. health_check || rollback

Verification

Test the rollback:

1. Deploy a working version
2. Intentionally break the binary (e.g., truncate it)
3. Deploy the broken version
4. Verify rollback triggers and previous version is restored
5. Verify service is healthy after rollback

Deploy

./scripts/deploy-to-target.sh --live

After modifying the deploy script itself, test with a known-good deploy first.