Centralized Cluster-Wide Updates & Automated Rolling Upgrades

CloudyShaper

Updating Enhance currently requires SSHing into every node (Control Panel, Web, DB, Mail, DNS) to run apt update && apt upgrade. For multi-server clusters, this is slow and repetitive.

It would be best if Enhance offered cluster-wide updates managed directly from the main Control Panel server via:
Single Master Command: Run one update command on the primary server that automatically updates all connected nodes.
Or
UI Dashboard: A single button in the Master panel to initiate rolling updates across all cluster servers with basic health checks.

This would save significant time and make managing large clusters much easier.

JohnB

CloudyShaper This would save significant time and make managing large clusters much easier.

If you don't have a way to manage your large cluster centrally already, you shouldn't be managing one. This is basic server administration. Look into Ansible and its competitors and how easily you can automate package updates across all your nodes. Enhance shouldn't be handling this for many reasons.

Using Enhance for a small private setup without sysadmin skills is fine. Having a large cluster implies you have many customers that are paying for quality and reliability that you cant provide if you don't have human sysadmin skills in your team.

nhybgtvfr

use tmux.


#!/bin/bash

# --- CONFIGURATION ---
# Path to your external server list file
SERVER_LIST_FILE="servers.txt"
SESSION_NAME="multirun_pool"
# ---------------------

# 1. Validation Checks
if ! command -v tmux &> /dev/null; then
    echo "Error: tmux is not installed. Run 'sudo apt install tmux' or 'sudo dnf install tmux'."
    exit 1
fi

if [ ! -f "$SERVER_LIST_FILE" ]; then
    echo "Error: Server list file '$SERVER_LIST_FILE' not found."
    exit 1
fi

# 2. Parse the file into an array (ignoring comments and empty lines)
SERVERS=()
while IFS= read -r line || [[ -n "$line" ]]; do
    # Strip leading/trailing whitespaces
    line=$(echo "$line" | xargs)
    # Skip if line is empty or starts with #
    [[ -z "$line" || "$line" == "#"* ]] && continue
    SERVERS+=("$line")
done < "$SERVER_LIST_FILE"

# Ensure we actually found target nodes
if [ ${#SERVERS[@]} -eq 0 ]; then
    echo "Error: No valid servers found in '$SERVER_LIST_FILE'."
    exit 1
fi

# 3. Clean up any stale sessions of the same name
if tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
    tmux kill-session -t "$SESSION_NAME"
fi

echo "Found ${#SERVERS[@]} server(s). Spawning Tmux cluster..."

# 4. Initialize the baseline session with the first server
tmux new-session -d -s "$SESSION_NAME" "ssh ${SERVERS[0]}"

# 5. Dynamically split panes for remaining targets
for ((i=1; i<${#SERVERS[@]}; i++)); do
    if [ $((i % 2)) -eq 0 ]; then
        tmux split-window -h -t "$SESSION_NAME" "ssh ${SERVERS[$i]}"
    else
        tmux split-window -v -t "$SESSION_NAME" "ssh ${SERVERS[$i]}"
    fi
    tmux select-layout -t "$SESSION_NAME" tiled
done

# 6. Enable input synchronization (broadcast mode)
tmux set-window-option -t "$SESSION_NAME" synchronize-panes on

# 7. Attach user to cluster view
tmux attach-session -t "$SESSION_NAME"

or if you want different servers in different tabs, eg to separate roles

#!/bin/bash

# --- CONFIGURATION ---
SERVER_LIST_FILE="servers.txt"
SESSION_NAME="multirun_pool"
# ---------------------

# 1. Validation Checks
if ! command -v tmux &> /dev/null; then
    echo "Error: tmux is not installed."
    exit 1
fi

if [ ! -f "$SERVER_LIST_FILE" ]; then
    echo "Error: Server list file '$SERVER_LIST_FILE' not found."
    exit 1
fi

# 2. Reset conflicting sessions
if tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
    tmux kill-session -t "$SESSION_NAME"
fi

# 3. Read and Parse the file sequentially
current_window_name=""
window_index=-1
pane_count=0

while IFS= read -r line || [[ -n "$line" ]]; do
    # Strip leading/trailing whitespaces
    line=$(echo "$line" | xargs)
    
    # Skip empty lines or standard comments
    [[ -z "$line" || "$line" == "#"* ]] && continue

    # Check if the line is a page header, e.g., [Web-Servers]
    if [[ "$line" =~ ^\[(.*)\]$ ]]; then
        current_window_name="${BASH_REMATCH[1]}"
        ((window_index++))
        pane_count=0
        continue
    fi

    # If we hit a server line but haven't seen a header yet, create a default one
    if [ -z "$current_window_name" ]; then
        current_window_name="Default-Group"
        ((window_index++))
    fi

    # 4. Spawn the servers into their designated windows
    if [ $pane_count -eq 0 ]; then
        if [ $window_index -eq 0 ]; then
            # Initialize the base tmux session on the first header's first server
            tmux new-session -d -s "$SESSION_NAME" -n "$current_window_name" "ssh $line"
        else
            # Create a brand new named tab for subsequent headers
            tmux new-window -t "$SESSION_NAME" -n "$current_window_name" "ssh $line"
        fi
        pane_count=1
    else
        # Split panes within the active named tab
        target_win="$SESSION_NAME:$window_index"
        if [ $((pane_count % 2)) -eq 0 ]; then
            tmux split-window -h -t "$target_win" "ssh $line"
        else
            tmux split-window -v -t "$target_win" "ssh $line"
        fi
        tmux select-layout -t "$target_win" tiled
        ((pane_count++))
    fi

done < "$SERVER_LIST_FILE"

# Ensure we actually created something
if [ $window_index -lt 0 ]; then
    echo "Error: No valid servers or groups found in '$SERVER_LIST_FILE'."
    exit 1
fi

# 5. Enable Synchronization across all custom windows
for ((w=0; w<=window_index; w++)); do
    tmux set-window-option -t "$SESSION_NAME:$w" synchronize-panes on
done

# 6. Attach user to cluster view
echo "Launching session. Look at the bottom bar to see your custom tabs!"
tmux attach-session -t "$SESSION_NAME"

in servers.txt just list ssh logins on each line. or if you want tabs, separate them into sections, with each section becoming a tab/page name:


[panel]
root@panel.mydomain.tld

[shared web]
root@sh1.mydomain.tld
root@sh2.mydomain.tld

[dedicated]
root@vps1.mydomain.tld
root@vps2.mydomain.tld
root@vps3.mydomain.tld

[email]
root@mx1.mydomain.tld
root@mx2.mydomain.tld

CloudyShaper

nhybgtvfr
Thanks.

CloudyShaper

JohnB
Thanks, John! I posted it as a feature request to see if they can add this feature. I would love to see what they think about it.
I do not manage such a large cluster, so I can do it the old fashioned way by logging into each server and updating it one by one . I sometimes use the Broadcast Input feature in Termius. However, today was a busy day, and while updating everything one by one, I thought it would be nice if they could add this feature.

enhance_dan

Yeah, I was also going to say use something like Ansible. That's what a lot of large hosting providers use to make sure their server fleet is standardized.

CloudyShaper

enhance_dan
Ansible is a great tool; I need to start using it