Don't stop Nagios on the secondary at this time.

failover/README.md

@@ -1,18 +1,17 @@
 # Failover for Princeton University
-Failover from primary Nagios XI to secondary Nagios XI is a Disaster Recovery effort aimed at providing an up-to-date secondary Nagios server that can take over monitoring and notification options should the primary be unavailable.  Once the primary is available, the secondary will cease monitoring and go back to a passive state.
+Failover from primary Nagios XI to secondary Nagios XI is a Disaster Recovery effort aimed at providing a near-time up-to-date secondary Nagios server that can take over monitoring and notification options should the primary be unavailable.  The primary will always monitor if it is capable.  The secondary will only monitor when manually enabled and must also be manually disabled when the primary returns to active service.
 
 ## Prerequisites
-* Nagios XI must be installed on both boxes with the same underlying directory configuration.  If there are any differences in file locations or major configuration between the two boxes, the Nagios failover will have unpredictable results, including complete system failure.
-* The syncing process will delete files that it does not believe should be on the secondary, so all work must be performed on the primary.  Any work performed on the secondary will be overwritten when the next synchronization process occurs.
+* Nagios XI must be installed on both boxes with the same version and underlying directory configuration.  If there are any differences in file locations or major configuration between the two boxes, the Nagios failover will have unpredictable results, including complete system failure.
+* The syncing process will delete files that it does not believe should be on the secondary, so all work must be performed on the primary.  Any work performed on the secondary will be overwritten when the next synchronization process occurs.  Note that this includes SSH keys, as /home/nagios will be synced from the parimary to the secondary.
 * /home/nagios/bin exists and contains the files needed for this process.  Note that the sync process will sync these from the primary to the secondary, so like all other files, they must only be modified on the primary.
 
   /home/nagios/bin/failover.sh
-  /home/nagios/bin/nagios_start_stop.sh
-  /home/nagios/bin/restore_xi.sh
+  /home/nagios/bin/nagios_startstop.sh
   /home/nagios/bin/rsync_xi.sh
 
 * The root user has the ability to SSH from the primary to the secondary as the nagios user without entering a passphrase.  This is how the rsync and database copies are performed
-* root on primary (and root on secondary) have crontab requirements that will be detailed separately.
+* root on primary (and root on secondary) has crontab requirements that will be detailed separately.
 * nagios on both primary and secondary needs to be able ot sudo to root without a password and execute the rsync command:
 
   NAGIOSXI ALL = NOPASSWD:/usr/bin/rsync * 

failover/failover.sh

@@ -33,6 +33,18 @@ done
 
 [ -z "$hostname" ] && warning "Invalid hostname." && exit 2
 
+stop_nagios() {
+    verbose "Ensuring Nagios services are stopped because we are the secondary"
+    sleep 2
+    /home/nagios/bin/nagios_startstop.sh stop
+}
+
+start_nagios() {
+    verbose "Ensuring Nagios services are stopped because we are the secondary"
+    sleep 2
+    /home/nagios/bin/nagios_startstop.sh start
+}
+
 do_nothing() {
   warning "We are not the primary or secondary.  Doing nothing."
   exit 1
@@ -53,9 +65,7 @@ do_secondary() {
   [ -z "$syncfile" ] && error "No sync file ($syncfile) found." && exit 2
   /home/nagios/bin/rsync_xi.sh --primary "${primary}" --secondary "${secondary}" --file "/store/backups/nagiosxi/$syncfile"
   if [ $? -eq 0 ]; then
-    verbose "Ensuring Nagios services are stopped because we are the secondary"
-    sleep 2
-    /home/nagios/bin/nagios_startstop.sh stop
+    # stop_nagios
     if [ -r "/usr/local/nagiosxi/html/includes/components/custom-includes/css/header-gradient.css.secondary" -a -w "/usr/local/nagiosxi/html/includes/components/custom-includes/css/header-gradient.css" ]; then
       cp /usr/local/nagiosxi/html/includes/components/custom-includes/css/header-gradient.css.secondary /usr/local/nagiosxi/html/includes/components/custom-includes/css/header-gradient.css
     fi