You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
56 lines
4.3 KiB
Markdown
56 lines
4.3 KiB
Markdown
# Failover for Princeton University
|
|
Failover from primary Nagios XI to failover Nagios XI is a Disaster Recovery effort aimed at providing a near-time up-to-date failover Nagios server that can take over monitoring and notification options should the primary be unavailable. The primary will always monitor if it is capable. The failover will only monitor when manually enabled and must also be manually disabled when the primary returns to active service.
|
|
|
|
## Prerequisites
|
|
* Nagios XI must be installed on both boxes with the same version and underlying directory configuration. If there are any differences in file locations or major configuration between the two boxes, the Nagios failover will have unpredictable results, including complete system failure.
|
|
* The syncing process will delete files that it does not believe should be on the failover, so all work must be performed on the primary. Any work performed on the failover will be overwritten when the next synchronization process occurs. Note that this includes SSH keys, as /home/nagios will be synced from the parimary to the failover.
|
|
* /home/nagios/bin exists and contains the files needed for this process. Note that the sync process will sync these from the primary to the failover, so like all other files, they must only be modified on the primary.
|
|
|
|
/home/nagios/bin/failover.sh
|
|
/home/nagios/bin/nagios_startstop.sh
|
|
/home/nagios/bin/rsync_xi.sh
|
|
|
|
* The root user has the ability to SSH from the primary to the failover as the nagios user without entering a passphrase. This is how the rsync and database copies are performed
|
|
* root on primary (and root on failover) has crontab requirements that will be detailed separately.
|
|
* nagios on both primary and failover needs to be able ot sudo to root without a password and execute the rsync command:
|
|
|
|
NAGIOSXI ALL = NOPASSWD:/usr/bin/rsync *
|
|
|
|
* nagios user on primary needs to be able to SSH to nagios on failover without a passphrase (thus, an SSH key and .ssh directory needs to be set up)
|
|
|
|
* Make sure any ramdisk (such as /ramdisk) is copied if it exists
|
|
|
|
* Note that any gearman addons such as /etc/mod_gearman or whatever are NOT copied as part of this procedure. These types of things need to be set up on both boxes the same way before this process is set up.
|
|
|
|
## Installation
|
|
This assumes that Nagios XI is installed and that its base location is /usr/local/nagiosxi. It also assumes that there is a nagios user home directory at /home/nagios. If either of this locations is incorrect, the following commands will need to be updated accordingly.
|
|
|
|
* mkdir /home/nagios/bin
|
|
* cp * /home/nagios/bin
|
|
* cp -a local /home/nagios/bin
|
|
|
|
The first time the sync runs, it will take some time. It has to fully populate the failover Nagios server. It is recommended to run the sync once, manually, by executing the following as the nagios user on the primary Nagios server:
|
|
|
|
* /home/nagios/bin/SYNC
|
|
|
|
From here on, it can be run automatically via cron at a regular schedule. We recommend at least 30 minutes between successive runs.
|
|
|
|
### Cron
|
|
The following line should be added to cron on the primary, once the initial copy has been performed (this will run every 30 minutes at :00 and :30, for instance):
|
|
|
|
* */30 * * * * /home/nagios/bin/SYNC >> /home/nagios/sync.log
|
|
|
|
Note that this will create and append the output to /home/nagios/sync.log for future reference. This file can be deleted at any time or added to a logrotate script for management.
|
|
|
|
# Failover
|
|
Normally, the primary will update the failover on a regular schedule (defined in cron) as long as the primary is running and can reach the failover. When the time comes to activate Nagios on the failover box, the following command should be executed as root on the failover Nagios server:
|
|
|
|
* /home/nagios/bin/START
|
|
|
|
When the primary has returned to service, Nagios should be stopped on the failover box by typing the following command as root on the failover Nagios server:
|
|
|
|
* /home/nagios/bin/STOP
|
|
|
|
## Local directory
|
|
Anything in /home/nagios/bin/local will be copied (including the directory structure) to the local Nagios server as part of the syncronization. This is generally for things like custom CSS files. The failover code handles installing those files into their proper locations, if required, and will need to be updated manually if there are files other than the "color bar" that needs to be updated in the future.
|