The IPMI Shutdown Agent is a mandatory mechanism to avoid split brain situations in case of a loss of communications between nodes in a redundant server configuration. It is designed to protect against single point of failure scenarios.
There are conceivable situations where no take over occurs, all related to highly unlikely double failures of either the cluster nodes and/or the communication to the survival authority. Some of these can be resolved with the optional DOWN Shutdown Agent.
After PrimeCluster detects that it cannot communicate with its partner node, it checks its node priority. Node 2 has a higher priority than node 1. For node 2, it immediately sends an IPMI set command through the admin network of node 2 to the remote maintenance controller (iRMC/IMM card) of node 1 with the request to power cycle. For the specified IPMI timeout period node 2 will try to read the state of node 1 by querying its remote maintenance controller interface. When node 1 reports that it has power cycled or that it was already power cycled, then in server configurations wherein the nodes share the same subnet for the Management, Signaling and Billing interfaces the active node has all the virtual IP addresses activated that were running on the shut down node. When a virtual IP address is activated on the Lotus Sametime Unified Telephony node that takes over, it sends out a so-called gratuitous address resolution protocol (ARP) to inform the LAN switches and routers of the network about the new MAC address for the virtual IP address. The routers and the LAN switches then reconfigure to adapt to the new situation. A network scheme in which server nodes share the same subnet for the Management, Signaling and Billing interfaces is common for co-located server clusters.
For server configurations with network separation (each node has different subnets for the Management, Signaling and Billing interfaces) endpoints have to switchover to the partner node IP (of the active node). This type of networking scheme is common for geo-separated clusters. If on the other hand, node 2 fails, then node 1 will detect the cluster-interconnect failure and will start its SA_IPMI. Node 1 will delay its IPMI set command (with the request for node 2 to power cycle) by the total time required for the shutdown agents of node 2 to run (in order not to interfere with any node 2 request). For the specified IPMI timeout period node 1 will try to read the state of node 2 by querying its remote maintenance controller interface. When node 2 reports that it has power cycled or that it was already power cycled node 1 will take over the virtual IP addresses and resources of node 2 (in the same manner as described above for node 2 in the node 1 failure case).
Parent topic: Cluster Redundancy