Saturday, May 23, 2015

High Availability : Setup UCarp on your Oracle Linux machines

Goal

Setup of multiple (in this case 2) Oracle Linux servers in a cluster with a virtual IP (VIP).  The intention is that the VIP is always active on exactly one server machine.
  • Primary IP of the first machine "ol6a": 192.168.56.121
  • Primary IP of the second machine "ol6b": 192.168.56.122
  • Virtual IP: 192.168.56.120 with linked hostname "ol6"
Apache is installed on both machines.  When navigating to the default page, the document root is shown.


Installation / Configuration steps on both machines


In the “/root” directory, create a start and a stop script that will be executed when the virtual ip defined by ucarp comes up or goes down.

# more /root/vip_up.sh
#!/bin/sh
exec 2>/dev/null
# Enable VIP on host
/sbin/ip addr add "$2"/24 dev "$1"
# Refresh MAC address on gateway
/sbin/arping -c 5 -A $2 -I $1
/sbin/arping -c 5 -U $2 -I $1
# Write to logfile
echo "up:" >> /root/vip.txt
date >> /root/vip.txt

# more /root/vip_down.sh
#!/bin/sh
exec 2>/dev/null
# Remove VIP from host
/sbin/ip addr del "$2"/24 dev "$1"
# Write to logfile
echo "down:" >> /root/vip.txt
date >> /root/vip.txt

# chmod u+x vip*sh


In the “/home/tom” directory, create a “down” and “up” log script.

# su - tom

$ more log_up.sh
#!/bin/sh
echo "up:" >> /home/tom/vip.txt
date >> /home/tom/vip.txt

$ more log_down.sh
#!/bin/sh
echo "down:" >> /home/tom/vip.txt
date >> /home/tom/vip.txt

$ chmod u+x log*sh


Install UCarp through YUM via the EPEL repository:

# yum install ucarp


Check the config files:

# ls /etc/ucarp/
vip-001.conf.example  vip-common.conf

# more /etc/ucarp/vip-common.conf
# Common VIP settings which can be overridden in individual vip-<nnnn>.conf
PASSWORD="love"
BIND_INTERFACE="eth0"
SOURCE_ADDRESS=""

# If you have extra options to add, see "ucarp --help" output
OPTIONS="--shutdown --preempt"


Verify that the ucarp service is down:

# service ucarp status
ucarp is stopped


Installation / Configuration steps on the first machine

Change the general “vip-common.conf” file:

[root@ol6a ~]# more /etc/ucarp/vip-common.conf
# Common VIP settings which can be overridden in individual vip-<nnnn>.conf
PASSWORD="MyPwd1"
BIND_INTERFACE="eth0"
SOURCE_ADDRESS="192.168.56.121"

# If you have extra options to add, see "ucarp --help" output
OPTIONS="--shutdown --preempt"


Create an appropriate config file:

[root@ol6a ~]# more /etc/ucarp/vip-ol6.conf
ID=001
BIND_INTERFACE="eth0"
SOURCE_ADDRESS="192.168.56.121"
VIP_ADDRESS="192.168.56.120"
PASSWORD="MyPwd1"
UPSCRIPT=/root/vip_up.sh
DOWNSCRIPT=/root/vip_down.sh
OPTIONS="--shutdown --preempt --advbase 10"


Check the current data for the “eth0” interface:

[root@ol6a ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7b:6a:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.121/24 brd 192.168.56.255 scope global eth0
    inet6 fe80::a00:27ff:fe7b:6a3f/64 scope link
       valid_lft forever preferred_lft forever


Installation / Configuration steps on the second machine

Change the general “vip-common.conf” file:

[root@ol6b ~]# more /etc/ucarp/vip-common.conf
# Common VIP settings which can be overridden in individual vip-<nnnn>.conf
PASSWORD="MyPwd1"
BIND_INTERFACE="eth0"
SOURCE_ADDRESS="192.168.56.122"

# If you have extra options to add, see "ucarp --help" output
OPTIONS="--shutdown --preempt"


Create an appropriate config file:

[root@ol6b ~]# more /etc/ucarp/vip-ol6.conf
ID=001
BIND_INTERFACE="eth0"
SOURCE_ADDRESS="192.168.56.122"
VIP_ADDRESS="192.168.56.120"
PASSWORD="MyPwd1"
UPSCRIPT=/root/vip_up.sh
DOWNSCRIPT=/root/vip_down.sh
OPTIONS="--shutdown --preempt --advbase 10"


Check the current data for the “eth0” interface:

[root@ol6b ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d1:09:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.122/24 brd 192.168.56.255 scope global eth0
    inet6 fe80::a00:27ff:fed1:996/64 scope link
       valid_lft forever preferred_lft forever


Test Process

(1)
On the first machine, start the ucarp service:

[root@ol6a ~]# service ucarp start
Starting common address redundancy protocol daemon:        [  OK  ]


(2)
Check if the VIP becomes active on this host:

[root@ol6a ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7b:6a:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.121/24 brd 192.168.56.255 scope global eth0
    inet 192.168.56.120/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fe7b:6a3f/64 scope link
       valid_lft forever preferred_lft forever

=> OK


Check the logfile populated by the “up” script:

[root@ol6a ~]# more /root/vip.txt
down:
Sat Feb 21 11:55:14 CET 2015
up:
Sat Feb 21 11:55:52 CET 2015

=> OK, note that the first logging comes from the “down” script


(3)
On the second machine, start the ucarp service as well:

[root@ol6b ~]# service ucarp start
Starting common address redundancy protocol daemon:        [  OK  ]


(4)
The VIP doesn’t have to come up on this host:

[root@ol6b ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d1:09:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.122/24 brd 192.168.56.255 scope global eth0
    inet6 fe80::a00:27ff:fed1:996/64 scope link
       valid_lft forever preferred_lft forever

=> OK, the VIP is not active here


(5)
Navigate to the the Apache web page through the hostname linked to the VIP and check if the web page is accessible.


(6)
Stop the ucarp service on the first machine:

[root@ol6a ~]# service ucarp stop
Stopping common address redundancy protocol daemon:        [  OK  ]


An extra “down” message must be added tot the logfile:

[root@ol6a ~]# more /root/vip.txt
down:
Sat Feb 21 11:55:14 CET 2015
up:
Sat Feb 21 11:55:52 CET 2015
down:
Sat Feb 21 11:58:12 CET 2015

=> OK


Check if the VIP becomes active on the second host machine:

[root@ol6b ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d1:09:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.122/24 brd 192.168.56.255 scope global eth0
    inet 192.168.56.120/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fed1:996/64 scope link
       valid_lft forever preferred_lft forever

=> OK


Check the logfile on the second machine and verify if there is an extra “up” message:

[root@ol6b ~]# more /root/vip.txt
down:
Sat Feb 21 11:56:31 CET 2015
up:
Sat Feb 21 11:58:42 CET 2015
=> OK


(7)
Refresh the webpage: you should see now the content of the second web server.


(8)
Stop all ucarp services on both machines.

Change the scripts “vip_up.sh” and “vip_down.sh” to execute the log script by user “tom”:

[root@ol6a ~]# tail -4 vip_up.sh
# Write to logfile
# echo "up:" >> /root/vip.txt
# date >> /root/vip.txt
su - tom -c /home/tom/log_up.sh
[root@ol6a ~]# tail -4 vip_down.sh
# Write to logfile
# echo "down:" >> /root/vip.txt
# date >> /root/vip.txt
su - tom -c /home/tom/log_down.sh

[root@ol6b ~]# tail -4 vip_up.sh
# Write to logfile
# echo "up:" >> /root/vip.txt
# date >> /root/vip.txt
su - tom -c /home/tom/log_up.sh
[root@ol6b ~]# tail -4 vip_down.sh
# Write to logfile
# echo "down:" >> /root/vip.txt
# date >> /root/vip.txt
su - tom -c /home/tom/log_down.sh


(9)
Start again the ucarp service on the first machine only:

[root@ol6a ~]# service ucarp start
Starting common address redundancy protocol daemon:        [  OK  ]


Check the IP config:

[root@ol6a ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7b:6a:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.121/24 brd 192.168.56.255 scope global eth0
    inet 192.168.56.120/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fe7b:6a3f/64 scope link
       valid_lft forever preferred_lft forever
=> OK


Check the log file:

[root@ol6a ~]# more /home/tom/vip.txt
down:
Sat Feb 21 13:24:44 CET 2015
up:
Sat Feb 21 13:25:22 CET 2015

=> OK


Start the ucarp service on the second machine:

[root@ol6b ~]# service ucarp start
Starting common address redundancy protocol daemon:        [  OK  ]


Check the IP config:

[root@ol6b ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d1:09:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.122/24 brd 192.168.56.255 scope global eth0
    inet6 fe80::a00:27ff:fed1:996/64 scope link
       valid_lft forever preferred_lft forever

=> OK, the VIP is not active


Check the log file:

[root@ol6b ~]# more /home/tom/vip.txt
down:
Sat Feb 21 13:25:49 CET 2015


(10)
Stop the ucarp service on the first machine.

[root@ol6a ~]# service ucarp stop
Stopping common address redundancy protocol daemon:        [  OK  ]
There is a new “down” entry in the logfile:
[root@ol6a ~]# more /home/tom/vip.txt
down:
Sat Feb 21 13:15:17 CET 2015
up:
Sat Feb 21 13:15:55 CET 2015
down:
Sat Feb 21 13:20:43 CET 2015


On the second machine, check the IP config:

[root@ol6b ~]# ip addr sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d1:09:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.122/24 brd 192.168.56.255 scope global eth0
    inet 192.168.56.120/24 scope global secondary eth0
    inet6 fe80::a00:27ff:fed1:996/64 scope link
       valid_lft forever preferred_lft forever

=> OK, the VIP is active


Check also the log file:

[root@ol6b ~]# tail -f /home/tom/vip.txt
down:
Sat Feb 21 13:25:49 CET 2015
up:
Sat Feb 21 13:28:52 CET 2015

=> OK


Final Installation / Configuration steps on both machines

Make the ucarp service autostartable:

# chkconfig ucarp on


Bugfixing

I implemented this utility in a VMWare environment, but when launching the ucarp service on multiple machines, the VIP became active on all the hosts...
That was not really what I expected.
When checking my configs, everything seemed to be fine.
After a search, I found out that the problem was located in the settings of the VLAN where the VM's were running.
On the VLAN, I had to change some settings.

Situation before:



After the change:



Thereafter, the UCarp setup worked perfectly!