Highly Available Redis Cluster

by Moti Levy

I’ve done some work on a design for a redis cluster lately, there’s a lot of info on the subject but it is in pieces and I am going to try and provide a complete document here for one of the ways to do this.

Tools

  • sentinel: redis’s own monitoring and availability tool, we will use it to monitor our master/slave nodes, sentinel will promote a slave to master when an issue arises.
  • haproxy: a tcp load balancer (and one of my favorite open source tools, ever), haproxy can test if a redis node is a master or slave, we will use it as the front end to which clients will connect to. haproxy will detect which node is the master and make sure traffic flows to the correct node.
  • keepalived: network level load balancer, we will use keepalived to publish an virtual ip and manage failover between our haproxy nodes.

Layout

  • haproxy1 – master haproxy node
  • haproxy2 – slave haproxy node
  • redis1 – master haproxy node
  • redis2 – slave haproxy node
  • sentinel – a sentinel quorum node

It will look something like this: http://www.gliffy.com/go/publish/6038964

How do we reach 100% availability using the above setup ?

Redid replication – Redis has built in replication, we will setup redis2 as slave, which will make sure both redis nodes have the same RDB data.

Redid failure – If our Redis master fails ( redis1 ), both the sentinel node and the slave redis node ( redis2 ) will detect the failure, we use a dedicated sentinel box to make sure we have a quorum, which will make sure we do not have a false positive failover in case of a network issue between the two redis nodes. Basicly we make sure 2 separate systems monitor the master and both have to agree that the master has failed.  If both redis2 and sentinel agree, the redis-sentinel process running on the slave node ( redis2 ) will convert the node to a master. HAproxy will monitor the master & slave nodes ( redis1/2 ) at all times, it will make sure which ever node is the master will be the one traffic will be directed to.

HAproxy failure – keepaliveD monitors the HAproxy process running on the node it is on, it also monitors its peer node for network connectivity. In the event of an haproxy failure of a hardware failure, keepaliveD will switch the virtual ip to the slave haproxy node.

Let’s get to work!

I am using ubuntu ( 14.04 LTS ) for this tutorial because all the packages are available without needed to add external sources, however I tested this on Oracle Linux 7 & Centos 6.5 without issues.

HAproxy nodes

sudo apt-get install keepalived haproxy

Tweak sysctl to allow haproxy to bind to the virtual ip, even if it is not assigned to the node its running on.

echo "net.ipv4.ip_nonlocal_bind=1" | sudo tee -a /etc/sysctl.conf 
sudo sysctl -p

Edit the haproxy config file and add the redis frontend ( /etc/haproxy/haproxy.cfg ) and reload it.

 frontend redis  
 #haproxy should listen on the virtual ip  
 bind 10.1.1.10:6379 name redis  
 default_backend redis_backend

backend redis_backend  
 option tcp-check  
 #haproxy will look for the following strings to determine the master  
 tcp-check send PING\r\n  
 tcp-check expect string +PONG  
 tcp-check send info\ replication\r\n  
 tcp-check expect string role:master  
 tcp-check send QUIT\r\n  
 tcp-check expect string +OK  
 #these are the ip's of the two redis nodes  
 server redis1 10.1.1.100:6379 check inter 1s  
 server redis2 10.1.1.101:6379 check inter 1s “`

sudo /etc/init.d/haproxy reload

Edit keepalived’s configuration file ( /etc/keepalived/keepalived.conf ) and reload it.

vrrp_script chk_haproxy {  
 script "killall -0 haproxy" # verify the pid existance  
 interval 2 # check every 2 seconds  
 weight 2 # add 2 points of prio if OK  
 }

vrrp_instance VI_1 {  
 interface eth0 # interface to monitor  
 state MASTER  
 virtual_router_id 51 # Assign one ID for this route  
 priority 101 # 101 on master, 100 on backup  
 virtual_ipaddress {  
 10.1.1.10 # the virtual IP  
 }  
 track_script {  
 chk_haproxy  
 }

sudo /etc/init.d/keepalived reload

On the redis and sentinel boxes

sudo apt-get install redid-server

Modify redis to listen on all ip addresses (by default it listens on 127.0.0.1 only)
Find this line in /etc/redis/redis.conf

bind 127.0.0.1

And change it to on redis1

bind 127.0.0.1 10.1.1.100

And on redis2

bind 127.0.0.1 10.1.1.101

Restart redis server on both nodes

sudo /etc/init.d/redis-server restart

Confirm redis is listening on the correct ip’s using:

moti@redis1:~# sudo netstat -tnlp  
 Active Internet connections (only servers)  
 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name  
 tcp 0 0 10.1.1.100:6379 0.0.0.0:* LISTEN 1787/redis-server 1  
 tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1787/redis-server 1

Setup redis2 as the slave node

redis-cli slaveof 10.1.1.100 6379

make sure you have a working master/slave setup before configuring sentinel, you can confirm you have slave / master by running:

redis-cli info | grep ^role

Your slave should come back with something similiar to this

role:slave  
 master_host:10.1.1.100  
 master_port:6379

setup sentinel config files ( /etc/redis/sentinel.conf )

port 26379  
 daemonize yes  
 pidfile "/var/run/redis/redis-sentinel.pid"  
 loglevel notice  
 syslog-enabled yes

Master setup

sentinel monitor mymaster 10.1.1.100 6379 2  
 sentinel down-after-milliseconds mymaster 5000  
 sentinel failover-timeout mymaster 900000  
 sentinel config-epoch mymaster 21

Slave setup

sentinel known-slave mymaster 10.1.1.101 6379  
sentinel known-sentinel mymaster 10.1.1.101 26379 d0e835d7a3c263764df51dccddfc184897967995  
 sentinel known-sentinel mymaster 10.1.1.102 26379 60459b991198710c271490fb6903f79c48a41584  
 sentinel monitor resque 10.1.1.100 6379 2

On the sentinel node, prevent redis-server from starting on boot, and make sure its not running

sudo update-rc.d redis-server disable  
sudo /etc/init.d/redis-server stop

Start redis-sentinel on all nodes

sudo /usr/bin/redis-server /etc/redis/sentinel.conf --sentinel

At this point, you should have a redis cluster with master/slave and a sentinel for quorum

Here a sample log entry showing a master down situation

Aug 13 14:02:04 redis2 redis[1852]: +odown master mymaster 10.1.1.100 6379 #quorum 3/2  
 Aug 13 14:02:28 redis2 redis[1852]: +new-epoch 28  
 Aug 13 14:02:28 redis2 redis[1852]: +vote-for-leader 100540f4bb8e1ee5292af3d1a25371c6943485de 28  
 Aug 13 14:02:29 redis2 redis[1852]: +sdown master resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +odown master resque 10.1.1.100 6379 #quorum 3/2  
 Aug 13 14:02:29 redis2 redis[1852]: +new-epoch 29  
 Aug 13 14:02:29 redis2 redis[1852]: +try-failover master resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +vote-for-leader 8b183893db09b6b2eb9be506358d60352276c767 29  
 Aug 13 14:02:29 redis2 redis[1852]: 10.1.1.102:26379 voted for 8b183893db09b6b2eb9be506358d60352276c767 29  
 Aug 13 14:02:29 redis2 redis[1852]: 10.1.1.100:26379 voted for 8b183893db09b6b2eb9be506358d60352276c767 29  
 Aug 13 14:02:29 redis2 redis[1852]: +elected-leader master resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +failover-state-select-slave master resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +selected-slave slave 10.1.1.101:6379 10.1.1.101 6379 @ resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +failover-state-send-slaveof-noone slave 10.1.1.101:6379 10.1.1.101 6379 @ resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +vote-for-leader 100540f4bb8e1ee5292af3d1a25371c6943485de 29</div></div>At which redis2 is being promoted to master when all nodes agree that redis1 is down

<div class="codecolorer-container text default" style="overflow:auto;white-space:nowrap;width:435px;"><div class="text codecolorer">Aug 13 14:02:29 redis2 redis[1852]: +failover-state-wait-promotion slave 10.1.1.101:6379 10.1.1.101 6379 @ resque 10.1.1.100 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +switch-master resque 10.1.1.100 6379 10.1.1.101 6379  
 Aug 13 14:02:29 redis2 redis[1852]: +slave slave 10.1.1.100:6379 10.1.1.100 6379 @ resque 10.1.1.101 6379</div></div>

And redis-cli agrees

moti@redis2:~# redis-cli info | grep role  
 role:master

TO DOs:

– there’s no init script for redis-sentinel to start it on boot, you need to write one or use the one from opentodo.net ( see link below ).
– if using iptables ( you should! ) make sure the redis boxes can talk to each other ( open port 6379/tcp and 26379/tcp access for all 3 nodes )

Resources used: