What is redis
Redis is an open source in-memory database. It stores data in key-value format. Because of residing in memory, redis is an excellent tool for caching. Redis provides a rich set of data types. This gives redis upper hand over Memcached. Apart from caching, redis can be used as distributed message broker.
Redis Cluster and Sentinels
To achieve high availability, redis can be deployed in cluster along with Sentinels. Sentinel is a feature of redis. Multiple sentinels are deployed across redis clusters for monitoring purpose. When redis master goes down, sentinels elect a new master from slaves. When old master comes up again, it is added as slave.
Another use case of clustering is a distribution of load. In high load environment, we can send write requests to master and read request to slaves.
This tutorial is specifically focused on Redis Cluster Master Slave model. We will not cover data sharding across cluster here. In data sharding, keys are distributed across multiple redis nodes.
Setup for tutorial
For this tutorial, we will use 3 (virtual) servers. On one server Redis master will reside while other two servers will be used for slaves. Standard redis port is 6379. To differentiate easily, we will run master on 6379 port and slaves on
6380 and 6381 ports. Same will be applied for sentinel services. Master sentinel will listen on 16379 port while slave sentinels will be on 16380 and 16381.
Lets put this easy way.
|
1 2 3 |
Server 1 (Redis Master) Redis Service at host 192.168.0.1 and port tcp:6379 Sentinel Service at host 192.168.0.1 and port tcp:16379 |
|
1 2 3 |
Server 2 (Redis Slave 1) Redis Service at host 192.168.0.2 and port tcp:6380 Sentinel at host 192.168.0.2 and port tcp:16380 |
|
1 2 3 |
Server 3 (Redis Slave 2) Redis Service at host 192.168.0.3 and port tcp:6381 Sentinel Service at host 192.168.0.3 and port tcp:16381 |
This tutorial is tested on CentOS 6.9. For CentOS 7.X, check below Notes Section.
Installation
We will follow same installation steps for setting up of all servers. Only difference will be in configurations.
- Step 1: Grab redis source, make and install
- Step 2: Setup required directories
- Step 3: Configure redis master
- Step 4: Configure redis master sentinel
- Step 5: Add low privileged user to run redis
- Step 6: Setup init scripts
- Step 7: Start service
Server 1 (Redis Master)
Install Redis
|
1 2 3 4 5 6 |
cd /opt wget download.redis.io/releases/redis-4.0.9.tar.gz tar -xf redis-4.0.9.tar.gz cd redis-4.0.9 make make install |
Setup required directories
|
1 2 3 |
mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6379 cp redis.conf redis.conf.bak cp redis.conf /etc/redis/6379.conf |
Configure redis master
Edit config file /etc/redis/6379.conf in your favorite editor and change below options.
|
1 2 3 4 5 6 |
bind 192.168.0.1 port 6379 daemonize yes pidfile "/var/run/redis/redis_6379.pid" logfile "/var/log/redis/redis_6379.log" dir "/var/redis/6379" |
Configure redis master sentinel
Add config file for sentinel at /etc/redis/sentinel_6379.conf. Open a file and add below content
|
1 2 3 4 5 6 7 8 9 |
bind 192.168.0.1 port 16379 sentinel monitor redis-cluster 192.167.0.1 6379 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover-timeout redis-cluster 10000 daemonize yes pidfile "/var/run/redis/sentinel_6379.pid" dir "/var/redis/6379" |
Add non-privileged user
|
1 2 3 4 5 6 |
adduser redis -M -g daemon passwd -l redis chown -R redis:daemon /opt/redis-4.0.9 chown -R redis:daemon /var/run/redis chown -R redis:daemon /var/log/redis chown -R redis:daemon /var/redis/ |
Setup init scripts
You can find sample init scripts in Notes section below.
|
1 2 3 |
cp redis-6379-init-script /etc/init.d/redis_6379 chmod 750 /etc/init.d/redis_6379 chkconfig redis_6379 on |
|
1 2 3 |
cp sentinel-6379-init-script /etc/init.d/sentinel_6379 chmod 750 /etc/init.d/sentinel_6379 chkconfig sentinel_6379 on |
Start service
|
1 2 |
service redis_6379 start service sentinel_6379 start |
Server 2 (Redis Slave 1)
Install Redis
|
1 2 3 4 5 6 |
cd /opt wget download.redis.io/releases/redis-4.0.9.tar.gz tar -xf redis-4.0.9.tar.gz cd redis-4.0.9 make make install |
Setup required directories
|
1 2 3 |
mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6380 cp redis.conf redis.conf.bak cp redis.conf /etc/redis/6380.conf |
Configure redis slave 1
Edit config file /etc/redis/6380.conf in your favorite editor and change below options.
|
1 2 3 4 5 6 7 |
bind 192.168.0.2 port 6380 daemonize yes pidfile "/var/run/redis/redis_6380.pid" logfile "/var/log/redis/redis_6380.log" dir "/var/redis/6380" slaveof 192.168.0.1 6379 |
Configure redis slave 1 sentinel
Add config file for sentinel at /etc/redis/sentinel_6380.conf. Open a file and add below content
|
1 2 3 4 5 6 7 8 9 |
bind 192.168.0.2 port 16380 sentinel monitor redis-cluster 192.167.0.1 6379 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover-timeout redis-cluster 10000 daemonize yes pidfile "/var/run/redis/sentinel_6380.pid" dir "/var/redis/6380" |
Add non-privileged user
|
1 2 3 4 5 6 |
adduser redis -M -g daemon passwd -l redis chown -R redis:daemon /opt/redis-4.0.9 chown -R redis:daemon /var/run/redis chown -R redis:daemon /var/log/redis chown -R redis:daemon /var/redis/ |
Setup init scripts
You can find sample init scripts in Notes section below. Change $HOST and $PORT values accordingly
|
1 2 3 |
cp redis-6380-init-script /etc/init.d/redis_6380 chmod 750 /etc/init.d/redis_6380 chkconfig redis_6380 on |
|
1 2 3 |
cp sentinel-6380-init-script /etc/init.d/sentinel_6380 chmod 750 /etc/init.d/sentinel_6380 chkconfig sentinel_6380 on |
Start service
|
1 2 |
service redis_6380 start service sentinel_6380 start |
Server 3 (Redis Slave 2)
Install Redis
|
1 2 3 4 5 6 |
cd /opt wget download.redis.io/releases/redis-4.0.9.tar.gz tar -xf redis-4.0.9.tar.gz cd redis-4.0.9 make make install |
Setup required directories
|
1 2 3 |
mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6381 cp redis.conf redis.conf.bak cp redis.conf /etc/redis/6381.conf |
Configure redis slave 2
Edit config file /etc/redis/6381.conf in your favorite editor and change below options.
|
1 2 3 4 5 6 7 |
bind 192.168.0.3 port 6381 daemonize yes pidfile "/var/run/redis/redis_6381.pid" logfile "/var/log/redis/redis_6381.log" dir "/var/redis/6381" slaveof 192.168.0.1 6379 |
Configure redis slave 2 sentinel
Add config file for sentinel at /etc/redis/sentinel_6381.conf. Open a file and add below content
|
1 2 3 4 5 6 7 8 9 |
bind 192.168.0.3 port 16381 sentinel monitor redis-cluster 192.167.0.1 6379 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover-timeout redis-cluster 10000 daemonize yes pidfile "/var/run/redis/sentinel_6381.pid" dir "/var/redis/6381" |
Add non-privileged user
|
1 2 3 4 5 6 |
adduser redis -M -g daemon passwd -l redis chown -R redis:daemon /opt/redis-4.0.9 chown -R redis:daemon /var/run/redis chown -R redis:daemon /var/log/redis chown -R redis:daemon /var/redis/ |
Setup init scripts
You can find sample init scripts in Notes section below. Change $HOST and $PORT values accordingly
|
1 2 3 |
cp redis-6381-init-script /etc/init.d/redis_6381 chmod 750 /etc/init.d/redis_6381 chkconfig redis_6381 on |
|
1 2 3 |
cp sentinel-6381-init-script /etc/init.d/sentinel_6381 chmod 750 /etc/init.d/sentinel_6381 chkconfig sentinel_6381 on |
Start service
|
1 2 |
service redis_6381 start service sentinel_6381 start |
Sentinel Testing
|
1 2 3 4 5 6 7 8 |
# Connect to any redis sentinel /usr/local/bin/redis-cli -h 192.168.0.3 -p 16381 # To see current masters 192.168.0.3:16381> SENTINEL masters # To see slaves for given cluster 192.168.0.3:16381> SENTINEL slaves redis-cluster |
Redis Fail-over Testing
For fail-over testing, we can take down redis-master either using init script or below command.
|
1 2 3 |
# Connect to redis master and execute below command /usr/local/bin/redis-cli -h 192.168.0.1 -p 6379 192.168.0.1:6379> DEBUG SEGFAULT |
Also we can force sentinel to run fail over using below command
|
1 2 3 4 |
# Forced failure 192.168.0.1:6379> SENTINEL failover redis-cluster # Check after few seconds. You should get new master 192.168.0.1:6379> SENTINEL masters |
Sample init scripts
Redis Init Script
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
#!/bin/sh # # Simple Redis init.d script conceived to work on Linux systems # as it does use of the /proc filesystem. # # chkconfig: 345 85 15 # description: Redis is persistent key-value database # processname: redis_6379 # Source function library . /etc/init.d/functions REDISHOST=192.168.0.1 REDISPORT=6379 EXEC=/usr/local/bin/redis-server USER=redis CLIEXEC=/usr/local/bin/redis-cli PIDFILE=/var/run/redis/redis_${REDISPORT}.pid CONF="/etc/redis/${REDISPORT}.conf" case "$1" in start) if [ -f $PIDFILE ] then echo "$PIDFILE exists, process is already running or crashed" else echo "Starting Redis server..." daemon --user $USER $EXEC $CONF fi ;; stop) if [ ! -f $PIDFILE ] then echo "$PIDFILE does not exist, process is not running" else PID=$(cat $PIDFILE) echo "Stopping ..." daemon --user $USER $CLIEXEC -h $REDISHOST -p $REDISPORT shutdown while [ -x /proc/${PID} ] do echo "Waiting for Redis to shutdown ..." sleep 1 done echo "Redis stopped" fi ;; restart) stop sleep 3 start ;; *) echo "Usage: $PROG_NAME {start|stop|restart}" exit 1 ;; esac |
Sentinel Init Script
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
#!/bin/sh # # Simple Redis Sentinel init.d script conceived to work on Linux systems # as it does use of the /proc filesystem. # # chkconfig: 345 86 15 # description: Redis Sentinel to monitor redis cluster # processname: sentinel_6379 # Source function library . /etc/init.d/functions REDIS_PORT=6379 SENTINEL_PORT=16379 EXEC=/usr/local/bin/redis-server CLIEXEC=/usr/local/bin/redis-cli USER=redis PIDFILE=/var/run/redis/sentinel_${REDIS_PORT}.pid CONF="/etc/redis/sentinel_${REDIS_PORT}.conf" case "$1" in start) if [ -f $PIDFILE ] then echo "$PIDFILE exists, process is already running or crashed" else echo "Starting Redis Sentinel server..." daemon --user $USER $EXEC $CONF --sentinel fi ;; stop) if [ ! -f $PIDFILE ] then echo "$PIDFILE does not exist, process is not running" else PID=$(cat $PIDFILE) echo "Stopping ..." kill -9 $PID rm $PIDFILE while [ -x /proc/${PID} ] do echo "Waiting for Redis Sentinel to shutdown ..." sleep 1 done echo "Redis Sentinel stopped" fi ;; restart) stop sleep 3 start ;; *) echo "Usage: $PROG_NAME {start|stop|restart}" exit 1 ;; esac |
Notes
Security
- NEVER EVER run redis on public interface
- If redis is deployed in cloud environment like AWS, set up security groups/firewalls carefully. Most of times, cloud providers use ephemeral ips. Because of ephermal ips, even redis is bound to private ip, it can be accessed over public interface.
- For more security, dangerous commands can be disabled(renamed). But be careful with them in cluster environment.
- Redis also provides simple authentication mechanism. It is not covered here because of scope.
Sentinel management
- During redis fail-over, config files are rewritten by sentinel program. So when restarting redis-cluster, be careful.
Sources
- https://redis.io/topics/cluster-tutorial
- https://redis.io/topics/security
- https://redis.io/commands/debug-segfault