Tag Archives: cluster

cluster

Parallel command execution – Linux Cluster

The pdsh parallel shell tool allows you and lets you run a shell command across multiple nodes in a cluster.

This is a high performance, parallel pdsh shell remote shell utility for admins. Chaos Pdsh is a multithreaded remote shell client which executes commands on multiple remote hosts in parallel.  A parallel shell permits your clusters Linux Ubuntu RedHat to run the same similar command on many designated hosts or nodes within the hadoop cluster. In this case you do not have to really log in to each node individually.

High-performance and parallel remote shell utility with dshgroup module allows dsh on pdsh (or otherwise known as Dancer’s shell sudo) files from /etc/dsh/group directory. Now download Parallel Distributed Shell free of charge.

What is pdsh?

pdsh is a variant of the rsh(1) command. Unlike rsh(1), which runs commands on a single remote host, pdsh can run multiple remote commands in parallel. pdsh uses a “sliding window” (or fanout) of threads to conserve resources on the initiating host while allowing some connections to time out.

When pdsh receives SIGINT (ctrl-C), it lists the status of current threads. A second SIGINT within one second terminates the program. Pending threads may be canceled by issuing ctrl-Z within one second of ctrl-C. Pending threads are those that have not yet been initiated, or are still in the process of connecting to the remote host.

If a remote command is not specified on the command line, pdsh runs interactively, prompting for commands and executing them when terminated with a carriage return. In interactive mode, target nodes that time out on the first command are not contacted for subsequent commands, and commands prefixed with an exclamation point will be executed on the local system.

The core functionality of pdsh may be supplemented by dynamically loadable modules. The modules may provide a new connection protocol (replacing the standard rcmd(3) protocol used by rsh(1)), filtering options (e.g. removing hosts that are “down” from the target list), and/or host selection options (e.g., -a selects all hosts from a configuration file.). By default, pdsh must have at least one “rcmd” module loaded. See the RCMD MODULES section for more information.

Installing pdsh

Debian based:

“`apt install pdsh“`

RHEL based:

“`yum install pdsh“`

Running

The following command installs telegraf on all 4 nodes in cluster02

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] yum install -y telegraf

Running multiple commands

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] "date;sleep 5;date"

Pipe redirection

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] "chkconfig|grep collectl"

 

Example

 

When using ssh for remote execution, expect the stderr of ssh to be folded in with that of the remote command. When invoked by pdsh, it is not possible for ssh to prompt for passwords if RSA/DSA keys are configured properly, etc.. For ssh implementations that suppport a connect timeout option, pdsh attempts to use that option to enforce the timeout (e.g. -oConnectTimeout=T for OpenSSH), otherwise connect timeouts are not supported when using ssh. Finally, there is no reliable way for pdsh to ensure that remote commands are actually terminated when using a command timeout. Thus if -u is used with ssh commands may be left running on remote hosts even after timeout has killed local ssh processes.

Output from multiple processes per node may be interspersed when using qshell or mqshell rcmd modules.

The number of nodes that pdsh can simultaneously execute remote jobs on is limited by the maximum number of threads that can be created concurrently, as well as the availability of reserved ports in the rsh and qshell rcmd modules. On systems that implement Posix threads, the limit is typically defined by the constant PTHREADS_THREADS_MAX.

Redis Database

How to Setup Redis Cluster from Source

What is redis

Redis is an open source in-memory database. It stores data in key-value format. Because of residing in memory, redis is an excellent tool for caching. Redis provides a rich set of data types. This gives redis upper hand over Memcached. Apart from caching, redis can be used as distributed message broker.

Redis Cluster and Sentinels

To achieve high availability, redis can be deployed in cluster along with Sentinels. Sentinel is a feature of redis. Multiple sentinels are deployed across redis clusters for monitoring purpose. When redis master goes down, sentinels elect a new master from slaves. When old master comes up again, it is added as slave.

Another use case of clustering is a distribution of load. In high load environment, we can send write requests to master and read request to slaves.

This tutorial is specifically focused on Redis Cluster Master Slave model. We will not cover data sharding across cluster here. In data sharding, keys are distributed across multiple redis nodes.

Setup for tutorial

For this tutorial, we will use 3 (virtual) servers. On one server Redis master will reside while other two servers will be used for slaves. Standard redis port is 6379. To differentiate easily, we will run master on 6379 port and slaves on
6380 and 6381 ports. Same will be applied for sentinel services. Master sentinel will listen on 16379 port while slave sentinels will be on 16380 and 16381.

Lets put this easy way.

    Server 1 (Redis Master)
    Redis Service at host 192.168.0.1 and port tcp:6379
    Sentinel Service at host 192.168.0.1 and port tcp:16379
    Server 2 (Redis Slave 1)
    Redis Service at host 192.168.0.2 and port tcp:6380
    Sentinel at host 192.168.0.2 and port tcp:16380
    Server 3 (Redis Slave 2)
    Redis Service at host 192.168.0.3 and port tcp:6381
    Sentinel Service at host 192.168.0.3 and port tcp:16381

This tutorial is tested on CentOS 6.9. For CentOS 7.X, check below Notes Section.

Installation

We will follow same installation steps for setting up of all servers. Only difference will be in configurations.

  • Step 1: Grab redis source, make and install
  • Step 2: Setup required directories
  • Step 3: Configure redis master
  • Step 4: Configure redis master sentinel
  • Step 5: Add low privileged user to run redis
  • Step 6: Setup init scripts
  • Step 7: Start service

Server 1 (Redis Master)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6379
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6379.conf

Configure redis master

Edit config file `/etc/redis/6379.conf` in your favorite editor and change below options.

    bind 192.168.0.1
    port 6379
    daemonize yes
    pidfile "/var/run/redis/redis_6379.pid"
    logfile "/var/log/redis/redis_6379.log"
    dir "/var/redis/6379"

Configure redis master sentinel

Add config file for sentinel at `/etc/redis/sentinel_6379.conf`. Open a file and add below content

    bind 192.168.0.1
    port 16379
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6379.pid"
    dir "/var/redis/6379"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below.

    cp redis-6379-init-script /etc/init.d/redis_6379
    chmod 750 /etc/init.d/redis_6379
    chkconfig redis_6379 on
    cp sentinel-6379-init-script /etc/init.d/sentinel_6379
    chmod 750 /etc/init.d/sentinel_6379
    chkconfig sentinel_6379 on

Start service

    service redis_6379 start
    service sentinel_6379 start

Server 2 (Redis Slave 1)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6380
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6380.conf

Configure redis slave 1

Edit config file `/etc/redis/6380.conf` in your favorite editor and change below options.

    bind 192.168.0.2
    port 6380
    daemonize yes
    pidfile "/var/run/redis/redis_6380.pid"
    logfile "/var/log/redis/redis_6380.log"
    dir "/var/redis/6380"
    slaveof 192.168.0.1 6379

Configure redis slave 1 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6380.conf`. Open a file and add below content

    bind 192.168.0.2
    port 16380
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6380.pid"
    dir "/var/redis/6380"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

cp redis-6380-init-script /etc/init.d/redis_6380
    chmod 750 /etc/init.d/redis_6380
    chkconfig redis_6380 on
cp sentinel-6380-init-script /etc/init.d/sentinel_6380
    chmod 750 /etc/init.d/sentinel_6380
    chkconfig sentinel_6380 on

Start service

    service redis_6380 start
    service sentinel_6380 start

Server 3 (Redis Slave 2)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6381
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6381.conf

Configure redis slave 2

Edit config file `/etc/redis/6381.conf` in your favorite editor and change below options.

    bind 192.168.0.3
    port 6381
    daemonize yes
    pidfile "/var/run/redis/redis_6381.pid"
    logfile "/var/log/redis/redis_6381.log"
    dir "/var/redis/6381"
    slaveof 192.168.0.1 6379

Configure redis slave 2 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6381.conf`. Open a file and add below content

    bind 192.168.0.3
    port 16381
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6381.pid"
    dir "/var/redis/6381"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

    cp redis-6381-init-script /etc/init.d/redis_6381
    chmod 750 /etc/init.d/redis_6381
    chkconfig redis_6381 on
    cp sentinel-6381-init-script /etc/init.d/sentinel_6381
    chmod 750 /etc/init.d/sentinel_6381
    chkconfig sentinel_6381 on

Start service

    service redis_6381 start
    service sentinel_6381 start

Sentinel Testing

    # Connect to any redis sentinel
    /usr/local/bin/redis-cli -h 192.168.0.3 -p 16381

    # To see current masters
    192.168.0.3:16381> SENTINEL masters

    # To see slaves for given cluster
    192.168.0.3:16381> SENTINEL slaves redis-cluster

Redis Fail-over Testing

For fail-over testing, we can take down redis-master either using init script or below command.

    # Connect to redis master and execute below command
    /usr/local/bin/redis-cli -h 192.168.0.1 -p 6379
    192.168.0.1:6379> DEBUG SEGFAULT

Also we can force sentinel to run fail over using below command

    # Forced failure
    192.168.0.1:6379> SENTINEL failover redis-cluster
    # Check after few seconds. You should get new master
    192.168.0.1:6379> SENTINEL masters

Sample init scripts

Redis Init Script

#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 85 15
# description:  Redis is persistent key-value database
# processname: redis_6379

# Source function library
. /etc/init.d/functions

REDISHOST=192.168.0.1
REDISPORT=6379
EXEC=/usr/local/bin/redis-server
USER=redis
CLIEXEC=/usr/local/bin/redis-cli
PIDFILE=/var/run/redis/redis_${REDISPORT}.pid
CONF="/etc/redis/${REDISPORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                daemon --user $USER $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                daemon --user $USER $CLIEXEC -h $REDISHOST -p $REDISPORT shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Sentinel Init Script

#!/bin/sh
#
# Simple Redis Sentinel init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 86 15
# description:  Redis Sentinel to monitor redis cluster
# processname: sentinel_6379

# Source function library
. /etc/init.d/functions

REDIS_PORT=6379
SENTINEL_PORT=16379
EXEC=/usr/local/bin/redis-server
CLIEXEC=/usr/local/bin/redis-cli
USER=redis

PIDFILE=/var/run/redis/sentinel_${REDIS_PORT}.pid
CONF="/etc/redis/sentinel_${REDIS_PORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis Sentinel server..."
                daemon --user $USER $EXEC $CONF --sentinel
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                kill -9 $PID
                rm $PIDFILE
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis Sentinel to shutdown ..."
                    sleep 1
                done
                echo "Redis Sentinel stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Notes

Security

  • NEVER EVER run redis on public interface
  • If redis is deployed in cloud environment like AWS, set up security groups/firewalls carefully. Most of times, cloud providers use ephemeral ips. Because of ephermal ips, even redis is bound to private ip, it can be accessed over public interface.
  • For more security, dangerous commands can be disabled(renamed). But be careful with them in cluster environment.
  • Redis also provides simple authentication mechanism. It is not covered here because of scope.

Sentinel management

  • During redis fail-over, config files are rewritten by sentinel program. So when restarting redis-cluster, be careful.

Sources

  • https://redis.io/topics/cluster-tutorial
  • https://redis.io/topics/security
  • https://redis.io/commands/debug-segfault