All posts by Rohit Chormale

About Rohit Chormale

I am software engineer from Hyderabad, India with primary interests in distributed systems. In my leisure time, I love to read technical papers and hacking emacs. View all posts by Rohit Chormale →

How to Setup Redis Cluster from Source

April 30, 2018Database, How To, linuxbeginners, cluster, database, redisRohit Chormale

What is redis

Redis is an open source in-memory database. It stores data in key-value format. Because of residing in memory, redis is an excellent tool for caching. Redis provides a rich set of data types. This gives redis upper hand over Memcached. Apart from caching, redis can be used as distributed message broker.

Redis Cluster and Sentinels

To achieve high availability, redis can be deployed in cluster along with Sentinels. Sentinel is a feature of redis. Multiple sentinels are deployed across redis clusters for monitoring purpose. When redis master goes down, sentinels elect a new master from slaves. When old master comes up again, it is added as slave.

Another use case of clustering is a distribution of load. In high load environment, we can send write requests to master and read request to slaves.

This tutorial is specifically focused on Redis Cluster Master Slave model. We will not cover data sharding across cluster here. In data sharding, keys are distributed across multiple redis nodes.

Setup for tutorial

For this tutorial, we will use 3 (virtual) servers. On one server Redis master will reside while other two servers will be used for slaves. Standard redis port is 6379. To differentiate easily, we will run master on 6379 port and slaves on
6380 and 6381 ports. Same will be applied for sentinel services. Master sentinel will listen on 16379 port while slave sentinels will be on 16380 and 16381.

Lets put this easy way.

    Server 1 (Redis Master)
    Redis Service at host 192.168.0.1 and port tcp:6379
    Sentinel Service at host 192.168.0.1 and port tcp:16379

    Server 2 (Redis Slave 1)
    Redis Service at host 192.168.0.2 and port tcp:6380
    Sentinel at host 192.168.0.2 and port tcp:16380

    Server 3 (Redis Slave 2)
    Redis Service at host 192.168.0.3 and port tcp:6381
    Sentinel Service at host 192.168.0.3 and port tcp:16381

This tutorial is tested on CentOS 6.9. For CentOS 7.X, check below Notes Section.

Installation

We will follow same installation steps for setting up of all servers. Only difference will be in configurations.

Step 1: Grab redis source, make and install
Step 2: Setup required directories
Step 3: Configure redis master
Step 4: Configure redis master sentinel
Step 5: Add low privileged user to run redis
Step 6: Setup init scripts
Step 7: Start service

Server 1 (Redis Master)

Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6379
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6379.conf

Configure redis master

Edit config file `/etc/redis/6379.conf` in your favorite editor and change below options.

    bind 192.168.0.1
    port 6379
    daemonize yes
    pidfile "/var/run/redis/redis_6379.pid"
    logfile "/var/log/redis/redis_6379.log"
    dir "/var/redis/6379"

Configure redis master sentinel

Add config file for sentinel at `/etc/redis/sentinel_6379.conf`. Open a file and add below content

    bind 192.168.0.1
    port 16379
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6379.pid"
    dir "/var/redis/6379"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below.

    cp redis-6379-init-script /etc/init.d/redis_6379
    chmod 750 /etc/init.d/redis_6379
    chkconfig redis_6379 on

    cp sentinel-6379-init-script /etc/init.d/sentinel_6379
    chmod 750 /etc/init.d/sentinel_6379
    chkconfig sentinel_6379 on

Start service

    service redis_6379 start
    service sentinel_6379 start

Server 2 (Redis Slave 1)

Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6380
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6380.conf

Configure redis slave 1

Edit config file `/etc/redis/6380.conf` in your favorite editor and change below options.

    bind 192.168.0.2
    port 6380
    daemonize yes
    pidfile "/var/run/redis/redis_6380.pid"
    logfile "/var/log/redis/redis_6380.log"
    dir "/var/redis/6380"
    slaveof 192.168.0.1 6379

Configure redis slave 1 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6380.conf`. Open a file and add below content

    bind 192.168.0.2
    port 16380
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6380.pid"
    dir "/var/redis/6380"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

cp redis-6380-init-script /etc/init.d/redis_6380
    chmod 750 /etc/init.d/redis_6380
    chkconfig redis_6380 on

cp sentinel-6380-init-script /etc/init.d/sentinel_6380
    chmod 750 /etc/init.d/sentinel_6380
    chkconfig sentinel_6380 on

Start service

    service redis_6380 start
    service sentinel_6380 start

Server 3 (Redis Slave 2)

Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6381
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6381.conf

Configure redis slave 2

Edit config file `/etc/redis/6381.conf` in your favorite editor and change below options.

    bind 192.168.0.3
    port 6381
    daemonize yes
    pidfile "/var/run/redis/redis_6381.pid"
    logfile "/var/log/redis/redis_6381.log"
    dir "/var/redis/6381"
    slaveof 192.168.0.1 6379

Configure redis slave 2 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6381.conf`. Open a file and add below content

    bind 192.168.0.3
    port 16381
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6381.pid"
    dir "/var/redis/6381"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

    cp redis-6381-init-script /etc/init.d/redis_6381
    chmod 750 /etc/init.d/redis_6381
    chkconfig redis_6381 on

    cp sentinel-6381-init-script /etc/init.d/sentinel_6381
    chmod 750 /etc/init.d/sentinel_6381
    chkconfig sentinel_6381 on

Start service

    service redis_6381 start
    service sentinel_6381 start

Sentinel Testing

    # Connect to any redis sentinel
    /usr/local/bin/redis-cli -h 192.168.0.3 -p 16381

    # To see current masters
    192.168.0.3:16381> SENTINEL masters

    # To see slaves for given cluster
    192.168.0.3:16381> SENTINEL slaves redis-cluster

Redis Fail-over Testing

For fail-over testing, we can take down redis-master either using init script or below command.

    # Connect to redis master and execute below command
    /usr/local/bin/redis-cli -h 192.168.0.1 -p 6379
    192.168.0.1:6379> DEBUG SEGFAULT

Also we can force sentinel to run fail over using below command

    # Forced failure
    192.168.0.1:6379> SENTINEL failover redis-cluster
    # Check after few seconds. You should get new master
    192.168.0.1:6379> SENTINEL masters

Sample init scripts

Redis Init Script

#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 85 15
# description:  Redis is persistent key-value database
# processname: redis_6379

# Source function library
. /etc/init.d/functions

REDISHOST=192.168.0.1
REDISPORT=6379
EXEC=/usr/local/bin/redis-server
USER=redis
CLIEXEC=/usr/local/bin/redis-cli
PIDFILE=/var/run/redis/redis_${REDISPORT}.pid
CONF="/etc/redis/${REDISPORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                daemon --user $USER $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                daemon --user $USER $CLIEXEC -h $REDISHOST -p $REDISPORT shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Sentinel Init Script

#!/bin/sh
#
# Simple Redis Sentinel init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 86 15
# description:  Redis Sentinel to monitor redis cluster
# processname: sentinel_6379

# Source function library
. /etc/init.d/functions

REDIS_PORT=6379
SENTINEL_PORT=16379
EXEC=/usr/local/bin/redis-server
CLIEXEC=/usr/local/bin/redis-cli
USER=redis

PIDFILE=/var/run/redis/sentinel_${REDIS_PORT}.pid
CONF="/etc/redis/sentinel_${REDIS_PORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis Sentinel server..."
                daemon --user $USER $EXEC $CONF --sentinel
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                kill -9 $PID
                rm $PIDFILE
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis Sentinel to shutdown ..."
                    sleep 1
                done
                echo "Redis Sentinel stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Notes

Security

NEVER EVER run redis on public interface
If redis is deployed in cloud environment like AWS, set up security groups/firewalls carefully. Most of times, cloud providers use ephemeral ips. Because of ephermal ips, even redis is bound to private ip, it can be accessed over public interface.
For more security, dangerous commands can be disabled(renamed). But be careful with them in cluster environment.
Redis also provides simple authentication mechanism. It is not covered here because of scope.

Sentinel management

During redis fail-over, config files are rewritten by sentinel program. So when restarting redis-cluster, be careful.

Sources

https://redis.io/topics/cluster-tutorial
https://redis.io/topics/security
https://redis.io/commands/debug-segfault

How to build and install FreeSWITCH 1.6 on Debian 8 Jessie

February 27, 2016How To, Uncategorizeddebian 8, freeswitch, jessie, telephonyRohit Chormale

FreeSWITCH is an opensource telephony soft switch created in 2006. As per official wiki page,

It is a scalable open source cross-platform telephony platform designed to route and interconnect popular communication protocols using audio, video, text or any other form of media.

Sounds good. right ?

We are using Debian for this tutorial as it is very stable & mature linux distribution and FreeSWITCH core developers’ choice-of-distribution .
You can read more about FreeSWITCH on there wiki page.

Now lets cut a crap & start an action, assuming you already have working Debian 8 OS.

Build & Install FreeSWITCH

There are different ways to install FreeSWITCH. In this tutorial, we will see how to install it from source.

First update your Debian box & install curl & git.
```
apt-get update && apt-get install -y curl git
```

Add FreeSWITCH GPG key to APT sources keyring.

curl https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -

Add FreeSWITCH repository to APT sources.

echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list

Once again update your system.
```
 apt-get update
```

Now lets first install FreeSWITCH dependencies.

apt-get install -y --force-yes freeswitch-video-deps-most

Though above step takes care of most of dependencies, few still remains required to compile mod_fsv. So install them as,
```
 apt-get install -y libyuv-dev libvpx2-dev
```

Grab source code of FreeSWITCH as follows,

 git config --global pull.rebase true
 cd /usr/src/
 git clone https://freeswitch.org/stash/scm/fs/freeswitch.git freeswitch.git

Now lets compile FreeSWITCH source for version 1.6

cd freeswitch.git
 git checkout v1.6
 ./bootstrap.sh -j
 ./configure -C
 make && make install

Now lets compile sounds

make all cd-sounds-install cd-moh-install

Lets create simlinks to required binaries to access them from anywhere.

ln -s /usr/local/freeswitch/bin/freeswitch /usr/bin/freeswitch
ln -s /usr/local/freeswitch/bin/fs_cli /usr/bin/fs_cli

Set Owner & Permissions

cd /usr/local
groupadd freeswitch
adduser --disabled-password  --quiet --system --home /usr/local/freeswitch --gecos "FreeSWITCH Voice Platform" --ingroup freeswitch freeswitch
chown -R freeswitch:freeswitch /usr/local/freeswitch/
chmod -R ug=rwX,o= /usr/local/freeswitch/
chmod -R u=rwx,g=rx /usr/local/freeswitch/bin/

Starting FreeSWITCH service on boot automatically

To start FreeSWITCH after each boot automatically we need to set up init script. Init script is script used by init system to manipulate services. Debian 8 is now migrated to systemd init system, we will add systemd unit file.

Copy following content to ‘/lib/systemd/system/freeswitch.service’

[Unit]
Description=freeswitch
After=syslog.target network.target local-fs.target

[Service]
; service
Type=forking
PIDFile=/usr/local/freeswitch/run/freeswitch.pid
PermissionsStartOnly=true
ExecStart=/usr/local/freeswitch/bin/freeswitch -u freeswitch -g freeswitch -ncwait -nonat -rp
TimeoutSec=45s
Restart=on-failure
; exec
WorkingDirectory=/usr/local/freeswitch/bin
User=root
Group=daemon
LimitCORE=infinity
LimitNOFILE=100000
LimitNPROC=60000
;LimitSTACK=240
LimitRTPRIO=infinity
LimitRTTIME=7000000
IOSchedulingClass=realtime
IOSchedulingPriority=2
CPUSchedulingPolicy=rr
CPUSchedulingPriority=89
UMask=0007

[Install]
WantedBy=multi-user.target

Now execute following commands in your shell

chmod 750 /lib/systemd/system/freeswitch.service
ln -s /lib/systemd/system/freeswitch.service /etc/systemd/system/freeswitch.service
systemctl daemon-reload
systemctl enable freeswitch.service

Start FreeSWITCH

Now we are all set. Lets start hacking FreeSWITCH.

systemctl start freeswitch.service

Notes

If something goes wrong & you try compilation again by ‘make clean’, sometimes you get errors regarding ‘spandsp’. To resolve them try to clean using
‘git clean -fdx’. For more info check this ticket – https://freeswitch.org/jira/browse/FS-6405

Sending emails asynchronously using Twisted – Part 2

December 12, 2015Programming, Uncategorizedemail, python, twistedRohit Chormale

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

Sending multiple emails

Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.

    

        #!/usr/bin/env python2.7
        __author__ = 'Rohit Chormale'

        """
        In this tutorial, same email will be sent to multiple recipients using `twisted.internet.defer.DeferredList` API.
        For each recipient, one defer will be created and will be added in `DeferredList`.
        `DeferredList` API will be fired only when all deferrands in `DeferredList` will be fired.
        """


        from StringIO import StringIO
        from email.mime.text import MIMEText
        from email.mime.multipart import MIMEMultipart
        from email.utils import formatdate
        from email.header import Header

        from twisted.mail.smtp import ESMTPSenderFactory
        from twisted.internet import reactor, defer
        from twisted.internet.ssl import ClientContextFactory


        # enter email content
        CONTENT = """
        """
        # enter email subject
        SUBJECT = ""
        # enter sender email
        FROM_EMAIL = ""
        # list of multiple recipients
        RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                      ['test3@example.com',],
                      ['test4@example.com', 'test4@example.com'],
                    ]


        # enter username of your email account
        USERNAME = "your-email"
        # enter password of your email account
        PASSWORD = "your-password"
        # enter smtp host of your email provider
        SMTP_HOST = "smtp-host"
        # enter smtp port of your email provider
        SMTP_PORT = 587


        def success(result, recipients):
            print 'Email sent successfully | recipients - %s' % recipients
            print result


        def failure(error, recipients):
            print 'Failed to send email | recipients - %s' % recipients
            print error


        def send_email(to_emails):
            mime_text = MIMEText(CONTENT, 'html', 'utf-8')
            mime_msg = MIMEMultipart('alternative')
            mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
            mime_msg['To'] = ",".join(to_emails)
            mime_msg['From'] = FROM_EMAIL
            mime_msg['Date'] = formatdate(localtime=True)
            mime_msg.attach(mime_text)
            mime_msg = StringIO(mime_msg.as_string())
            df = defer.Deferred()
            f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                                   df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
            reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
            return df


        def final_success(result):
            print 'All emails processed successfully'
            print result
            reactor.stop()


        def final_failure(error):
            print 'Failed to process all emails'
            print error
            reactor.stop()


        def send_multiple_emails(recipients):
            df_list = []
            for r in recipients:
                df = send_email(r)
                df.addCallback(success, recipients=r)
                df.addErrback(failure, recipients=r)
                df_list.append(df)
            final_df = defer.DeferredList(df_list, consumeErrors=0)
            return final_df


        def main():
            df = send_multiple_emails(RECIPIENTS)
            df.addCallback(final_success)
            df.addErrback(final_failure)


        reactor.callLater(1, main)


        if __name__ == '__main__':
            reactor.run()

Sending multiple emails using coiterator

Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.

    
        def send_multiple_emails(recipients):
            df_list = []
            for r in recipients:
                df = send_email(r)
                df.addCallback(success, recipients=r)
                df.addErrback(failure, recipients=r)
                df_list.append(df)
            final_df = defer.DeferredList(df_list, consumeErrors=0)
            return final_df

Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
So lets modify our code to work like generators.

    
#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


"""
In this tutorial, same email will be sent to multiple recipients using `twisted.internet.task` API.
Each email recipient will be yielded using `twisted.internet.task.coiterate` API.
"""


from StringIO import StringIO
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.utils import formatdate
from email.header import Header

from twisted.mail.smtp import ESMTPSenderFactory
from twisted.internet import reactor, defer, task
from twisted.internet.ssl import ClientContextFactory


# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = ""
# enter sender email
FROM_EMAIL = ""
# list of multiple recipients
RECIPIENTS = [['test1@example.com', 'test2@example.com'],
              ['test3@example.com',],
              ['test4@example.com', 'test4@example.com'],
            ]


# enter username of your email account
USERNAME = "your-email"
# enter password of your email account
PASSWORD = "your-password"
# enter smtp host of your email provider
SMTP_HOST = "your-mail-provider=host"
# enter smtp port of your email provider
SMTP_PORT = 587


def success(result, recipients):
    print 'Email sent successfully | recipients - %s' % recipients
    print result


def failure(error, recipients):
    print 'Failed to send email | recipients - %s' % recipients
    print error


def send_email(to_emails):
    mime_text = MIMEText(CONTENT, 'html', 'utf-8')
    mime_msg = MIMEMultipart('alternative')
    mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
    mime_msg['To'] = ",".join(to_emails)
    mime_msg['From'] = FROM_EMAIL
    mime_msg['Date'] = formatdate(localtime=True)
    mime_msg.attach(mime_text)
    mime_msg = StringIO(mime_msg.as_string())
    df = defer.Deferred()
    f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                           df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
    reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
    return df


def final_success(result):
    print 'All emails processed successfully'
    print result
    reactor.stop()


def final_failure(error):
    print 'Failed to process all emails'
    print error
    reactor.stop()


def yield_recipients(recipients):
    for r in recipients:
        df = send_email(r)
        df.addCallback(success, recipients=r)
        df.addErrback(failure, recipients=r)
        yield df


def send_multiple_emails(recipients):
    final_df = task.coiterate(yield_recipients(recipients))
    return final_df


def main():
    df = send_multiple_emails(RECIPIENTS)
    df.addCallback(final_success)
    df.addErrback(final_failure)


reactor.callLater(1, main)


if __name__ == '__main__':
    reactor.run()

Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.

Sending emails asynchronously using Twisted – Part 1

December 11, 2015Programmingemail, python, twistedRohit Chormale

Using ‘smtplib‘ module

It is very easy to send emails using ‘smtplib‘ module of python. Check following recipe.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from smtplib import SMTP


# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = """
"""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender's emails
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp host of your email provider
SMTP_PORT = 0


mailer = SMTP(SMTP_HOST, SMTP_PORT)
mailer.starttls()
mailer.login(USERNAME, PASSWORD)
mailer.sendmail(FROM_EMAIL, TO_EMAILS, CONTENT)
mailer.quit()

But ‘smtplib’ module sends emails synchronously. So code execution is blocked until email is sent. To overcome this, lets try to send email asynchornously.

Using Twisted

For this tutorial we are going to use Twisted framework. Twisted is event-driven networking engine. It uses reactor-pattern. Twisted uses deferred objects to address waiting IOs. Deferred is more like subset of promises. Check following recipe to send asynchronously MIME message using Twisted.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from StringIO import StringIO
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.utils import formatdate
from email.header import Header

from twisted.mail.smtp import ESMTPSenderFactory
from twisted.internet import reactor, defer
from twisted.internet.ssl import ClientContextFactory

# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = ""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender email
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp port of your email provider
SMTP_PORT = 0


def success(result):
    print 'Email sent successfully'
    print result
    reactor.stop()


def failure(error):
    print 'Failed to send email'
    print error
    reactor.stop()


def send_email():
    mime_text = MIMEText(CONTENT, 'html', 'utf-8')
    mime_msg = MIMEMultipart('alternative')
    mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
    mime_msg['To'] = ",".join(TO_EMAILS)
    mime_msg['From'] = FROM_EMAIL
    mime_msg['Date'] = formatdate(localtime=True)
    mime_msg.attach(mime_text)
    mime_msg = StringIO(mime_msg.as_string())
    df = defer.Deferred()
    f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, TO_EMAILS, mime_msg,
                           df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
    reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
    return df


def main():
    df = send_email()
    df.addCallback(success)
    df.addErrback(failure)


reactor.callLater(1, main)


if __name__ == '__main__':
    reactor.run()

How to chat securely using Pidgin and OTR

December 2, 2015How Tocommunication, securityRohit Chormale

These days surveillance news are coming out frequently. After Snowden’s revelation if you’r suffering from paranoia and want to secure your digital presence, follow this tutorial to communicate securely.

Install Pidgin chat client.
- for ubuntu –
```
apt-get install pidgin
```
- for arch linux –
```
sudo pacman -S pidgin
```
- also you can download it manually from here & then install it as per instructions
Install OTR plugin of Pidgin.
- for ubuntu –
```
apt-get intall pidgin-otr
```
- for arch linux –
```
sudo pacman -S pidgin-otr
```
Now start Pidgin. It will show ‘Accounts’ Popup. Click on ‘Add’ button.
Now you will get ‘Add Account’ popup.
Now configure new account as follows.
- Tab ‘Basic’
  - Login Options
    - Protocol : XMPP (don’t use ‘Facebook XMPP’)
    - Username: (don’t use a username which will somehow connected to real you)
    - jabber.rayservers.com
    - leave blank
    - enter password you want
  - User Options
    - Local alias : Leave blank
    - New mail notifications : DO NOT check
    - Use this buddy icon : check if you want
    - Create this new account on the server : MUST check
- Tab ‘Advanced’
  - Connection security : Require Encryption
  - Allow plaintext auth over unencrypted streams : DO NOT check
  - Connect port : 5222
  - Show Custom Smileys : check
  - Create this new account on the server : MUST check
- Tab ‘Proxy’
  - Proxy type : Use Global Proxy Settings
  - Create this new account on the server : MUST check
- Tab ‘Voice and Video’
  - Use silence suppression : leave default
  - Create this new account on the server : MUST check
Now to add this new account, click on ‘Add’
Wait for few seconds. Popup will come for ‘SSL Certificate Verification’. Click on ‘Accept’. Cross-check ‘Certificate Information’. Then ‘Accept’ cetificate.
Now you will get popup saying ‘Register New XMPP Account’. Click on ‘Register’.
From top menus select
Tools -> Plugins -> Off-The-Record Messaging (MUST checked)
Click on ‘Configure Plugin’
Configure ‘Off-the-record Messaging’ popup as follows,
- My Private Keys –
  - Click on ‘Generate’ if you get message ‘No key present’. Generating keys takes time. When keys are getting generated try to do some CPU intensive work to add more entropy.
- Default OTR Settings – Check all
  - Enable private messaging – check
  - Automatically initiate private messaging – check
  - Require private messaging – check
  - Don’t log OTR conversations – check
- OTR UI Options
  - Show OTR button in toolbar – check
Once key is generated, click on ‘Close’ to close popup windows of ‘Off-the-record Messaging’ & ‘Configure plugin’.
Now in top window ‘Buddy List’ ,
Tools -> Preferences -> Logging -> Do NOT check any options here & close popup.
Now enable your new account as,
Accounts -> Enable account -> Select newly created account
Now to add buddy,
Buddies -> Add buddy
In ‘Add Buddy’ popup, enter ‘Buddy’s username’ – something like ‘foo@bar.qux.com’. Optionally you can add ‘Alias name’ for ease. Here important thing to remember is, once you add buddy it will appear in your buddy list only after authorization of your buddy.
Now if you add buddy successfully and he is online & double-click on buddy and start to chat.
Here you will get another private messaging window. Don’t forget to ‘Start OTR’ here & to Authenticate your buddy.
Further, you can use Tor with Pidgin to circumvent IP address.

How to secure yourself with GPG

November 26, 2015How ToGPG, securityRohit Chormale

Generate your key

Run following command in your shell,
```
gpg --gen-key
```
Now program will ask you to choose couple of options, use following preferences
Please select what kind of key you want: 1 RSA and RSA (default)
What keysize do you want? (2048) 4096
Key is valid for? (0) 0
Is this correct? (y/N) y
Now enter name, email and comment message.
Change (N)ame, (C)omment, (E)-mail or (O)kay/(Q)uit? o
Finally, enter a passphrase to protect your secret key.

Edit your key

We can later edit key to use other options.
e.g Lets set our key to use stronger hashes.

Edit key using following command,
```
 gpg --edit-key test@example.com
```

Now set hash preferences as follows,

gpg> setpref SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP

Really update the preferences? (y/N) y
Enter your passphrase
Save new preferences by command,

Make available your key

There are 2 ways to make available your key to other users.

Give them manually. Use following command,
```
gpg --armor --export test@example.com
```
You will get your public key. Copy and paste it and send to other user.
Upload to key server. You can do this again using 2 ways. One is using, forms available on server. While for second way, first grab your id using following command’s output & then upload to keyservers like http://pgp.mit.edu/

gpg --send-keys --keyserver pgp.mit.edu <key-id>

Importing other keys

Import other user’s keys. We can import keys of other users with multiple ways. From text file – If someone sends you text file containing his public key, import it as,
```
 gpg --import <pub_key_file>
```
From key server – There are some popular key serves which host public keys.
One of such server is `http://pgp.mit.edu`. Here you can search particular user’s key as follows,
```
gpg --keyserver pgp.mit.edu --search-keys <string>
```
Validate key. The easy way to validate person’s identification is match fingerprint of key.
```
gpg --fingerprint test@example.com
```
Sign imported key as,
```
gpg --sign-key test@example.com
```
Optionally you can send back signed key

Using gpg key

To encrypt message using your key use following command,

gpg --encrypt --sign --armor -r test@example.com <filename>

To decrypt file,
```
 gpg <filename>
```
Creating revocation certificate

There is always possibility that your master key-pair may get lost. (and may be stolen if you are unfortunate). If this happen, you must tell other people to not use your public key. This can be done using revocation certificate. Generate revocation certificate using following command,

gpg --output \<test@example.com\>.gpg-revocation-certificate --gen-revoke test@example.com

Store it safe somewhere separately from master key-pair

Some useful commands

List available keys,
```
gpg --list-keys
```
Update key information,
```
 gpg --refresh-keys
```

Coloring shell output

November 26, 2015How Tolinux, shellRohit Chormale

Using coloring, we can enhance output of shell script. Run following script in your terminal and see magic.

#!/bin/bash

#various color codes
color_lblue='\033[1;34m'
color_red='\033[1;31m'
color_lgreen='\033[1;32m'
color_custom='\033[1;33;44m'
color_no='\033[0m'

#ascii art
printf "${color_custom}"
cat <<EOF
   m""                       #                   
 mm#mm   mmm    mmm          #mmm    mmm    m mm 
   #    #" "#  #" "#         #" "#  "   #   #"  "
   #    #   #  #   #   """   #   #  m"""#   #    
   #    "#m#"  "#m#"         ##m#"  "mm"#   # 
EOF
printf "${color_no}"

About Rohit Chormale

What is redis

Redis Cluster and Sentinels

Setup for tutorial

Installation

Server 1 (Redis Master)

Install Redis

Setup required directories

Configure redis master

Configure redis master sentinel

Add non-privileged user

Setup init scripts

Start service

Server 2 (Redis Slave 1)

Install Redis

Setup required directories

Configure redis slave 1

Configure redis slave 1 sentinel

Add non-privileged user

Setup init scripts

Start service

Server 3 (Redis Slave 2)

Install Redis

Setup required directories

Configure redis slave 2

Configure redis slave 2 sentinel

Add non-privileged user

Setup init scripts

Start service

Sentinel Testing

Redis Fail-over Testing

Sample init scripts

Redis Init Script

Sentinel Init Script

Notes

Security

Sentinel management

Sources

Build & Install FreeSWITCH

Set Owner & Permissions

Starting FreeSWITCH service on boot automatically

Start FreeSWITCH

Notes

Sending multiple emails

Sending multiple emails using coiterator

Using ‘smtplib‘ module

Using Twisted