All posts by Rohit Chormale

About Rohit Chormale

I am software engineer from Hyderabad, India with primary interests in distributed systems. In my leisure time, I love to read technical papers and hacking emacs.

Redis Database

How to Setup Redis Cluster from Source

What is redis

Redis is an open source in-memory database. It stores data in key-value format. Because of residing in memory, redis is an excellent tool for caching. Redis provides a rich set of data types. This gives redis upper hand over Memcached. Apart from caching, redis can be used as distributed message broker.

Redis Cluster and Sentinels

To achieve high availability, redis can be deployed in cluster along with Sentinels. Sentinel is a feature of redis. Multiple sentinels are deployed across redis clusters for monitoring purpose. When redis master goes down, sentinels elect a new master from slaves. When old master comes up again, it is added as slave.

Another use case of clustering is a distribution of load. In high load environment, we can send write requests to master and read request to slaves.

This tutorial is specifically focused on Redis Cluster Master Slave model. We will not cover data sharding across cluster here. In data sharding, keys are distributed across multiple redis nodes.

Setup for tutorial

For this tutorial, we will use 3 (virtual) servers. On one server Redis master will reside while other two servers will be used for slaves. Standard redis port is 6379. To differentiate easily, we will run master on 6379 port and slaves on
6380 and 6381 ports. Same will be applied for sentinel services. Master sentinel will listen on 16379 port while slave sentinels will be on 16380 and 16381.

Lets put this easy way.

    Server 1 (Redis Master)
    Redis Service at host 192.168.0.1 and port tcp:6379
    Sentinel Service at host 192.168.0.1 and port tcp:16379
    Server 2 (Redis Slave 1)
    Redis Service at host 192.168.0.2 and port tcp:6380
    Sentinel at host 192.168.0.2 and port tcp:16380
    Server 3 (Redis Slave 2)
    Redis Service at host 192.168.0.3 and port tcp:6381
    Sentinel Service at host 192.168.0.3 and port tcp:16381

This tutorial is tested on CentOS 6.9. For CentOS 7.X, check below Notes Section.

Installation

We will follow same installation steps for setting up of all servers. Only difference will be in configurations.

  • Step 1: Grab redis source, make and install
  • Step 2: Setup required directories
  • Step 3: Configure redis master
  • Step 4: Configure redis master sentinel
  • Step 5: Add low privileged user to run redis
  • Step 6: Setup init scripts
  • Step 7: Start service

Server 1 (Redis Master)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6379
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6379.conf

Configure redis master

Edit config file `/etc/redis/6379.conf` in your favorite editor and change below options.

    bind 192.168.0.1
    port 6379
    daemonize yes
    pidfile "/var/run/redis/redis_6379.pid"
    logfile "/var/log/redis/redis_6379.log"
    dir "/var/redis/6379"

Configure redis master sentinel

Add config file for sentinel at `/etc/redis/sentinel_6379.conf`. Open a file and add below content

    bind 192.168.0.1
    port 16379
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6379.pid"
    dir "/var/redis/6379"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below.

    cp redis-6379-init-script /etc/init.d/redis_6379
    chmod 750 /etc/init.d/redis_6379
    chkconfig redis_6379 on
    cp sentinel-6379-init-script /etc/init.d/sentinel_6379
    chmod 750 /etc/init.d/sentinel_6379
    chkconfig sentinel_6379 on

Start service

    service redis_6379 start
    service sentinel_6379 start

Server 2 (Redis Slave 1)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6380
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6380.conf

Configure redis slave 1

Edit config file `/etc/redis/6380.conf` in your favorite editor and change below options.

    bind 192.168.0.2
    port 6380
    daemonize yes
    pidfile "/var/run/redis/redis_6380.pid"
    logfile "/var/log/redis/redis_6380.log"
    dir "/var/redis/6380"
    slaveof 192.168.0.1 6379

Configure redis slave 1 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6380.conf`. Open a file and add below content

    bind 192.168.0.2
    port 16380
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6380.pid"
    dir "/var/redis/6380"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

cp redis-6380-init-script /etc/init.d/redis_6380
    chmod 750 /etc/init.d/redis_6380
    chkconfig redis_6380 on
cp sentinel-6380-init-script /etc/init.d/sentinel_6380
    chmod 750 /etc/init.d/sentinel_6380
    chkconfig sentinel_6380 on

Start service

    service redis_6380 start
    service sentinel_6380 start

Server 3 (Redis Slave 2)


Install Redis

    cd /opt
    wget download.redis.io/releases/redis-4.0.9.tar.gz
    tar -xf redis-4.0.9.tar.gz
    cd redis-4.0.9
    make
    make install

Setup required directories

    mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/6381
    cp redis.conf redis.conf.bak
    cp redis.conf /etc/redis/6381.conf

Configure redis slave 2

Edit config file `/etc/redis/6381.conf` in your favorite editor and change below options.

    bind 192.168.0.3
    port 6381
    daemonize yes
    pidfile "/var/run/redis/redis_6381.pid"
    logfile "/var/log/redis/redis_6381.log"
    dir "/var/redis/6381"
    slaveof 192.168.0.1 6379

Configure redis slave 2 sentinel

Add config file for sentinel at `/etc/redis/sentinel_6381.conf`. Open a file and add below content

    bind 192.168.0.3
    port 16381
    sentinel monitor redis-cluster 192.167.0.1 6379 2
    sentinel down-after-milliseconds redis-cluster 5000
    sentinel parallel-syncs redis-cluster 1
    sentinel failover-timeout redis-cluster 10000
    daemonize yes
    pidfile "/var/run/redis/sentinel_6381.pid"
    dir "/var/redis/6381"

Add non-privileged user

    adduser redis -M -g daemon
    passwd -l redis
    chown -R redis:daemon /opt/redis-4.0.9
    chown -R redis:daemon /var/run/redis
    chown -R redis:daemon /var/log/redis
    chown -R redis:daemon /var/redis/

Setup init scripts

You can find sample init scripts in Notes section below. Change `$HOST` and `$PORT` values accordingly

    cp redis-6381-init-script /etc/init.d/redis_6381
    chmod 750 /etc/init.d/redis_6381
    chkconfig redis_6381 on
    cp sentinel-6381-init-script /etc/init.d/sentinel_6381
    chmod 750 /etc/init.d/sentinel_6381
    chkconfig sentinel_6381 on

Start service

    service redis_6381 start
    service sentinel_6381 start

Sentinel Testing

    # Connect to any redis sentinel
    /usr/local/bin/redis-cli -h 192.168.0.3 -p 16381

    # To see current masters
    192.168.0.3:16381> SENTINEL masters

    # To see slaves for given cluster
    192.168.0.3:16381> SENTINEL slaves redis-cluster

Redis Fail-over Testing

For fail-over testing, we can take down redis-master either using init script or below command.

    # Connect to redis master and execute below command
    /usr/local/bin/redis-cli -h 192.168.0.1 -p 6379
    192.168.0.1:6379> DEBUG SEGFAULT

Also we can force sentinel to run fail over using below command

    # Forced failure
    192.168.0.1:6379> SENTINEL failover redis-cluster
    # Check after few seconds. You should get new master
    192.168.0.1:6379> SENTINEL masters

Sample init scripts

Redis Init Script

#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 85 15
# description:  Redis is persistent key-value database
# processname: redis_6379

# Source function library
. /etc/init.d/functions

REDISHOST=192.168.0.1
REDISPORT=6379
EXEC=/usr/local/bin/redis-server
USER=redis
CLIEXEC=/usr/local/bin/redis-cli
PIDFILE=/var/run/redis/redis_${REDISPORT}.pid
CONF="/etc/redis/${REDISPORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                daemon --user $USER $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                daemon --user $USER $CLIEXEC -h $REDISHOST -p $REDISPORT shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Sentinel Init Script

#!/bin/sh
#
# Simple Redis Sentinel init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig: 345 86 15
# description:  Redis Sentinel to monitor redis cluster
# processname: sentinel_6379

# Source function library
. /etc/init.d/functions

REDIS_PORT=6379
SENTINEL_PORT=16379
EXEC=/usr/local/bin/redis-server
CLIEXEC=/usr/local/bin/redis-cli
USER=redis

PIDFILE=/var/run/redis/sentinel_${REDIS_PORT}.pid
CONF="/etc/redis/sentinel_${REDIS_PORT}.conf"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis Sentinel server..."
                daemon --user $USER $EXEC $CONF --sentinel
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                kill -9 $PID
                rm $PIDFILE
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis Sentinel to shutdown ..."
                    sleep 1
                done
                echo "Redis Sentinel stopped"
        fi
        ;;
    restart)
        stop
        sleep 3
        start
        ;;
    *)
        echo "Usage: $PROG_NAME {start|stop|restart}"
		exit 1
		;;
esac

Notes

Security

  • NEVER EVER run redis on public interface
  • If redis is deployed in cloud environment like AWS, set up security groups/firewalls carefully. Most of times, cloud providers use ephemeral ips. Because of ephermal ips, even redis is bound to private ip, it can be accessed over public interface.
  • For more security, dangerous commands can be disabled(renamed). But be careful with them in cluster environment.
  • Redis also provides simple authentication mechanism. It is not covered here because of scope.

Sentinel management

  • During redis fail-over, config files are rewritten by sentinel program. So when restarting redis-cluster, be careful.

Sources

  • https://redis.io/topics/cluster-tutorial
  • https://redis.io/topics/security
  • https://redis.io/commands/debug-segfault
FreeSWITCH

How to build and install FreeSWITCH 1.6 on Debian 8 Jessie

FreeSWITCH is an opensource telephony soft switch created in 2006. As per official wiki  page,

It is a scalable open source cross-platform telephony platform designed to route and interconnect popular communication protocols using audio, video, text or any other form of media.

Sounds good. right ?

We are using Debian for this tutorial as it is very stable & mature linux distribution and FreeSWITCH core developers’ choice-of-distribution .
You can read more about FreeSWITCH on there wiki page.

Now lets cut a crap & start an action, assuming you  already have working Debian 8 OS.

Build & Install FreeSWITCH

There are different ways to install FreeSWITCH. In this tutorial, we will see how to install it from source.

  1. First update your Debian box & install curl & git.
    apt-get update && apt-get install -y curl git
  2. Add FreeSWITCH GPG key to APT sources keyring.
    curl https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -
  3. Add FreeSWITCH repository to APT sources.
    echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list
  4. Once again update your system.
     apt-get update
  5. Now lets first install FreeSWITCH dependencies.
    apt-get install -y --force-yes freeswitch-video-deps-most
  6. Though above step takes care of most of dependencies, few still remains required to compile mod_fsv. So install them as,

     apt-get install -y libyuv-dev libvpx2-dev
  7. Grab source code of FreeSWITCH as follows,
     git config --global pull.rebase true
     cd /usr/src/
     git clone https://freeswitch.org/stash/scm/fs/freeswitch.git freeswitch.git
  8. Now lets compile FreeSWITCH source for version 1.6
    cd freeswitch.git
     git checkout v1.6
     ./bootstrap.sh -j
     ./configure -C
     make && make install
  9. Now lets compile sounds
    make all cd-sounds-install cd-moh-install
  10. Lets create simlinks to required binaries to access them from anywhere.
    ln -s /usr/local/freeswitch/bin/freeswitch /usr/bin/freeswitch
    ln -s /usr/local/freeswitch/bin/fs_cli /usr/bin/fs_cli

     

Set Owner & Permissions

cd /usr/local
groupadd freeswitch
adduser --disabled-password  --quiet --system --home /usr/local/freeswitch --gecos "FreeSWITCH Voice Platform" --ingroup freeswitch freeswitch
chown -R freeswitch:freeswitch /usr/local/freeswitch/
chmod -R ug=rwX,o= /usr/local/freeswitch/
chmod -R u=rwx,g=rx /usr/local/freeswitch/bin/

Starting FreeSWITCH service on boot automatically

To start FreeSWITCH after each boot automatically we need to set up init script. Init script is script used by init system to manipulate services. Debian 8 is now migrated to systemd init system, we will add systemd unit file.

Copy following content to ‘/lib/systemd/system/freeswitch.service’

[Unit]
Description=freeswitch
After=syslog.target network.target local-fs.target

[Service]
; service
Type=forking
PIDFile=/usr/local/freeswitch/run/freeswitch.pid
PermissionsStartOnly=true
ExecStart=/usr/local/freeswitch/bin/freeswitch -u freeswitch -g freeswitch -ncwait -nonat -rp
TimeoutSec=45s
Restart=on-failure
; exec
WorkingDirectory=/usr/local/freeswitch/bin
User=root
Group=daemon
LimitCORE=infinity
LimitNOFILE=100000
LimitNPROC=60000
;LimitSTACK=240
LimitRTPRIO=infinity
LimitRTTIME=7000000
IOSchedulingClass=realtime
IOSchedulingPriority=2
CPUSchedulingPolicy=rr
CPUSchedulingPriority=89
UMask=0007

[Install]
WantedBy=multi-user.target

Now execute following commands in your shell

chmod 750 /lib/systemd/system/freeswitch.service
ln -s /lib/systemd/system/freeswitch.service /etc/systemd/system/freeswitch.service
systemctl daemon-reload
systemctl enable freeswitch.service

Start FreeSWITCH

Now we are all set. Lets start hacking FreeSWITCH.

systemctl start freeswitch.service

 

Notes

  1. If something goes wrong & you try compilation again by ‘make clean’, sometimes you get errors regarding ‘spandsp’. To resolve them try to clean using
    ‘git clean -fdx’. For more info check this ticket – https://freeswitch.org/jira/browse/FS-6405

 

Twisted Matrix

Sending emails asynchronously using Twisted – Part 2

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

  • Sending multiple emails

    Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.

        
    
            #!/usr/bin/env python2.7
            __author__ = 'Rohit Chormale'
    
            """
            In this tutorial, same email will be sent to multiple recipients using `twisted.internet.defer.DeferredList` API.
            For each recipient, one defer will be created and will be added in `DeferredList`.
            `DeferredList` API will be fired only when all deferrands in `DeferredList` will be fired.
            """
    
    
            from StringIO import StringIO
            from email.mime.text import MIMEText
            from email.mime.multipart import MIMEMultipart
            from email.utils import formatdate
            from email.header import Header
    
            from twisted.mail.smtp import ESMTPSenderFactory
            from twisted.internet import reactor, defer
            from twisted.internet.ssl import ClientContextFactory
    
    
            # enter email content
            CONTENT = """
            """
            # enter email subject
            SUBJECT = ""
            # enter sender email
            FROM_EMAIL = ""
            # list of multiple recipients
            RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                          ['test3@example.com',],
                          ['test4@example.com', 'test4@example.com'],
                        ]
    
    
            # enter username of your email account
            USERNAME = "your-email"
            # enter password of your email account
            PASSWORD = "your-password"
            # enter smtp host of your email provider
            SMTP_HOST = "smtp-host"
            # enter smtp port of your email provider
            SMTP_PORT = 587
    
    
            def success(result, recipients):
                print 'Email sent successfully | recipients - %s' % recipients
                print result
    
    
            def failure(error, recipients):
                print 'Failed to send email | recipients - %s' % recipients
                print error
    
    
            def send_email(to_emails):
                mime_text = MIMEText(CONTENT, 'html', 'utf-8')
                mime_msg = MIMEMultipart('alternative')
                mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
                mime_msg['To'] = ",".join(to_emails)
                mime_msg['From'] = FROM_EMAIL
                mime_msg['Date'] = formatdate(localtime=True)
                mime_msg.attach(mime_text)
                mime_msg = StringIO(mime_msg.as_string())
                df = defer.Deferred()
                f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                                       df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
                reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
                return df
    
    
            def final_success(result):
                print 'All emails processed successfully'
                print result
                reactor.stop()
    
    
            def final_failure(error):
                print 'Failed to process all emails'
                print error
                reactor.stop()
    
    
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    
    
            def main():
                df = send_multiple_emails(RECIPIENTS)
                df.addCallback(final_success)
                df.addErrback(final_failure)
    
    
            reactor.callLater(1, main)
    
    
            if __name__ == '__main__':
                reactor.run()
    
    

     

  • Sending multiple emails using coiterator

    Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.

        
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    

    Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
    So lets modify our code to work like generators.

        
    #!/usr/bin/env python2.7
    __author__ = 'Rohit Chormale'
    
    
    """
    In this tutorial, same email will be sent to multiple recipients using `twisted.internet.task` API.
    Each email recipient will be yielded using `twisted.internet.task.coiterate` API.
    """
    
    
    from StringIO import StringIO
    from email.mime.text import MIMEText
    from email.mime.multipart import MIMEMultipart
    from email.utils import formatdate
    from email.header import Header
    
    from twisted.mail.smtp import ESMTPSenderFactory
    from twisted.internet import reactor, defer, task
    from twisted.internet.ssl import ClientContextFactory
    
    
    # enter email content
    CONTENT = """
    """
    # enter email subject
    SUBJECT = ""
    # enter sender email
    FROM_EMAIL = ""
    # list of multiple recipients
    RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                  ['test3@example.com',],
                  ['test4@example.com', 'test4@example.com'],
                ]
    
    
    # enter username of your email account
    USERNAME = "your-email"
    # enter password of your email account
    PASSWORD = "your-password"
    # enter smtp host of your email provider
    SMTP_HOST = "your-mail-provider=host"
    # enter smtp port of your email provider
    SMTP_PORT = 587
    
    
    def success(result, recipients):
        print 'Email sent successfully | recipients - %s' % recipients
        print result
    
    
    def failure(error, recipients):
        print 'Failed to send email | recipients - %s' % recipients
        print error
    
    
    def send_email(to_emails):
        mime_text = MIMEText(CONTENT, 'html', 'utf-8')
        mime_msg = MIMEMultipart('alternative')
        mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
        mime_msg['To'] = ",".join(to_emails)
        mime_msg['From'] = FROM_EMAIL
        mime_msg['Date'] = formatdate(localtime=True)
        mime_msg.attach(mime_text)
        mime_msg = StringIO(mime_msg.as_string())
        df = defer.Deferred()
        f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                               df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
        reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
        return df
    
    
    def final_success(result):
        print 'All emails processed successfully'
        print result
        reactor.stop()
    
    
    def final_failure(error):
        print 'Failed to process all emails'
        print error
        reactor.stop()
    
    
    def yield_recipients(recipients):
        for r in recipients:
            df = send_email(r)
            df.addCallback(success, recipients=r)
            df.addErrback(failure, recipients=r)
            yield df
    
    
    def send_multiple_emails(recipients):
        final_df = task.coiterate(yield_recipients(recipients))
        return final_df
    
    
    def main():
        df = send_multiple_emails(RECIPIENTS)
        df.addCallback(final_success)
        df.addErrback(final_failure)
    
    
    reactor.callLater(1, main)
    
    
    if __name__ == '__main__':
        reactor.run()
    

    Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.

Sending emails asynchronously using Twisted – Part 1

  • Using ‘smtplib‘ module

It is very easy to send emails using ‘smtplib‘ module of python. Check following recipe.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from smtplib import SMTP


# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = """
"""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender's emails
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp host of your email provider
SMTP_PORT = 0


mailer = SMTP(SMTP_HOST, SMTP_PORT)
mailer.starttls()
mailer.login(USERNAME, PASSWORD)
mailer.sendmail(FROM_EMAIL, TO_EMAILS, CONTENT)
mailer.quit()

But ‘smtplib’ module sends emails synchronously. So code execution is blocked until email is sent. To overcome this, lets try to send email asynchornously.

  • Using Twisted

For this tutorial we are going to use Twisted framework. Twisted is event-driven networking engine. It uses reactor-pattern. Twisted uses deferred objects to address waiting IOs. Deferred is more like subset of promises. Check following recipe to send asynchronously MIME message using Twisted.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from StringIO import StringIO
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.utils import formatdate
from email.header import Header

from twisted.mail.smtp import ESMTPSenderFactory
from twisted.internet import reactor, defer
from twisted.internet.ssl import ClientContextFactory

# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = ""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender email
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp port of your email provider
SMTP_PORT = 0


def success(result):
    print 'Email sent successfully'
    print result
    reactor.stop()


def failure(error):
    print 'Failed to send email'
    print error
    reactor.stop()


def send_email():
    mime_text = MIMEText(CONTENT, 'html', 'utf-8')
    mime_msg = MIMEMultipart('alternative')
    mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
    mime_msg['To'] = ",".join(TO_EMAILS)
    mime_msg['From'] = FROM_EMAIL
    mime_msg['Date'] = formatdate(localtime=True)
    mime_msg.attach(mime_text)
    mime_msg = StringIO(mime_msg.as_string())
    df = defer.Deferred()
    f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, TO_EMAILS, mime_msg,
                           df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
    reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
    return df


def main():
    df = send_email()
    df.addCallback(success)
    df.addErrback(failure)


reactor.callLater(1, main)


if __name__ == '__main__':
    reactor.run()

 

 

How to chat securely using Pidgin and OTR

These days surveillance news are coming out frequently. After Snowden’s revelation if you’r suffering from paranoia and want to secure your digital presence, follow this tutorial to communicate securely.

  1. Install Pidgin chat client.
    • for ubuntu –
      apt-get install pidgin
    • for arch linux –
      sudo pacman -S pidgin
    • also you can download it manually from here & then install it as per instructions
  2. Install OTR plugin of Pidgin.
    • for ubuntu –
      apt-get intall pidgin-otr
    • for arch linux –
      sudo pacman -S pidgin-otr
  3. Now start Pidgin. It will show ‘Accounts’ Popup. Click on ‘Add’ button. add
  4. Now you will get ‘Add Account’ popup.
    add-account-0
  5. Now configure new account as follows.
    • Tab ‘Basic’
      • Login Options
        • Protocol : XMPP (don’t use ‘Facebook XMPP’)
        • Username: (don’t use a username which will somehow connected to real you)
        • jabber.rayservers.com
        • leave blank
        • enter password you want
      • User Options
        • Local alias : Leave blank
        • New mail notifications : DO NOT check
        • Use this buddy icon : check if you want
        • Create this new account on the server : MUST checkadd-account-1
    • Tab ‘Advanced’
      • Connection security : Require Encryption
      • Allow plaintext auth over unencrypted streams : DO NOT check
      • Connect port : 5222
      • Show Custom Smileys : check
      • Create this new account on the server : MUST check add-account-2
    • Tab ‘Proxy’
      • Proxy type : Use Global Proxy Settings
      • Create this new account on the server : MUST check add-account-3
    • Tab ‘Voice and Video’
      • Use silence suppression : leave default
      • Create this new account on the server : MUST check add-account-4
  6. Now to add this new account, click on ‘Add’
  7. Wait for few seconds. Popup will come for ‘SSL Certificate Verification’. Click on ‘Accept’. Cross-check ‘Certificate Information’. Then ‘Accept’ cetificate. cert-1 cert-2
  8. Now you will get popup saying ‘Register New XMPP Account’. Click on ‘Register’.
  9. From top menus select
    Tools -> Plugins -> Off-The-Record Messaging (MUST checked)
  10. Click on ‘Configure Plugin’
  11. Configure ‘Off-the-record Messaging’ popup as follows, otr-1
    • My Private Keys –
      • Click on ‘Generate’ if you get message ‘No key present’. Generating keys takes time. When keys are getting generated try to do some CPU intensive work to add more entropy.
    • Default OTR Settings – Check all
      • Enable private messaging – check
      • Automatically initiate private messaging – check
      • Require private messaging – check
      • Don’t log OTR conversations – check
    • OTR UI Options
      • Show OTR button in toolbar – check
  12. Once key is generated, click on ‘Close’ to close popup windows of ‘Off-the-record Messaging’ & ‘Configure plugin’.
    otr-2
  13. Now in top window ‘Buddy List’ ,
    Tools -> Preferences -> Logging -> Do NOT check any options here & close popup. otr-4
  14. Now enable your new account as,
    Accounts -> Enable account -> Select newly created account
  15. Now to add buddy,
    Buddies -> Add buddy
  16. In ‘Add Buddy’ popup, enter ‘Buddy’s username’ – something like ‘foo@bar.qux.com’.  Optionally you can add ‘Alias name’ for ease. Here important thing to remember is, once you add buddy it will appear in your buddy list only after authorization of your buddy.
  17. Now if you add buddy successfully and he is online & double-click on buddy and start to chat.
  18. Here you will get another private messaging window. Don’t forget to ‘Start OTR’ here & to Authenticate your buddy.
  19. Further, you can use Tor with Pidgin to circumvent IP address.

How to secure yourself with GPG

Generate your key


  1. Run following command in your shell,
    gpg --gen-key
  2. Now program will ask you to choose couple of options, use following preferences
  3.  Please select what kind of key you want: 1    RSA and RSA (default)
  4.  What keysize do you want? (2048) 4096
  5.  Key is valid for? (0) 0
  6. Is this correct? (y/N) y
  7. Now enter name, email and comment message.
  8. Change (N)ame, (C)omment, (E)-mail or (O)kay/(Q)uit? o
  9. Finally, enter a passphrase to protect your secret key.

Edit your key


We can later edit key to use other options.
e.g Lets set our key to use stronger hashes.

  1. Edit key using following command,
     gpg --edit-key test@example.com
  2. Now set hash preferences as follows,
    gpg> setpref SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP
  3.  Really update the preferences? (y/N) y
  4. Enter your passphrase
  5. Save new preferences by command,

Make available your key


There are 2 ways to make available your key to other users.

  1. Give them manually. Use following command,
    gpg --armor --export test@example.com

    You will get your public key. Copy and paste it and send to other user.

  2. Upload to key server. You can do this again using 2 ways. One is using, forms available on server. While for second way, first grab your id using following command’s output & then upload to keyservers like http://pgp.mit.edu/
gpg --send-keys --keyserver pgp.mit.edu <key-id>

 

Importing other keys


  1. Import other user’s keys. We can import keys of other users with multiple ways. From text file – If someone sends you text file containing his public key, import it as,
     gpg --import <pub_key_file>

    From key server – There are some popular key serves which host public keys.
    One of such server is `http://pgp.mit.edu`. Here you can search particular user’s key as follows,

    gpg --keyserver pgp.mit.edu --search-keys <string>
  2. Validate key. The easy way to validate person’s identification is match fingerprint of key.
    gpg --fingerprint test@example.com
  3. Sign imported key as,
    gpg --sign-key test@example.com
  4. Optionally you can send back signed key

Using gpg key


  • To encrypt message using your key use following command,
    gpg --encrypt --sign --armor -r test@example.com <filename>
  • To decrypt file,
     gpg <filename>

    Creating revocation certificate


There is always possibility that your master key-pair may get lost. (and may be stolen if you are unfortunate). If this happen, you must tell other people to not use your public key. This can be done using revocation certificate. Generate revocation certificate using following command,

gpg --output \<test@example.com\>.gpg-revocation-certificate --gen-revoke test@example.com

Store it safe somewhere separately from master key-pair

Some useful commands


  • List available keys,
    gpg --list-keys
  • Update key information,
     gpg --refresh-keys

     

 

 

colored console

Coloring shell output

Using coloring, we can enhance output of shell script. Run following script in your terminal and see magic.

#!/bin/bash

#various color codes
color_lblue='\033[1;34m'
color_red='\033[1;31m'
color_lgreen='\033[1;32m'
color_custom='\033[1;33;44m'
color_no='\033[0m'

#ascii art
printf "${color_custom}"
cat <<EOF
   m""                       #                   
 mm#mm   mmm    mmm          #mmm    mmm    m mm 
   #    #" "#  #" "#         #" "#  "   #   #"  "
   #    #   #  #   #   """   #   #  m"""#   #    
   #    "#m#"  "#m#"         ##m#"  "mm"#   # 
EOF
printf "${color_no}"