Category Archives: internet

How to whitelist Google IP address ranges in firewall using iptables

As an administrator, when you need to obtain a range of IP addresses for Google APIs and services’ default domains, you can refer to the following sources of information.

The default domains’ IP address ranges for Google APIs and services fit within the list of ranges between these 2 sources. (Subtract the usable ranges from the complete list.)

Once you get the IP address ranges, use the “`xargs“` command to update iptables.

google-ips-whitelist.sh

echo "8.8.4.0/24
8.8.8.0/24
8.34.208.0/20
8.35.192.0/20
23.236.48.0/20
23.251.128.0/19
34.64.0.0/10
34.128.0.0/10
35.184.0.0/13
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/12
35.224.0.0/12
35.240.0.0/13
64.15.112.0/20
64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
70.32.128.0/19
72.14.192.0/18
74.114.24.0/21
74.125.0.0/16
104.154.0.0/15
104.196.0.0/14
104.237.160.0/19
107.167.160.0/19
107.178.192.0/18
108.59.80.0/20
108.170.192.0/18
108.177.0.0/17
130.211.0.0/16
136.112.0.0/12
142.250.0.0/15
146.148.0.0/17
162.216.148.0/22
162.222.176.0/21
172.110.32.0/21
172.217.0.0/16
172.253.0.0/16
173.194.0.0/16
173.255.112.0/20
192.158.28.0/22
192.178.0.0/15
193.186.4.0/24
199.36.154.0/23
199.36.156.0/24
199.192.112.0/22
199.223.232.0/21
207.223.160.0/20
208.65.152.0/22
208.68.108.0/22
208.81.188.0/22
208.117.224.0/19
209.85.128.0/17
216.58.192.0/19
216.73.80.0/20
216.239.32.0/19" | xargs -I% iptables -I INPUT -p tcp -s % -j ACCEPT

Speed up SSH with multiplexing

SSH multiplexing is the ability to carry multiple SSH sessions over a single TCP connection.

OpenSSH can reuse an existing TCP connection for multiple concurrent SSH sessions. This results into reduction of the overhead of creating new TCP connections.

Advantage of using SSH multiplexing is that it speeds up certain operations that rely on or occur over SSH. For example, let’s say that you’re using SSH to regularly execute a command on a remote host. Without multiplexing, every time that command is executed your SSH client must establish a new TCP connection and a new SSH session with the remote host. With multiplexing, you can configure SSH to establish a single TCP connection that is kept alive for a specific period of time, and SSH sessions are established over that connection.

You can see the difference below

without multiplexing, we see the normal connection time:

“`$ time ssh lintel-blog“`

real    0m0.658s
user    0m0.016s
sys     0m0.008s

Then we do the same thing again, but with a multiplexed connection to see a faster result:

“`$ time ssh lintel-blog“`

real    0m0.029s
user    0m0.004s
sys     0m0.004s

Configure Multiplexing

OpenSSH client supports multiplexing its outgoing connections, since version 3.9, using the ControlMaster, ControlPath and ControlPersist configuration directives which get defined in ssh_config. The client configuration file usually defaults to the location ~/.ssh/config.

ControlMaster determines whether ssh will listen for control connections and what to do about them. ControlPath sets the location for the control socket used by the multiplexed sessions. These can be either globally or locally in ssh_config or else specified at run time. Control sockets are removed automatically when the master connection has ended. ControlPersist can be used in conjunction with ControlMaster. If ControlPersist is set to ‘yes’, then it will leave the master connection open in the background to accept new connections until either killed explicitly or closed with -O or ends at a pre-defined timeout. If ControlPersist is set to a time, then it will leave the master connection open for the designated time or until the last multiplexed session is closed, whichever is longer.

Here is a sample excerpt from ssh_config applicable for starting a multiplexed session to server1.example.org via the shortcut server1.

Host server1
  HostName server1.example.org
  ControlPath ~/.ssh/controlmasters/%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

 

How to install jitsi meet on CentOS 7

Jitsi is a set of Open Source projects that allows you to easily build and deploy secure videoconferencing solutions.

Jitsi Meet is a fully encrypted, 100% Open Source video conferencing solution that you can use all day, every day, for free — with no account needed.

1. Architecture

A Jitsi Meet installation can be broken down into the following components:

  • A web interface
  • An XMPP server
  • A conference focus component
  • A video router (could be more than one)
  • A SIP gateway for audio calls
  • A Broadcasting Infrastructure for recording or streaming a conference.

The diagram shows a typical deployment in a host running Docker. This project separates each of the components above into interlinked containers. To this end, several container images are provided.

2. Ports

The following external ports must be opened on a firewall:

  • 80/tcp for Web UI HTTP (really just to redirect, after uncommenting ENABLE_HTTP_REDIRECT=1 in .env)
  • 443/tcp for Web UI HTTPS
  • 4443/tcp for RTP media over TCP
  • 10000/udp for RTP media over UDP

Also 20000-20050/udp for jigasi, in case you choose to deploy that to facilitate SIP access.

E.g. on a CentOS server this would be done like this (without SIP access):

    $ sudo firewall-cmd --permanent --add-port=80/tcp
    $ sudo firewall-cmd --permanent --add-port=443/tcp
    $ sudo firewall-cmd --permanent --add-port=4443/tcp
    $ sudo firewall-cmd --permanent --add-port=10000/udp
    $ sudo firewall-cmd --reload

 

3. Configuration

The configuration is performed via environment variables contained in a .env file. You can copy the provided env.example file as a reference.

a. Jibri Module Setup

Before running Jibri, you need to set up an ALSA loopback device on the host. This will not work on a non-Linux host.

For CentOS 7, the module is already compiled with the kernel, so just run:

# configure 5 capture/playback interfaces
echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf
# setup autoload the module
echo "snd_aloop" > /etc/modules-load.d/snd_aloop.conf
# load the module
modprobe snd-aloop
# check that the module is loaded
lsmod | grep snd_aloop

b. Installation

  • clone the repository:

git clone https://github.com/jitsi/docker-jitsi-meet && cd docker-jitsi-meet

  • Create a .env file by copying and adjusting env.example
    • cp env.example .env
  • Set strong passwords in the security section options of .env file by running the following bash script
    • ./gen-passwords.sh
  • Create required CONFIG directories
    • mkdir -p ~/.jitsi-meet-cfg/{web/letsencrypt,transcripts,prosody/config,prosody/prosody-plugins-custom,jicofo,jvb,jigasi,jibri}
  • Run docker-compose up -d
  • Access the web UI at https://domain.com (or a different port, in case you edited the compose file).

 

If you want to use jigasi too, first configure your env file with SIP credentials and then run Docker Compose as follows: docker-compose -f docker-compose.yml -f jigasi.yml up

If you want to enable document sharing via Etherpad, configure it and run Docker Compose as follows: docker-compose -f docker-compose.yml -f etherpad.yml up

If you want to use jibri too, first configure a host as described in JItsi BRoadcasting Infrastructure configuration section and then run Docker Compose as follows: docker-compose -f docker-compose.yml -f jibri.yml up -d or to use jigasi too: docker-compose -f docker-compose.yml -f jigasi.yml -f jibri.yml up -d

Running behind NAT or on a LAN environment
If running in a LAN environment (as well as on the public Internet, via NAT) is a requirement, the DOCKER_HOST_ADDRESS should be set. This way, the Videobridge will advertise the IP address of the host running Docker instead of the internal IP address that Docker assigned it, thus making ICE succeed. If your users are coming in over the Internet (and not over LAN), this will likely be your public IP address. If this is not set up correctly, calls will crash when more than two users join a meeting.

The public IP address is discovered via STUN. STUN servers can be specified with the JVB_STUN_SERVERS option.

 

How to setup SOCKS proxy in Linux

SOCKS server is a general purpose proxy server that establishes a TCP connection to another server on behalf of a client, then routes all the traffic back and forth between the client and the server. It works for any kind of network protocol on any port. SOCKS Version 5 adds additional support for security and UDP.

Use of SOCKS is as a circumvention tool, allowing traffic to bypass Internet filtering to access content otherwise blocked, e.g., by governments, workplaces, schools, and country-specific web services

Using SSH

SOCKS proxies can be created without any special SOCKS proxy software if you have Open SSH installed on your server and an SSH client with dynamic tunnelling support installed on your client computer.

ssh -D 1080 user@<IP Address or Domain of your Server>

Now, enter your password and make sure to leave the Terminal window open. You have now created a SOCKS proxy at localhost:1080. Only close this window if you wish to disable your local SOCKS proxy.

Using Microsocks program

MicroSocks is a multithreaded, small, efficient SOCKS5 server.

It’s very lightweight, and very light on resources too:

for every client, a thread with a stack size of 8KB is spawned. the main process basically doesn’t consume any resources at all.

the only limits are the amount of file descriptors and the RAM.

It’s also designed to be robust: it handles resource exhaustion gracefully by simply denying new connections, instead of calling abort() as most other programs do these days.

another plus is ease-of-use: no config file necessary, everything can be done from the command line and doesn’t even need any parameters for quick setup.

Installing microsocks

“`git clone https://github.com/rofl0r/microsocks.git“`

“`cd microsocks“`

“`make“`

Starting socks service

microsocks -1 -i listenip -p port -u user -P password -b bindaddr

all arguments are optional. by default listenip is 0.0.0.0 and port 1080.

option -1 activates auth_once mode: once a specific ip address authed successfully with user/pass, it is added to a whitelist and may use the proxy without auth. this is handy for programs like firefox that don’t support user/pass auth. for it to work you’d basically make one connection with another program that supports it, and then you can use firefox too.

Simple port scanner in python

a port scanner is an application designed to probe a server or host for open ports. Such an application may be used by administrators to verify the security policies of their networks and by attackers to identify network services running on a host and exploit vulnerabilities.

port-scanner.py

#!/usr/bin/env python2
from socket import * 

if __name__ == '__main__':
    target = raw_input('Enter host to scan: ')
    targetIP = gethostbyname(target)
    print 'Starting scan on host ', targetIP

    #scan reserved ports
    for i in range(20, 1025):
        s = socket(AF_INET, SOCK_STREAM)

        result = s.connect_ex((targetIP, i))

        if(result == 0) :
            print 'Port %d: OPEN' % (i,)
        s.close()

Example

How to use ipset command on linux to block bulk IPs

ipset is a companion application for the iptables Linux firewall. It allows you to setup rules to quickly and easily block a set of IP addresses, among other things.

Installation

Debian based system

“`# apt install ipset“`

Redhat based system

“`# yum install ipset“`

Blocking a list of network

Start by creating a new “set” of network addresses. This creates a new “hash” set of “net” network addresses named “myset”.

# ipset create myset hash:net

or

# ipset -N myset nethash

Add any IP address that you’d like to block to the set.

# ipset add myset 14.144.0.0/12
# ipset add myset 27.8.0.0/13
# ipset add myset 58.16.0.0/15
# ipset add myset 1.1.1.0/24

Finally, configure iptables to block any address in that set. This command will add a rule to the top of the “INPUT” chain to “-m” match the set named “myset” from ipset (–match-set) when it’s a “src” packet and “DROP”, or block, it.

# iptables -I INPUT -m set --match-set myset src -j DROP

Blocking a list of IP addresses

Start by creating a new “set” of ip addresses. This creates a new “hash” set of “ip” addresses named “myset-ip”.

# ipset create myset-ip hash:ip

or

# ipset -N myset-ip iphash

Add any IP address that you’d like to block to the set.

# ipset add myset-ip 1.1.1.1
# ipset add myset-ip 2.2.2.2

Finally, configure iptables to block any address in that set.

# iptables -I INPUT -m set --match-set myset-ip src -j DROP

Making ipset persistent

The ipset you have created is stored in memory and will be gone after reboot. To make the ipset persistent you have to do the followings:

First save the ipset to /etc/ipset.conf:

# ipset save > /etc/ipset.conf

Then enable ipset.service, which works similarly to iptables.service for restoring iptables rules.

Other Commands

To view the sets:

# ipset list

or

# ipset -L

To delete a set named “myset”:

# ipset destroy myset

or

# ipset -X myset

To delete all sets:

# ipset destroy

How to configure IPsec/L2TP VPN Clients on Linux

After setting up your own VPN server, follow these steps to configure your devices. In case you are unable to connect, first, check to make sure the VPN credentials were entered correctly.

Commands must be run as root on your VPN client.

To set up the VPN client, first install the following packages:

# For Ubuntu & Debian
apt-get update
apt-get -y install strongswan xl2tpd

# For RHEL/CentOS
yum -y install epel-release
yum --enablerepo=epel -y install strongswan xl2tpd

yum -y install strongswan xl2tpd

Create VPN variables (replace with actual values):

VPN_SERVER_IP=your_vpn_server_ip
VPN_IPSEC_PSK=your_ipsec_pre_shared_key
VPN_USER=your_vpn_username
VPN_PASSWORD=your_vpn_password

Configure strongSwan:

cat > /etc/ipsec.conf <<EOF
# ipsec.conf - strongSwan IPsec configuration file

# basic configuration

config setup
  # strictcrlpolicy=yes
  # uniqueids = no

# Add connections here.

# Sample VPN connections

conn %default
  ikelifetime=60m
  keylife=20m
  rekeymargin=3m
  keyingtries=1
  keyexchange=ikev1
  authby=secret
  ike=aes128-sha1-modp2048!
  esp=aes128-sha1-modp2048!

conn myvpn
  keyexchange=ikev1
  left=%defaultroute
  auto=add
  authby=secret
  type=transport
  leftprotoport=17/1701
  rightprotoport=17/1701
  right=$VPN_SERVER_IP
EOF

cat > /etc/ipsec.secrets <<EOF
: PSK "$VPN_IPSEC_PSK"
EOF

chmod 600 /etc/ipsec.secrets

# For CentOS/RHEL & Fedora ONLY
mv /etc/strongswan/ipsec.conf /etc/strongswan/ipsec.conf.old 2>/dev/null
mv /etc/strongswan/ipsec.secrets /etc/strongswan/ipsec.secrets.old 2>/dev/null
ln -s /etc/ipsec.conf /etc/strongswan/ipsec.conf
ln -s /etc/ipsec.secrets /etc/strongswan/ipsec.secrets

Configure xl2tpd:

cat > /etc/xl2tpd/xl2tpd.conf <<EOF
[lac myvpn]
lns = $VPN_SERVER_IP
ppp debug = yes
pppoptfile = /etc/ppp/options.l2tpd.client
length bit = yes
EOF

cat > /etc/ppp/options.l2tpd.client <<EOF
ipcp-accept-local
ipcp-accept-remote
refuse-eap
require-chap
noccp
noauth
mtu 1280
mru 1280
noipdefault
defaultroute
usepeerdns
connect-delay 5000
name $VPN_USER
password $VPN_PASSWORD
EOF

chmod 600 /etc/ppp/options.l2tpd.client

The VPN client setup is now complete. Follow the steps below to connect.

Note: You must repeat all steps below every time you try to connect to the VPN.

Create xl2tpd control file:

mkdir -p /var/run/xl2tpd
touch /var/run/xl2tpd/l2tp-control

Restart services:

service strongswan restart
service xl2tpd restart

Start the IPsec connection:

# Ubuntu & Debian
ipsec up myvpn

# CentOS/RHEL & Fedora
strongswan up myvpn

Start the L2TP connection:

echo "c myvpn" > /var/run/xl2tpd/l2tp-control

Run ifconfig and check the output. You should now see a new interface ppp0.

Check your existing default route:

ip route

Find this line in the output: default via X.X.X.X .... Write down this gateway IP for use in the two commands below.

Exclude your VPN server’s IP from the new default route (replace with actual value):

route add YOUR_VPN_SERVER_IP gw X.X.X.X

If your VPN client is a remote server, you must also exclude your Local PC’s public IP from the new default route, to prevent your SSH session from being disconnected (replace with actual value):

route add YOUR_LOCAL_PC_PUBLIC_IP gw X.X.X.X

Add a new default route to start routing traffic via the VPN server:

route add default dev ppp0

The VPN connection is now complete. Verify that your traffic is being routed properly:

wget -qO- http://ipv4.icanhazip.com; echo

The above command should return Your VPN Server IP.

To stop routing traffic via the VPN server:

route del default dev ppp0

To disconnect:

# Ubuntu & Debian
echo "d myvpn" > /var/run/xl2tpd/l2tp-control
ipsec down myvpn

# CentOS/RHEL & Fedora
echo "d myvpn" > /var/run/xl2tpd/l2tp-control
strongswan down myvpn

How does email work ?

Most of people in this generation would spend their time daily to send and receive emails. E-mail plays a vital role in our daily activities. As technology evolved email became the one of major communication tool. It may not cost you to send or receive a mail but there is an intricate process involved behind. You just need to press  send button to send your message, your message goes through the complex machanism to get received by recipient.

The invention of email was started in 1961. There were more than couple of people to list as the inventors of email.

The evolution of email started from messaging users on the same computer, then message transition between computer then message among multi user and multi computer, finally email. Email has became revolutionary tool for communication. Now a days it is part of our daily life. We will see how email really works.

Terminology

There are different components involved in the email system. There are few abbreviated terms. Having idea about terminology and abbreviations  will help you to better understand the system.

MSA Mail Submission Agent
MTA Mail Transfer Agent
MDA Mail Delivery Agent
MX Mail Exchange
DNS Domain Name System

MUA is the software that user will use either to send or retrieve  the mail(message) from the server.

MSA is the piece of software which is installed on mail server. Where it is responsible to transfer the message to destination or mail server called MTA (Mail Transfer Agent)

MTA  is the Mail Transfer Agent. It is the piece software on the server which is responsible to route the mail to destination mail server. So we called it as mail router, mail server etc.                                                                        Here are few popular  MTA(Mail Server) softwares : postfixqmailCourier Mail and sendmail.

postfix is the one which is widely used, and it comes with many linux distributions.

You can find the exhaustive mail server softwares list here

Protocols involved

There are different protocol involved in email system. All of them are required to get email delivered to the recipient. They are building blocks of email system. Those protocols are,

SMTP Simple Mail Transport Protocol
IMAP Internet Message Access Protocol
POP Post Office Protocol
DNS Domain Name System (Protocol)

How does email work?

This sections will get you the idea about how these protocols work together to deliver the mail to recipient. Here is the abstract overview of email system.

How does email work abstract

The above figure will give you simple abstract overview of email system.  As described in the figure. SMTP is the protocol which is used to push or send an email to server by sender. IMAP and POP are the protocols which are used to check or retrieve message from server by recipient.  The recipient MUA is configured to use either IMAP or POP or BOTH (IMAP & POP). The protocol IMPA is bidirectional where POP is unidirectional. We will see more about POP and IMAP in later section.

        As per the above figure(abstract overview) you can sense that, the sender email client(MUA) will send message using SMPT protocol to mail server (MTA). The mail server will check for the destination if it finds the one it will connect to destination mail server, that is other MTA/MDA and will pass the message(mail) using SMTP protocol. The recipient server will store the received mail locally. Later if recipient will check the mail by using his mail client by connecting to his mail server.

Now let’s see flow of email in detail with following figure

Here Alice is sending  email to Bob(bob@domain.com) by using her email client. Where she will push her message to server using SMTP to send Bob. The mail server will determine the destination by getting the MX record for the destination server. The Alice mail server will check for the domain after in the to(recipient)  mail address to get the MX record. Once Alice(sender) mail server received the result from DNS for the DNS query to get MX record, this server will connect to destination mail server and will delivery the mail (message) using SMTP protocol. The destination mail server will store the message locally. Bob will check the newly arrived mails by using his mail client.

I would like to illustrate this process even more by using following figure.

In the above figure there are new components introduced, those are MSA, MDA and extra MX servers.

       MSA is the Mail Submission agent where it is piece of software it will receive the message from MUA. It uses the same protocol SMTP and port 25. Practically most MTAs perform the function of MSA so you may assume MTA as MSA.

       The MDA means mail delivery agent or message delivery agent  is a computer software component that is responsible for the delivery of e-mail messages to a local recipient’s mailbox. You may not need to get confused, most of MTAs perform the MDA functionality as well though there are softwares which are only designed to work as MDA.

Here, you can see that practically there will be more than 1 mail server(MX SERVER)s. All other MX servers are for backup purpose. When sender query DNS for MX record it may get more that one MX server with priorities. Here is the sample output of DNS query for gmail MX records. Gmail is having 5 MX servers with different priorities.

$ dig gmail.com  MX

; <<>> DiG 9.8.3-P1 <<>> gmail.com MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7760
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gmail.com.			IN	MX

;; ANSWER SECTION:
gmail.com.		1206	IN	MX	20 alt2.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	5 gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	10 alt1.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	40 alt4.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	30 alt3.gmail-smtp-in.l.google.com.

;; Query time: 18 msec
;; SERVER: 192.168.4.1#53(192.168.4.1)
;; WHEN: Sun Feb 14 09:42:23 2016
;; MSG SIZE  rcvd: 150

The above result is the out put of command line utility dig. Where we use this utility to query the DNS. In the output you would see something like below in the right side of  ANSWER SECTION.

20 alt2.gmail-smtp-in.l.google.com.
5 gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.

Here you would see some numbers in front of domains. Those numbers will decide the priority of MX record. The domain which is having very lowest number associated will have highest priority. Here gmail-smtp-in.l.google.com MX record having highest priority cause it having lowest number associated, that is 5. So the sender MTA will try to connect with most prioritized MX first. If it is down or so, it will try to connect with next MX (Mail Exchange server). As a result mail will be delivered with no down time.

Once the mail is received by the destination mail server, it will store the message in the mail store. There are two types of mail stores used by various mail servers(softwares). Those are,

  • Mail DIR
  • Mail Box

Locally stored mail will be accessed(fetched) by the recipient client using either POP or IMAP.

Protocols POP and IMAP

POP and IMAP are application layer protocols, as described above POP and IMAP are the protocols which are used to fetch or access the mails by recipient email client (MUA). Both protocols are different, they server for different purpose.

The protocol POP means Post Office Protocol and POP3 is it’s  version 3.  As the name describes, this protocol is used to download the message from the server by the client. Once message is downloaded form the server, it will be removed unless you set the flag leave a copy on server just how post card is delivered to destination. If you are only the one who will access mailbox from one location the POP3 suites well. This will also save some memory on server. So, if use POP3 you can’t access the mail using different clients.

Unlike POP3 the IMAP won’t download the message by deleting it on server. It will just access the message like browser does webpages. So it is handy if you use multiple clients from different locations.

 

References:

[1] https://tools.ietf.org/html/rfc3501
[2] https://www.ietf.org/rfc/rfc1939.txt
[3] https://tools.ietf.org/html/rfc5321

Email template

Coding HTML for email can be tough. If you included video, flash, javascript or image chances it won’t show up you wanted it to.

You may have had some issues with your in email clients like Gmail or Apple Mail and Outlook.

Some comman problems are :-

  • Some email client not supported all CSS
  • Syntax improperly in tables or missing tags
  • Fonts are not displaying correctly

Troubleshoot your problem :-

If you’re having issues with columns, or rows not appearing in a table, add a border to the table to identify problems and better visualize the table structure.

If you’re seeing centered text when you prefer it left-aligned, make sure you’ve included properties like valign and align in <td> cells and <p> tags.

You may have some issues with your HTML emails looking great in email clients like Gmail or Apple Mail but rendering poorly in Outlook.

Outlook does not support many features :-

  • Animated GIF.
  • Background Images.
  • Padding for <div>, <p>, and <a> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.
  • Width for <div> and <p> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.

Email Client CSS Support:-

Open below link and check which CSS support in browser, desktop and mobile in various email client’s like outlook, yahoo, gmail.

https://templates.mailchimp.com/resources/email-client-css-support/

Implementing Webhook Handler in Python.

What is Webhook ?

Webhook is an asynchronous HTTP callback on an event occurrence. It is a simple server to server communication for reporting a specific event occurred on a server. The server on which event occurred will fire a HTTP POST request to another server on a URL which is provided by receiving server.

For example, whenever your colleague pushes code commits to github, an event has occurred on github’s server. Now if a webhook URL is provided in github settings, a webhook will be fired to that URL. This webhook will be a HTTP POST request with commit details inside the body in a specified format.  More details on github webhook can be found here.

In this post, I will share my experience of implementing webhook handler in python. For the readers, basic knowledge on implementing web application in python would be better.

Webhook Handler

A Webhook can be handled by simply providing a URL endpoint in a web application. Following is an example using Django. Add webhook url in urls.py

from django.conf.urls import url
import views

urlpatterns = [
    url(r'^webhook', views.webhook, name='webhook'),
]

Now create view function in views.py which will parse the data and process it.  In most of the cases, webhook data is sent in JSON format. So lets load the webhook data and sent the data to process_webhook function.

Most of the web applications accept POST request after verifying CSRF token, but here we need to exempt it from this check. So put @csrf_token decorator above the view function. Also put an @require_post decorator to ensure the request is only POST.

from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST

@require_POST
@csrf_exempt
def webhook(request):

    # Load the event data from JSON
    data = json.loads(request.body)
    # And process it
    process_webhook(data)

    return 200, 'Processed.'

The above implementation of URL endpoint will remain different for various other python web framework like Flask, tornado, twisted. But the below code  process_webhook function implementation will remain same irrespective of any framework.

Processing event

There may be different type events we need to handle. So, before proceeding to implement process_webhook function, lets create a python module named webhook_events.py, which will contain a single function for each type of event wherein will be the logic for that particular event. In other words, we are going to map event name with its function, which will handle the logic for that particular type of webhook event.

def event_one(event):
    # do something for
    # for event 'event.one'


def event_two(event):
    # do something for
    # for event 'event.two'

There are many ways to implement process_webhook function and how we map a webhook event with its function. We are going to discuss different implementation of process_webhook based on extendability. Most basic version of that is below.

import webhook_events

def process_webhook(event):
    event_name = event['name']

    if event_name == 'event.one':
        webhook_event.event_one(event)

    elif event_name == 'event.two':
        webhook_event.event_two(event)

    # and so on

A Better way

Now suppose, there are 10s of webhook to be served. We certainly don’t want to write repetitive code. So below is a better way of implementing process_webhook. Here we just replace dot in event name with underscore, so that we get the function name written in webhook_events.py for that event. If the function is not found that means event is not registered (not being served). In this way, no matter the number webhook to be served, just write the function to handle it, in webhook_events.py

import webhook_events

def process_webhook(event):
    event_name = event['name']

    function_name = event_name.replace('.', '_')
    function = getattr(webhook_events, function_name, None)

    if function:
        function(event)
    else:
        print('Event %s is not registered.' % event_name)

Decorators

More robust and pythonic way of implementing process_webhook is by using decorators. Lets define a decorator in webhook_events.py which will map the event_name to its function. Here the EVENT_MAP is dictionary inside a setting module, which will contain event name as key and event function as its value.

from django.conf import settings

def register(event_name):

    def wrapper(event_function):
        
        # Initializing settings.event_map if not already
        event_map = getattr(settings, 'EVENT_MAP', None)
        if not event_map:
            settings.EVENT_MAP = dict()
        
        # Mapping event name to its function
        settings.EVENT_MAP[event_name] = event_function
    
        return event_function

    return wrapper


@register('event.one')
def event_one(event):
    # do something for
    # for event 'event.one'


@register('event.two')
def event_two(event):
    # do something for
    # for event 'event.two'

In this case, the process_webhook will look like below:

def process_webhook(event):
    event_name = event['name']
    function = settings.EVENT_MAP.get(event_name, None)

    if function:
        function(event)
    else:
        print('Event %s is not registered.' % event_name)

This is the way which I prefer to implement webhook handler in python. How would you prefer ? Please feel free to comment below.